Compare commits

..
Author SHA1 Message Date
Michael S. Molina 961eba6d97 chore: Updates CHANGELOG.md for 3.0.2 (rc2) 2023-11-20 16:40:41 -03:00
Daniel Vaz Gaspar 08604cc686 fix: update FAB to 4.3.10, Azure user info fix (#26037)
(cherry picked from commit 628cd345f2)
2023-11-20 16:27:34 -03:00
JUST.in DO IT 49661bcc59 fix(native filters): rendering performance improvement by reduce overrendering (#25901)
(cherry picked from commit e1d73d5420)
2023-11-20 16:19:24 -03:00
John Bodley da06206ea6 chore: Optimize fetching samples logic (#25995)
(cherry picked from commit 326ac4a6c4)
2023-11-20 16:19:24 -03:00
JUST.in DO IT e3fbb01bb8 fix(explore): redandant force param (#25985)
(cherry picked from commit e7a1876807)
2023-11-20 16:19:24 -03:00
Jack Fragassi ee1ba7e172 fix: Make Select component fire onChange listener when a selection is pasted in (#25993)
(cherry picked from commit 5fccf67cdc)
2023-11-16 18:30:35 -03:00
yousoph 8d873e6da6 fix(rls): Update text from tables to datasets in RLS modal (#25997)
(cherry picked from commit 210f1f8f95)
2023-11-16 17:28:25 -03:00
josedev-union 1d2a564d4e fix(helm): Restart all related deployments when bootstrap script changed (#25703) 2023-11-16 17:28:08 -03:00
Hugh A. Miles II 1c287dfc74 fix: naming denomalized to denormalized in helpers.py (#25973)
(cherry picked from commit 5def416f63)
2023-11-16 17:26:51 -03:00
John Bodley fb1919a483 chore(colors): Updating Airbnb brand colors (#23619)
(cherry picked from commit 6d8424c104)
2023-11-16 17:26:51 -03:00
Hugh A. Miles II e07eed10a2 fix: always denorm column value before querying values (#25919) 2023-11-16 17:26:15 -03:00
Giacomo BaroneandVille Brofeldt a7fbdd607a fix: update flask-caching to avoid breaking redis cache, solves #25339 (#25947)
Co-authored-by: Ville Brofeldt <33317356+villebro@users.noreply.github.com>
2023-11-16 17:22:25 -03:00
JUST.in DO IT 6da18f8451 fix(sqllab): invalid sanitization on comparison symbol (#25903)
(cherry picked from commit 581d3c7108)
2023-11-16 17:21:04 -03:00
John Bodley eea6a8ed4f fix(table): Double percenting ad-hoc percentage metrics (#25857)
(cherry picked from commit 784a478268)
2023-11-16 17:21:04 -03:00
FGrobelny 078b78f30b fix(trino): allow impersonate_user flag to be imported (#25872)
Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>
(cherry picked from commit 458be8c848)
2023-11-16 17:21:04 -03:00
Michael S. Molina c655a3e7dd chore: Updates CHANGELOG.md for 3.0.2 2023-11-13 16:44:20 -03:00
Beto Dealmeida d265bd2ffc fix: trino cursor (#25897)
(cherry picked from commit cdb18e04ff)
2023-11-08 09:56:38 -03:00
Beto Dealmeida 8c099a3f6f fix: database version field (#25898)
(cherry picked from commit 06ffcd29e2)
2023-11-08 09:56:38 -03:00
Kamil Gabryjelski 81f7c63763 fix: Saving Mixed Chart with dashboard filter applied breaks adhoc_filter_b (#25877)
(cherry picked from commit 268c1dcdad)
2023-11-08 09:56:37 -03:00
Antonio Rivero 756324d713 fix(charts): Time grain is None when dataset uses Jinja (#25842)
(cherry picked from commit 7536dd12cd)
2023-11-08 09:56:37 -03:00
Beto Dealmeida 5198279a2b fix: remove update_charts_owners (#25843) 2023-11-06 10:18:36 -03:00
Arko 4534a070df fix(table chart): Show Cell Bars correctly #25625 (#25707)
(cherry picked from commit 916f7bcbba)
2023-11-06 09:40:48 -03:00
mapledan d5901140a7 fix: the temporal x-axis results in a none time_range. (#25429)
Co-authored-by: Elizabeth Thompson <eschutho@gmail.com>
(cherry picked from commit ae619b169c)
2023-11-06 09:40:48 -03:00
Michael S. Molina 925c63d4a6 fix: Fires onChange when clearing all values of single select (#25853)
(cherry picked from commit 8061d5cce9)
2023-11-06 09:40:48 -03:00
JUST.in DO IT 28272527fc fix(sqllab): infinite fetching status after results are landed (#25814)
(cherry picked from commit 3f28eebb20)
2023-11-01 09:08:19 -03:00
Ross Mabbett 2d574963f0 fix(SQL field in edit dataset modal): display full sql query (#25768)
(cherry picked from commit 1eba7121aa)
2023-11-01 09:08:19 -03:00
John Bodley 04c11b477b fix: Resolve issue #24195 (#25804)
(cherry picked from commit 8737a8a546)
2023-11-01 09:08:19 -03:00
John Bodley c216b3efdf fix: Revert "fix: Apply normalization to all dttm columns (#25147)" (#25801) 2023-10-31 11:47:45 -03:00
Beto Dealmeida 1d403dab98 fix: DB-specific quoting in Jinja macro (#25779)
(cherry picked from commit 5659c87ed2)
2023-10-31 11:22:13 -03:00
Elizabeth Thompson 2f468900c8 fix: allow for backward compatible errors (#25640) 2023-10-31 11:18:24 -03:00
JUST.in DO IT fbe7e6265d fix(sqllab): slow pop datasource query (#25741)
(cherry picked from commit 2a2bc82a8b)
2023-10-31 10:47:06 -03:00
Beto Dealmeida 01d3ac20c7 fix: dataset update uniqueness (#25756)
(cherry picked from commit c7f8d11a7e)
2023-10-31 10:47:06 -03:00
Geido fd2c2725d4 fix: Revert "fix(Charts): Set max row limit + removed the option to use an empty row limit value" (#25753)
(cherry picked from commit e2fe967788)
2023-10-31 10:47:06 -03:00
Ross Mabbett 9b31d97ac3 fix(horizontal filter label): show full tooltip with ellipsis (#25732)
(cherry picked from commit e4173d90c8)
2023-10-31 10:47:06 -03:00
Daniel Vaz Gaspar 8da27eda40 fix: bump to FAB 4.3.9 remove CSP exception (#25712)
(cherry picked from commit 8fb0c8da56)
2023-10-31 10:37:08 -03:00
Stepan 8483ab6c42 fix(chore): dashboard requests to database equal the number of slices it has (#24709)
(cherry picked from commit 75a7431379)
2023-10-31 10:37:08 -03:00
Igor Khrol 315e75811f fix: remove unnecessary redirect (#25679)
(cherry picked from commit da42bf2dbb)
2023-10-31 10:37:07 -03:00
Rob Moore 5293f5521d fix(sqllab): reinstate "Force trino client async execution" (#25680) 2023-10-31 10:36:30 -03:00
OskarNS 293568ad5a fix(dremio): Fixes issue with Dremio SQL generation for Charts with Series Limit (#25657)
(cherry picked from commit be82657940)
2023-10-19 10:05:46 -03:00
JUST.in DO IT b380495516 fix: warning of nth-child (#23638)
(cherry picked from commit 16cc089b19)
2023-10-19 09:59:46 -03:00
Daniel Vaz Gaspar b0f229ea7e fix: improve upload ZIP file validation (#25658) 2023-10-19 09:59:24 -03:00
Jack b95ff2da23 fix(header navlinks): link navlinks to path prefix (#25495)
(cherry picked from commit 51c56dd2a0)
2023-10-19 09:56:44 -03:00
Hugh A. Miles II 236aef8126 fix: permalink save/overwrites in explore (#25112)
Co-authored-by: Elizabeth Thompson <eschutho@gmail.com>
(cherry picked from commit e58a3aba54)
2023-10-19 09:56:44 -03:00
Jack Fragassi af1e71352a fix(import): Make sure query context is overwritten for overwriting imports (#25493)
(cherry picked from commit a0a0d8043f)
2023-10-19 09:56:43 -03:00
Igor Khrol ec3bed709e fix: avoid 500 errors with SQLLAB_BACKEND_PERSISTENCE (#25553)
(cherry picked from commit 99f79f5143)
2023-10-19 09:56:43 -03:00
JUST.in DO IT 701ee30d1e fix(sqllab): template validation error within comments (#25626)
(cherry picked from commit b370c66308)
2023-10-19 09:56:43 -03:00
JUST.in DO IT d7cbdca081 fix(sqllab): Mistitled for new tab after rename (#25523)
(cherry picked from commit a520124a78)
2023-10-19 09:56:43 -03:00
Michael S. Molina 890bf59ce4 chore: Updates 3.0.1 CHANGELOG 2023-10-13 10:52:07 -03:00
Ville Brofeldt cd1b7a4c06 fix: revert fix(sqllab): Force trino client async execution (#24859) (#25541)
(cherry picked from commit e56e0de458)
2023-10-13 09:10:46 -03:00
Beto Dealmeida c44f1a3299 fix: finestTemporalGrainFormatter (#25618)
(cherry picked from commit 62bffaf935)
2023-10-13 09:10:46 -03:00
Fabien a0b2dc4266 fix(window): unavailable localStorage and sessionStorage (#25599) 2023-10-13 09:10:16 -03:00
Corbin Bullard 732c5b1f08 fix(Charts): Set max row limit + removed the option to use an empty row limit value (#25579)
(cherry picked from commit f556ef53f3)
2023-10-13 09:04:41 -03:00
Rui Zhao 254cc36b17 fix(Presto): catch DatabaseError when testing Presto views (#25559)
Co-authored-by: Rui Zhao <zhaorui@dropbox.com>
(cherry picked from commit be3714e131)
2023-10-13 09:04:41 -03:00
Igor Khrol 53b84b9664 fix: thubmnails loading - Talisman default config (#25486)
(cherry picked from commit 52f631a038)
2023-10-13 09:04:41 -03:00
Corbin Bullard 69c2378747 fix(RLS): Fix Info Tooltip + Button Alignment on RLS Modal (#25400)
(cherry picked from commit a6d0e6f37a)
2023-10-11 08:36:08 -03:00
Daniel Vaz Gaspar 286c095506 fix: REST API CSRF exempt list (#25590)
(cherry picked from commit 549abb542b)
2023-10-11 08:36:08 -03:00
Kamil Gabryjelski dd769eb7a0 fix: Apply normalization to all dttm columns (#25147)
(cherry picked from commit 58fcd292a9)
2023-10-09 11:45:56 -03:00
Igor Khrol 8b66603566 fix: tags permissions error message (#25516)
(cherry picked from commit 50b0816e37)
2023-10-09 11:27:46 -03:00
JUST.in DO IT 220dc58fe0 chore: Expand error detail on screencapture (#25519)
(cherry picked from commit ba541e8022)
2023-10-04 15:18:35 -03:00
JUST.in DO IT 1367d7b954 fix(sqllab): Broken query containing 'children' (#25490)
(cherry picked from commit b92957e510)
2023-10-04 15:18:35 -03:00
mapledan ae700d13cf fix: Unable to sync columns when database or dataset name contains + (#25390)
(cherry picked from commit dbe0838f8f)
2023-10-04 15:18:35 -03:00
Michael S. Molina 4ad2a05333 chore: Adds 3.0.1 CHANGELOG 2023-10-03 13:13:47 -03:00
John Bodley 9d1ab460c3 fix: Address Mypy issue which is causing CI to fail (#25494)
(cherry picked from commit 36ed617090)
2023-10-03 10:52:16 -03:00
ʈᵃᵢ 455b3d8a67 fix(sqllab): error with lazy_gettext for tab titles (#25469)
(cherry picked from commit ddde178e3b)
2023-10-02 11:58:07 -03:00
Kamil Gabryjelski 0dd1a3bea5 fix: Styles not loading because of faulty CSP setting (#25468)
(cherry picked from commit 0cebffd59a)
2023-10-02 11:03:05 -03:00
Ville Brofeldt 615d7f5ccc fix(mysql): handle string typed decimal results (#24241)
(cherry picked from commit 7eab59af51)
2023-10-02 11:03:05 -03:00
Celalettin Calis f682dbae52 fix(helm chart): set chart appVersion to 3.0.0 (#25373) 2023-10-02 11:02:40 -03:00
Gyuil Han d8e87aa3cc fix: update the SQLAlchemy model definition at json column for Log table (#25445)
(cherry picked from commit e83a76a586)
2023-10-02 11:01:38 -03:00
Michael S. Molina 58778a78b2 fix: Duplicate items when pasting into Select (#25447)
(cherry picked from commit 7cf96cd843)
2023-09-28 14:38:25 -03:00
Corbin Bullard 731cd65111 fix(SqlLab): make icon placement even (#25372)
(cherry picked from commit 11b49a6ceb)
2023-09-28 11:46:32 -03:00
Stepan 7b2b696a19 fix(nativeFilters): Speed up native filters by removing unnecessary rerenders (#25282)
Co-authored-by: JUST.in DO IT <justin.park@airbnb.com>
(cherry picked from commit a0eeb4d767)
2023-09-28 11:46:32 -03:00
JUST.in DO IT b83bd5dc1c fix(sqllab): invalid start date (#25437) 2023-09-28 11:45:58 -03:00
Beto Dealmeida 8dfe95f89d fix: smarter date formatter (#25404)
(cherry picked from commit f0080f9c55)
2023-09-28 11:37:17 -03:00
Daniel Vaz Gaspar 0c6db230af fix: swagger UI CSP error (#25368)
(cherry picked from commit 1716b9f8f6)
2023-09-28 11:37:17 -03:00
Beto Dealmeida 1757ce49a9 fix: chart import (#25425)
(cherry picked from commit a4d8f36863)
2023-09-28 11:37:17 -03:00
Jack 4a65d41ce8 fix: preventing save button from flickering in SQL Lab (#25106)
(cherry picked from commit 296ff17f19)
2023-09-28 11:37:17 -03:00
John Bodley eacdbdd877 fix: Rename on_delete parameter to ondelete (#25424)
(cherry picked from commit 893b45feef)
2023-09-26 16:29:47 -07:00
JUST.in DO IT 721db8a1a9 fix(sqllab): invalid persisted tab state (#25308) (#25398) 2023-09-26 11:19:05 -03:00
Michael S. Molina c508a335e6 fix: Workaround for Cypress ECONNRESET error (#25399)
(cherry picked from commit d76ff39766)
2023-09-25 17:42:57 -03:00
Beto Dealmeida 88e6f22180 fix: datetime with timezone excel export (#25318)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
(cherry picked from commit 5ebcd2a5f6)
2023-09-25 10:42:16 -03:00
Michael S. Molina 5390e2d826 fix: DashboardRoles cascade operation (#25349)
(cherry picked from commit a971a28a34)
2023-09-25 10:02:37 -03:00
Jack Fragassi 0d53446562 fix: Improve the reliability of alerts & reports (#25239)
(cherry picked from commit f672d5da5c)
2023-09-25 09:52:08 -03:00
Jack Fragassi 61dcc70db4 fix: Use RLS clause instead of ID for cache key (#25229)
(cherry picked from commit fba66c6250)
2023-09-25 09:52:08 -03:00
Zef LinandKamil Gabryjelski 8ca49d4e6f fix(chart): Supporting custom SQL as temporal x-axis column with filter (#25126)
Co-authored-by: Kamil Gabryjelski <kamil.gabryjelski@gmail.com>
2023-09-25 09:51:41 -03:00
John Bodley 2d1f1e3d71 fix: Add explicit ON DELETE CASCADE for dashboard_roles (#25320)
(cherry picked from commit d54e827bb9)
2023-09-25 09:48:33 -03:00
Beto Dealmeida 807a027a5f fix: is_select with UNION (#25290)
(cherry picked from commit bb002d6147)
2023-09-25 09:48:33 -03:00
Michael S. Molina 40d9c4c81f chore: Updates CHANGELOG.md 2023-09-13 08:46:07 -03:00
John Bodley f55962fbb8 chore: Remove the ability to switch to filter-box chart when DASHBOARD_NATIVE_FILTERS feature is enabled (#25275)
(cherry picked from commit 8eff5a75b4)
2023-09-13 08:41:12 -03:00
Lily Kuang 28e944ae86 fix: inability to remove chart filter when dashboard time filter is applied (#25217)
(cherry picked from commit a9512c1eef)
2023-09-12 11:20:26 -03:00
Kamil Gabryjelski 1498c21d42 fix: Add line height to metadata bar (#25268)
(cherry picked from commit 242921bb4c)
2023-09-12 11:01:02 -03:00
Hugo Sjöberg 69db484d2d fix(dockefile): broken docker image (#25251) 2023-09-12 11:00:33 -03:00
Kamil Gabryjelski 39dcb29a69 fix: Currency formatting in Table raw mode (#25248)
(cherry picked from commit ea21e800a7)
2023-09-11 11:51:17 -03:00
Kamil Gabryjelski 6d1b969602 fix: Don't apply number formatting to the label in Treemap (#25249)
(cherry picked from commit 894f250229)
2023-09-11 11:07:48 -03:00
Michael S. Molina 2f1ce7f721 fix: Clearing the currency format has no effect on the chart (#25238)
(cherry picked from commit 6f4e63162f)
2023-09-11 11:07:48 -03:00
Ville Brofeldt 65a2ca9e6f chore(trino): remove unnecessary index checks (#25211)
(cherry picked from commit 0668d12e3b)
2023-09-11 11:07:48 -03:00
Kamil Gabryjelski 10e781d0ff feat: Add currencies controls in control panels (#24718) 2023-09-11 11:06:49 -03:00
Michael S. Molina dfd699f440 fix: Cypress test to force mouseover (follow-up) (#25223)
(cherry picked from commit 0e17e4b06f)
2023-09-07 14:48:33 -03:00
Beto Dealmeida 2ae9d2ef05 fix: granularity_sqla and GENERIC_CHART_AXES (#25213) 2023-09-07 10:30:14 -03:00
Rob Moore 80df8bc558 fix(sqllab): Force trino client async execution (#24859)
(cherry picked from commit cfda30c81a)
2023-09-07 10:24:22 -03:00
Michael S. Molina dac3009cc6 fix: Cypress test to force mouseover (#25209)
(cherry picked from commit 47518cb002)
2023-09-07 10:24:22 -03:00
Beto Dealmeida 2ac03c3bcc fix: is_select (#25189)
(cherry picked from commit 2f68010729)
2023-09-07 10:24:22 -03:00
Michael S. Molina d8c72b86bc fix: All values being selected in Select (#25202)
(cherry picked from commit e605d6dddf)
2023-09-06 10:47:21 -03:00
Hugh A. Miles II 408708be62 fix: DML failures in SQL Lab (#25190)
(cherry picked from commit d8496425e2)
2023-09-06 10:47:21 -03:00
Sandeep Patel e77bc066bd fix: Issue #24493; Resolved report selection menu in chart and dashboard page (#25157)
(cherry picked from commit d0305445b2)
2023-09-06 10:47:21 -03:00
Michael S. Molina 90a66ee76c chore: Add a note about adhoc subqueries in UPDATING.md (#25161) 2023-09-05 09:15:19 -03:00
Lily Kuang 331fefd3dd fix: Applying Dashboard Time Range Filters to Overwritten Charts (#25156)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
(cherry picked from commit f2523b26fa)
2023-09-05 09:13:16 -03:00
JUST.in DO IT c56948e1f1 fix(sqllab): Invalid start date (#25133)
(cherry picked from commit 8b2a408dea)
2023-09-05 09:13:16 -03:00
Michael S. Molina b53042a7be fix: Sunburst chart error when secondary metric is null (#25159)
(cherry picked from commit aea916782d)
2023-09-05 09:13:15 -03:00
aehanno f778b62712 fix: Fixing untranslated FR strings (#20078)
(cherry picked from commit 2c4629a77d)
2023-09-01 10:21:17 -03:00
KSPT-taylorjohn 76d7bf9011 fix: Chart series limit doesn't work for some databases (#25150)
(cherry picked from commit bbfaeb074e)
2023-09-01 10:21:17 -03:00
John Bodley af8f074919 fix: Handle Python date format for GENERIC_CHART_AXES feature (#25135)
(cherry picked from commit de9515c294)
2023-09-01 10:21:17 -03:00
Ethan M Lewis dd0475d63c fix: add format for timestamp in crdb to correctly use times values (#24711)
Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>
(cherry picked from commit 321d968504)
2023-09-01 10:21:17 -03:00
John Bodley 9837dabb2c docs: Fix typo in UPDATING.md (#25127) 2023-09-01 10:20:37 -03:00
Michael S. Molina 372004d0e6 fix: Bumps Flask Caching to fix RCE vulnerability (#25090) 2023-08-31 08:55:54 -03:00
Sam Firke 46e6d32260 refactor(frontend): make "Search" box the first filter for charts and datasets (#25129)
(cherry picked from commit eeecd59c9d)
2023-08-31 08:54:32 -03:00
Vitor Avila e2f89d7aa8 fix(DB Connection): Update placeholder values for Snowflake connection (#25119)
(cherry picked from commit 46a0a6e08a)
2023-08-31 08:54:31 -03:00
John BodleyandMichael S. Molina 798b493f3a docs: Update UPDATING.md regarding potential breaking change to ab_user.email column (#25115)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
2023-08-30 09:16:55 -03:00
Michael S. Molina caa3b6d5ba fix: Date format when importing international timestamps (#25113)
(cherry picked from commit 00550d7c02)
2023-08-30 09:13:40 -03:00
JUST.in DO IT 90e7e769ce fix(sqllab): error while removing a referenced table (#25114)
(cherry picked from commit 29355577f1)
2023-08-30 09:13:39 -03:00
JUST.in DO IT 429ff9b0f8 fix(sqllab): rendering performance regression by resultset (#25091) 2023-08-29 09:27:33 -03:00
Jack Fragassi 6a461260fc fix: Allow embedded guest user datasource access with dashboard context (#25081)
(cherry picked from commit 2b8d8da22a)
2023-08-29 08:48:48 -03:00
Jack Fragassi f34e21be69 fix(assets import): Ensure old datasource ids are not referenced in imported charts (#25086)
(cherry picked from commit b240b795b5)
2023-08-29 08:48:48 -03:00
Michael S. Molina 9ceba619c3 fix: Filter names overflow wrap (#25087)
(cherry picked from commit b5bac6c876)
2023-08-29 08:48:48 -03:00
Erich 76da1b59f9 refactor(pinot): The python_date_format for a temporal column was not being passed to get_timestamp_expr (#24942)
(cherry picked from commit c2a21d2da0)
2023-08-29 08:48:48 -03:00
Zef Lin 7d5cd72e43 fix: dataset update permission out of sync (#25043) 2023-08-29 08:48:28 -03:00
Jack Fragassi 0caaad7b0a fix(embedded sdk): Remove trailing slash from passed superset domain if there is one (#25020)
(cherry picked from commit 7402379346)
2023-08-29 08:39:58 -03:00
Lily Kuang 55c57b9277 chore: remove CssTemplate and Annotation access from gamma role (#24826)
(cherry picked from commit 6ac906f388)
2023-08-29 08:39:58 -03:00
Michael S. Molina 2c99366333 chore: Updates CHANGELOG.md 2023-08-24 14:11:50 -03:00
Michael S. Molina e4affbfc95 fix: Cross filters initial scope (#25074)
(cherry picked from commit f584c8462b)
2023-08-24 13:06:53 -03:00
Stepan f5d2075ace fix(table): condition formatting can't formate 0 values (#24008)
(cherry picked from commit 0d5be8e3f6)
2023-08-24 13:06:53 -03:00
Rémy DUBOIS 2554a89cc9 fix(cli): import-directory is missing the application context (#21255)
(cherry picked from commit 80fcbfab7c)
2023-08-24 13:06:53 -03:00
John Bodley ad89ea549b fix: Ensure SQLAlchemy sessions are closed (#25031)
(cherry picked from commit adaab3550c)
2023-08-24 13:06:53 -03:00
Daniel Vaz Gaspar 931e1b2139 fix: dataset safe URL for explore_url (#24686)
(cherry picked from commit a9efd4b2e3)
2023-08-23 09:55:50 -03:00
Michael S. Molina b5f7f54c7f fix: Error when using the legacy dataset editor (#25057)
Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>
(cherry picked from commit c92a975e4b)
2023-08-23 09:55:50 -03:00
John Bodley 1af6df3190 fix: Native filter dashboard RBAC aware dataset permission (#25029)
(cherry picked from commit 60889d27ed)
2023-08-22 17:39:55 -03:00
Michael S. Molina 8cb5142f87 fix: docker-compose non-dev (#25055)
(cherry picked from commit 7317d9c0b2)
2023-08-22 17:39:55 -03:00
Daniel Vaz Gaspar 6003aa2485 fix: extend user email size (#25053)
(cherry picked from commit 6975084ea5)
2023-08-22 17:39:55 -03:00
Sebastian Liebscher 80f1eaf6d7 chore: use os.getenv to simplify superset_config.py (#25016)
(cherry picked from commit 969cd664cb)
2023-08-22 17:39:55 -03:00
Yuval Moshe 34bc86a484 fix(mssql): avoid trying to return a resultset for DML queries with not resultset (#24999)
(cherry picked from commit 66eabc253f)
2023-08-21 11:12:57 -03:00
Daniel Vaz Gaspar 994fd2301f fix: CTE queries with non-SELECT statements (#25014)
(cherry picked from commit 357986103b)
2023-08-21 10:46:15 -03:00
Michael S. Molina a5a027d8d1 fix: Dashboard fullscreen is removing custom URL params (#25028)
(cherry picked from commit 0be1754666)
2023-08-21 10:46:15 -03:00
John Bodley fad872fffb fix: Address regression introduced in #24789 (#25008)
(cherry picked from commit 3f93755be2)
2023-08-21 10:46:15 -03:00
Michael S. Molina 88383ded80 chore: Pass the dashboard id when requesting filter values (#25025)
(cherry picked from commit 52c7186b56)
2023-08-21 10:46:15 -03:00
Jack Fragassi 9ff1a63c3b fix: Don't let users see dashboards only because it's favorited (#24991)
(cherry picked from commit 258e56285a)
2023-08-18 10:32:39 -03:00
Michael S. Molina 696917905e fix: Downgrades Prophet to 1.1.1 and Holidays to 0.23 (#25017) 2023-08-18 10:28:11 -03:00
Michael S. Molina f63cb47f35 fix: Date column in Heatmap is displayed as unix timestamp (#25009) 2023-08-17 16:38:59 -03:00
Daniel Vaz Gaspar c2c5f232c8 chore: isolate examples database by default (#25003)
(cherry picked from commit 269c99293f)
2023-08-17 16:36:42 -03:00
Elizabeth Thompson ea27cf13d3 fix: update permalink schema (#24970)
(cherry picked from commit bc1c5c2f84)
2023-08-17 16:36:42 -03:00
Elizabeth Thompson e20c2967c4 fix: Revert "fix(chart): Time Series set showMaxLabel as null for time xAxis (#20627) (#24995)
(cherry picked from commit 2b63577046)
2023-08-17 16:36:42 -03:00
Ville Brofeldt 387549f69c fix(snowflake): opt-in denormalization of column names (#24982) 2023-08-16 11:29:03 -03:00
1569f0177f chore: modify UPDATING.md for logs table migrations (#24923)
Co-authored-by: John Bodley <4567245+john-bodley@users.noreply.github.com>
Co-authored-by: Evan Rusackas <evan@preset.io>
2023-08-16 10:58:09 -03:00
JUST.in DO IT 8d3a919f5e fix(sqllab): scroll position after run current sql (#24965)
(cherry picked from commit 155cf54c15)
2023-08-16 10:56:53 -03:00
Michael S. Molina 3ffc5b69f8 fix: Calendar Heatmap day offset (#24989)
(cherry picked from commit 025e4d4772)
2023-08-16 10:56:53 -03:00
Michael S. Molina 4b07b5d628 feat: Adds options to show subtotals in Pivot Table (#24960)
(cherry picked from commit be11556799)
2023-08-14 09:26:21 -03:00
Michael S. Molina dd53b334d6 fix: Duplicated options in Select when using numerical values (#24906)
(cherry picked from commit b621ee92c9)
2023-08-14 09:26:21 -03:00
Michael S. Molina 5c931b1951 fix: Tooltips don't disappear on the Heatmap chart (#24959)
(cherry picked from commit 9703490129)
2023-08-14 09:26:21 -03:00
Beto Dealmeida 52319201f9 chore: rate limit requests (#24324)
(cherry picked from commit 4bc46003b5)
2023-08-14 09:26:21 -03:00
Beto Dealmeida ff2ec23102 fix: calls to _get_sqla_engine (#24953)
(cherry picked from commit 6f24a4e7a8)
2023-08-14 09:26:20 -03:00
Rob Moore 5d8c65ae6f fix(charts): View in SQL Lab with relevant perm (#24903)
(cherry picked from commit ce65a3b9cd)
2023-08-14 09:26:20 -03:00
Multazim Deshmukh 7a7fa748f5 fix: remove unused file (#24946)
(cherry picked from commit bcd24936bc)
2023-08-10 13:37:21 -03:00
Michael S. Molina 2574e11544 chore: Removes duplicated featureFlags.ts (#24935)
(cherry picked from commit 284c12697b)
2023-08-10 13:15:58 -03:00
John BodleyandMichael S. Molina 1e20c0bf8a chore: Add explicit ON DELETE CASCADE for embedded_dashboards (#24939)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
2023-08-10 13:14:56 -03:00
John Bodley 9b3ec806cd chore: Add explicit ON DELETE CASCADE for dashboard_slices (#24938) 2023-08-10 13:14:08 -03:00
John Bodley 309582516d fix: Dashboard aware RBAC "Save as" menu item (#24806)
(cherry picked from commit f6c3f0cbbb)
2023-08-10 10:12:49 -03:00
John Bodley dba72c4197 chore: Refine native dashboard cleanup logic (#24864)
(cherry picked from commit 34586648a5)
2023-08-10 10:12:48 -03:00
John Bodley 804cc36080 chore: Refactor dashboard security access (#24804)
(cherry picked from commit 5522facdc6)
2023-08-10 10:12:48 -03:00
Michael S. Molina df92cc2d55 fix: Tooltip of area chart shows undefined total (#24916)
(cherry picked from commit ec9e9a46f2)
2023-08-10 10:12:48 -03:00
JUST.in DO IT 42451880a8 fix(explore): double resize triggered (#24886)
(cherry picked from commit 340bfd88ae)
2023-08-10 10:12:48 -03:00
Gyuil Han ed56375d5f fix(logs): increase json field for logs table (#24911)
(cherry picked from commit eb7c14561e)
2023-08-10 10:12:48 -03:00
Jack Fragassi abead484e1 fix: Migration to fix out of sync schema_perm in charts and datasets (#24884)
(cherry picked from commit 07992c11e7)
2023-08-07 16:29:30 -03:00
John Bodley c8c7539ff1 fix: Dashboard aware RBAC dataset permission (#24789)
(cherry picked from commit 7397ab36f2)
2023-08-07 16:19:56 -03:00
John Bodley 215b3b5a4b fix: revert "fix(embedded): adding logic to check dataset used by filters (#24808) (#24892)
(cherry picked from commit 9f7f2c60d6)
2023-08-07 16:19:56 -03:00
Michael S. Molina 7fe61ccf7d fix: Pylint errors from cherry-picking 2023-08-07 16:17:10 -03:00
Elizabeth Thompson e46f10a8a6 chore: add talisman env var to config (#24774)
(cherry picked from commit d23b20ea75)
2023-08-04 14:12:35 -03:00
JUST.in DO IT b272814ff5 fix(dataset): resizable dataset layout left column (#24829)
(cherry picked from commit 6ff7fae0b0)
2023-08-04 14:12:35 -03:00
JUST.in DO IT 84035badab fix(explore): invalid "No Filter" applied (#24876)
(cherry picked from commit 371bffbfea)
2023-08-04 14:12:35 -03:00
John Bodley 21764f9ae3 fix(annotation): Address regression from #24694 (#24874)
(cherry picked from commit f05638ba84)
2023-08-04 14:12:35 -03:00
Hugh A. Miles II 8cf702bf3d fix: validation errors appearing after ssh tunnel switch (#24849)
(cherry picked from commit b71541fb7f)
2023-08-04 14:12:35 -03:00
Michael S. Molina e47377e576 fix: Explore misleading save action (#24862)
(cherry picked from commit bf1b1a4c46)
2023-08-04 14:12:35 -03:00
JUST.in DO IT 21ded992ad fix(sqllab): Add docText for long keyword (#24847)
(cherry picked from commit 1a9c559a8f)
2023-08-04 14:12:35 -03:00
JUST.in DO IT 29528e9783 fix(legacy-chart): corrupted raw chart data (#24850)
(cherry picked from commit 1c5971d3af)
2023-08-02 08:38:09 -03:00
Michael S. Molina 34adeb4f4f fix: Links in tooltips of dashboard chart cards (#24846)
(cherry picked from commit ea17dd637c)
2023-08-01 10:54:41 -03:00
Vitor Avila bbe4e016d8 fix(embedded): adding logic to check dataset used by filters (#24808)
(cherry picked from commit 7f9b0380e0)
2023-08-01 10:54:41 -03:00
EugeneTorap 161e05445c fix: Python3.11 (str, Enum) issue (#24803) 2023-08-01 10:54:20 -03:00
Stepan b5df3f9e4e fix(datasets): give possibility to add dataset with slashes in name (#24796) 2023-07-31 09:25:24 -03:00
Jack Fragassi a9b8c8e3ec fix: Allow chart import to update the dataset an existing chart points to (#24821)
(cherry picked from commit 77889b29fb)
2023-07-31 09:23:03 -03:00
John Bodley 1a0d270e5b fix(migration): Ensure cascadeParentIds key exists (#24831)
(cherry picked from commit caffe3cb1f)
2023-07-31 09:23:03 -03:00
Michael S. Molina 7d2da96e81 fix: Removes unnecessary query on filters (#24814)
(cherry picked from commit 5bb8e0da89)
2023-07-31 09:23:02 -03:00
Beto Dealmeida b89d7387a2 fix: pass schema on dataset creation (#24815)
(cherry picked from commit ba508a786c)
2023-07-31 09:23:02 -03:00
Michael S. Molina 571f33536e chore: Updates CHANGELOG.md 2023-07-26 19:35:07 -03:00
Michael S. Molina 651c13b934 chore: Updates helm files 2023-07-26 18:39:24 -03:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> af8c813cb6 build(deps): bump tox from 4.6.3 to 4.6.4 in /requirements (#24613)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 18:03:46 -03:00
Michael S. Molina ad1a425269 fix: Filters alert width (#24801)
(cherry picked from commit 4b1f1d49d6)
2023-07-26 16:50:57 -03:00
JUST.in DO IT 84488b7c5e fix(sqllab): Replace autocomplete logic by a hook (#24677)
(cherry picked from commit 77505173ae)
2023-07-26 16:50:57 -03:00
Daniel Vaz Gaspar d86ae30958 fix: docs invalid client redirect (#24816)
(cherry picked from commit d1eb9ea982)
2023-07-26 16:50:57 -03:00
Daniel Vaz Gaspar d92f9a73d2 docs: update security policy and add CVE info (#24769)
(cherry picked from commit 165afee55a)
2023-07-26 16:50:57 -03:00
xavier-GitHub76 98ec90938d fix: Department names fixed for CountryMap of France (#23988)
(cherry picked from commit a9c4472d25)
2023-07-26 16:50:57 -03:00
Gyuil Han 25d42538d1 fix(dashboard): increase json_metadata field (#24510)
(cherry picked from commit ff7c1528db)
2023-07-26 16:50:57 -03:00
Stepan a85da81815 fix(chore): switching between languages should be without timeout 60 seconds (#24798)
(cherry picked from commit 761fec2574)
2023-07-26 16:50:57 -03:00
JUST.in DO IT d95241df9d fix(sqllab): Replace margin style by gap on query results (#24772)
(cherry picked from commit 4a81284056)
2023-07-25 15:24:40 -03:00
Michael S. Molina d336431559 fix: Select onChange is being fired without explicit selection (#24698)
Co-authored-by: JUST.in DO IT <justin.park@airbnb.com>
(cherry picked from commit 6089b5fdae)
2023-07-25 15:24:40 -03:00
Zef Lin e3d8ecb478 chore(view_api): return application/json as content-type for api/v1/form_data endpoint (#24758)
(cherry picked from commit 0631a8086c)
2023-07-24 09:47:23 -03:00
Elizabeth Thompson a066ebbb5e fix: update svgr webpack config to use svg dimensions (#24747) 2023-07-24 09:47:04 -03:00
Beto Dealmeida 6947983d5f feat: migrate charts on import (#24703)
(cherry picked from commit abb8e28e49)
2023-07-24 09:43:52 -03:00
Evan Rusackas 00516c2088 chore(docs): Adding link to additional visualization content (#24759)
(cherry picked from commit e210da963d)
2023-07-24 09:43:52 -03:00
mattitoo 2e978bba14 docs: Fix STANDARD_ROLES.md for Gamma role permissions on tableschemaview (#24766)
(cherry picked from commit d642288092)
2023-07-24 09:43:52 -03:00
karsten-wagner a8f808f7a8 fix(docs): respect no_proxy environment variable (#23816)
(cherry picked from commit a80ec15f4c)
2023-07-24 09:43:52 -03:00
Michael S. Molina dd002a8cf6 fix: Tooltip no longer highlights hovered data series (#24756)
(cherry picked from commit ac19f58cf6)
2023-07-24 09:43:52 -03:00
Maxime Beauchemin 157cc621c5 fix: column/metric type icons look too small (#24740)
(cherry picked from commit 341b8d41c5)
2023-07-24 09:43:52 -03:00
Maxime Beauchemin 299dbb7012 chore: make control panel sub sections look better (#24736)
(cherry picked from commit 22a0fe555c)
2023-07-24 09:43:51 -03:00
Beto Dealmeida 2e222865bf fix: search_path in RDS (#24739)
(cherry picked from commit 7675e0db10)
2023-07-24 09:43:51 -03:00
JUST.in DO IT d87724a6e3 fix(datasets): Replace left panel layout by TableSelector (#24599)
Co-authored-by: Justin Park <justinpark@apache.org>
(cherry picked from commit b2831b419e)
2023-07-24 09:43:51 -03:00
Stepan 6281f50bab fix(pvt2): migrations from legacy pivot table error when form_data have pieces of pvt2 (#24710)
(cherry picked from commit df106aa708)
2023-07-24 09:43:51 -03:00
JUST.in DO IT 06258a06ed chore(native filters): Expandable filter config modal (#24559)
Co-authored-by: Justin Park <justinpark@apache.org>
(cherry picked from commit 05e724f3d7)
2023-07-24 09:43:51 -03:00
Michael S. Molina a7221c6dd4 fix: Dashboard time grain in Table (#24746)
(cherry picked from commit 317aa989c2)
2023-07-24 09:43:51 -03:00
JUST.in DO IT 66792beb8b fix(native filter): clean deleted parent filter ids (#24749)
Co-authored-by: John Bodley <john.bodley@gmail.com>
(cherry picked from commit 4086514fa5)
2023-07-24 09:43:51 -03:00
Maxime Beauchemin aa5f0b1019 chore: make antd table font size same as data table (#24741)
(cherry picked from commit d1d5ff6f9f)
2023-07-24 09:43:51 -03:00
Daniel Vaz Gaspar cc7983cd4a fix: import database engine validation (#24697)
(cherry picked from commit cb9b865a53)
2023-07-20 12:00:26 -03:00
Elizabeth Thompson 1625fe4104 chore: turn off talisman for ephemeral environments in ci (#24627)
(cherry picked from commit 1a9724582f)
2023-07-20 12:00:26 -03:00
91919569ba feat: use Scarf Gateway for Superset npm downloads (#24433)
Co-authored-by: Arjun Devarajan <arjun.devarajan@scarf.sh>
Co-authored-by: Evan Rusackas <evan@preset.io>
Co-authored-by: Evan Rusackas <evan@rusackas.com>
2023-07-20 12:00:04 -03:00
Stepan befe41df14 fix(range-slider): removed localization of metric key (#24716)
(cherry picked from commit 2d58dddbdc)
2023-07-20 11:55:54 -03:00
John Bodley a3138f2bbe fix(cache): Add cache warmup for non-legacy charts (#24671)
(cherry picked from commit 5f49e0fdd0)
2023-07-20 11:55:54 -03:00
John Bodley 82311edaf6 fix(druid): Delete obsolete Druid NoSQL slice parameters (#24737)
(cherry picked from commit 4c5ada421c)
2023-07-20 11:55:53 -03:00
Arjun Devarajan 757741942d feat: use Scarf Gateway for Superset helm charts/Docker compose downloads (#24432) 2023-07-19 10:56:54 -03:00
John Bodley 65822cab13 chore: Bump pyyaml bounds (#24731) 2023-07-19 10:53:05 -03:00
John Bodley 4390968dd7 chore: Remove obsolete legacy visualizations (#24694)
(cherry picked from commit 1b5a6790f0)
2023-07-19 10:50:30 -03:00
Lily Kuang 2e913492c9 fix: color collision in dashboard with tabs (#24670)
(cherry picked from commit 0328dd2704)
2023-07-19 10:50:30 -03:00
Vitor Avila c94bee47aa fix(dataset-import): support empty strings for extra fields (#24663)
(cherry picked from commit 65fb8e10ba)
2023-07-19 10:50:30 -03:00
Daniel Vaz Gaspar c6ecbc81ac chore: move deprecated default version to 4.0.0 (#24689)
(cherry picked from commit ca68250adf)
2023-07-19 10:50:30 -03:00
Beto Dealmeida 60385ed059 fix: embedded dashboard check (#24690)
(cherry picked from commit 9844b15e07)
2023-07-19 10:50:30 -03:00
Michael S. Molina b0c90cb261 fix: Dashboard time grain in Pivot Table (#24665)
(cherry picked from commit 6e59f11f4c)
2023-07-19 10:50:30 -03:00
John Bodley 49605e763c chore(command): Condense delete/bulk-delete operations (#24607)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
(cherry picked from commit a156816064)
2023-07-13 13:36:20 -03:00
Arkadii Yakovets 7243332082 chore: Bump holidays to 0.28 (#24647)
(cherry picked from commit 767afef398)
2023-07-13 13:36:20 -03:00
John Bodley 1872a651da chore: Bump/relax Flask libs (#24652)
(cherry picked from commit 462418ba82)
2023-07-12 11:20:05 -03:00
JUST.in DO IT f60ab456f3 feat(sqllab): add shortcut for run current sql (#24329)
Co-authored-by: Justin Park <justinpark@apache.org>
(cherry picked from commit 1473d97055)
2023-07-12 11:20:05 -03:00
John Bodley 96cf6406f1 chore(dao): Add explicit ON DELETE CASCADE for ownership (#24628) 2023-07-12 11:19:43 -03:00
John Bodley 71eff954ba chore(dao): Condense delete/bulk-delete operations (#24466)
Co-authored-by: Michael S. Molina <70410625+michael-s-molina@users.noreply.github.com>
(cherry picked from commit 7409289762)
2023-07-12 11:17:04 -03:00
JUST.in DO IT 79a42c6509 fix(sqllab): missing column meta on autocomplete (#24611)
Co-authored-by: Justin Park <justinpark@apache.org>
(cherry picked from commit ca8c8d2ad3)
2023-07-12 11:03:47 -03:00
Jack Fragassi d8bf955515 fix: Chart can be added to dashboard by non-owner via save as option (#24630)
(cherry picked from commit 4caf33b41d)
2023-07-12 11:03:47 -03:00
Kamil Gabryjelski a2fdc84073 fix: Ensure config is a dict in cross filters scope migration (#24661)
(cherry picked from commit 2e4d9f2e2a)
2023-07-12 11:03:47 -03:00
Kamil Gabryjelski 5a4996c226 fix: Incorrect initial global scoping of cross filters (#24642)
(cherry picked from commit bbffc4c1f8)
2023-07-12 11:03:46 -03:00
Michael S. Molina 359bbe26f7 fix: Validation of out of scope filters and interaction with Clear All (#24610)
(cherry picked from commit 0efb88479e)
2023-07-12 11:03:46 -03:00
John Bodley 27dcc3e025 chore: Drop rouge constraints and tables (#24629)
(cherry picked from commit 65291a00c6)
2023-07-12 11:03:46 -03:00
Daniel Vaz Gaspar d6296c1ad1 fix: CSP, bump FAB to 4.3.4 (#24646)
(cherry picked from commit 4881328fbf)
2023-07-12 11:03:46 -03:00
Hugh A. Miles II 846d3c03aa fix: fix extra insert for count on dataset creation (#24625)
(cherry picked from commit e6e8276a20)
2023-07-12 11:03:46 -03:00
Kamil Gabryjelski 2bd9ca3c94 fix: Ensure metrics is an array in Mixed Chart (#24643)
(cherry picked from commit fe2c14ff3a)
2023-07-12 11:03:46 -03:00
Daniel Vaz Gaspar a15e809c0f fix: remove spaces and EOL from nonce attr (#24644)
(cherry picked from commit b809815436)
2023-07-12 11:03:46 -03:00
Michael S. Molina b81ee82d3c fix: Wrong positioning of filter tooltips on scroll (#24617)
(cherry picked from commit c53b249998)
2023-07-10 10:27:25 -03:00
Kamil Gabryjelski 6b366a2cff feat: Implement support for currencies in more charts (#24594)
(cherry picked from commit d74d7eca23)
2023-07-10 10:27:25 -03:00
Hugh A. Miles II 9a96d8cf8b fix: change naming convention for count metric on Dataset creation (#24609)
(cherry picked from commit c573cfcd12)
2023-07-10 10:27:25 -03:00
Gyuil Han 744cf2ec29 chore: remove underscore from builtin_time_grains label (#24567)
(cherry picked from commit a6e749da87)
2023-07-10 10:27:25 -03:00
Michael S. Molina 26909bf517 fix: Incorrect dependency between filters related feature flags (#24608)
(cherry picked from commit 781a20423a)
2023-07-07 09:48:45 -03:00
Beto Dealmeida 967ca0470b fix(report): edit without custom width (#24612)
(cherry picked from commit 93ba4ade09)
2023-07-07 09:48:45 -03:00
Hugh A. Miles II 0976b5e51e fix(ssh): Editting Database w/ SSH Tunneling (#24552)
(cherry picked from commit d041648ad4)
2023-07-07 09:48:45 -03:00
jwilliams-ocient 67008f2c35 fix: set max version for pyocient (#24556)
(cherry picked from commit 64d728f80f)
2023-07-07 09:48:45 -03:00
John Bodley af2689d501 fix: Bump prophet, re-enable tests, and remedy column eligibility logic (#24129)
(cherry picked from commit 383dac6a5e)
2023-07-07 09:48:45 -03:00
JUST.in DO IT a5f3cfc726 chore(sqllab): Log current local storage usage (#24554)
Co-authored-by: Justin Park <justinpark@apache.org>
(cherry picked from commit 0836000f9f)
2023-07-07 09:48:45 -03:00
Michael S. Molina 9ab61c1103 fix: Clicking on a tag in the dashboard and charts lists does not work (#24593)
(cherry picked from commit 04ae259f3d)
2023-07-05 13:02:50 -03:00
Ville Brofeldt 6de6f47728 chore(metastore-cache): add codec support (#24586)
(cherry picked from commit a4880cabd4)
2023-07-05 11:50:50 -03:00
Hugh A. Miles II 1b41653d3b fix: SSH Tunnel creation with dynamic form (#24196)
(cherry picked from commit 226c7f807d)
2023-07-05 11:50:50 -03:00
Michael S. Molina eb42fa7a2e chore: Updates CHANGELOG.md and UPDATING.md 2023-06-30 09:27:24 -03:00
11453 changed files with 701025 additions and 1967194 deletions
+11 -35
View File
@@ -17,16 +17,7 @@
# https://cwiki.apache.org/confluence/display/INFRA/.asf.yaml+features+for+git+repositories
---
notifications:
commits: commits@superset.apache.org
issues: notifications@superset.apache.org
pullrequests: notifications@superset.apache.org
discussions: notifications@superset.apache.org
github:
pull_requests:
del_branch_on_merge: true
allow_update_branch: true
description: "Apache Superset is a Data Visualization and Data Exploration Platform"
homepage: https://superset.apache.org/
labels:
@@ -56,39 +47,31 @@ github:
projects: true
# Enable wiki for documentation
wiki: true
# Enable discussions
discussions: true
enabled_merge_buttons:
squash: true
merge: false
rebase: false
ghp_branch: gh-pages
ghp_path: /
protected_branches:
master:
required_status_checks:
# strict means "Require branches to be up to date before merging".
strict: false
# contexts are the names of checks that must pass
# unfortunately AFAICT for `matrix:` jobs, we have to itemize every
# combination here.
contexts:
- lint-check
- cypress-matrix-required
- dependency-review
- enforce-single-migration-head
- check
- cypress-matrix (1, chrome)
- cypress-matrix (2, chrome)
- cypress-matrix (3, chrome)
- docker-build
- frontend-build
- playwright-tests-required
- pre-commit (current)
- test-mysql
- test-postgres-required
- test-postgres-hive
- test-postgres-presto
- test-sqlite
- unit-tests-required
- pre-commit (3.9)
- python-lint (3.9)
- test-mysql (3.9)
- test-postgres (3.9)
- test-postgres (3.10)
- test-sqlite (3.9)
required_pull_request_reviews:
dismiss_stale_reviews: false
@@ -96,10 +79,3 @@ github:
required_approving_review_count: 1
required_signatures: false
gh-pages:
required_pull_request_reviews:
dismiss_stale_reviews: false
require_code_owner_reviews: true
required_approving_review_count: 1
required_signatures: false
-10
View File
@@ -1,10 +0,0 @@
# JavaScript to TypeScript Migration Command
## Usage
```
/js-to-ts <core-filename>
```
- `<core-filename>` - Path to CORE file relative to `superset-frontend/` (e.g., `src/utils/common.js`, `src/middleware/loggerMiddleware.js`)
## Agent Instructions
**See:** [../projects/js-to-ts/AGENT.md](../projects/js-to-ts/AGENT.md) for complete migration guide.
-684
View File
@@ -1,684 +0,0 @@
# JavaScript to TypeScript Migration Agent Guide
**Complete technical reference for converting JavaScript/JSX files to TypeScript/TSX in Apache Superset frontend.**
**Agent Role:** Atomic migration unit - migrate the core file + ALL related tests/mocks as one cohesive unit. Use `git mv` to preserve history, NO `git commit`. NO global import changes. Report results upon completion.
---
## 🎯 Migration Principles
1. **Atomic migration units** - Core file + all related tests/mocks migrate together
2. **Zero `any` types** - Use proper TypeScript throughout
3. **Leverage existing types** - Reuse established definitions
4. **Type inheritance** - Derivatives extend base component types
5. **Strategic placement** - File types for maximum discoverability
6. **Surgical improvements** - Enhance existing types during migration
---
## Step 0: Dependency Check (MANDATORY)
**Command:**
```bash
grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" superset-frontend/{filename}
```
**Decision:**
- ✅ No matches → Proceed with atomic migration (core + tests + mocks)
- ❌ Matches found → EXIT with dependency report (see format below)
---
## Step 1: Identify Related Files (REQUIRED)
**Atomic Migration Scope:**
For core file `src/utils/example.js`, also migrate:
- `src/utils/example.test.js` / `src/utils/example.test.jsx`
- `src/utils/example.spec.js` / `src/utils/example.spec.jsx`
- `src/utils/__mocks__/example.js`
- Any other related test/mock files found by pattern matching
**Find all related test and mock files:**
```bash
# Pattern-based search for related files
basename=$(basename {filename} .js)
dirname=$(dirname superset-frontend/{filename})
# Find test files
find "$dirname" -name "${basename}.test.js" -o -name "${basename}.test.jsx"
find "$dirname" -name "${basename}.spec.js" -o -name "${basename}.spec.jsx"
# Find mock files
find "$dirname" -name "__mocks__/${basename}.js"
find "$dirname" -name "${basename}.mock.js"
```
**Migration Requirement:** All discovered related files MUST be migrated together as one atomic unit.
**Test File Creation:** If NO test files exist for the core file, CREATE a minimal test file using the following pattern:
- Location: Same directory as core file
- Name: `{basename}.test.ts` (e.g., `DebouncedMessageQueue.test.ts`)
- Content: Basic test structure importing and testing the main functionality
- Use proper TypeScript types in test file
---
## 🗺️ Type Reference Map
### From `@superset-ui/core`
```typescript
// Data & Query
QueryFormData, QueryData, JsonObject, AnnotationData, AdhocMetric
LatestQueryFormData, GenericDataType, DatasourceType, ExtraFormData
DataMaskStateWithId, NativeFilterScope, NativeFiltersState, NativeFilterTarget
// UI & Theme
FeatureFlagMap, LanguagePack, ColorSchemeConfig, SequentialSchemeConfig
```
### From `@superset-ui/chart-controls`
```typescript
Dataset, ColumnMeta, ControlStateMapping
```
### From Local Types (`src/types/`)
```typescript
// Authentication
User, UserWithPermissionsAndRoles, BootstrapUser, PermissionsAndRoles
// Dashboard
Dashboard, DashboardState, DashboardInfo, DashboardLayout, LayoutItem
ComponentType, ChartConfiguration, ActiveFilters
// Charts
Chart, ChartState, ChartStatus, ChartLinkedDashboard, Slice, SaveActionType
// Data
Datasource, Database, Owner, Role
// UI Components
TagType, FavoriteStatus, Filter, ImportResourceName
```
### From Domain Types
```typescript
// src/dashboard/types.ts
RootState, ChartsState, DatasourcesState, FilterBarOrientation
ChartCrossFiltersConfig, ActiveTabs, MenuKeys
// src/explore/types.ts
ExplorePageInitialData, ExplorePageState, ExploreResponsePayload, OptionSortType
// src/SqlLab/types.ts
[SQL Lab specific types]
```
---
## 🏗️ Type Organization Strategy
### Type Placement Hierarchy
1. **Component-Colocated** (90% of cases)
```typescript
// Same file as component
interface MyComponentProps {
title: string;
onClick: () => void;
}
```
2. **Feature-Shared**
```typescript
// src/[domain]/components/[Feature]/types.ts
export interface FilterConfiguration {
filterId: string;
targets: NativeFilterTarget[];
}
```
3. **Domain-Wide**
```typescript
// src/[domain]/types.ts
export interface ExploreFormData extends QueryFormData {
viz_type: string;
}
```
4. **Global**
```typescript
// src/types/[TypeName].ts
export interface ApiResponse<T> {
result: T;
count?: number;
}
```
### Type Discovery Commands
```bash
# Search existing types before creating
find superset-frontend/src -name "types.ts" -exec grep -l "[TypeConcept]" {} \;
grep -r "interface.*Props\|type.*Props" superset-frontend/src/
```
### Derivative Component Patterns
**Rule:** Components that extend others should extend their type interfaces.
```typescript
// ✅ Base component type
interface SelectProps {
value: string | number;
options: SelectOption[];
onChange: (value: string | number) => void;
disabled?: boolean;
}
// ✅ Derivative extends base
interface ChartSelectProps extends SelectProps {
charts: Chart[];
onChartSelect: (chart: Chart) => void;
}
// ✅ Derivative with modified props
interface DatabaseSelectProps extends Omit<SelectProps, 'value' | 'onChange'> {
value: number; // Narrowed type
onChange: (databaseId: number) => void; // Specific signature
}
```
**Common Patterns:**
- **Extension:** `extends BaseProps` - adds new props
- **Omission:** `Omit<BaseProps, 'prop'>` - removes props
- **Modification:** `Omit<BaseProps, 'prop'> & { prop: NewType }` - changes prop type
- **Restriction:** Override with narrower types (union → specific)
---
## 📋 Migration Recipe
### Step 2: File Conversion
```bash
# Use git mv to preserve history
git mv component.js component.ts
git mv Component.jsx Component.tsx
```
### Step 3: Import & Type Setup
```typescript
// Import order (enforced by linting)
import { FC, ReactNode } from 'react';
import { JsonObject, QueryFormData } from '@superset-ui/core';
import { Dataset } from '@superset-ui/chart-controls';
import type { Dashboard } from 'src/types/Dashboard';
```
### Step 4: Function & Component Typing
```typescript
// Functions with proper parameter/return types
export function processData(
data: Dataset[],
config: JsonObject
): ProcessedData[] {
// implementation
}
// Component props with inheritance
interface ComponentProps extends BaseProps {
data: Chart[];
onSelect: (id: number) => void;
}
const Component: FC<ComponentProps> = ({ data, onSelect }) => {
// implementation
};
```
### Step 5: State & Redux Typing
```typescript
// Hooks with specific types
const [data, setData] = useState<Chart[]>([]);
const [selected, setSelected] = useState<number | null>(null);
// Redux with existing RootState
const mapStateToProps = (state: RootState) => ({
charts: state.charts,
user: state.user,
});
```
---
## 🧠 Type Debugging Strategies (Real-World Learnings)
### The Evolution of Type Approaches
When you hit type errors, follow this debugging evolution:
#### 1. ❌ Idealized Union Types (First Attempt)
```typescript
// Looks clean but doesn't match reality
type DatasourceInput = Datasource | QueryEditor;
```
**Problem**: Real calling sites pass variations, not exact types.
#### 2. ❌ Overly Precise Types (Second Attempt)
```typescript
// Tried to match exact calling signatures
type DatasourceInput =
| IDatasource // From DatasourcePanel
| (QueryEditor & { columns: ColumnMeta[] }); // From SaveQuery
```
**Problem**: Too rigid, doesn't handle legacy variations.
#### 3. ✅ Flexible Interface (Final Solution)
```typescript
// Captures what the function actually needs
interface DatasourceInput {
name?: string | null; // Allow null for compatibility
datasource_name?: string | null; // Legacy variations
columns?: any[]; // Multiple column types accepted
database?: { id?: number };
// ... other optional properties
}
```
**Success**: Works with all calling sites, focuses on function needs.
### Type Debugging Process
1. **Start with compilation errors** - they show exact mismatches
2. **Examine actual usage** - look at calling sites, not idealized types
3. **Build flexible interfaces** - capture what functions need, not rigid contracts
4. **Iterate based on downstream validation** - let calling sites guide your types
---
## 🚨 Anti-Patterns to Avoid
```typescript
// ❌ Never use any
const obj: any = {};
// ✅ Use proper types
const obj: Record<string, JsonObject> = {};
// ❌ Don't recreate base component props
interface ChartSelectProps {
value: string; // Duplicated from SelectProps
onChange: () => void; // Duplicated from SelectProps
charts: Chart[]; // New prop
}
// ✅ Inherit and extend
interface ChartSelectProps extends SelectProps {
charts: Chart[]; // Only new props
}
// ❌ Don't create ad-hoc type variations
interface UserInfo {
name: string;
email: string;
}
// ✅ Extend existing types (DRY principle)
import { User } from 'src/types/bootstrapTypes';
type UserDisplayInfo = Pick<User, 'firstName' | 'lastName' | 'email'>;
// ❌ Don't create overly rigid unions
type StrictInput = ExactTypeA | ExactTypeB;
// ✅ Create flexible interfaces for function parameters
interface FlexibleInput {
// Focus on what the function actually needs
commonProperty: string;
optionalVariations?: any; // Allow for legacy variations
}
```
## 📍 DRY Type Guidelines (WHERE TYPES BELONG)
### Type Placement Rules
**CRITICAL**: Type variations must live close to where they belong, not scattered across files.
#### ✅ Proper Type Organization
```typescript
// ❌ Don't create one-off interfaces in utility files
// src/utils/datasourceUtils.ts
interface DatasourceInput { /* custom interface */ } // Wrong!
// ✅ Use existing types or extend them in their proper domain
// src/utils/datasourceUtils.ts
import { IDatasource } from 'src/explore/components/DatasourcePanel';
import { QueryEditor } from 'src/SqlLab/types';
// Create flexible interface that references existing types
interface FlexibleDatasourceInput {
// Properties that actually exist across variations
}
```
#### Type Location Hierarchy
1. **Domain Types**: `src/{domain}/types.ts` (dashboard, explore, SqlLab)
2. **Component Types**: Co-located with components
3. **Global Types**: `src/types/` directory
4. **Utility Types**: Only when they truly don't belong elsewhere
#### ✅ DRY Type Patterns
```typescript
// ✅ Extend existing domain types
interface SaveQueryData extends Pick<QueryEditor, 'sql' | 'dbId' | 'catalog'> {
columns: ColumnMeta[]; // Add what's needed
}
// ✅ Create flexible interfaces for cross-domain utilities
interface CrossDomainInput {
// Common properties that exist across different source types
name?: string | null; // Accommodate legacy null values
// Only include properties the function actually uses
}
```
---
## 🎯 PropTypes Auto-Generation (Elegant Approach)
**IMPORTANT**: Superset has `babel-plugin-typescript-to-proptypes` configured to automatically generate PropTypes from TypeScript interfaces. Use this instead of manual PropTypes duplication!
### ❌ Manual PropTypes Duplication (Avoid This)
```typescript
export interface MyComponentProps {
title: string;
count?: number;
}
// 8+ lines of manual PropTypes duplication 😱
const propTypes = PropTypes.shape({
title: PropTypes.string.isRequired,
count: PropTypes.number,
});
export default propTypes;
```
### ✅ Auto-Generated PropTypes (Use This)
```typescript
import { InferProps } from 'prop-types';
export interface MyComponentProps {
title: string;
count?: number;
}
// Single validator function - babel plugin auto-generates PropTypes! ✨
export default function MyComponentValidator(props: MyComponentProps) {
return null; // PropTypes auto-assigned by babel-plugin-typescript-to-proptypes
}
// Optional: For consumers needing PropTypes type inference
export type MyComponentPropsInferred = InferProps<typeof MyComponentValidator>;
```
### Migration Pattern for Type-Only Files
**When migrating type-only files with manual PropTypes:**
1. **Keep the TypeScript interfaces** (single source of truth)
2. **Replace manual PropTypes** with validator function
3. **Remove PropTypes imports** and manual shape definitions
4. **Add InferProps import** if type inference needed
**Example Migration:**
```typescript
// Before: 25+ lines with manual PropTypes duplication
export interface AdhocFilterType { /* ... */ }
const adhocFilterTypePropTypes = PropTypes.oneOfType([...]);
// After: 3 lines with auto-generation
export interface AdhocFilterType { /* ... */ }
export default function AdhocFilterValidator(props: { filter: AdhocFilterType }) {
return null; // Auto-generated PropTypes by babel plugin
}
```
### Component PropTypes Pattern
**For React components, the babel plugin works automatically:**
```typescript
interface ComponentProps {
title: string;
onClick: () => void;
}
const MyComponent: FC<ComponentProps> = ({ title, onClick }) => {
// Component implementation
};
// PropTypes automatically generated by babel plugin - no manual work needed!
export default MyComponent;
```
### Auto-Generation Benefits
- ✅ **Single source of truth**: TypeScript interfaces drive PropTypes
- ✅ **No duplication**: Eliminate 15-20 lines of manual PropTypes code
- ✅ **Automatic updates**: Changes to TypeScript automatically update PropTypes
- ✅ **Type safety**: Compile-time checking ensures PropTypes match interfaces
- ✅ **Backward compatibility**: Existing JavaScript components continue working
### Babel Plugin Configuration
The plugin is already configured in `babel.config.js`:
```javascript
['babel-plugin-typescript-to-proptypes', { loose: true }]
```
**No additional setup required** - just use TypeScript interfaces and the plugin handles the rest!
---
## 🧪 Test File Migration Patterns
### Test File Priority
- **Always migrate test files** alongside production files
- **Test files are often leaf nodes** - good starting candidates
- **Create tests if missing** - Leverage new TypeScript types for better test coverage
### Test-Specific Type Patterns
```typescript
// Mock interfaces for testing
interface MockStore {
getState: () => Partial<RootState>; // Partial allows minimal mocking
}
// Type-safe mocking for complex objects
const mockDashboardInfo: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
};
// Sinon stub typing
let postStub: sinon.SinonStub;
beforeEach(() => {
postStub = sinon.stub(SupersetClient, 'post');
});
// Use stub reference instead of original method
expect(postStub.callCount).toBe(1);
expect(postStub.getCall(0).args[0].endpoint).toMatch('/api/');
```
### Test Migration Recipe
1. **Migrate production file first** (if both need migration)
2. **Update test imports** to point to `.ts/.tsx` files
3. **Add proper mock typing** using `Partial<T> as T` pattern
4. **Fix stub typing** - Use stub references, not original methods
5. **Verify all tests pass** with TypeScript compilation
---
## 🔧 Type Conflict Resolution
### Multiple Type Definitions Issue
**Problem**: Same type name defined in multiple files causes compilation errors.
**Example**: `DashboardInfo` defined in both:
- `src/dashboard/reducers/types.ts` (minimal)
- `src/dashboard/components/Header/types.ts` (different shape)
- `src/dashboard/types.ts` (complete - used by RootState)
### Resolution Strategy
1. **Identify the authoritative type**:
```bash
# Find which type is used by RootState/main interfaces
grep -r "DashboardInfo" src/dashboard/types.ts
```
2. **Use import from authoritative source**:
```typescript
// ✅ Import from main domain types
import { RootState, DashboardInfo } from 'src/dashboard/types';
// ❌ Don't import from component-specific files
import { DashboardInfo } from 'src/dashboard/components/Header/types';
```
3. **Mock complex types in tests**:
```typescript
// For testing - provide minimal required fields
const mockInfo: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
// Only provide fields actually used in test
};
```
### Type Hierarchy Discovery Commands
```bash
# Find all definitions of a type
grep -r "interface.*TypeName\|type.*TypeName" src/
# Find import usage patterns
grep -r "import.*TypeName" src/
# Check what RootState uses
grep -A 10 -B 10 "TypeName" src/*/types.ts
```
---
## Agent Constraints (CRITICAL)
1. **Use git mv** - Run `git mv file.js file.ts` to preserve git history, but NO `git commit`
2. **NO global import changes** - Don't update imports across codebase
3. **Type files OK** - Can modify existing type files to improve/align types
4. **Single-File TypeScript Validation** (CRITICAL) - tsc has known issues with multi-file compilation:
- **Core Issue**: TypeScript's `tsc` has documented problems validating multiple files simultaneously in complex projects
- **Solution**: ALWAYS validate files one at a time using individual `tsc` calls
- **Command Pattern**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {single-file-path}`
- **Why**: Multi-file validation can produce false positives, miss real errors, and conflict during parallel agent execution
5. **Downstream Impact Validation** (CRITICAL) - Your migration affects calling sites:
- **Find downstream files**: `find superset-frontend/src -name "*.tsx" -o -name "*.ts" | xargs grep -l "your-core-filename" 2>/dev/null || echo "No files found"`
- **Validate each downstream file individually**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}`
- **Fix type mismatches** you introduced in calling sites
- **NEVER ignore downstream errors** - they indicate your types don't match reality
6. **Avoid Project-Wide Validation During Migration**:
- **NEVER use `npm run type`** during parallel agent execution - produces unreliable results
- **Single-file validation is authoritative** - trust individual file checks over project-wide scans
6. **ESLint validation** - Run `npm run eslint -- --fix {file}` for each migrated file to auto-fix formatting/linting issues
6. Zero `any` types - use proper TypeScript types
7. Search existing types before creating new ones
8. Follow patterns from this guide
---
## Success Report Format
```
SUCCESS: Atomic Migration of {core-filename}
## Files Migrated (Atomic Unit)
- Core: {core-filename} → {core-filename.ts/tsx}
- Tests: {list-of-test-files} → {list-of-test-files.ts/tsx} OR "CREATED: {basename}.test.ts"
- Mocks: {list-of-mock-files} → {list-of-mock-files.ts}
- Type files modified: {list-of-type-files}
## Types Created/Improved
- {TypeName}: {location} ({scope}) - {rationale}
- {ExistingType}: enhanced in {location} - {improvement-description}
## Documentation Recommendations
- ADD_TO_DIRECTORY: {TypeName} - {reason}
- NO_DOCUMENTATION: {TypeName} - {reason}
## Quality Validation
- **Single-File TypeScript Validation**: ✅ PASS - Core files individually validated
- Core file: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {core-file}`
- Test files: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {test-file}` (if exists)
- **Downstream Impact Check**: ✅ PASS - Found {N} files importing this module, all validate individually
- Downstream files: {list-of-files-that-import-your-module}
- Individual validation: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}`
- **ESLint validation**: ✅ PASS (using `npm run eslint -- --fix {files}` to auto-fix formatting)
- **Zero any types**: ✅ PASS
- **Local imports resolved**: ✅ PASS
- **Functionality preserved**: ✅ PASS
- **Tests pass** (if test file): ✅ PASS
- **Follow-up action required**: {YES/NO}
## Validation Strategy Notes
- **Single-file approach used**: Avoided multi-file tsc validation due to known TypeScript compilation issues
- **Project-wide validation skipped**: `npm run type` not used during parallel migration to prevent false positives
## Migration Learnings
- Type conflicts encountered: {describe any multiple type definitions}
- Mock patterns used: {describe test mocking approaches}
- Import hierarchy decisions: {note authoritative type sources used}
- PropTypes strategy: {AUTO_GENERATED via babel plugin | MANUAL_DUPLICATION_REMOVED | N/A}
## Improvement Suggestions for Documentation
- AGENT.md enhancement: {suggest additions to migration guide}
- Common pattern identified: {note reusable patterns for future migrations}
```
---
## Dependency Block Report Format
```
DEPENDENCY_BLOCK: Cannot migrate {filename}
## Blocking Dependencies
- {path}: {type} - {usage} - {priority}
## Impact Analysis
- Estimated types: {number}
- Expected locations: {list}
- Cross-domain: {YES/NO}
## Recommended Order
{ordered-list}
```
---
## 📚 Quick Reference
**Type Utilities:**
- `Record<K, V>` - Object with specific key/value types
- `Partial<T>` - All properties optional
- `Pick<T, K>` - Subset of properties
- `Omit<T, K>` - Exclude specific properties
- `NonNullable<T>` - Exclude null/undefined
**Event Types:**
- `MouseEvent<HTMLButtonElement>`
- `ChangeEvent<HTMLInputElement>`
- `FormEvent<HTMLFormElement>`
**React Types:**
- `FC<Props>` - Functional component
- `ReactNode` - Any renderable content
- `CSSProperties` - Style objects
---
**Remember:** Every type should add value and clarity. The goal is meaningful type safety that catches bugs and improves developer experience.
-199
View File
@@ -1,199 +0,0 @@
# JS-to-TS Coordinator Workflow
**Role:** Strategic migration coordination - select leaf-node files, trigger agents, review results, handle integration, manage dependencies.
---
## 1. Core File Selection Strategy
**Target ONLY Core Files**: Coordinators identify core files (production code), agents handle related tests/mocks atomically.
**File Analysis Commands**:
```bash
# Find CORE files with no JS/JSX dependencies (exclude tests/mocks) - SIZE PRIORITIZED
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | xargs wc -l | sort -n | head -20
# Alternative: Get file sizes in lines with paths
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | while read file; do
lines=$(wc -l < "$file")
echo "$lines $file"
done | sort -n | head -20
# Check dependencies for core files only (start with smallest)
for file in <core-files-sorted-by-size>; do
echo "=== $file ($(wc -l < "$file") lines) ==="
grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" || echo "✅ LEAF CANDIDATE"
done
# Identify heavily imported files (migrate last)
grep -r "from.*utils/common" superset-frontend/src/ | wc -l
# Quick leaf analysis with size priority
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | head -30 | while read file; do
deps=$(grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" | wc -l)
lines=$(wc -l < "$file")
if [ "$deps" -eq 0 ]; then
echo "✅ LEAF: $lines lines - $file"
fi
done | sort -n
```
**Priority Order** (Smallest files first for easier wins):
1. **Small leaf files** (<50 lines) - No JS/JSX imports, quick TypeScript conversion
2. **Medium leaf files** (50-200 lines) - Self-contained utilities and helpers
3. **Small dependency files** (<100 lines) - Import only already-migrated files
4. **Larger components** (200+ lines) - Complex but well-contained functionality
5. **Core foundational files** (utils/common.js, controls.jsx) - migrate last regardless of size
**Size-First Benefits**:
- Faster completion builds momentum
- Earlier validation of migration patterns
- Easier rollback if issues arise
- Better success rate for agent learning
**Migration Unit**: Each agent call migrates:
- 1 core file (primary target)
- All related `*.test.js/jsx` files
- All related `*.mock.js` files
- All related `__mocks__/` files
---
## 2. Task Creation & Agent Control
### Task Triggering
When triggering the `/js-to-ts` command:
- **Task Title**: Use the core filename as the task title (e.g., "DebouncedMessageQueue.js migration", "hostNamesConfig.js migration")
- **Task Description**: Include the full relative path to help agent locate the file
- **Reference**: Point agent to [AGENT.md](./AGENT.md) for technical instructions
### Post-Processing Workflow
After each agent completes:
1. **Review Agent Report**: Always read and analyze the complete agent report
2. **Share Summary**: Provide user with key highlights from agent's work:
- Files migrated (core + tests/mocks)
- Types created or improved
- Any validation issues or coordinator actions needed
3. **Quality Assessment**: Evaluate agent's TypeScript implementation against criteria:
-**Type Usage**: Proper types used, no `any` types
-**Type Filing**: Types placed in correct hierarchy (component → feature → domain → global)
-**Side Effects**: No unintended changes to other files
-**Import Alignment**: Proper .ts/.tsx import extensions
4. **Integration Decision**:
- **COMMIT**: If agent work is complete and high quality
- **FIX & COMMIT**: If minor issues need coordinator fixes
- **ROLLBACK**: If major issues require complete rework
5. **Next Action**: Ask user preference - commit this work or trigger next migration
---
## 3. Integration Decision Framework
**Automatic Integration** ✅:
- `npm run type` passes without errors
- Agent created clean TypeScript with proper types
- Types appropriately filed in hierarchy
**Coordinator Integration** (Fix Side-Effects) 🔧:
- `npm run type` fails BUT agent's work is high quality
- Good type usage, proper patterns, well-organized
- Side-effects are manageable TypeScript compilation errors
- **Coordinator Action**: Integrate the change, then fix global compilation issues
**Rollback Only** ❌:
- Agent introduced `any` types or poor type choices
- Types poorly organized or conflicting with existing patterns
- Fundamental approach issues requiring complete rework
**Integration Process**:
1. **Review**: Agent already used `git mv` to preserve history
2. **Fix Side-Effects**: Update dependent files with proper import extensions
3. **Resolve Types**: Fix any cascading type issues across codebase
4. **Validate**: Ensure `npm run type` passes after fixes
---
## 4. Common Integration Patterns
**Common Side-Effects (Expect These)**:
- **Type import conflicts**: Multiple definitions of same type name
- **Mock object typing**: Tests need complete type satisfaction
- **Stub method references**: Use stub vars instead of original methods
**Coordinator Fixes (Standard Process)**:
1. **Import Resolution**:
```bash
# Find authoritative type source
grep -r "TypeName" src/*/types.ts
# Import from domain types (src/dashboard/types.ts) not component types
```
2. **Test Mock Completion**:
```typescript
// Use Partial<T> as T pattern for minimal mocking
const mockDashboard: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
};
```
3. **Stub Reference Fixes**:
```typescript
// ✅ Use stub variable
expect(postStub.callCount).toBe(1);
// ❌ Don't use original method
expect(SupersetClient.post.callCount).toBe(1);
```
4. **Validation Commands**:
```bash
npm run type # TypeScript compilation
npm test -- filename # Test functionality
git status # Should show rename, not add/delete
```
---
## 5. File Categories for Planning
### Leaf Files (Start Here)
**Self-contained files with minimal JS/JSX dependencies**:
- Test files (80 files) - Usually only import the file being tested
- Utility files without internal dependencies
- Components importing only external libraries
### Heavily Imported Files (Migrate Last)
**Core files that many others depend on**:
- `utils/common.js` - Core utility functions
- `utils/reducerUtils.js` - Redux helpers
- `@superset-ui/core` equivalent files
- Major state management files (`explore/store.js`, `dashboard/actions/`)
### Complex Components (Middle Priority)
**Large files requiring careful type analysis**:
- `components/Datasource/DatasourceEditor.jsx` (1,809 lines)
- `explore/components/controls/AnnotationLayerControl/AnnotationLayer.jsx` (1,031 lines)
- `explore/components/ExploreViewContainer/index.jsx` (911 lines)
---
## 6. Success Metrics & Continuous Improvement
**Per-File Gates**:
- ✅ `npm run type` passes after each migration
- ✅ Zero `any` types introduced
- ✅ All imports properly typed
- ✅ Types filed in correct hierarchy
**Linear Scheduling**:
When agents report `DEPENDENCY_BLOCK`:
- Queue dependencies in linear order
- Process one file at a time to avoid conflicts
- Handle cascading type changes between files
**After Each Migration**:
1. **Update guides** with new patterns discovered
2. **Document coordinator fixes** that become common
3. **Enhance agent instructions** based on recurring issues
4. **Track success metrics** - automatic vs coordinator integration rates
-76
View File
@@ -1,76 +0,0 @@
# JavaScript to TypeScript Migration Project
Progressive migration of 219 JS/JSX files to TypeScript in Apache Superset frontend.
## 📁 Project Documentation
- **[AGENT.md](./AGENT.md)** - Complete technical migration guide for agents (includes type reference, patterns, validation)
- **[COORDINATOR.md](./COORDINATOR.md)** - Strategic workflow for coordinators (file selection, task management, integration)
## 🎯 Quick Start
**For Agents:** Read [AGENT.md](./AGENT.md) for complete migration instructions
**For Coordinators:** Read [COORDINATOR.md](./COORDINATOR.md) for workflow and [AGENT.md](./AGENT.md) for supervision
**Command:** `/js-to-ts <filename>` - See [../../commands/js-to-ts.md](../../commands/js-to-ts.md)
## 📊 Migration Progress
**Scope**: 219 files total (112 JS + 107 JSX)
- Production files: 139 (63%)
- Test files: 80 (37%)
**Strategy**: Leaf-first migration with dependency-aware coordination
### Completed Migrations ✅
1. **roundDecimal** - `plugins/legacy-plugin-chart-map-box/src/utils/roundDecimal.js`
- Migrated core + test files
- Added proper TypeScript function signature with optional precision parameter
- All tests pass
2. **timeGrainSqlaAnimationOverrides** - `src/explore/controlPanels/timeGrainSqlaAnimationOverrides.js`
- Migrated to TypeScript with ControlPanelState and Dataset types
- Added TimeGrainOverrideState interface for return type
- Used type guards for safe property access
3. **DebouncedMessageQueue** - `src/utils/DebouncedMessageQueue.js`
- Migrated to TypeScript with proper generics
- Created DebouncedMessageQueueOptions interface
- **CREATED test file** with 4 comprehensive test cases
- Excellent class property typing with private/readonly modifiers
**Files Migrated**: 3/219 (1.4%)
**Tests Created**: 2 (roundDecimal had existing, DebouncedMessageQueue created)
### Next Candidates (Leaf Nodes) 🎯
**Identified leaf files with no JS/JSX dependencies:**
- `src/utils/hostNamesConfig.js` - Domain configuration utility
- `src/explore/controlPanels/Separator.js` - Control panel configuration
- `src/middleware/loggerMiddleware.js` - Logging middleware
**Migration Quality**: All completed migrations have:
- ✅ Zero `any` types
- ✅ Proper TypeScript compilation
- ✅ ESLint validation passed
- ✅ Test coverage (created where missing)
---
## 📈 Success Metrics
**Per-File Gates**:
-`npm run type` passes after each migration
- ✅ Zero `any` types introduced
- ✅ All imports properly typed
- ✅ Types filed in correct hierarchy
**Overall Progress**:
- **Automatic Integration Rate**: 100% (3/3 migrations required no coordinator fixes)
- **Test Coverage**: Improved (1 new test file created)
- **Type Safety**: Enhanced with proper interfaces and generics
---
*This is a claudette-managed progressive refactor. All documentation and coordination resources are organized under `.claude/projects/js-to-ts/`*
-15
View File
@@ -1,15 +0,0 @@
{
"hooks": {
"PreToolUse": [
{
"matcher": "Bash",
"hooks": [
{
"type": "command",
"command": "jq -r '.tool_input.command // \"\"' | grep -qE '^git commit' && cd \"$CLAUDE_PROJECT_DIR\" && echo '🔍 Running pre-commit before commit...' && pre-commit run || true"
}
]
}
]
}
}
-36
View File
@@ -1,36 +0,0 @@
# .coveragerc to control coverage.py
[run]
branch = True
source = superset
# omit = bad_file.py
[paths]
source =
superset/
*/site-packages/
[report]
# Regexes for lines to exclude from consideration
exclude_lines =
# Have to re-enable the standard pragma
pragma: no cover
# Don't complain about missing debug-only code:
def __repr__
if self\.debug
# Don't complain if tests don't hit defensive assertion code:
raise AssertionError
raise NotImplementedError
# Don't complain if non-runnable code isn't run:
if 0:
if __name__ == .__main__.:
# Ignore importlib backport
from importlib
if TYPE_CHECKING:
#fail_under = 100
show_missing = True
-125
View File
@@ -1,125 +0,0 @@
---
description: Apache Superset development standards and guidelines for Cursor IDE
globs: ["**/*.py", "**/*.ts", "**/*.tsx", "**/*.js", "**/*.jsx", "**/*.sql", "**/*.md"]
alwaysApply: true
---
# Apache Superset Development Standards for Cursor IDE
Apache Superset is a data visualization platform with Flask/Python backend and React/TypeScript frontend.
## ⚠️ CRITICAL: Ongoing Refactors (What NOT to Do)
**These migrations are actively happening - avoid deprecated patterns:**
### Frontend Modernization
- **NO `any` types** - Use proper TypeScript types
- **NO JavaScript files** - Convert to TypeScript (.ts/.tsx)
- **NO Enzyme** - Use React Testing Library/Jest (Enzyme fully removed)
- **Use @superset-ui/core** - Don't import Ant Design directly
### Testing Strategy Migration
- **Prefer unit tests** over integration tests
- **Prefer integration tests** over Cypress end-to-end tests
- **Cypress is last resort** - Actively moving away from Cypress
- **Use Jest + React Testing Library** for component testing
### Backend Type Safety
- **Add type hints** - All new Python code needs proper typing
- **MyPy compliance** - Run `pre-commit run mypy` to validate
- **SQLAlchemy typing** - Use proper model annotations
## Code Standards
### TypeScript Frontend
- **NO `any` types** - Use proper TypeScript
- **Functional components** with hooks
- **@superset-ui/core** for UI components (not direct antd)
- **Jest** for testing (NO Enzyme)
- **Redux** for global state, hooks for local
### Python Backend
- **Type hints required** for all new code
- **MyPy compliant** - run `pre-commit run mypy`
- **SQLAlchemy models** with proper typing
- **pytest** for testing
### Apache License Headers
- **New files require ASF license headers** - When creating new code files, include the standard Apache Software Foundation license header
- **LLM instruction files are excluded** - Files like LLMS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead
## Key Directory Structure
```
superset/
├── superset/ # Python backend (Flask, SQLAlchemy)
│ ├── views/api/ # REST API endpoints
│ ├── models/ # Database models
│ └── connectors/ # Database connections
├── superset-frontend/src/ # React TypeScript frontend
│ ├── components/ # Reusable components
│ ├── explore/ # Chart builder
│ ├── dashboard/ # Dashboard interface
│ └── SqlLab/ # SQL editor
├── superset-frontend/packages/
│ └── superset-ui-core/ # UI component library (USE THIS)
├── tests/ # Python/integration tests
├── docs/ # Documentation (UPDATE FOR CHANGES)
└── UPDATING.md # Breaking changes log
```
## Architecture Patterns
### Dataset-Centric Approach
Charts built from enriched datasets containing:
- Dimension columns with labels/descriptions
- Predefined metrics as SQL expressions
- Self-service analytics within defined contexts
### Security & Features
- **RBAC**: Role-based access via Flask-AppBuilder
- **Feature flags**: Control feature rollouts
- **Row-level security**: SQL-based data access control
## Test Utilities
### Python Test Helpers
- **`SupersetTestCase`** - Base class in `tests/integration_tests/base_tests.py`
- **`@with_config`** - Config mocking decorator
- **`@with_feature_flags`** - Feature flag testing
- **`login_as()`, `login_as_admin()`** - Authentication helpers
- **`create_dashboard()`, `create_slice()`** - Data setup utilities
### TypeScript Test Helpers
- **`superset-frontend/spec/helpers/testing-library.tsx`** - Custom render() with providers
- **`createWrapper()`** - Redux/Router/Theme wrapper
- **`selectOption()`** - Select component helper
- **React Testing Library** - NO Enzyme (removed)
## Pre-commit Validation
**Use pre-commit hooks for quality validation:**
```bash
# Install hooks
pre-commit install
# Quick validation (faster than --all-files)
pre-commit run # Staged files only
pre-commit run mypy # Python type checking
pre-commit run format # Code formatting
pre-commit run eslint # Frontend linting
```
## Development Guidelines
- **Documentation**: Update docs/ for any user-facing changes
- **Breaking Changes**: Add to UPDATING.md
- **Docstrings**: Required for new functions/classes
- **Follow existing patterns**: Mimic code style, use existing libraries and utilities
- **Type Safety**: This codebase is actively modernizing toward full TypeScript and type safety
- **Always run `pre-commit run`** to validate changes before committing
---
**Note**: This codebase is actively modernizing toward full TypeScript and type safety. Always run `pre-commit run` to validate changes. Follow the ongoing refactors section to avoid deprecated patterns.
-20
View File
@@ -1,20 +0,0 @@
# Keep this in sync with the base image in the main Dockerfile (ARG PY_VER)
FROM python:3.11.13-trixie AS base
# Install system dependencies that Superset needs
# This layer will be cached across Codespace sessions
RUN apt-get update && apt-get install -y \
libsasl2-dev \
libldap2-dev \
libpq-dev \
tmux \
gh \
&& rm -rf /var/lib/apt/lists/*
# Install uv for fast Python package management
# This will also be cached in the image
RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
echo 'export PATH="/root/.cargo/bin:$PATH"' >> /etc/bash.bashrc
# Set the cargo/bin directory in PATH for all users
ENV PATH="/root/.cargo/bin:${PATH}"
-5
View File
@@ -1,5 +0,0 @@
# Superset Development with GitHub Codespaces
For complete documentation on using GitHub Codespaces with Apache Superset, please see:
**[Setting up a Development Environment - GitHub Codespaces](https://superset.apache.org/docs/contributing/development#github-codespaces-cloud-development)**
-62
View File
@@ -1,62 +0,0 @@
# Superset Codespaces environment setup
# This file is appended to ~/.bashrc during Codespace setup
# Find the workspace directory (handles both 'superset' and 'superset-2' names)
WORKSPACE_DIR=$(find /workspaces -maxdepth 1 -name "superset*" -type d | head -1)
if [ -n "$WORKSPACE_DIR" ]; then
# Check if virtual environment exists
if [ -d "$WORKSPACE_DIR/.venv" ]; then
# Activate the virtual environment
source "$WORKSPACE_DIR/.venv/bin/activate"
echo "✅ Python virtual environment activated"
# Verify pre-commit is installed and set up
if command -v pre-commit &> /dev/null; then
echo "✅ pre-commit is available ($(pre-commit --version))"
# Install git hooks if not already installed
if [ -d "$WORKSPACE_DIR/.git" ] && [ ! -f "$WORKSPACE_DIR/.git/hooks/pre-commit" ]; then
echo "🪝 Installing pre-commit hooks..."
cd "$WORKSPACE_DIR" && pre-commit install
fi
else
echo "⚠️ pre-commit not found. Run: pip install pre-commit"
fi
else
echo "⚠️ Python virtual environment not found at $WORKSPACE_DIR/.venv"
echo " Run: cd $WORKSPACE_DIR && .devcontainer/setup-dev.sh"
fi
# Always cd to the workspace directory for convenience
cd "$WORKSPACE_DIR"
fi
# Add helpful aliases for Superset development
alias start-superset="$WORKSPACE_DIR/.devcontainer/start-superset.sh"
alias setup-dev="$WORKSPACE_DIR/.devcontainer/setup-dev.sh"
# Show helpful message on login
echo ""
echo "🚀 Superset Codespaces Environment"
echo "=================================="
# Check if Superset is running
if docker ps 2>/dev/null | grep -q "superset"; then
echo "✅ Superset is running!"
echo " - Check the 'Ports' tab for your live Superset URL"
echo " - Initial startup takes 10-20 minutes"
echo " - Login: admin/admin"
else
echo "⚠️ Superset is not running. Use: start-superset"
# Check if there's a startup log
if [ -f "/tmp/superset-startup.log" ]; then
echo " 📋 Startup log found: cat /tmp/superset-startup.log"
fi
fi
echo ""
echo "Quick commands:"
echo " start-superset - Start Superset with Docker Compose"
echo " setup-dev - Set up Python environment (if not already done)"
echo " pre-commit run - Run pre-commit checks on staged files"
echo ""
-20
View File
@@ -1,20 +0,0 @@
#!/bin/bash
# Script to build and push the devcontainer image to GitHub Container Registry
# This allows caching the image between Codespace sessions
# You'll need to run this with appropriate GitHub permissions
# gh auth login --scopes write:packages
REGISTRY="ghcr.io"
OWNER="apache"
REPO="superset"
TAG="devcontainer-base"
echo "Building devcontainer image..."
docker build -t $REGISTRY/$OWNER/$REPO:$TAG .devcontainer/
echo "Pushing to GitHub Container Registry..."
docker push $REGISTRY/$OWNER/$REPO:$TAG
echo "Done! Update .devcontainer/devcontainer.json to use:"
echo " \"image\": \"$REGISTRY/$OWNER/$REPO:$TAG\""
-19
View File
@@ -1,19 +0,0 @@
{
// Extend the base configuration
"extends": "../devcontainer-base.json",
"name": "Apache Superset Development (Default)",
// Forward ports for development
"forwardPorts": [9001],
"portsAttributes": {
"9001": {
"label": "Superset (via Webpack Dev Server)",
"onAutoForward": "notify",
"visibility": "public"
}
},
// Auto-start Superset on Codespace resume
"postStartCommand": ".devcontainer/start-superset.sh"
}
-38
View File
@@ -1,38 +0,0 @@
{
"name": "Apache Superset Development",
// Keep this in sync with the base image in Dockerfile (ARG PY_VER)
// Using the same base as Dockerfile, but non-slim for dev tools
"image": "python:3.11.13-bookworm",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
"moby": true,
"dockerDashComposeVersion": "v2"
},
"ghcr.io/devcontainers/features/node:1": {
"version": "20"
},
"ghcr.io/devcontainers/features/git:1": {},
"ghcr.io/devcontainers/features/common-utils:2": {
"configureZshAsDefaultShell": true
},
"ghcr.io/devcontainers/features/sshd:1": {
"version": "latest"
}
},
// Run commands after container is created
"postCreateCommand": "chmod +x .devcontainer/setup-dev.sh && .devcontainer/setup-dev.sh",
// VS Code customizations
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"charliermarsh.ruff",
"dbaeumer.vscode-eslint",
]
}
}
}
-65
View File
@@ -1,65 +0,0 @@
{
"name": "Apache Superset Development",
// Option 1: Use pre-built image directly
// "image": "ghcr.io/apache/superset:devcontainer-base",
// Option 2: Build from Dockerfile with cache (current approach)
"build": {
"dockerfile": "Dockerfile",
"context": ".",
// Cache from the Apache registry image
"cacheFrom": ["ghcr.io/apache/superset:devcontainer-base"]
},
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
"moby": false,
"dockerDashComposeVersion": "v2"
},
"ghcr.io/devcontainers/features/node:1": {
"version": "20"
},
"ghcr.io/devcontainers/features/git:1": {},
"ghcr.io/devcontainers/features/common-utils:2": {
"configureZshAsDefaultShell": true
},
"ghcr.io/devcontainers/features/sshd:1": {
"version": "latest"
}
},
// Forward ports for development
"forwardPorts": [9001],
"portsAttributes": {
"9001": {
"label": "Superset (via Webpack Dev Server)",
"onAutoForward": "notify",
"visibility": "public"
}
},
// Run commands after container is created
"postCreateCommand": "bash .devcontainer/setup-dev.sh || echo '⚠️ Setup had issues - run .devcontainer/setup-dev.sh manually'",
// Auto-start Superset after ensuring Docker is ready
// Run in foreground to see any errors, but don't block on failures
"postStartCommand": "bash -c 'echo \"Waiting 30s for services to initialize...\"; sleep 30; .devcontainer/start-superset.sh || echo \"⚠️ Auto-start failed - run start-superset manually\"'",
// Set environment variables
"remoteEnv": {
// Removed automatic venv activation to prevent startup issues
// The setup script will handle this
},
// VS Code customizations
"customizations": {
"vscode": {
"extensions": [
"ms-python.python",
"ms-python.vscode-pylance",
"charliermarsh.ruff",
"dbaeumer.vscode-eslint",
]
}
}
}
-32
View File
@@ -1,32 +0,0 @@
#!/bin/bash
# Setup script for Superset Codespaces development environment
echo "🔧 Setting up Superset development environment..."
# The universal image has most tools, just need Superset-specific libs
echo "📦 Installing Superset-specific dependencies..."
sudo apt-get update
sudo apt-get install -y \
libsasl2-dev \
libldap2-dev \
libpq-dev \
tmux \
gh
# Install uv for fast Python package management
echo "📦 Installing uv..."
curl -LsSf https://astral.sh/uv/install.sh | sh
# Add cargo/bin to PATH for uv
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc
echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.zshrc
# Install Claude Code CLI via npm
echo "🤖 Installing Claude Code..."
npm install -g @anthropic-ai/claude-code
# Make the start script executable
chmod +x .devcontainer/start-superset.sh
echo "✅ Development environment setup complete!"
echo "🚀 Run '.devcontainer/start-superset.sh' to start Superset"
-69
View File
@@ -1,69 +0,0 @@
#!/bin/bash
# Startup script for Superset in Codespaces
echo "🚀 Starting Superset in Codespaces..."
echo "🌐 Frontend will be available at port 9001"
# Check if MCP is enabled
if [ "$ENABLE_MCP" = "true" ]; then
echo "🤖 MCP Service will be available at port 5008"
fi
# Find the workspace directory (Codespaces clones as 'superset', not 'superset-2')
WORKSPACE_DIR=$(find /workspaces -maxdepth 1 -name "superset*" -type d | head -1)
if [ -n "$WORKSPACE_DIR" ]; then
cd "$WORKSPACE_DIR"
echo "📁 Working in: $WORKSPACE_DIR"
else
echo "📁 Using current directory: $(pwd)"
fi
# Check if docker is running
if ! docker info > /dev/null 2>&1; then
echo "⏳ Waiting for Docker to start..."
sleep 5
fi
# Clean up any existing containers
echo "🧹 Cleaning up existing containers..."
docker-compose -f docker-compose-light.yml --profile mcp down
# Start services
echo "🏗️ Building and starting services..."
echo ""
echo "📝 Once started, login with:"
echo " Username: admin"
echo " Password: admin"
echo ""
echo "📋 Running in foreground with live logs (Ctrl+C to stop)..."
# Run docker-compose and capture exit code
if [ "$ENABLE_MCP" = "true" ]; then
echo "🤖 Starting with MCP Service enabled..."
docker-compose -f docker-compose-light.yml --profile mcp up
else
docker-compose -f docker-compose-light.yml up
fi
EXIT_CODE=$?
# If it failed, provide helpful instructions
if [ $EXIT_CODE -ne 0 ] && [ $EXIT_CODE -ne 130 ]; then # 130 is Ctrl+C
echo ""
echo "❌ Superset startup failed (exit code: $EXIT_CODE)"
echo ""
echo "🔄 To restart Superset, run:"
echo " .devcontainer/start-superset.sh"
echo ""
echo "🔧 For troubleshooting:"
echo " # View logs:"
echo " docker-compose -f docker-compose-light.yml logs"
echo ""
echo " # Clean restart (removes volumes):"
echo " docker-compose -f docker-compose-light.yml down -v"
echo " .devcontainer/start-superset.sh"
echo ""
echo " # Common issues:"
echo " - Network timeouts: Just retry, often transient"
echo " - Port conflicts: Check 'docker ps'"
echo " - Database issues: Try clean restart with -v"
fi
-29
View File
@@ -1,29 +0,0 @@
{
// Extend the base configuration
"extends": "../devcontainer-base.json",
"name": "Apache Superset Development with MCP",
// Forward ports for development
"forwardPorts": [9001, 5008],
"portsAttributes": {
"9001": {
"label": "Superset (via Webpack Dev Server)",
"onAutoForward": "notify",
"visibility": "public"
},
"5008": {
"label": "MCP Service (Model Context Protocol)",
"onAutoForward": "notify",
"visibility": "private"
}
},
// Auto-start Superset with MCP on Codespace resume
"postStartCommand": "ENABLE_MCP=true .devcontainer/start-superset.sh",
// Environment variables
"containerEnv": {
"ENABLE_MCP": "true"
}
}
+1 -2
View File
@@ -34,6 +34,7 @@
**/*.sqllite
**/*.swp
**/.terser-plugin-cache/
**/.storybook/
**/node_modules/
tests/
@@ -41,8 +42,6 @@ docs/
install/
superset-frontend/cypress-base/
superset-frontend/coverage/
superset-frontend/.temp_cache/
superset/static/assets/
superset-websocket/dist/
venv
.venv
-41
View File
@@ -1,41 +0,0 @@
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Auto-configure Docker Compose for multi-instance support
# Requires direnv: https://direnv.net/
#
# Install: brew install direnv (or apt install direnv)
# Setup: Add 'eval "$(direnv hook bash)"' to ~/.bashrc (or ~/.zshrc)
# Allow: Run 'direnv allow' in this directory once
# Generate unique project name from directory
export COMPOSE_PROJECT_NAME=$(basename "$PWD" | tr '[:upper:]' '[:lower:]' | sed 's/[^a-z0-9]/-/g')
# Find available ports sequentially to avoid collisions
_is_free() { ! lsof -i ":$1" &>/dev/null 2>&1; }
_p=80; while ! _is_free $_p; do ((_p++)); done; export NGINX_PORT=$_p
_p=8088; while ! _is_free $_p; do ((_p++)); done; export SUPERSET_PORT=$_p
_p=9000; while ! _is_free $_p; do ((_p++)); done; export NODE_PORT=$_p
_p=8080; while ! _is_free $_p || [ $_p -eq $NGINX_PORT ]; do ((_p++)); done; export WEBSOCKET_PORT=$_p
_p=8081; while ! _is_free $_p || [ $_p -eq $WEBSOCKET_PORT ]; do ((_p++)); done; export CYPRESS_PORT=$_p
_p=5432; while ! _is_free $_p; do ((_p++)); done; export DATABASE_PORT=$_p
_p=6379; while ! _is_free $_p; do ((_p++)); done; export REDIS_PORT=$_p
unset _p _is_free
echo "🐳 Superset configured: http://localhost:$SUPERSET_PORT (dev: localhost:$NODE_PORT)"
-3
View File
@@ -1,4 +1 @@
docker/**/*.sh text eol=lf
*.svg binary
*.ipynb binary
*.geojson binary
+18 -18
View File
@@ -1,29 +1,29 @@
# Notify PMC members of changes to GitHub Actions
# Notify all committers of DB migration changes, per SIP-59
/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @kgabryje @sha174n @dpgaspar @sadpandajoe @hainenber
# https://github.com/apache/superset/issues/13351
# Notify PMC members of changes to CI-executed scripts (supply-chain risk:
# scripts/ files run directly in CI workflows and can execute arbitrary code)
/superset/migrations/ @apache/superset-committers
/scripts/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @kgabryje @dpgaspar @sha174n @sadpandajoe @hainenber
# Notify Preset team when ephemeral env settings are changed
# Notify PMC members of changes to required GitHub Actions
.github/workflows/ecs-task-definition.json @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
.github/workflows/docker-ephemeral-env.yml @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
.github/workflows/ephemeral*.yml @robdiciuccio @craig-rueda @rusackas @eschutho @dpgaspar @nytai @mistercrunch
/.asf.yaml @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @kgabryje @dpgaspar @sha174n @Antonio-RiveroMartnez
# Notify some committers of changes in the components
# Maps are fragile and political. GeoJson edits MUST be made in the Jupyter notebook or they'll be overwritten.
/superset-frontend/src/components/Select/ @michael-s-molina @geido @kgabryje
/superset-frontend/src/components/MetadataBar/ @michael-s-molina @geido @kgabryje
/superset-frontend/src/components/DropdownContainer/ @michael-s-molina @geido @kgabryje
**/*.geojson @villebro @rusackas
**/*.ipynb @villebro @rusackas
/superset-frontend/plugins/plugin-chart-country-map/ @villebro @rusackas
# Notify Helm Chart maintainers about changes in it
# Notify PMC members of changes to extension-related files
/helm/superset/ @craig-rueda @dpgaspar @villebro
/docs/developer_docs/extensions/ @michael-s-molina @villebro @rusackas
/superset-extensions-cli/ @michael-s-molina @villebro @rusackas @sadpandajoe
/superset/extensions/ @michael-s-molina @villebro @rusackas @sadpandajoe
/superset-frontend/src/extensions/ @michael-s-molina @villebro @rusackas @sadpandajoe
# Notify E2E test maintainers of changes
# Notify PMC members of config changes e.g. feature flags
/superset-frontend/cypress-base/ @jinghua-qa @geido @eschutho @rusackas @betodealmeida
/superset/config.py @michael-s-molina @villebro @rusackas @sadpandajoe
# Notify PMC members of changes to Github Actions
/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @craig-rueda @john-bodley @kgabryje
+49
View File
@@ -0,0 +1,49 @@
---
name: Bug report
about: "Create a report to help us improve Superset's stability! For feature requests please open a discussion [here](https://github.com/apache/superset/discussions/categories/ideas)."
labels: bug
---
A clear and concise description of what the bug is.
#### How to reproduce the bug
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
### Expected results
what you expected to happen.
### Actual results
what actually happens.
#### Screenshots
If applicable, add screenshots to help explain your problem.
### Environment
(please complete the following information):
- browser type and version:
- superset version: `superset version`
- python version: `python --version`
- node.js version: `node -v`
- any feature flags active:
### Checklist
Make sure to follow these steps before submitting your issue - thank you!
- [ ] I have checked the superset logs for python stacktraces and included it here as text if there are any.
- [ ] I have reproduced the issue with at least the latest released version of superset.
- [ ] I have checked the issue tracker for the same issue and I haven't found one similar.
### Additional context
Add any other context about the problem here.
-98
View File
@@ -1,98 +0,0 @@
name: Bug report
description: Report a bug to improve Superset's stability
labels: ["bug"]
body:
- type: markdown
attributes:
value: |
Hello Superset Community member! Please keep things tidy by putting your post in the proper place:
🚨 Reporting a security issue: send an email to security@superset.apache.org. DO NOT USE GITHUB ISSUES TO REPORT SECURITY PROBLEMS.
🐛 Reporting a bug: use this form.
🙏 Asking a question or getting help: post in the [Superset Slack chat](http://bit.ly/join-superset-slack) or [GitHub Discussions](https://github.com/apache/superset/discussions) under "Q&A / Help".
💡 Requesting a new feature: Search [GitHub Discussions](https://github.com/apache/superset/discussions) to see if it exists already. If not, add a new post there under "Ideas".
- type: textarea
id: bug-description
attributes:
label: Bug description
description: A clear description of what the bug is, including reproduction steps and expected behavior.
placeholder: |
The bug is that...
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
validations:
required: true
- type: textarea
id: screenshots-recordings
attributes:
label: Screenshots/recordings
description: If applicable, add screenshots or recordings to help explain your problem.
- type: markdown
attributes:
value: |
### Environment
Please specify your environment. If your environment does not match the alternatives, you need to upgrade your environment before submitting the issue as it may have already been fixed. For additional information about the releases, see [Release Process](https://github.com/apache/superset/wiki/Release-Process).
- type: dropdown
id: superset-version
attributes:
label: Superset version
options:
- master / latest-dev
- "6.1.0"
- "6.0.0"
validations:
required: true
- type: dropdown
id: python-version
attributes:
label: Python version
options:
- "3.11"
- "3.12"
- Not applicable
- I don't know
validations:
required: true
- type: dropdown
id: node-version
attributes:
label: Node version
options:
- "16"
- "17"
- "18 or greater"
- Not applicable
- I don't know
validations:
required: true
- type: dropdown
id: browser
attributes:
label: Browser
options:
- Chrome
- Firefox
- Safari
- Not applicable
validations:
required: true
- type: textarea
id: additional-context
attributes:
label: Additional context
description: |
Add any other context about the problem here such as the feature flags that you have enabled, any customizations you have made, the data source you are querying, etc.
- type: checkboxes
id: checklist
attributes:
label: Checklist
description: Make sure to follow these steps before submitting your issue - thank you!
options:
- label: I have searched Superset docs and Slack and didn't find a solution to my problem.
- label: I have searched the GitHub issue tracker and didn't find a similar bug report.
- label: I have checked Superset's logs for errors and if I found a relevant Python stacktrace, I included it here as text in the "additional context" section.
validations:
required: true
-12
View File
@@ -1,12 +0,0 @@
---
blank_issues_enabled: false
contact_links:
- name: Feature Request
url: https://github.com/apache/superset/discussions/new?category=ideas
about: Propose a feature request to the Superset community
- name: Q&A
url: https://github.com/apache/superset/discussions/new?category=q-a-help
about: Open a community Q&A thread on GitHub Discussions
- name: Slack
url: https://bit.ly/join-superset-slack
about: Join the Superset Community on Slack for other discussions and assistance
+3 -3
View File
@@ -1,13 +1,13 @@
---
name: SIP
about: "Superset Improvement Proposal. See SIP-0 (https://github.com/apache/superset/issues/5602) for details. A SIP introduces any major change into Apache Superset's code or process."
about: "Superset Improvement Proposal. See [here](https://github.com/apache/superset/issues/5602) for details."
labels: sip
title: "[SIP] Your Title Here (do not add SIP number)"
assignees: "apache/superset-committers"
asignees: "apache/superset-committers"
---
*Please make sure you are familiar with the SIP process documented*
[here](https://github.com/apache/superset/issues/5602). The SIP will be numbered by a committer upon acceptance.
(here)[https://github.com/apache/superset/issues/5602]. The SIP will be numbered by a committer upon acceptance.
## [SIP] Proposal for ...<title>
+38
View File
@@ -0,0 +1,38 @@
# Security Policy
This is a project of the [Apache Software Foundation](https://apache.org) and follows the
ASF [vulnerability handling process](https://apache.org/security/#vulnerability-handling).
## Reporting Vulnerabilities
**⚠️ Please do not file GitHub issues for security vulnerabilities as they are public! ⚠️**
Apache Software Foundation takes a rigorous standpoint in annihilating the security issues
in its software projects. Apache Superset is highly sensitive and forthcoming to issues
pertaining to its features and functionality.
If you have any concern or believe you have found a vulnerability in Apache Superset,
please get in touch with the Apache Security Team privately at
e-mail address [security@apache.org](mailto:security@apache.org).
More details can be found on the ASF website at
[ASF vulnerability reporting process](https://apache.org/security/#reporting-a-vulnerability)
We kindly ask you to include the following information in your report:
- Apache Superset version that you are using
- A sanitized copy of your `superset_config.py` file or any config overrides
- Detailed steps to reproduce the vulnerability
Note that Apache Superset is not responsible for any third-party dependencies that may
have security issues. Any vulnerabilities found in third-party dependencies should be
reported to the maintainers of those projects. Results from security scans of Apache
Superset dependencies found on its official Docker image can be remediated at release time
by extending the image itself.
**Your responsible disclosure and collaboration are invaluable.**
## Extra Information
- [Apache Superset documentation](https://superset.apache.org/docs/security)
- [Common Vulnerabilities and Exposures by release](https://superset.apache.org/docs/security/cves)
- [How Security Vulnerabilities are Reported & Handled in Apache Superset (Blog)](https://preset.io/blog/how-security-vulnerabilities-are-reported-and-handled-in-apache-superset/)
@@ -1,34 +0,0 @@
name: Change Detector
description: Detects file changes for pull request and push events
inputs:
token:
description: GitHub token for authentication
required: true
outputs:
python:
description: Whether Python-related files were changed
value: ${{ steps.change-detector.outputs.python }}
frontend:
description: Whether frontend-related files were changed
value: ${{ steps.change-detector.outputs.frontend }}
docker:
description: Whether docker-related files were changed
value: ${{ steps.change-detector.outputs.docker }}
docs:
description: Whether docs-related files were changed
value: ${{ steps.change-detector.outputs.docs }}
superset-extensions-cli:
description: Whether superset-extensions-cli package-related files were changed
value: ${{ steps.change-detector.outputs.superset-extensions-cli }}
runs:
using: composite
steps:
- name: Detect file changes
id: change-detector
run: |
python --version
python scripts/change_detector.py
shell: bash
env:
GITHUB_TOKEN: ${{ inputs.token }}
GITHUB_OUTPUT: ${{ github.output }}
@@ -1,23 +0,0 @@
name: Label Draft PRs
on:
pull_request:
types:
- opened
- converted_to_draft
jobs:
label-draft:
runs-on: ubuntu-latest
steps:
- name: Check if the PR is a draft
id: check-draft
uses: actions/github-script@v8
with:
script: |
const isDraft = context.payload.pull_request.draft;
core.setOutput('isDraft', isDraft);
- name: Add `review:draft` Label
if: steps.check-draft.outputs.isDraft == 'true'
uses: actions-ecosystem/action-add-labels@v1
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
labels: "review:draft"
-75
View File
@@ -1,75 +0,0 @@
name: 'Setup Python Environment'
description: 'Set up Python and install dependencies with optional configurations.'
inputs:
python-version:
description: 'Python version to set up. Accepts a version number, "current", or "next".'
required: true
default: 'current'
requirements-type:
description: 'Type of requirements to install. Options: base, development, default'
required: false
default: 'dev'
install-superset:
description: 'Whether to install Superset itself. If false, only python is installed'
required: false
default: 'true'
runs:
using: 'composite'
steps:
- name: Interpret Python Version
id: set-python-version
shell: bash
env:
INPUT_PYTHON_VERSION: ${{ inputs.python-version }}
run: |
if [ "$INPUT_PYTHON_VERSION" = "current" ]; then
RESOLVED_VERSION="3.11"
elif [ "$INPUT_PYTHON_VERSION" = "next" ]; then
# currently disabled in GHA matrixes because of library compatibility issues
RESOLVED_VERSION="3.12"
elif printf '%s' "$INPUT_PYTHON_VERSION" | grep -Eq '^[0-9]+\.[0-9]+(\.[0-9]+)?$'; then
RESOLVED_VERSION="$INPUT_PYTHON_VERSION"
else
echo "Invalid python-version: '$INPUT_PYTHON_VERSION'" >&2
exit 1
fi
echo "python-version=$RESOLVED_VERSION" >> "$GITHUB_OUTPUT"
- name: Set up Python ${{ steps.set-python-version.outputs.python-version }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
with:
python-version: ${{ steps.set-python-version.outputs.python-version }}
- name: Install uv
if: inputs.install-superset == 'true'
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
python-version: ${{ steps.set-python-version.outputs.python-version }}
enable-cache: true
- name: Update apt package lists
# cache-apt-pkgs-action assumes a fresh `apt-cache` index (true on GitHub-hosted
# runners, not on all self-hosted/custom runner images), so refresh it explicitly
# or package lookups silently resolve to an empty list.
if: inputs.install-superset == 'true'
shell: bash
run: sudo apt-get update
- name: Install apt packages
if: inputs.install-superset == 'true'
uses: awalsh128/cache-apt-pkgs-action@553a35bb8ebd9fcabcb1c9451aa4c98e1b4ca8a9 # v1.6.3
with:
packages: libldap2-dev libsasl2-dev
version: 1.0
- name: Install dependencies
env:
INPUT_INSTALL_SUPERSET: ${{ inputs.install-superset }}
INPUT_REQUIREMENTS_TYPE: ${{ inputs.requirements-type }}
run: |
if [ "$INPUT_INSTALL_SUPERSET" = "true" ]; then
if [ "$INPUT_REQUIREMENTS_TYPE" = "dev" ]; then
uv pip install --system -r requirements/development.txt
elif [ "$INPUT_REQUIREMENTS_TYPE" = "base" ]; then
uv pip install --system -r requirements/base.txt
fi
uv pip install --system -e .
fi
shell: bash
-78
View File
@@ -1,78 +0,0 @@
name: "Setup Docker Environment"
description: "Reusable steps for setting up QEMU, Docker Buildx, DockerHub login, Supersetbot, and optionally Docker Compose"
inputs:
build:
description: "Used for building?"
required: false
default: "false"
dockerhub-user:
description: "DockerHub username"
required: false
dockerhub-token:
description: "DockerHub token"
required: false
install-docker-compose:
description: "Flag to install Docker Compose"
required: false
default: "true"
login-to-dockerhub:
description: "Whether you want to log into dockerhub"
required: false
default: "true"
outputs: {}
runs:
using: "composite"
steps:
- name: Set up QEMU
if: ${{ inputs.build == 'true' }}
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
with:
# Pin the binfmt image to a specific QEMU release. The default
# (`tonistiigi/binfmt:latest`) is a moving target, and drift across
# QEMU's x86_64→aarch64 translator has been the proximate cause of
# intermittent `exit code: 132` (SIGILL) failures during the arm64
# leg of the multi-platform docker build — newer Node native modules
# emit instructions QEMU's user-mode emulation occasionally drops on
# the floor. Pinning a known-good release stabilises that path.
image: tonistiigi/binfmt:qemu-v8.1.5
- name: Set up Docker Buildx
if: ${{ inputs.build == 'true' }}
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
- name: Try to login to DockerHub
if: ${{ inputs.login-to-dockerhub == 'true' }}
continue-on-error: true
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ inputs.dockerhub-user }}
password: ${{ inputs.dockerhub-token }}
- name: Install Docker Compose
if: ${{ inputs.install-docker-compose == 'true' }}
shell: bash
run: |
sudo apt-get update
sudo apt-get install -y ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
# Download and save the Docker GPG key in the correct format
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
# Ensure the key file is readable
sudo chmod a+r /etc/apt/keyrings/docker.gpg
# Add the Docker repository using the correct key
echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
$(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
# Update package lists and install Docker Compose plugin
sudo apt update
sudo apt install -y docker-compose-plugin
- name: Docker Version Info
shell: bash
run: docker info
@@ -1,43 +0,0 @@
name: 'Setup supersetbot'
description: 'Sets up supersetbot npm lib from the repo or npm'
inputs:
from-npm:
description: 'Install from npm instead of local setup'
required: false
default: 'true' # Defaults to using the local setup
runs:
using: 'composite'
steps:
- name: Setup Node Env
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
with:
node-version: '20'
- name: Install supersetbot from npm
if: ${{ inputs.from-npm == 'true' }}
shell: bash
# zizmor: ignore[adhoc-packages] - supersetbot is a first-party Apache CLI (apache-superset/supersetbot) installed globally as a tool; a global CLI install has no application manifest/lockfile context
run: npm install -g supersetbot
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: ${{ inputs.from-npm == 'false' }}
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
with:
persist-credentials: false
repository: apache-superset/supersetbot
path: supersetbot
- name: Setup supersetbot from repo
if: ${{ inputs.from-npm == 'false' }}
shell: bash
working-directory: supersetbot
# zizmor: ignore[adhoc-packages] - installs the locally packed supersetbot tarball built from the trusted apache-superset/supersetbot checkout; no lockfile applies to a global CLI install
run: |
# simple trick to install globally with dependencies
npm pack
npm install -g ./supersetbot*.tgz
- name: echo supersetbot version
shell: bash
run: supersetbot version
-1
View File
@@ -1 +0,0 @@
../AGENTS.md
+4 -113
View File
@@ -1,139 +1,35 @@
version: 2
enable-beta-ecosystems: true
updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
cooldown:
default-days: 7
- package-ecosystem: "npm"
ignore:
# TODO: remove below entries once the application supports React >= 19.0.0
- dependency-name: "react"
update-types: ["version-update:semver-major"]
- dependency-name: "react-dom"
update-types: ["version-update:semver-major"]
- dependency-name: "@types/react"
update-types: ["version-update:semver-major"]
- dependency-name: "@types/react-dom"
update-types: ["version-update:semver-major"]
- dependency-name: "react-icons"
# JSDOM v30 doesn't play well with Jest v30
# Source: https://jestjs.io/blog#known-issues
# GH thread: https://github.com/jsdom/jsdom/issues/3492
- dependency-name: "jest-environment-jsdom"
# `@swc/plugin-transform-imports` doesn't work with current Webpack-SWC hybrid setup
# See https://github.com/apache/superset/pull/37384#issuecomment-3793991389
# TODO: remove the plugin once Lodash usage has been migrated to a more readily tree-shakeable alternative
- dependency-name: "@swc/plugin-transform-imports"
# deck.gl and luma.gl share strict peer constraints across the root and
# plugin workspaces, and root overrides pin their transitive versions.
# Upgrade both families together in a manually validated change.
- dependency-name: "@deck.gl/*"
- dependency-name: "@luma.gl/*"
# `just-handlerbars-helpers` library in plugin-chart-handlebars requires `currencyformatter`` to be < 2
- dependency-name: "currencyformatter.js"
update-types: ["version-update:semver-major"]
# TODO: remove below clause once https://github.com/pmmmwh/react-refresh-webpack-plugin/pull/940 lands onto a future release
# and confirm the issue https://github.com/apache/superset/issues/39600 is fixed
- dependency-name: "react-checkbox-tree"
update-types: ["version-update:semver-major"]
# Babel 8 (7.x -> 8.x) is blocked on the surrounding ecosystem: @emotion/babel-plugin
# (NodePath#hoist), babel-plugin-jsx-remove-data-test-id (t.jSXOpeningElement), and
# ts-jest all rely on Babel APIs removed in v8 and have not shipped Babel 8 support.
# Ignore the coordinated major bump until the ecosystem catches up; it must be done
# as a single manual upgrade anyway. TODO: remove when Babel 8 support is viable.
- dependency-name: "@babel/*"
update-types: ["version-update:semver-major"]
# v2.0.0 renamed ZSTDDecompress to decompress and made it async, breaking
# the webpack dev proxy (see #38662, #39138, #39139). Dependabot reopened
# the same bump in #39369 after the first revert, so pin it here instead
# of relying on a package.json comment (package.json is JSON and can't
# hold comments). Remove this once the proxy code is updated to await
# the async decompress() API.
- dependency-name: "simple-zstd"
directory: "/superset-frontend/"
schedule:
interval: "daily"
labels:
- npm
- dependabot
groups:
rjsf:
patterns:
- "@rjsf/*"
typescript-eslint:
patterns:
- "@typescript-eslint/*"
- "typescript-eslint"
babel:
patterns:
- "@babel/*"
storybook:
patterns:
- "@storybook/*"
- "storybook"
- "eslint-plugin-storybook"
ag-grid:
patterns:
- "ag-grid-react"
- "ag-grid-community"
open-pull-requests-limit: 30
versioning-strategy: increase
cooldown:
default-days: 7
- package-ecosystem: "pip"
directory: "/"
open-pull-requests-limit: 10
# Bump the lower bound to the new version, not just widen the upper
# bound. Without this, a `sqlglot>=28.10.0, <29` constraint upgraded
# to `<30` would keep the stale lower bound forever, dragging
# transitively-resolved versions with it. See #40186 (review thread).
versioning-strategy: increase
directory: "/requirements/"
schedule:
interval: "weekly"
interval: "daily"
labels:
- pip
- dependabot
cooldown:
default-days: 7
- package-ecosystem: "npm"
directory: ".github/actions"
schedule:
interval: "daily"
open-pull-requests-limit: 10
open-pull-requests-limit: 0
versioning-strategy: increase
cooldown:
default-days: 7
- package-ecosystem: "npm"
directory: "/docs/"
schedule:
interval: "daily"
groups:
storybook:
patterns:
- "@storybook/*"
- "storybook"
docusaurus:
patterns:
- "@docusaurus/*"
docusaurus-openapi:
patterns:
- "docusaurus-plugin-openapi-docs"
- "docusaurus-theme-openapi-docs"
typescript-eslint:
patterns:
- "@typescript-eslint/*"
- "typescript-eslint"
open-pull-requests-limit: 10
open-pull-requests-limit: 0
versioning-strategy: increase
cooldown:
default-days: 7
- package-ecosystem: "npm"
directory: "/superset-websocket/"
@@ -143,8 +39,6 @@ updates:
- npm
- dependabot
versioning-strategy: increase
cooldown:
default-days: 7
- package-ecosystem: "npm"
directory: "/superset-websocket/utils/client-ws-app/"
@@ -153,7 +47,4 @@ updates:
labels:
- npm
- dependabot
open-pull-requests-limit: 10
versioning-strategy: increase
cooldown:
default-days: 7
-173
View File
@@ -1,173 +0,0 @@
# TODO (if we can)
# - Label PRs in need of codeowner review
# - viz:charts:xyz labels
# component/design system areas
# - storybook(s)
# - f/e and b/e test changes?
# - product areas (SQL Lab, Explore, Dashboard, etc.)
# - database areas (SQLAlchemy, labelind DBs by driver, etc.)
############################################
# General workflow warnings
# full list of labels is here: https://github.com/apache/superset/labels
############################################
"risk:db-migration":
- changed-files:
- any-glob-to-any-file:
- 'superset/migrations/**'
"risk:ci-script":
- changed-files:
- any-glob-to-any-file:
- 'scripts/**'
############################################
# Dependencies
############################################
"dependencies:python":
- changed-files:
- any-glob-to-any-file:
- 'superset/requirements/**'
- 'superset/translations/requirements.txt'
- 'RELEASING/requirements.txt'
"dependencies:npm":
- changed-files:
- any-glob-to-any-file:
- 'superset-frontend/package.json'
- 'superset-frontend/package-lock.json'
- 'superset-embedded-sdk/package.json'
- 'superset-embedded-sdk/package-lock.json'
- 'superset-websocket/package.json'
- 'superset-websocket/package-lock.json'
- 'superset-frontend/cypress-base/package.json'
- 'superset-frontend/cypress-base/package-lock.json'
- 'superset-frontend/packages/**/package.json'
- 'superset-frontend/plugins/**/package.json'
############################################
# Areas of the main codebase
############################################
"doc":
- changed-files:
- any-glob-to-any-file:
- 'docs/**'
"api":
- changed-files:
- any-glob-to-any-file:
- 'superset/**/api.py'
- 'superset/views/core.py'
"i18n":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/**'
"i18n:brazilian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/pt_BR/**'
"i18n:chinese":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/zh/**'
"i18n:czech":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/cs/**'
"i18n:traditional-chinese":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/zh_TW/**'
"i18n:dutch":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/nl/**'
"i18n:french":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/fr/**'
"i18n:italian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/it/**'
"i18n:japanese":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/ja/**'
"i18n:korean":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/ko/**'
"i18n:portuguese":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/pt/**'
"i18n:russian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/ru/**'
"i18n:slovak":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/sk/**'
"i18n:latvian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/lv/**'
"i18n:ukrainian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/uk/**'
"i18n:spanish":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/es/**'
"i18n:persian":
- changed-files:
- any-glob-to-any-file:
- 'superset/translations/fa/**'
############################################
# Sub-projects and monorepo packages
############################################
"plugins":
- changed-files:
- any-glob-to-any-file:
- 'superset-frontend/plugins/**'
"packages":
- changed-files:
- any-glob-to-any-file:
- 'superset-frontend/packages/**'
"embedded":
- changed-files:
- any-glob-to-any-file:
- 'superset-embedded-sdk/**'
"github_actions":
- changed-files:
- any-glob-to-any-file:
- '.github/actions/**'
- '.github/workflows/**'
+102 -248
View File
@@ -20,6 +20,10 @@ set -e
GITHUB_WORKSPACE=${GITHUB_WORKSPACE:-.}
ASSETS_MANIFEST="$GITHUB_WORKSPACE/superset/static/assets/manifest.json"
# Rounded job start time, used to create a unique Cypress build id for
# parallelization so we can manually rerun a job after 20 minutes
NONCE=$(echo "$(date "+%Y%m%d%H%M") - ($(date +%M)%20)" | bc)
# Echo only when not in parallel mode
say() {
if [[ $(echo "$INPUT_PARALLEL" | tr '[:lower:]' '[:upper:]') != 'TRUE' ]]; then
@@ -27,6 +31,20 @@ say() {
fi
}
# default command to run when the `run` input is empty
default-setup-command() {
apt-get-install
pip-upgrade
}
apt-get-install() {
say "::group::apt-get install dependencies"
sudo apt-get update && sudo apt-get install --yes \
libsasl2-dev \
libldap2-dev
say "::endgroup::"
}
pip-upgrade() {
say "::group::Upgrade pip"
pip install --upgrade pip
@@ -55,15 +73,6 @@ build-assets() {
say "::endgroup::"
}
build-embedded-sdk() {
cd "$GITHUB_WORKSPACE/superset-embedded-sdk"
say "::group::Build embedded SDK bundle for E2E tests"
npm ci
npm run build
say "::endgroup::"
}
build-instrumented-assets() {
cd "$GITHUB_WORKSPACE/superset-frontend"
@@ -94,8 +103,6 @@ EOF
setup-mysql() {
say "::group::Initialize database"
mysql -h 127.0.0.1 -P 13306 -u root --password=root <<-EOF
SET GLOBAL transaction_isolation='READ-COMMITTED';
SET GLOBAL TRANSACTION ISOLATION LEVEL READ COMMITTED;
DROP DATABASE IF EXISTS superset;
CREATE DATABASE superset DEFAULT CHARACTER SET utf8 COLLATE utf8_unicode_ci;
DROP DATABASE IF EXISTS sqllab_test_db;
@@ -114,7 +121,7 @@ testdata() {
say "::group::Load test data"
# must specify PYTHONPATH to make `tests.superset_test_config` importable
export PYTHONPATH="$GITHUB_WORKSPACE"
uv pip install --system -e .
pip install -e .
superset db upgrade
superset load_test_users
superset load_examples --load-test-data
@@ -122,44 +129,9 @@ testdata() {
say "::endgroup::"
}
playwright_testdata() {
cd "$GITHUB_WORKSPACE"
say "::group::Load all examples for Playwright tests"
# must specify PYTHONPATH to make `tests.superset_test_config` importable
export PYTHONPATH="$GITHUB_WORKSPACE"
uv pip install --system -e .
superset db upgrade
superset load_test_users
superset load_examples
superset init
# Enable DML on the examples database so Playwright tests can create/drop
# temporary tables via SQL Lab without depending on external data sources.
superset shell <<'PYEOF'
import sys
from superset.extensions import db
from superset.models.core import Database
examples_db = db.session.query(Database).filter_by(database_name='examples').first()
if not examples_db:
sys.exit('ERROR: examples database not found. load_examples may have failed.')
examples_db.allow_dml = True
db.session.commit()
print('Enabled allow_dml on examples database')
PYEOF
say "::endgroup::"
}
celery-worker() {
cd "$GITHUB_WORKSPACE"
say "::group::Start Celery worker"
# must specify PYTHONPATH to make `tests.superset_test_config` importable
export PYTHONPATH="$GITHUB_WORKSPACE"
celery \
--app=superset.tasks.celery_app:app \
worker \
--concurrency=2 \
--detach \
--optimization=fair
codecov() {
say "::group::Upload code coverage"
bash ".github/workflows/codecov.sh" "$@"
say "::endgroup::"
}
@@ -175,199 +147,71 @@ cypress-install() {
cache-save cypress
}
cypress-run-all() {
local USE_DASHBOARD=$1
local APP_ROOT=$2
# Run Cypress and upload coverage reports
cypress-run() {
cd "$GITHUB_WORKSPACE/superset-frontend/cypress-base"
# Start the Superset backend via gunicorn (not `flask run`). The Flask
# development server is single-threaded and has no crash-recovery, so
# heavy tests (dashboard import/export, SQL Lab) can knock it offline
# for the rest of the run — surfacing as `ECONNREFUSED` / `socket hang up`
# / `Missing CSRF token` cascades. Gunicorn gives us multiple workers,
# a request timeout, and worker-recycling under load.
local serverlog="${HOME}/superset-cypress.log"
local port=8081
CYPRESS_BASE_URL="http://localhost:${port}"
if [ -n "$APP_ROOT" ]; then
export SUPERSET_APP_ROOT=$APP_ROOT
CYPRESS_BASE_URL=${CYPRESS_BASE_URL}${APP_ROOT}
fi
export CYPRESS_BASE_URL
local page=$1
local group=${2:-Default}
local cypress="./node_modules/.bin/cypress run"
local browser=${CYPRESS_BROWSER:-chrome}
# Mirrors the args in docker/entrypoints/run-server.sh (1 worker × 20
# gthread threads) to keep parity with production. Multi-worker
# configurations expose timing-sensitive races in the SQL Lab → Explore
# navigation flow under E2E. We diverge from the entrypoint on:
# --timeout 120: heavy dashboard import/export specs exceed the 60s
# default
# superset.app:create_app(): explicit factory so we don't depend on
# FLASK_APP being exported
#
# No --max-requests, matching the entrypoint's default of 0 (recycling
# off). With a single worker a recycle takes the whole backend offline for
# the graceful-timeout drain — browser keep-alive connections hold it open
# for the full 30s — plus ~5s of app boot. A run issues ~3800 requests in
# ~8 minutes, so recycling every 500 produced seven ~35s outages per run
# and flaked whichever specs happened to navigate into one. Lowering
# --graceful-timeout is not enough: a dashboard load plus chart render
# needs 6-10s, which still lands inside the window.
nohup gunicorn \
--bind "127.0.0.1:$port" \
--workers 1 \
--worker-class gthread \
--threads 20 \
--timeout 120 \
--access-logfile - \
--error-logfile - \
"superset.app:create_app()" \
>"$serverlog" 2>&1 </dev/null &
local serverPid=$!
export TERM="xterm"
# Ensure the backend is cleaned up and its log is emitted even when the
# test runner fails under `set -e`.
trap '
echo "::group::gunicorn log for Cypress run"
cat "'"$serverlog"'" || true
echo "::endgroup::"
kill '"$serverPid"' 2>/dev/null || true
' EXIT
# Wait for the backend to be ready before launching Cypress; otherwise
# the first spec can race the server bind and see connection errors.
local timeout=60
say "Waiting for gunicorn server to start on port $port..."
while [ $timeout -gt 0 ]; do
if curl -f "http://localhost:${port}${APP_ROOT}/health" >/dev/null 2>&1; then
say "gunicorn server is ready"
break
fi
sleep 1
timeout=$((timeout - 1))
done
if [ $timeout -eq 0 ]; then
echo "::error::gunicorn server failed to start within 60 seconds"
echo "::group::Server startup log"
cat "$serverlog"
echo "::endgroup::"
return 1
fi
USE_DASHBOARD_FLAG=''
if [ "$USE_DASHBOARD" = "true" ]; then
USE_DASHBOARD_FLAG='--use-dashboard'
fi
# UNCOMMENT the next few commands to monitor memory usage
# monitor_memory & # Start memory monitoring in the background
# memoryMonitorPid=$!
python ../../scripts/cypress_run.py --retries 5 $USE_DASHBOARD_FLAG
# kill $memoryMonitorPid
}
playwright-install() {
cd "$GITHUB_WORKSPACE/superset-frontend"
say "::group::Install Playwright browsers"
npx playwright install --with-deps chromium
# Create output directories for test results and debugging
mkdir -p playwright-results
mkdir -p test-results
say "::endgroup::"
}
playwright-run() {
local APP_ROOT=$1
local TEST_PATH=$2
# Start the Superset backend via gunicorn from the project root.
# See cypress-run-all() above for the rationale — the Flask dev server
# cannot survive the dashboard import/export tests under load.
cd "$GITHUB_WORKSPACE"
local serverlog="${HOME}/superset-playwright.log"
local port=8081
# Use 127.0.0.1 explicitly: `flask run` binds IPv4 only, and Node's DNS
# resolution for `localhost` can return `::1` first (IPv6), which then
# refuses against the IPv4 listener and surfaces as
# `connect ECONNREFUSED ::1:<port>` in API helpers driven from Node
# (e.g., the embedded test app's exposed token fetcher).
PLAYWRIGHT_BASE_URL="http://127.0.0.1:${port}"
if [ -n "$APP_ROOT" ]; then
export SUPERSET_APP_ROOT=$APP_ROOT
PLAYWRIGHT_BASE_URL=${PLAYWRIGHT_BASE_URL}${APP_ROOT}/
fi
export PLAYWRIGHT_BASE_URL
# See cypress-run-all() above for the args rationale (1 worker × 20
# gthread threads matching docker/entrypoints/run-server.sh, a 120s
# timeout for heavy E2E load, and why worker recycling is off).
nohup gunicorn \
--bind "127.0.0.1:$port" \
--workers 1 \
--worker-class gthread \
--threads 20 \
--timeout 120 \
--access-logfile - \
--error-logfile - \
"superset.app:create_app()" \
>"$serverlog" 2>&1 </dev/null &
local serverPid=$!
# Ensure cleanup on exit (and emit the server log on failure)
trap '
echo "::group::gunicorn log for Playwright run"
cat "'"$serverlog"'" || true
echo "::endgroup::"
kill '"$serverPid"' 2>/dev/null || true
' EXIT
# Wait for server to be ready with health check
local timeout=60
say "Waiting for gunicorn server to start on port $port..."
while [ $timeout -gt 0 ]; do
if curl -f ${PLAYWRIGHT_BASE_URL}/health >/dev/null 2>&1; then
say "gunicorn server is ready"
break
fi
sleep 1
timeout=$((timeout - 1))
done
if [ $timeout -eq 0 ]; then
echo "::error::gunicorn server failed to start within 60 seconds"
echo "::group::Server startup log"
cat "$serverlog"
echo "::endgroup::"
return 1
fi
# Change to frontend directory for Playwright execution
cd "$GITHUB_WORKSPACE/superset-frontend"
say "::group::Run Playwright tests"
echo "Running Playwright with baseURL: ${PLAYWRIGHT_BASE_URL}"
if [ -n "$TEST_PATH" ]; then
# Check if there are any test files in the specified path
if ! find "playwright/tests/${TEST_PATH}" -name "*.spec.ts" -type f 2>/dev/null | grep -q .; then
echo "No test files found in ${TEST_PATH} - skipping test run"
say "::endgroup::"
return 0
fi
echo "Running tests: ${TEST_PATH}"
# Set INCLUDE_EXPERIMENTAL=true to allow experimental tests to run
export INCLUDE_EXPERIMENTAL=true
npx playwright test "${TEST_PATH}" --output=playwright-results
local status=$?
# Unset to prevent leaking into subsequent commands
unset INCLUDE_EXPERIMENTAL
say "::group::Run Cypress for [$page]"
if [[ -z $CYPRESS_KEY ]]; then
$cypress --spec "cypress/e2e/$page" --browser "$browser"
else
echo "Running all required tests (experimental/ excluded via playwright.config.ts)"
npx playwright test --output=playwright-results
local status=$?
export CYPRESS_RECORD_KEY=$(echo $CYPRESS_KEY | base64 --decode)
# additional flags for Cypress dashboard recording
$cypress --spec "cypress/e2e/$page" --browser "$browser" \
--record --group "$group" --tag "${GITHUB_REPOSITORY},${GITHUB_EVENT_NAME}" \
--parallel --ci-build-id "${GITHUB_SHA:0:8}-${NONCE}"
fi
# don't add quotes to $record because we do want word splitting
say "::endgroup::"
}
cypress-run-all() {
# Start Flask and run it in background
# --no-debugger means disable the interactive debugger on the 500 page
# so errors can print to stderr.
local flasklog="${HOME}/flask.log"
local port=8081
export CYPRESS_BASE_URL="http://localhost:${port}"
nohup flask run --no-debugger -p $port >"$flasklog" 2>&1 </dev/null &
local flaskProcessId=$!
cypress-run "*/**/*"
# After job is done, print out Flask log for debugging
say "::group::Flask log for default run"
cat "$flasklog"
say "::endgroup::"
return $status
# Rerun SQL Lab tests with backend persist disabled
export SUPERSET_CONFIG=tests.integration_tests.superset_test_config_sqllab_backend_persist_off
# Restart Flask with new configs
kill $flaskProcessId
nohup flask run --no-debugger -p $port >"$flasklog" 2>&1 </dev/null &
local flaskProcessId=$!
cypress-run "sqllab/*" "Backend persist"
# Upload code coverage separately so each page can have separate flags
# -c will clean existing coverage reports, -F means add flags
# || true to prevent CI failure on codecov upload
codecov -c -F "cypress" || true
say "::group::Flask log for backend persist"
cat "$flasklog"
say "::endgroup::"
# make sure the program exits
kill $flaskProcessId
}
eyes-storybook-dependencies() {
@@ -376,17 +220,27 @@ eyes-storybook-dependencies() {
say "::endgroup::"
}
monitor_memory() {
# This is a small utility to monitor memory usage. Useful for debugging memory in GHA.
# To use wrap your command as follows
#
# monitor_memory & # Start memory monitoring in the background
# memoryMonitorPid=$!
# YOUR_COMMAND_HERE
# kill $memoryMonitorPid
while true; do
echo "$(date) - Top 5 memory-consuming processes:"
ps -eo pid,comm,%mem --sort=-%mem | head -n 6 # First line is the header, next 5 are top processes
sleep 2
done
cypress-run-applitools() {
cd "$GITHUB_WORKSPACE/superset-frontend/cypress-base"
local flasklog="${HOME}/flask.log"
local port=8081
local cypress="./node_modules/.bin/cypress run"
local browser=${CYPRESS_BROWSER:-chrome}
export CYPRESS_BASE_URL="http://localhost:${port}"
nohup flask run --no-debugger -p $port >"$flasklog" 2>&1 </dev/null &
local flaskProcessId=$!
$cypress --spec "cypress/e2e/*/**/*.applitools.test.ts" --browser "$browser" --headless --config ignoreTestFiles="[]"
codecov -c -F "cypress" || true
say "::group::Flask log for default run"
cat "$flasklog"
say "::endgroup::"
# make sure the program exits
kill $flaskProcessId
}
-83
View File
@@ -1,83 +0,0 @@
name: Bump Python Package
on:
# Can be triggered manually
workflow_dispatch:
inputs:
package:
required: false
description: The python package to bump (all if empty)
group:
required: false
description: The optional dependency group to bump (as defined in pyproject.toml)
limit:
required: true
description: Max number of PRs to open (0 for no limit)
default: 5
extra-flags:
required: false
default: --only-base
description: Additional flags to pass to the bump-python command
#schedule:
# - cron: '0 0 * * *' # Runs daily at midnight UTC
jobs:
bump-python-package:
runs-on: ubuntu-slim
permissions:
actions: write
contents: write
pull-requests: write
checks: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: true
ref: master
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
- name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.11"
- name: Install uv
uses: astral-sh/setup-uv@c771a70e6277c0a99b617c7a806ffedaca235ff9 # v9.0.0
with:
python-version: "3.11"
enable-cache: true
- name: supersetbot bump-python -p "${{ github.event.inputs.package }}"
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
INPUT_PACKAGE: ${{ github.event.inputs.package }}
INPUT_GROUP: ${{ github.event.inputs.group }}
INPUT_EXTRA_FLAGS: ${{ github.event.inputs.extra-flags }}
INPUT_LIMIT: ${{ github.event.inputs.limit }}
run: |
git config --global user.email "action@github.com"
git config --global user.name "GitHub Action"
PACKAGE_OPT=""
if [ -n "${INPUT_PACKAGE}" ]; then
PACKAGE_OPT="-p ${INPUT_PACKAGE}"
fi
GROUP_OPT=""
if [ -n "${INPUT_GROUP}" ]; then
GROUP_OPT="-g ${INPUT_GROUP}"
fi
EXTRA_FLAGS="${INPUT_EXTRA_FLAGS}"
supersetbot bump-python \
--verbose \
--use-current-repo \
--include-subpackages \
--limit ${INPUT_LIMIT} \
$PACKAGE_OPT \
$GROUP_OPT \
$EXTRA_FLAGS
+43
View File
@@ -0,0 +1,43 @@
name: Cancel Duplicates
on:
workflow_run:
workflows:
- "Miscellaneous"
types:
- requested
jobs:
cancel-duplicate-runs:
name: Cancel duplicate workflow runs
runs-on: ubuntu-20.04
permissions:
actions: write
contents: read
steps:
- name: Check number of queued tasks
id: check_queued
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPO: ${{ github.repository }}
run: |
get_count() {
echo $(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$GITHUB_REPO/actions/runs?status=$1" | \
jq ".total_count")
}
count=$(( `get_count queued` + `get_count in_progress` ))
echo "Found $count unfinished jobs."
echo "::set-output name=count::$count"
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: steps.check_queued.outputs.count >= 20
uses: actions/checkout@v3
- name: Cancel duplicate workflow runs
if: steps.check_queued.outputs.count >= 20
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
pip install click requests typing_extensions python-dateutil
python ./scripts/cancel_github_workflows.py
@@ -1,77 +0,0 @@
name: Check DB migration conflict
on:
push:
paths:
- "superset/migrations/**"
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
paths:
- "superset/migrations/**"
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
check_db_migration_conflict:
name: Check DB migration conflict
runs-on: ubuntu-slim
permissions:
contents: read
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check and notify
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ github.token }}
script: |
// API reference: https://octokit.github.io/rest.js
const currentBranch = context.ref.replace('refs/heads/', '');
// Find all pull requests to current branch
const opts = github.rest.pulls.list.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
base: context.ref,
state: 'open',
sort: 'updated',
per_page: 100,
});
const pulls = await github.paginate(opts);
if (pulls.length > 0) {
console.log(`Found ${pulls.length} open PRs for base branch "${currentBranch}"`)
}
for (const pull of pulls) {
const listFilesOpts = await github.rest.pulls.listFiles.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pull.number,
});
const files = await github.paginate(listFilesOpts);
if (
files.some(x => x.contents_url.includes('/contents/superset/migrations'))
) {
console.log(`PR #${pull.number} "${pull.title}" also added db migration`)
await github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pull.number,
body:
`# 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️` +
`❗ @${pull.user.login} Your base branch \`${currentBranch}\` has ` +
'also updated `superset/migrations`.\n' +
'\n' +
'**Please consider rebasing your branch and [resolving potential db migration conflicts](https://superset.apache.org/docs/contributing/development#merging-db-migrations).**',
});
}
}
-81
View File
@@ -1,81 +0,0 @@
name: Check python dependencies
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
pull-requests: read
actions: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
check-python-deps:
runs-on: ubuntu-26.04
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
fetch-depth: 1
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
if: steps.check.outputs.python
uses: ./.github/actions/setup-backend/
# Authenticate the Docker daemon so the python:slim pull in
# uv-pip-compile.sh uses our (much higher) authenticated rate limit
# instead of the shared-runner anonymous one. Best-effort: on fork PRs the
# secrets are unavailable, so this no-ops and the pull falls back to
# anonymous (covered by the retry loop in the script).
- name: Login to Docker Hub
if: steps.check.outputs.python
continue-on-error: true
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Run uv
if: steps.check.outputs.python
run: ./scripts/uv-pip-compile.sh
- name: Check for uncommitted changes
if: steps.check.outputs.python
run: |
echo "Full diff (for logging/debugging):"
git diff
echo "Filtered diff (excluding comments and whitespace):"
filtered_diff=$(git diff -U0 | grep '^[-+]' | grep -vE '^[-+]{3}' | grep -vE '^[-+][[:space:]]*#' | grep -vE '^[-+][[:space:]]*$' || true)
echo "$filtered_diff"
if [[ -n "$filtered_diff" ]]; then
echo
echo "ERROR: The pinned dependencies are not up-to-date."
echo "Please run './scripts/uv-pip-compile.sh' and commit the changes."
echo "More info: https://github.com/apache/superset/tree/master/requirements"
exit 1
else
echo "Pinned dependencies are up-to-date."
fi
@@ -0,0 +1,63 @@
name: Check DB migration conflict
on:
push:
paths:
- "superset/migrations/**"
jobs:
check_db_migration_conflict:
name: Check DB migration conflict
runs-on: ubuntu-20.04
permissions:
contents: read
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
- name: Check and notify
uses: actions/github-script@v3
with:
github-token: ${{ github.token }}
script: |
// API reference: https://octokit.github.io/rest.js
const currentBranch = context.ref.replace('refs/heads/', '');
// Find all pull requests to current branch
const opts = github.pulls.list.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
base: context.ref,
state: 'open',
sort: 'updated',
per_page: 100,
});
const pulls = await github.paginate(opts);
if (pulls.length > 0) {
console.log(`Found ${pulls.length} open PRs for base branch "${currentBranch}"`)
}
for (const pull of pulls) {
const listFilesOpts = await github.pulls.listFiles.endpoint.merge({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: pull.number,
});
const files = await github.paginate(listFilesOpts);
if (
files.some(x => x.contents_url.includes('/contents/superset/migrations'))
) {
console.log(`PR #${pull.number} "${pull.title}" also added db migration`)
await github.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: pull.number,
body:
`# 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️ 🙅‍♂️` +
`❗ @${pull.user.login} Your base branch \`${currentBranch}\` has ` +
'also updated `superset/migrations`.\n' +
'\n' +
'**Please consider rebasing your branch and [resolving potential db migration conflicts](https://github.com/apache/superset/blob/master/CONTRIBUTING.md#merging-db-migrations).**',
});
}
}
+72
View File
@@ -0,0 +1,72 @@
# .github/workflows/chromatic.yml
# seee https://www.chromatic.com/docs/github-actions
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Workflow name
name: 'Chromatic Storybook Master'
# Event for the workflow
# Only run if changes were made in superset-frontend folder of repo on merge to Master
on:
# This will trigger when a branch merges to master when the PR has changes in the frontend folder updating the chromatic baseline
push:
branches:
- master
paths:
- "superset-frontend/**"
# List of jobs
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.CHROMATIC_PROJECT_TOKEN != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
chromatic-deployment:
needs: config
if: needs.config.outputs.has-secrets
# Operating System
runs-on: ubuntu-latest
# Job steps
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0 # 👈 Required to retrieve git history
- name: Install dependencies
run: npm ci
working-directory: superset-frontend
# 👇 Build and publish Storybook to Chromatic
- name: Build and publish Storybook to Chromatic
id: chromatic-master
uses: chromaui/action@v1
# Required options for the Chromatic GitHub Action
with:
# 👇 Location of package.json from root of mono-repo
workingDir: superset-frontend
# 👇 Chromatic projectToken, refer to the manage page to obtain it.
projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
exitZeroOnChanges: true # 👈 Option to prevent the workflow from failing
autoAcceptChanges: true # 👈 Option to accept all changes when merging to master
+1903
View File
File diff suppressed because it is too large Load Diff
+12 -42
View File
@@ -2,69 +2,39 @@ name: "CodeQL"
on:
push:
branches: ["master", "[0-9].[0-9]*"]
branches: [ "master" ]
paths:
- 'superset/**'
pull_request:
# The branches below must be a subset of the branches above
branches: ["master"]
branches: [ "master" ]
paths:
- 'superset/**'
schedule:
- cron: "0 4 * * *"
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
- cron: '0 4 * * *'
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
analyze:
name: Analyze
needs: changes
# Skip on PRs that touch neither code group (e.g. docs-only) so the
# analysis runners don't spin up. push/schedule runs always proceed:
# the change-detector returns "all changed" for non-PR events.
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 30
runs-on: ubuntu-22.04
permissions:
actions: read
contents: read
pull-requests: read
security-events: write
strategy:
fail-fast: false
matrix:
language: ["python", "javascript"]
language: [ 'python', 'javascript' ]
# Learn more about CodeQL language support at https://aka.ms/codeql-docs/language-support
steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
uses: actions/checkout@v3
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6
uses: github/codeql-action/init@v2
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -75,6 +45,6 @@ jobs:
# queries: security-extended,security-and-quality
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@5595ccaf912efad79be6eef63a5619ff05969be3 # v4.37.6
uses: github/codeql-action/analyze@v2
with:
category: "/language:${{matrix.language}}"
@@ -1,63 +0,0 @@
name: Auto-approve Dependabot patch bumps
# Posts an approving review on Dependabot PRs that only bump a patch
# version, using the same trigger/guard convention already proven to work
# for Dependabot PRs in sync-requirements-for-python-dep-upgrade-pr.yml
# (plain `pull_request` gets a working, write-capable GITHUB_TOKEN here
# because Dependabot pushes branches directly to this repo, not a fork).
#
# This does NOT auto-merge anything - repo-wide auto-merge is disabled
# (Settings > General > Pull Requests > "Allow auto-merge" is off), and
# flipping that is a separate, repo-wide decision this workflow doesn't
# make on its own. Branch protection also still requires 1 approving
# review; this just means that review can already exist by the time a
# human looks at the PR, for the (large majority of) ecosystems whose
# files aren't matched by any CODEOWNERS pattern. One ecosystem - the npm
# bump under .github/actions - matches the /.github/ CODEOWNERS entry, so
# those PRs will still need a human owner's approval regardless of this
# workflow; it posts a review there too, but that alone won't satisfy the
# code-owner requirement.
on:
pull_request:
types: [opened, synchronize]
# Cancel a superseded run if Dependabot pushes to the same PR again before
# the previous run finished (matches the pattern used elsewhere in
# superset-docs-verify.yml).
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
cancel-in-progress: true
permissions: {}
jobs:
approve-patch-bump:
name: Approve patch-level bump
# Mirrors the guard in sync-requirements-for-python-dep-upgrade-pr.yml:
# limited to (1) PRs authored by Dependabot and (2) the upstream repo,
# since forked PRs don't get a write-capable token here anyway.
if: >
github.repository == 'apache/superset' &&
github.event.pull_request.user.login == 'dependabot[bot]' &&
github.event.pull_request.head.repo.fork == false
runs-on: ubuntu-latest
permissions:
pull-requests: write # to post the approving review via `gh pr review`
steps:
- name: Fetch Dependabot metadata
id: metadata
# This exact SHA is on ASF Infra's action allowlist
# (apache/infrastructure-actions approved_patterns.yml) as of this
# writing. Do not bump without opening an Infra ticket to allow
# the new SHA first!
uses: dependabot/fetch-metadata@25dd0e34f4fe68f24cc83900b1fe3fe149efef98 # v3.1.0
- name: Approve patch-level bump
if: steps.metadata.outputs.update-type == 'version-update:semver-patch'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
PR_URL: ${{ github.event.pull_request.html_url }}
DEPENDENCY_NAMES: ${{ steps.metadata.outputs.dependency-names }}
run: |
gh pr review --approve "$PR_URL" \
--body "Auto-approved: patch-level bump only ($DEPENDENCY_NAMES)."
+8 -58
View File
@@ -4,72 +4,22 @@
#
# Source repository: https://github.com/actions/dependency-review-action
# Public documentation: https://docs.github.com/en/code-security/supply-chain-security/understanding-your-software-supply-chain/about-dependency-review#dependency-review-enforcement
name: "Dependency Review"
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
name: 'Dependency Review'
on: [pull_request]
permissions:
contents: read
actions: read
jobs:
dependency-review:
if: github.event_name == 'pull_request'
runs-on: ubuntu-slim
runs-on: ubuntu-latest
steps:
- name: "Checkout Repository"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: 'Checkout Repository'
uses: actions/checkout@v3
- name: 'Dependency Review'
uses: actions/dependency-review-action@v2
with:
persist-credentials: false
- name: "Dependency Review"
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
with:
fail-on-severity: critical
fail-on-severity: high
# compatible/incompatible licenses addressed here: https://www.apache.org/legal/resolved.html
# find SPDX identifiers here: https://spdx.org/licenses/
deny-licenses: MS-LPL, BUSL-1.1, QPL-1.0, Sleepycat, SSPL-1.0, CPOL-1.02, AGPL-3.0, GPL-1.0+, BSD-4-Clause-UC, NPL-1.0, NPL-1.1, JSON
# pkg:npm/store2@2.14.2
# adding an exception for an ambigious license on store2, which has been resolved in
# the latest version. It's MIT: https://github.com/nbubna/store/blob/master/LICENSE-MIT
# pkg:npm/node-forge@1.3.1
# selecting BSD-3-Clause licensing terms for node-forge to ensure compatibility with Apache
allow-dependencies-licenses: pkg:npm/rgbcolor, pkg:npm/jszip@3.10.1
python-dependency-liccheck:
# NOTE: Configuration for liccheck lives in our pyproject.yml.
# You cannot use a liccheck.ini file in this workflow.
runs-on: ubuntu-slim
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: "Checkout Repository"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
requirements-type: base
- name: "Set up liccheck"
run: |
uv pip install --system liccheck
- name: "Run liccheck"
run: |
# run the checks
liccheck -R output.txt
# Print the report
cat output.txt
+100
View File
@@ -0,0 +1,100 @@
name: Push ephemeral env image
on:
workflow_run:
workflows: ["Docker"]
types:
- completed
jobs:
config:
runs-on: "ubuntu-latest"
if: github.event.workflow_run.event == 'pull_request' && github.event.workflow_run.conclusion == 'success'
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.AWS_ACCESS_KEY_ID != '' &&
secrets.AWS_ACCESS_KEY_ID != '' &&
secrets.AWS_SECRET_ACCESS_KEY != '' &&
secrets.AWS_SECRET_ACCESS_KEY != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
echo "has secrets!"
else
echo "has-secrets=0" >> "$GITHUB_OUTPUT"
echo "no secrets!"
fi
docker_ephemeral_env:
needs: config
if: needs.config.outputs.has-secrets
name: Push ephemeral env Docker image to ECR
runs-on: ubuntu-latest
steps:
- name: "Download artifact"
uses: actions/github-script@v3.1.0
with:
script: |
const artifacts = await github.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: ${{ github.event.workflow_run.id }},
});
core.info('*** artifacts')
core.info(JSON.stringify(artifacts))
const matchArtifact = artifacts.data.artifacts.filter((artifact) => {
return artifact.name == "build"
})[0];
if(!matchArtifact) return core.setFailed("Build artifacts not found")
const download = await github.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: matchArtifact.id,
archive_format: 'zip',
});
var fs = require('fs');
fs.writeFileSync('${{github.workspace}}/build.zip', Buffer.from(download.data));
- run: unzip build.zip
- name: Display downloaded files (debug)
run: ls -la
- name: Get SHA
id: get-sha
run: echo "::set-output name=sha::$(cat ./SHA)"
- name: Get PR
id: get-pr
run: echo "::set-output name=num::$(cat ./PR-NUM)"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Load, tag and push image to ECR
id: push-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: superset-ci
SHA: ${{ steps.get-sha.outputs.sha }}
IMAGE_TAG: pr-${{ steps.get-pr.outputs.num }}
run: |
docker load < $SHA.tar.gz
docker tag $SHA $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG
docker tag $SHA $ECR_REGISTRY/$ECR_REPOSITORY:$SHA
docker push -a $ECR_REGISTRY/$ECR_REPOSITORY
+38
View File
@@ -0,0 +1,38 @@
name: Docker
on:
release:
types: [published]
jobs:
config:
runs-on: "ubuntu-latest"
if: github.event.pull_request.draft == false
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.DOCKERHUB_USER != '' && secrets.DOCKERHUB_TOKEN != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
docker-release:
needs: config
if: needs.config.outputs.has-secrets
name: docker-release
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.ref }}
- shell: bash
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
.github/workflows/docker_build_push.sh
+41 -255
View File
@@ -1,275 +1,61 @@
name: Build & publish docker images
name: Docker
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
- 'master'
pull_request:
branches:
- "master"
permissions:
contents: read
pull-requests: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
types: [synchronize, opened, reopened, ready_for_review]
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
config:
runs-on: "ubuntu-latest"
if: github.event.pull_request.draft == false
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
docker: ${{ steps.check.outputs.docker }}
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
- name: "Check for secrets"
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
setup_matrix:
runs-on: ubuntu-slim
timeout-minutes: 5
outputs:
matrix_config: ${{ steps.set_matrix.outputs.matrix_config }}
steps:
- id: set_matrix
run: |
MATRIX_CONFIG=$(if [ "${{ github.event_name }}" == "pull_request" ]; then echo '["dev", "lean"]'; else echo '["dev", "lean", "websocket", "dockerize", "py311", "py312"]'; fi)
echo "matrix_config=${MATRIX_CONFIG}" >> $GITHUB_OUTPUT
echo $GITHUB_OUTPUT
# Runs unconditionally (no dependency on `changes`, and no build-preset
# matrix restriction) so a regression in the PY_VER override logic is
# always caught on PRs. Without this, the real docker-build job only runs
# when the change detector flags docker/python/frontend changes (a
# workflow-only edit like this one does not), and even then the PR build
# matrix never includes the "py311"/"py312" presets that logic protects -
# so a break here would otherwise first surface on a push to master.
pyver-override-check:
name: verify docker build PY_VER override
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
- name: Assert PY_VER override applies to every preset except py311/py312
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
set -euo pipefail
# Asserts against the actual buildx command line `supersetbot docker
# --dry-run` would run, not just this repo's own extra-flags helper,
# so a regression in supersetbot itself (dropping the py311/py312
# PY_VER pin, or reordering args so our override no longer lands
# last) is caught here too, instead of only surfacing on master.
assert_effective_py_ver() {
local preset="$1" expected="$2" extra_flags command actual
extra_flags="$(scripts/docker-build-extra-flags.sh "$preset" dummy-tag)"
command="$(supersetbot docker --preset "$preset" --platform linux/amd64 --extra-flags "$extra_flags" --dry-run)"
# docker buildx keeps the LAST value of a repeated --build-arg key.
actual="$(grep -oE -- '--build-arg PY_VER=[^[:space:]]+' <<<"$command" | tail -1)"
if [ "$actual" != "--build-arg PY_VER=$expected" ]; then
echo "::error::preset '$preset' expected effective --build-arg PY_VER=$expected, got: ${actual:-<none>} (full command: $command)"
exit 1
fi
}
for preset in dev lean websocket dockerize; do
assert_effective_py_ver "$preset" "3.11.14-slim-trixie"
done
assert_effective_py_ver py311 "3.11-slim-bookworm"
assert_effective_py_ver py312 "3.12-slim-bookworm"
echo "PY_VER override logic verified against the assembled buildx command for all build presets"
docker-build:
name: docker-build
needs: [setup_matrix, changes]
if: >-
needs.changes.outputs.python == 'true' ||
needs.changes.outputs.frontend == 'true' ||
needs.changes.outputs.docker == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 60
strategy:
matrix:
build_preset: ${{fromJson(needs.setup_matrix.outputs.matrix_config)}}
fail-fast: false
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
IMAGE_TAG: apache/superset:GHA-${{ matrix.build_preset }}-${{ github.run_id }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Free up disk space
shell: bash
run: |
# Reclaim large preinstalled toolchains we don't use. The image
# build, and especially the docker-compose sanity check (which
# rebuilds from scratch whenever the registry cache image
# apache/superset-cache is unavailable), can otherwise exhaust the
# runner's root disk and fail with "no space left on device".
echo "Disk before cleanup:"; df -h /
sudo rm -rf \
/usr/share/dotnet \
/usr/local/lib/android \
/opt/ghc \
/usr/local/.ghcup \
/opt/hostedtoolcache/CodeQL \
/usr/local/share/boost || true
echo "Disk after cleanup:"; df -h /
- name: Setup Docker Environment
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
build: "true"
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
- name: Build Docker Image
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_PRESET: ${{ matrix.build_preset }}
run: |
# Single platform builds in pull_request context to speed things up
if [ "$GITHUB_EVENT_NAME" = "push" ]; then
PLATFORM_ARG="--platform linux/arm64 --platform linux/amd64"
# can only --load images in single-platform builds
PUSH_OR_LOAD="--push"
elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
PLATFORM_ARG="--platform linux/amd64"
PUSH_OR_LOAD="--load"
if [ -n "${{ (secrets.DOCKERHUB_USER != '' && secrets.DOCKERHUB_TOKEN != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
echo "has secrets!"
else
echo "has-secrets=0" >> "$GITHUB_OUTPUT"
echo "no secrets!"
fi
# Retry to absorb transient Docker Hub registry errors (base-image
# pull timeouts, 504/401 on push, ECONNRESET) that otherwise fail
# the whole job. buildx reuses the buildkit layer cache from the
# failed attempt, so a retry mostly re-does just the failed push.
#
# See scripts/docker-build-extra-flags.sh for why "py311"/"py312"
# are excluded from the PY_VER override applied to every other
# preset; that logic is also exercised on every PR by the
# always-on pyver-override-check job below, since this job itself
# only runs when the change detector flags docker/python/frontend
# changes and the PR build matrix never includes py311/py312.
EXTRA_FLAGS="$(scripts/docker-build-extra-flags.sh "$BUILD_PRESET" "$IMAGE_TAG")"
for attempt in 1 2 3; do
if supersetbot docker \
$PUSH_OR_LOAD \
--preset "$BUILD_PRESET" \
--context "$EVENT" \
--context-ref "$RELEASE" $FORCE_LATEST \
--extra-flags "$EXTRA_FLAGS" \
$PLATFORM_ARG; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "::error::supersetbot docker build failed after 3 attempts"
exit 1
fi
echo "::warning::Build attempt ${attempt} failed; retrying in 30s..."
sleep 30
done
# in the context of push (using multi-platform build), we need to pull the image locally
- name: Docker pull
if: github.event_name == 'push'
run: |
for i in 1 2 3; do
docker pull $IMAGE_TAG && break
[ $i -lt 3 ] && sleep 30
done
- name: Print docker stats
run: |
echo "SHA: ${{ github.sha }}"
echo "IMAGE: $IMAGE_TAG"
docker images $IMAGE_TAG
docker history $IMAGE_TAG
- name: docker-compose sanity check
if: matrix.build_preset == 'dev'
shell: bash
env:
BUILD_PRESET: ${{ matrix.build_preset }}
run: |
export SUPERSET_BUILD_TARGET=$BUILD_PRESET
# This should reuse the CACHED image built in the previous steps
docker compose build superset-init --build-arg DEV_MODE=false --build-arg INCLUDE_CHROMIUM=false
docker compose up superset-init --exit-code-from superset-init
docker-compose-image-tag:
# Run this job only on pushes to master (not for PRs)
# goal is to check that building the latest image works, not required for all PR pushes
needs: changes
if: github.event_name == 'push' && github.ref == 'refs/heads/master' && needs.changes.outputs.docker == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 30
docker-build:
needs: config
if: needs.config.outputs.has-secrets
name: docker-build
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
- name: Free up disk space
shell: bash
run: |
# The sanity check rebuilds the image from scratch whenever the
# registry cache image apache/superset-cache is unavailable, which
# can exhaust the runner's root disk ("no space left on device").
echo "Disk before cleanup:"; df -h /
sudo rm -rf \
/usr/share/dotnet \
/usr/local/lib/android \
/opt/ghc \
/usr/local/.ghcup \
/opt/hostedtoolcache/CodeQL \
/usr/local/share/boost || true
echo "Disk after cleanup:"; df -h /
- name: Setup Docker Environment
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
build: "false"
install-docker-compose: "true"
- name: docker-compose sanity check
shell: bash
run: |
docker compose -f docker-compose-image-tag.yml up superset-init --exit-code-from superset-init
actions-timeline:
needs: [docker-build, docker-compose-image-tag]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
- shell: bash
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
.github/workflows/docker_build_push.sh
- name: Build ephemeral env image
if: github.event_name == 'pull_request'
run: |
mkdir -p ./build
echo ${{ github.sha }} > ./build/SHA
echo ${{ github.event.pull_request.number }} > ./build/PR-NUM
DOCKER_BUILDKIT=1 docker build --target ci -t ${{ github.sha }} -t "pr-${{ github.event.pull_request.number }}" .
docker save ${{ github.sha }} | gzip > ./build/${{ github.sha }}.tar.gz
- name: Upload build artifacts
if: github.event_name == 'pull_request'
uses: actions/upload-artifact@v3
with:
expand-composite-actions: true
name: build
path: build/
+118
View File
@@ -0,0 +1,118 @@
#!/usr/bin/env bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -eo pipefail
SHA=$(git rev-parse HEAD)
REPO_NAME="apache/superset"
if [[ "${GITHUB_EVENT_NAME}" == "pull_request" ]]; then
REFSPEC=$(echo "${GITHUB_HEAD_REF}" | sed 's/[^a-zA-Z0-9]/-/g' | head -c 40)
PR_NUM=$(echo "${GITHUB_REF}" | sed 's:refs/pull/::' | sed 's:/merge::')
LATEST_TAG="pr-${PR_NUM}"
elif [[ "${GITHUB_EVENT_NAME}" == "release" ]]; then
REFSPEC=$(echo "${GITHUB_REF}" | sed 's:refs/tags/::' | head -c 40)
LATEST_TAG="${REFSPEC}"
else
REFSPEC=$(echo "${GITHUB_REF}" | sed 's:refs/heads/::' | sed 's/[^a-zA-Z0-9]/-/g' | head -c 40)
LATEST_TAG="${REFSPEC}"
fi
if [[ "${REFSPEC}" == "master" ]]; then
LATEST_TAG="latest"
fi
cat<<EOF
Rolling with tags:
- ${REPO_NAME}:${SHA}
- ${REPO_NAME}:${REFSPEC}
- ${REPO_NAME}:${LATEST_TAG}
EOF
#
# Build the "lean" image
#
DOCKER_BUILDKIT=1 docker build --target lean \
-t "${REPO_NAME}:${SHA}" \
-t "${REPO_NAME}:${REFSPEC}" \
-t "${REPO_NAME}:${LATEST_TAG}" \
--label "sha=${SHA}" \
--label "built_at=$(date)" \
--label "target=lean" \
--label "build_actor=${GITHUB_ACTOR}" \
.
#
# Build the "lean310" image
#
DOCKER_BUILDKIT=1 docker build --target lean \
-t "${REPO_NAME}:${SHA}-py310" \
-t "${REPO_NAME}:${REFSPEC}-py310" \
-t "${REPO_NAME}:${LATEST_TAG}-py310" \
--build-arg PY_VER="3.10-slim-bookworm"\
--label "sha=${SHA}" \
--label "built_at=$(date)" \
--label "target=lean310" \
--label "build_actor=${GITHUB_ACTOR}" \
.
#
# Build the "websocket" image
#
DOCKER_BUILDKIT=1 docker build \
-t "${REPO_NAME}:${SHA}-websocket" \
-t "${REPO_NAME}:${REFSPEC}-websocket" \
-t "${REPO_NAME}:${LATEST_TAG}-websocket" \
--label "sha=${SHA}" \
--label "built_at=$(date)" \
--label "target=websocket" \
--label "build_actor=${GITHUB_ACTOR}" \
superset-websocket
#
# Build the dev image
#
DOCKER_BUILDKIT=1 docker build --target dev \
-t "${REPO_NAME}:${SHA}-dev" \
-t "${REPO_NAME}:${REFSPEC}-dev" \
-t "${REPO_NAME}:${LATEST_TAG}-dev" \
--label "sha=${SHA}" \
--label "built_at=$(date)" \
--label "target=dev" \
--label "build_actor=${GITHUB_ACTOR}" \
.
#
# Build the dockerize image
#
DOCKER_BUILDKIT=1 docker build \
-t "${REPO_NAME}:dockerize" \
--label "sha=${SHA}" \
--label "built_at=$(date)" \
--label "build_actor=${GITHUB_ACTOR}" \
-f dockerize.Dockerfile \
.
if [ -z "${DOCKERHUB_TOKEN}" ]; then
# Skip if secrets aren't populated -- they're only visible for actions running in the repo (not on forks)
echo "Skipping Docker push"
else
# Login and push
docker logout
docker login --username "${DOCKERHUB_USER}" --password "${DOCKERHUB_TOKEN}"
docker push --all-tags "${REPO_NAME}"
fi
+23 -23
View File
@@ -3,36 +3,36 @@ name: Embedded SDK Release
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
permissions:
contents: read
- 'master'
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.NPM_TOKEN != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
build:
# Publishing uses npm trusted publishing (OIDC), so there is no NPM_TOKEN to
# gate on. Restrict to the canonical repo: forks cannot mint a valid OIDC
# token for this package and must not publish.
if: github.repository == 'apache/superset'
runs-on: ubuntu-slim
permissions:
contents: read
id-token: write # required for npm trusted publishing (OIDC)
needs: config
if: needs.config.outputs.has-secrets
runs-on: ubuntu-20.04
defaults:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
persist-credentials: false
# Note: registry-url is intentionally omitted. When set, actions/setup-node
# writes an .npmrc with `_authToken=${NODE_AUTH_TOKEN}` and a placeholder
# token, which makes npm attempt token auth and skip the OIDC
# trusted-publishing exchange. With no .npmrc auth line, npm authenticates
# via OIDC against the default registry (registry.npmjs.org).
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./superset-embedded-sdk/.nvmrc"
node-version: "16"
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm run ci:release
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+6 -15
View File
@@ -6,28 +6,19 @@ on:
- "superset-embedded-sdk/**"
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
embedded-sdk-test:
runs-on: ubuntu-slim
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
defaults:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v3
- uses: actions/setup-node@v3
with:
persist-credentials: false
- uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./superset-embedded-sdk/.nvmrc"
registry-url: "https://registry.npmjs.org"
node-version: "16"
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm test
- run: npm run build
@@ -1,82 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
name: Enforce single Alembic migration head
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# No `paths:` filter on purpose: this job is a required status check, and a
# required check that never runs for a given PR blocks that PR from merging
# forever. It has to fire on every PR so it always reports a status; whether
# migrations changed is decided inside the job, not the trigger.
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
enforce-single-migration-head:
runs-on: ubuntu-26.04
permissions:
contents: read
pull-requests: read
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for migration file changes
id: check
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ github.token }}
script: |
if (context.eventName === 'push') {
core.setOutput('changed', 'true');
return;
}
const files = await github.paginate(github.rest.pulls.listFiles, {
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.issue.number,
});
const changed = files.some((f) => f.filename.startsWith('superset/migrations/'));
core.setOutput('changed', String(changed));
- name: Setup Python
if: steps.check.outputs.changed == 'true'
uses: ./.github/actions/setup-backend/
with:
requirements-type: base
- name: Assert a single Alembic head
if: steps.check.outputs.changed == 'true'
env:
SUPERSET__SQLALCHEMY_DATABASE_URI: "sqlite:///:memory:"
run: |
heads="$(superset db heads)"
echo "$heads"
head_count=$(printf '%s\n' "$heads" | grep -c .)
if [ "$head_count" -ne 1 ]; then
echo "::error::superset/migrations resolves to $head_count Alembic heads (expected exactly 1)."
echo "Another migration already landed with the same down_revision this branch was cut from."
echo "Add a no-op merge revision joining the heads: https://superset.apache.org/docs/contributing/development#merging-db-migrations"
exit 1
fi
@@ -0,0 +1,75 @@
name: Cleanup ephemeral envs (PR close)
on:
pull_request_target:
types: [closed]
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.AWS_ACCESS_KEY_ID != '' && secrets.AWS_SECRET_ACCESS_KEY != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
ephemeral-env-cleanup:
needs: config
if: needs.config.outputs.has-secrets
name: Cleanup ephemeral envs
runs-on: ubuntu-latest
permissions:
pull-requests: write
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Describe ECS service
id: describe-services
run: |
echo "::set-output name=active::$(aws ecs describe-services --cluster superset-ci --services pr-${{ github.event.number }}-service | jq '.services[] | select(.status == "ACTIVE") | any')"
- name: Delete ECS service
if: steps.describe-services.outputs.active == 'true'
id: delete-service
run: |
aws ecs delete-service \
--cluster superset-ci \
--service pr-${{ github.event.number }}-service \
--force
- name: Login to Amazon ECR
if: steps.describe-services.outputs.active == 'true'
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Delete ECR image tag
if: steps.describe-services.outputs.active == 'true'
id: delete-image-tag
run: |
aws ecr batch-delete-image \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-${{ github.event.number }}
- name: Comment (success)
if: steps.describe-services.outputs.active == 'true'
uses: actions/github-script@v3
with:
github-token: ${{github.token}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: 'Ephemeral environment shutdown and build artifacts deleted.'
})
+214
View File
@@ -0,0 +1,214 @@
name: Ephemeral env workflow
on:
issue_comment:
types: [created]
jobs:
config:
runs-on: "ubuntu-latest"
if: github.event.issue.pull_request
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.AWS_ACCESS_KEY_ID != '' && secrets.AWS_SECRET_ACCESS_KEY != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
ephemeral_env_comment:
needs: config
if: needs.config.outputs.has-secrets
name: Evaluate ephemeral env comment trigger (/testenv)
runs-on: ubuntu-latest
permissions:
pull-requests: write
outputs:
slash-command: ${{ steps.eval-body.outputs.result }}
feature-flags: ${{ steps.eval-feature-flags.outputs.result }}
steps:
- name: Debug
run: |
echo "Comment on PR #${{ github.event.issue.number }} by ${{ github.event.issue.user.login }}, ${{ github.event.comment.author_association }}"
- name: Eval comment body for /testenv slash command
uses: actions/github-script@v3
id: eval-body
with:
result-encoding: string
script: |
const pattern = /^\/testenv (up|down)/
const result = pattern.exec(context.payload.comment.body)
return result === null ? 'noop' : result[1]
- name: Eval comment body for feature flags
uses: actions/github-script@v3
id: eval-feature-flags
with:
script: |
const pattern = /FEATURE_(\w+)=(\w+)/g;
let results = [];
[...context.payload.comment.body.matchAll(pattern)].forEach(match => {
const config = {
name: `SUPERSET_FEATURE_${match[1]}`,
value: match[2],
};
results.push(config);
});
return results;
- name: Limit to committers
if: >
steps.eval-body.outputs.result != 'noop' &&
github.event.comment.author_association != 'MEMBER' &&
github.event.comment.author_association != 'OWNER'
uses: actions/github-script@v3
with:
github-token: ${{github.token}}
script: |
const errMsg = '@${{ github.event.comment.user.login }} Ephemeral environment creation is currently limited to committers.'
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: errMsg
})
core.setFailed(errMsg)
ephemeral_env_up:
needs: ephemeral_env_comment
if: needs.ephemeral_env_comment.outputs.slash-command == 'up'
name: Spin up an ephemeral environment
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@v3
with:
persist-credentials: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v1
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@v1
- name: Check target image exists in ECR
id: check-image
continue-on-error: true
run: |
aws ecr describe-images \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-${{ github.event.issue.number }}
- name: Fail on missing container image
if: steps.check-image.outcome == 'failure'
uses: actions/github-script@v3
with:
github-token: ${{github.token}}
script: |
const errMsg = '@${{ github.event.comment.user.login }} Container image not yet published for this PR. Please try again when build is complete.'
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: errMsg
})
core.setFailed(errMsg)
- name: Fill in the new image ID in the Amazon ECS task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@v1
with:
task-definition: .github/workflows/ecs-task-definition.json
container-name: superset-ci
image: ${{ steps.login-ecr.outputs.registry }}/superset-ci:pr-${{ github.event.issue.number }}
- name: Update env vars in the Amazon ECS task definition
run: |
cat <<< "$(jq '.containerDefinitions[0].environment += ${{ needs.ephemeral_env_comment.outputs.feature-flags }}' < ${{ steps.task-def.outputs.task-definition }})" > ${{ steps.task-def.outputs.task-definition }}
- name: Describe ECS service
id: describe-services
run: |
echo "::set-output name=active::$(aws ecs describe-services --cluster superset-ci --services pr-${{ github.event.issue.number }}-service | jq '.services[] | select(.status == "ACTIVE") | any')"
- name: Create ECS service
if: steps.describe-services.outputs.active != 'true'
id: create-service
env:
ECR_SUBNETS: subnet-0e15a5034b4121710,subnet-0e8efef4a72224974
ECR_SECURITY_GROUP: sg-092ff3a6ae0574d91
run: |
aws ecs create-service \
--cluster superset-ci \
--service-name pr-${{ github.event.issue.number }}-service \
--task-definition superset-ci \
--launch-type FARGATE \
--desired-count 1 \
--platform-version LATEST \
--network-configuration "awsvpcConfiguration={subnets=[$ECR_SUBNETS],securityGroups=[$ECR_SECURITY_GROUP],assignPublicIp=ENABLED}" \
--tags key=pr,value=${{ github.event.issue.number }} key=github_user,value=${{ github.actor }}
- name: Deploy Amazon ECS task definition
id: deploy-task
uses: aws-actions/amazon-ecs-deploy-task-definition@v1
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: pr-${{ github.event.issue.number }}-service
cluster: superset-ci
wait-for-service-stability: true
wait-for-minutes: 10
- name: List tasks
id: list-tasks
run: |
echo "::set-output name=task::$(aws ecs list-tasks --cluster superset-ci --service-name pr-${{ github.event.issue.number }}-service | jq '.taskArns | first')"
- name: Get network interface
id: get-eni
run: |
echo "::set-output name=eni::$(aws ecs describe-tasks --cluster superset-ci --tasks ${{ steps.list-tasks.outputs.task }} | jq '.tasks | .[0] | .attachments | .[0] | .details | map(select(.name=="networkInterfaceId")) | .[0] | .value')"
- name: Get public IP
id: get-ip
run: |
echo "::set-output name=ip::$(aws ec2 describe-network-interfaces --network-interface-ids ${{ steps.get-eni.outputs.eni }} | jq -r '.NetworkInterfaces | first | .Association.PublicIp')"
- name: Comment (success)
if: ${{ success() }}
uses: actions/github-script@v3
with:
github-token: ${{github.token}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '@${{ github.event.comment.user.login }} Ephemeral environment spinning up at http://${{ steps.get-ip.outputs.ip }}:8080. Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping and startup.'
})
- name: Comment (failure)
if: ${{ failure() }}
uses: actions/github-script@v3
with:
github-token: ${{github.token}}
script: |
github.issues.createComment({
issue_number: ${{ github.event.issue.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '@${{ github.event.comment.user.login }} Ephemeral environment creation failed. Please check the Actions logs for details.'
})
@@ -1,135 +0,0 @@
name: Frontend bundle size (nightly baseline + analyzer)
# Refreshes the bundle-size baseline that superset-frontend.yml's `bundle-size`
# job compares PRs against, and publishes a browsable bundle-analyzer treemap
# report of the same build. Deliberately NOT triggered on every push to
# master: a day-old baseline/report is fine for catching relative
# regressions on PRs and for browsing what's actually in the bundle, and
# building the production bundle on every one of the many pushes master
# gets per day would burn CI time for no benefit a nightly refresh doesn't
# already cover.
on:
schedule:
- cron: "0 6 * * *"
workflow_dispatch: {}
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: true
env:
TAG: apache/superset:bundle-size-nightly-${{ github.run_id }}
permissions:
contents: read
jobs:
refresh-baseline:
runs-on: ubuntu-26.04
timeout-minutes: 30
env:
NETLIFY_SITE_ID: ${{ secrets.NETLIFY_BUNDLE_ANALYZER_SITE_ID }}
steps:
- name: "Checkout master"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: master
- name: Build Docker Image
run: |
docker buildx build \
-t $TAG \
--cache-from=type=registry,ref=apache/superset-cache:3.11-slim-trixie \
--target superset-node-ci \
.
# Same cache the PR-time bundle-size job restores/writes -- webpack's
# persistent filesystem cache turns a warm production build into ~20s
# instead of several minutes. See superset-frontend.yml for the
# matching restore step and why it's keyed this way.
- name: Restore webpack build cache
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: superset-frontend/.temp_cache
key: >-
webpack-prod-cache-${{ hashFiles('superset-frontend/package-lock.json',
'superset-frontend/babel.config.js', 'superset-frontend/tsconfig.json',
'superset-frontend/webpack.config.js') }}
# Only ever pull the last recorded data point off the cache, keyed by
# run ID -- `restore-keys` prefix-matches the most recently created
# entry. Absent on the very first run ever; benchmark-action starts a
# fresh history in that case.
- name: Restore bundle size history
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: bundle-size-history.json
key: bundle-size-history-${{ github.run_id }}
restore-keys: |
bundle-size-history-
# BUNDLE_ANALYZER rides along in the same build as BUNDLE_SIZE_STATS --
# they're independent env-gated additions in webpack.config.js (one
# sets `config.stats`, the other pushes plugins), so one production
# build produces both the numeric stats.json and the analyzer's
# report.html. Only report.html is mounted out, not
# BUNDLE_ANALYZER's sibling `statistics.html` sunburst -- that file is
# documented in webpack.config.js as routinely exceeding 100MB for
# this app (it's .gitignore'd for exactly that reason), too large to
# publish as a static site page.
- name: Build production bundle with stats and analyzer report
run: |
mkdir -p ${{ github.workspace }}/superset-frontend/bundle-stats
mkdir -p ${{ github.workspace }}/superset-frontend/.temp_cache
mkdir -p ${{ github.workspace }}/superset/static/assets
docker run \
-v ${{ github.workspace }}/superset-frontend/bundle-stats:/app/superset-frontend/bundle-stats \
-v ${{ github.workspace }}/superset-frontend/.temp_cache:/app/superset-frontend/.temp_cache \
-v ${{ github.workspace }}/superset/static/assets:/app/superset/static/assets \
--rm $TAG \
bash -c \
"npm i && BUNDLE_SIZE_STATS=true BUNDLE_ANALYZER=true npm run build -- --json=bundle-stats/stats.json"
- name: Summarize bundle size
run: |
node superset-frontend/scripts/bundle-size-summary.js \
superset-frontend/bundle-stats/stats.json > bundle-size-summary.json
rm -rf superset-frontend/bundle-stats
# No PR to comment on here, so comment-on-alert is off -- the job
# summary (summary-always) is the only surface for this run.
- name: Update bundle size baseline
uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1
with:
tool: customSmallerIsBetter
output-file-path: bundle-size-summary.json
external-data-json-path: bundle-size-history.json
fail-on-alert: false
summary-always: true
- name: Save bundle size history
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: bundle-size-history.json
key: bundle-size-history-${{ github.run_id }}
# Publishes the treemap to Netlify (the same host already used for
# superset-storybook.netlify.app and docs previews, reusing the
# existing NETLIFY_AUTH_TOKEN). Skipped until
# NETLIFY_BUNDLE_ANALYZER_SITE_ID exists -- create a new (free)
# Netlify site named superset-bundle-analyzer and add its site ID as
# that secret to turn this on; nothing else in this workflow depends
# on it.
- name: Publish bundle analyzer report to Netlify
if: ${{ env.NETLIFY_SITE_ID != '' }}
env:
NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
run: |
mkdir -p netlify-publish
cp superset/static/assets/report.html netlify-publish/index.html
# zizmor: ignore[adhoc-packages] - netlify-cli is a one-shot CI deploy
# tool, not an application dependency; a global/npx install has no
# lockfile context. Version pinned above the floor set by other
# ad-hoc installs in this repo (bump deliberately when upgrading).
npx --yes netlify-cli@27.0.1 deploy --prod --dir=netlify-publish
@@ -1,70 +0,0 @@
name: Generate FOSSA report
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
permissions:
contents: read
jobs:
config:
runs-on: ubuntu-slim
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${FOSSA_API_KEY}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
FOSSA_API_KEY: ${{ (secrets.FOSSA_API_KEY != '' ) || '' }}
license_check:
needs: config
if: needs.config.outputs.has-secrets
name: Generate Report
runs-on: ubuntu-slim
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
with:
distribution: "temurin"
java-version: "11"
- name: Generate fossa report
env:
FOSSA_API_KEY: ${{ secrets.FOSSA_API_KEY }}
run: |
set -eo pipefail
if [[ "${{github.event_name}}" != "pull_request" ]]; then
./scripts/fossa.sh
exit 0
fi
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
FILES=$(curl -s -X GET -G $URL | jq -r '.[] | .filename')
cat<<EOF
CHANGED FILES:
$FILES
EOF
if [[ "${FILES}" =~ (.*package*\.json|requirements\/[a-z_-]+\.txt|setup\.py) ]]; then
echo "Detected dependency changes... running fossa check"
./scripts/fossa.sh
else
echo "No dependency changes... skiping fossa check"
fi
shell: bash
@@ -1,28 +0,0 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Inspired from https://github.com/mpalmer/action-validator?tab=readme-ov-file#pre-commit-hook-example
echo "Running pre-commit hook for GitHub Actions: https://github.com/mpalmer/action-validator"
for action in $(git ls-files .github/ | grep -E '^\.github/(workflows|actions)/.*\.ya?ml$'); do
if action-validator "$action"; then
echo "$action"
else
echo "$action"
exit 1
fi
done
@@ -1,52 +0,0 @@
name: Validate All GitHub Actions
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
branches:
- "**"
permissions:
contents: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
validate-all-ghas:
runs-on: ubuntu-26.04
permissions:
contents: read
# Required for the zizmor action to upload its SARIF results to
# GitHub code scanning (advanced-security is enabled by default).
security-events: write
steps:
- name: Checkout Repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: "20"
- name: Install Dependencies
# Versions are pinned to avoid ad-hoc, unpinned package installs.
# Bump deliberately when upgrading.
# zizmor: ignore[adhoc-packages] - @action-validator is a global CLI tool installed to validate the repo's workflows; a global CLI install has no application manifest/lockfile context, and the versions are pinned above
run: npm install -g @action-validator/core@0.6.0 @action-validator/cli@0.6.0
- name: Run Script
run: bash .github/workflows/github-action-validator.sh
- name: Test docs-deploy freshness gate
run: bash .github/workflows/scripts/check-docs-deploy-freshness.test.sh
- name: Check for security issues on GHA workflows
uses: zizmorcore/zizmor-action@3dc1ecc9bcb9e94e9b2c709687979e1298497054 # v0.6.2
-115
View File
@@ -1,115 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
name: Hold Label CI Gate
on:
pull_request:
types: [labeled, unlabeled]
permissions: {}
jobs:
cancel-on-hold:
name: Cancel CI runs when hold label applied
if: github.event.action == 'labeled' && startsWith(github.event.label.name, 'hold')
runs-on: ubuntu-26.04
permissions:
actions: write
pull-requests: read
steps:
- name: Cancel in-progress workflow runs
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const { owner, repo } = context.repo;
const sha = context.payload.pull_request.head.sha;
const { data: { workflow_runs: runs } } =
await github.rest.actions.listWorkflowRunsForRepo({
owner,
repo,
head_sha: sha,
per_page: 100,
});
const active = runs.filter(r =>
['in_progress', 'queued', 'waiting', 'requested', 'pending'].includes(r.status)
);
core.info(`Found ${active.length} active run(s) to cancel for SHA ${sha}`);
for (const run of active) {
try {
await github.rest.actions.cancelWorkflowRun({
owner,
repo,
run_id: run.id,
});
core.info(`Cancelled run ${run.id} (${run.name})`);
} catch (err) {
core.warning(`Could not cancel run ${run.id}: ${err.message}`);
}
}
rerun-on-unhold:
name: Re-run CI when hold label removed
if: github.event.action == 'unlabeled' && startsWith(github.event.label.name, 'hold')
runs-on: ubuntu-24.04
permissions:
actions: write
pull-requests: read
steps:
- name: Re-trigger cancelled workflow runs
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const { owner, repo } = context.repo;
const sha = context.payload.pull_request.head.sha;
// Check that no other hold* labels remain on the PR
const labels = context.payload.pull_request.labels.map(l => l.name);
const stillHeld = labels.some(l => l.startsWith('hold'));
if (stillHeld) {
core.info('PR still has a hold label — skipping re-run.');
return;
}
const { data: { workflow_runs: runs } } =
await github.rest.actions.listWorkflowRunsForRepo({
owner,
repo,
head_sha: sha,
per_page: 100,
});
const cancelled = runs.filter(r => r.conclusion === 'cancelled');
core.info(`Found ${cancelled.length} cancelled run(s) to re-trigger for SHA ${sha}`);
for (const run of cancelled) {
try {
await github.rest.actions.reRunWorkflow({
owner,
repo,
run_id: run.id,
});
core.info(`Re-triggered run ${run.id} (${run.name})`);
} catch (err) {
core.warning(`Could not re-run ${run.id}: ${err.message}`);
}
}
-33
View File
@@ -1,33 +0,0 @@
name: supersetbot orglabel based on author
on:
issues:
types: [opened, edited]
pull_request:
types: [opened, edited]
jobs:
superbot-orglabel:
runs-on: ubuntu-slim
permissions:
contents: read
pull-requests: write
issues: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
- name: Execute supersetbot orglabel command
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Label the issue with the appropriate org using supersetbot
# - this requires for the author to be publicly associated with their org
# - and for the org to be listed in `supersetbot/src/metadata.js`
supersetbot orglabel --issue ${{ github.event.number || github.event.issue.number }} --repo ${{ github.repository }} || true
@@ -1,54 +0,0 @@
name: Label Merge Conflicts
# Sweeps every open PR and labels the ones GitHub reports as CONFLICTING with
# `requires:rebase` (removing it once a rebase makes the PR mergeable again),
# so the label can be used to filter the PR backlog for the ones that need a
# rebase before they can be reviewed/merged.
#
# The action itself always re-checks *every* open PR via GraphQL on each run
# regardless of what triggered it (see eps1lon/actions-label-merge-conflict's
# sources/main.ts) - there's no way to scope it to "just this PR". The
# project's own README suggests triggering on `push` (to the default branch)
# plus `pull_request_target: [synchronize]`, but on a repo with Superset's PR
# volume that combination would re-sweep the entire open-PR list on every
# merge to master *and* every push to *any* open PR - many times an hour.
# A schedule bounds that to a fixed, predictable cadence instead; adjust it
# if 2 hours turns out to be too slow or too chatty in practice.
on:
schedule:
- cron: "0 */2 * * *"
workflow_dispatch:
# Avoid two full backlog sweeps racing (a manual workflow_dispatch landing
# mid-schedule-tick, say); queue rather than cancel so an in-progress
# paginated sweep always runs to completion.
concurrency:
group: ${{ github.workflow }}
cancel-in-progress: false
permissions: {}
jobs:
label-merge-conflicts:
# Scheduled/dispatch workflows still run on forks that carry this file;
# skip anywhere but the canonical repo.
if: github.repository == 'apache/superset'
name: Label Merge Conflicts
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write # to add/remove requires:rebase and need:merge
steps:
# ASF Infra allowlists this whole action via a wildcard
# (eps1lon/actions-label-merge-conflict@*), so any pinned SHA/version
# is already fine here - no Infra ticket needed for future bumps.
- uses: eps1lon/actions-label-merge-conflict@0273be72a0bbd58fcd71d0d6c02c209b50d1e5e1 # v3.1.0
with:
dirtyLabel: "requires:rebase"
# A conflicting PR isn't actually ready to merge; strip that signal
# if it was previously set so reviewers don't act on a stale one.
removeOnDirtyLabel: "need:merge"
repoToken: ${{ secrets.GITHUB_TOKEN }}
# Intentionally no commentOnDirty/commentOnClean: the label alone is
# the signal (matches the label's existing description, and avoids
# a one-time comment storm across the whole backlog on first run).
-26
View File
@@ -1,26 +0,0 @@
name: "Pull Request Labeler"
on:
- pull_request_target
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
labeler:
permissions:
contents: read
pull-requests: write
runs-on: ubuntu-slim
steps:
- uses: actions/labeler@bf12e9b00b37c5c0ca2b87b79b2daf7891dbda13 # v7.0.0
with:
sync-labels: true
# TODO: run scripts based on labels!
# - id: run-translation-scripts
# if: contains(steps.label-the-PR.outputs.all-labels, 'i18n')
# run: |
# echo "Running translation scripts"
# # Generate .pot -> .po -> .json files
+18 -25
View File
@@ -6,34 +6,27 @@ on:
jobs:
latest-release:
name: Add/update tag to new release
runs-on: ubuntu-slim
runs-on: ubuntu-latest
permissions:
contents: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check for latest tag
id: latest-tag
env:
RELEASE_TAG_NAME: ${{ github.event.release.tag_name }}
run: |
source ./scripts/tag_latest_release.sh "$RELEASE_TAG_NAME" --dry-run
- name: Check for latest tag
id: latest-tag
run: |
source ./scripts/tag_latest_release.sh $(echo ${{ github.event.release.tag_name }}) --dry-run
- name: Configure Git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Run latest-tag
uses: ./.github/actions/latest-tag
if: steps.latest-tag.outputs.SKIP_TAG != 'true'
with:
description: Superset latest release
tag-name: latest
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Run latest-tag
uses: ./.github/actions/latest-tag
if: (! ${{ steps.latest-tag.outputs.SKIP_TAG }} )
with:
description: Superset latest release
tag-name: latest
env:
GITHUB_TOKEN: ${{ github.token }}
+50 -15
View File
@@ -1,31 +1,66 @@
name: License Template Check
name: License Check
on:
push:
branches-ignore:
- "dependabot/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.FOSSA_API_KEY != '' ) || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
license_check:
needs: config
if: needs.config.outputs.has-secrets
name: License Check
runs-on: ubuntu-slim
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
uses: actions/setup-java@v1
with:
distribution: "temurin"
java-version: "11"
java-version: 8
- name: Generate fossa report
env:
FOSSA_API_KEY: ${{ secrets.FOSSA_API_KEY }}
run: |
set -eo pipefail
if [[ "${{github.event_name}}" != "pull_request" ]]; then
./scripts/fossa.sh
exit 0
fi
URL="https://api.github.com/repos/${{ github.repository }}/pulls/${{ github.event.pull_request.number }}/files"
FILES=$(curl -s -X GET -G $URL | jq -r '.[] | .filename')
cat<<EOF
CHANGED FILES:
$FILES
EOF
if [[ "${FILES}" =~ (.*package*\.json|requirements\/[a-z_-]+\.txt|setup\.py) ]]; then
echo "Detected dependency changes... running fossa check"
./scripts/fossa.sh
else
echo "No dependency changes... skiping fossa check"
fi
shell: bash
- name: Run license check
run: ./scripts/check_license.sh
-113
View File
@@ -1,113 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Mirror the Docker Hub service-container images that CI relies on into the
# repository's GitHub Container Registry (GHCR) namespace.
#
# WHY: CI jobs declare Postgres/MySQL/Redis/Presto as `services:` containers
# pulled anonymously from Docker Hub. Anonymous pulls share the runner's IP
# rate limit, which causes intermittent timeouts / 429s / 502s on `master`
# and same-repo PRs. The obvious fix — adding `credentials:` to the service
# blocks — breaks fork PRs hard: forks can't read secrets, so the templated
# username/password resolve to '' and GitHub rejects the workflow at parse
# time ("Unexpected value ''"), failing every fork job at "Set up job".
#
# Mirroring to GHCR sidesteps both problems: public GHCR images are pulled
# without Docker Hub's anonymous rate limit AND without any credentials, so
# the consuming workflows need no `credentials:` block and forks work
# unchanged.
#
# ONE-TIME BOOTSTRAP (maintainer, after this lands on the default branch):
# 1. Run this workflow once (Actions tab → "Mirror service images to GHCR"
# → Run workflow), or wait for the weekly schedule.
# 2. In the org's Packages settings, set each mirrored package's visibility
# to **public** (apache/superset → ci/postgres, ci/mysql, ci/redis,
# ci/presto). Public visibility is what lets fork CI pull without auth.
# 3. Only then merge the follow-up that repoints the `services.*.image`
# refs at these GHCR copies and drops the `credentials:` blocks.
#
# NOTE: this mirrors only the images declared as `services:` containers (the
# ones that broke forks). The `bde2020` hive-metastore image pulled via
# `docker compose` in the Presto/Hive job is a separate path and is left for
# a follow-up.
name: Mirror service images to GHCR
on:
schedule:
# Weekly, Monday 06:00 UTC — keeps the mirror fresh as upstream tags move.
- cron: "0 6 * * 1"
workflow_dispatch: {}
concurrency:
group: mirror-service-images
cancel-in-progress: false
permissions:
contents: read
packages: write
jobs:
mirror:
# Never run on forks: they lack both the secrets and write access to the
# apache GHCR namespace, so a scheduled run there would only ever fail.
if: github.repository == 'apache/superset'
runs-on: ubuntu-latest
strategy:
fail-fast: false
matrix:
# Keep this list in sync with the `services.*.image` refs in
# superset-e2e.yml, superset-python-integrationtest.yml, and
# superset-python-presto-hive.yml.
image:
- postgres:17-alpine
- redis:7-alpine
- mysql:8.0
- starburstdata/presto:350-e.6
steps:
- name: Log in to Docker Hub (authenticated source pulls)
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Log in to GHCR (push target)
uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@bb05f3f5519dd87d3ba754cc423b652a5edd6d2c # v4.2.0
- name: Copy image to GHCR
env:
# Pass the matrix value through the environment rather than
# interpolating it into the shell, to avoid template injection.
SRC_IMAGE: ${{ matrix.image }}
run: |
set -euo pipefail
# Destination keeps the image's short name (drop any namespace),
# under a `ci/` prefix in this repo's GHCR namespace.
name="${SRC_IMAGE##*/}"
dst="ghcr.io/${GITHUB_REPOSITORY}/ci/${name}"
echo "Mirroring docker.io/${SRC_IMAGE} -> ${dst}"
# imagetools copies the full (multi-arch) manifest registry-to-
# registry without a local pull/retag/push round trip.
docker buildx imagetools create --tag "${dst}" "docker.io/${SRC_IMAGE}"
echo "- \`docker.io/${SRC_IMAGE}\` → \`${dst}\`" >> "${GITHUB_STEP_SUMMARY}"
-31
View File
@@ -1,31 +0,0 @@
name: Hold Label Check
on:
pull_request:
types: [labeled, unlabeled, opened, reopened, synchronize]
permissions:
pull-requests: read
# Let each label event run to completion. Cancelling in-progress runs leaves
# CANCELLED entries in the PR's check-suite rollup, which poisons GitHub's
# `status:success` search filter even though all real CI passed. The job is
# a tiny no-op github-script call, so the wasted compute is negligible.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: false
jobs:
check-hold-label:
runs-on: ubuntu-slim
steps:
- name: Check for 'hold' label
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |
const payload = context.payload.pull_request
const holdLabelPresent = !!payload.labels.find(label => label.name.includes('hold'))
if (holdLabelPresent) {
core.setFailed('Hold label is present, merge is blocked.')
}
+5 -15
View File
@@ -8,26 +8,15 @@ on:
# Possible values: https://help.github.com/en/actions/reference/events-that-trigger-workflows#pull-request-event-pull_request
types: [opened, edited, reopened, synchronize]
# Serialize runs per PR without cancelling: the `edited` trigger means a PR
# opened and then edited has two queued runs for the same head SHA. On
# first-time-contributor PRs those runs start together when a maintainer
# approves workflows, and cancel-in-progress lets the older run cancel the
# newer one — leaving a permanently-cancelled required check on the head SHA
# that blocks merging until manually re-run. This job takes seconds, so let
# queued runs complete instead.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: false
jobs:
lint-check:
runs-on: ubuntu-slim
check:
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
@@ -37,5 +26,6 @@ jobs:
on-failed-regex-fail-action: true
on-failed-regex-request-changes: false
on-failed-regex-create-review: false
on-failed-regex-comment: "Please format your PR title to match: `%regex%`!"
on-failed-regex-comment:
"Please format your PR title to match: `%regex%`!"
repo-token: "${{ github.token }}"
-209
View File
@@ -1,209 +0,0 @@
name: pre-commit checks
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# Nightly full-tree sweep. Per-PR runs only lint changed files, so a change
# that invalidates an untouched file (e.g. a type change that breaks an
# importing test) can pass every PR yet leave master red. This catches that.
schedule:
- cron: "0 6 * * *"
permissions:
contents: read
actions: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
pre-commit:
runs-on: ubuntu-26.04
timeout-minutes: 20
strategy:
matrix:
# Run the full version spread on push (master/release) and nightly,
# but only the current version on PRs — lint/format/type results
# rarely differ across patch versions, so 3x per PR is wasteful.
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "next"]') }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
# Full history so we can diff a PR/push against its base commit to
# determine changed files (see "Determine changed files" below).
fetch-depth: 0
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
python-version: ${{ matrix.python-version }}
- name: Setup Go
uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0
- name: Install helm-docs
run: go install github.com/norwoodj/helm-docs/cmd/helm-docs@v1.14.2
# Spike: run the existing .pre-commit-config.yaml through prek (a Rust
# reimplementation of pre-commit) instead of pre-commit itself, to see
# whether it's viable to speed up this job. CI-only — contributors keep
# installing/running `pre-commit` locally exactly as documented; nothing
# here changes that.
- name: Install prek
run: |
curl --proto '=https' --tlsv1.2 -LsSf https://github.com/j178/prek/releases/download/v0.4.11/prek-installer.sh | sh
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
- name: Install Frontend Dependencies
run: |
cd superset-frontend
npm ci
- name: Install Docs Dependencies
run: |
cd docs
yarn install --immutable
- name: Cache prek environments
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: ~/.cache/prek
key: prek-v1-${{ runner.os }}-py${{ matrix.python-version }}-${{ hashFiles('.pre-commit-config.yaml') }}
restore-keys: |
prek-v1-${{ runner.os }}-py${{ matrix.python-version }}-
- name: Determine changed files
id: changed_files
env:
EVENT_NAME: ${{ github.event_name }}
BEFORE_SHA: ${{ github.event.before }}
run: |
set -euo pipefail
# Scheduled runs check the whole tree (see the pre-commit step).
if [ "${EVENT_NAME}" = "schedule" ]; then
echo "mode=all" >> "$GITHUB_OUTPUT"
exit 0
fi
# Resolve the commit to diff against.
base=""
if [ "${EVENT_NAME}" = "pull_request" ]; then
# HEAD is the PR merge commit, so its first parent is the current
# tip of the base branch. github.event.pull_request.base.sha is
# NOT that: GitHub freezes it at PR creation, so on a long-lived
# PR it points at the original branch point and the diff picks up
# all of the base branch's churn since then — thousands of paths,
# enough for the CHANGED_FILES env var below to exceed the
# kernel's per-variable size limit and kill the step with
# "Argument list too long" before bash even starts.
base="$(git rev-parse HEAD^1 2>/dev/null || true)"
elif [ -n "${BEFORE_SHA:-}" ] && \
[ "${BEFORE_SHA}" != "0000000000000000000000000000000000000000" ]; then
base="${BEFORE_SHA}"
fi
# Fail closed: if the diff base can't be resolved, check every file
# instead of silently checking nothing. Previously an empty file list
# made `pre-commit run --files` a no-op that still reported success,
# which let unlinted code reach master.
if [ -z "${base}" ] || ! git cat-file -e "${base}^{commit}" 2>/dev/null; then
echo "::notice::Could not resolve a diff base; falling back to --all-files."
echo "mode=all" >> "$GITHUB_OUTPUT"
exit 0
fi
# Files present in HEAD that changed since the base (drop deletions).
files="$(git diff --name-only --diff-filter=ACMRT "${base}...HEAD")"
# Env vars have a hard per-variable size limit (E2BIG at step
# start). A PR that legitimately touches thousands of files is
# better served by --all-files anyway.
if [ "$(printf '%s' "${files}" | wc -c)" -gt 100000 ]; then
echo "::notice::Changed-file list too large to pass via env; falling back to --all-files."
echo "mode=all" >> "$GITHUB_OUTPUT"
exit 0
fi
if [ -z "${files}" ]; then
echo "mode=none" >> "$GITHUB_OUTPUT"
else
echo "mode=files" >> "$GITHUB_OUTPUT"
{
echo "files<<__CHANGED_FILES_EOF__"
echo "${files}"
echo "__CHANGED_FILES_EOF__"
} >> "$GITHUB_OUTPUT"
fi
- name: pre-commit (via prek)
env:
MODE: ${{ steps.changed_files.outputs.mode }}
CHANGED_FILES: ${{ steps.changed_files.outputs.files }}
run: |
set +e # Don't exit immediately on failure
export SKIP=type-checking-frontend
case "${MODE}" in
all)
echo "️ Running prek on all files."
prek run --all-files
;;
files)
echo "️ Running prek on changed files:"
echo "${CHANGED_FILES}"
# shellcheck disable=SC2086
prek run --files ${CHANGED_FILES}
;;
none)
echo "️ No source files changed; nothing for prek to check."
exit 0
;;
*)
echo "⚠️ Unrecognized changed-files mode '${MODE}'; checking all files."
prek run --all-files
;;
esac
PRE_COMMIT_EXIT_CODE=$?
git diff --quiet --exit-code
GIT_DIFF_EXIT_CODE=$?
if [ "${PRE_COMMIT_EXIT_CODE}" -ne 0 ] || [ "${GIT_DIFF_EXIT_CODE}" -ne 0 ]; then
if [ "${PRE_COMMIT_EXIT_CODE}" -ne 0 ]; then
echo "❌ Pre-commit check failed (exit code: ${PRE_COMMIT_EXIT_CODE})."
echo "🔍 Modified files:"
git diff --name-only
else
echo "❌ Git working directory is dirty."
echo "📌 This likely means that pre-commit made changes that were not committed."
echo "🔍 Modified files:"
git diff --name-only
fi
echo "🚒 To prevent/address this CI issue, please install/use pre-commit locally."
echo "📖 More details here: https://superset.apache.org/docs/contributing/development#git-hooks"
exit 1
fi
actions-timeline:
needs: pre-commit
if: always()
runs-on: ubuntu-26.04
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
+59
View File
@@ -0,0 +1,59 @@
name: Prefer Typescript
on:
push:
branches-ignore:
- "dependabot/**"
pull_request:
jobs:
prefer_typescript:
if: github.ref == 'ref/heads/master' && github.event_name == 'pull_request'
name: Prefer Typescript
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Get changed files
id: changed
uses: ./.github/actions/file-changes-action
with:
githubToken: ${{ github.token }}
- name: Determine if a .js or .jsx file was added
id: check
run: |
js_files_added() {
jq -r '
map(
select(
endswith(".js") or endswith(".jsx")
)
) | join("\n")
' ${HOME}/files_added.json
}
echo ::set-output name=js_files_added::$(js_files_added)
- if: steps.check.outputs.js_files_added
name: Add Comment to PR
uses: ./.github/actions/comment-on-pr
continue-on-error: true
env:
GITHUB_TOKEN: ${{ github.token }}
with:
msg: |
### WARNING: Prefer TypeScript
Looks like your PR contains new `.js` or `.jsx` files:
```
${{steps.check.outputs.js_files_added}}
```
As decided in [SIP-36](https://github.com/apache/superset/issues/9101), all new frontend code should be written in TypeScript. Please convert above files to TypeScript then re-request review.
+17 -20
View File
@@ -3,15 +3,11 @@ name: release-workflow
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
permissions:
contents: read
- 'master'
jobs:
config:
runs-on: ubuntu-slim
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
@@ -19,23 +15,24 @@ jobs:
id: check
shell: bash
run: |
if [ -n "${NPM_TOKEN}" ]; then
if [ -n "${{ (secrets.NPM_TOKEN != '' && secrets.GH_PERSONAL_ACCESS_TOKEN != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
NPM_TOKEN: ${{ (secrets.NPM_TOKEN != '' && secrets.GH_PERSONAL_ACCESS_TOKEN != '') || '' }}
build:
needs: config
if: needs.config.outputs.has-secrets
name: Bump version and publish package(s)
runs-on: ubuntu-26.04
permissions:
contents: write
runs-on: ubuntu-20.04
strategy:
matrix:
node-version: [16]
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- uses: actions/checkout@v3
with:
persist-credentials: false
# pulls all commits (needed for lerna / semantic release to correctly version)
fetch-depth: 0
- name: Get tags and filter trigger tags
@@ -48,15 +45,15 @@ jobs:
git fetch --prune --unshallow
git tag -d `git tag | grep -E '^trigger-'`
- name: Install Node.js
- name: Use Node.js ${{ matrix.node-version }}
if: env.HAS_TAGS
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
uses: actions/setup-node@v3
with:
node-version-file: "./superset-frontend/.nvmrc"
node-version: ${{ matrix.node-version }}
- name: Cache npm
if: env.HAS_TAGS
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
uses: actions/cache@v1
with:
path: ~/.npm # npm cache files are stored in `~/.npm` on Linux/macOS
key: ${{ runner.OS }}-node-${{ hashFiles('**/package-lock.json') }}
@@ -67,10 +64,10 @@ jobs:
- name: Get npm cache directory path
if: env.HAS_TAGS
id: npm-cache-dir-path
run: echo "dir=$(npm config get cache)" >> $GITHUB_OUTPUT
run: echo "::set-output name=dir::$(npm config get cache)"
- name: Cache npm
if: env.HAS_TAGS
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
uses: actions/cache@v1
id: npm-cache # use this to check for `cache-hit` (`steps.npm-cache.outputs.cache-hit != 'true'`)
with:
path: ${{ steps.npm-cache-dir-path.outputs.dir }}
@@ -1,187 +0,0 @@
name: Scheduled Docker image refresh
# Re-runs the Docker image build against the latest published release on a
# weekly cadence. The code being built doesn't change — but the base image
# layers (python:*-slim-trixie and its OS packages) DO get upstream
# security patches between Superset releases, and those patches don't
# reach our published images unless we rebuild.
#
# Without this workflow, `apache/superset:<latest>` lags behind upstream
# Debian/Python base patches by whatever interval falls between Superset
# releases (typically 36 weeks). With it, the lag drops to at most one
# week regardless of release cadence.
#
# This is a security-hygiene cron, not a release. It overwrites the
# existing tags for the most recent release (e.g. `apache/superset:5.0.0`
# and `apache/superset:latest`) with bit-for-bit-equivalent contents
# layered on a refreshed base. Image digests change; everything users
# actually pin against (image content, code, deps) does not.
on:
schedule:
# Mondays at 06:00 UTC — gives the weekend for upstream patches to
# settle and surfaces failures at the start of the work week so a
# human can react.
- cron: "0 6 * * 1"
# Manual trigger so operators can force a refresh on demand (e.g.
# immediately after a high-severity base-image CVE drops).
workflow_dispatch: {}
permissions:
contents: read
# Serialize with itself and with the release publisher (tag-release.yml) —
# both push to the same Docker Hub tags, so a race could end with stale
# layers winning. Both workflows must declare this group for the lock to work.
concurrency:
group: docker-publish-latest-release
cancel-in-progress: false
jobs:
config:
runs-on: ubuntu-26.04
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
latest-release: ${{ steps.latest.outputs.tag }}
force-latest: ${{ steps.latest.outputs.force-latest }}
steps:
- name: Check for Docker Hub secrets
id: check
shell: bash
run: |
if [ -n "${DOCKERHUB_USER}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
DOCKERHUB_USER: ${{ (secrets.DOCKERHUB_USER != '' && secrets.DOCKERHUB_TOKEN != '') || '' }}
- name: Look up latest published release
id: latest
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPOSITORY: ${{ github.repository }}
run: |
# `releases/latest` returns the latest non-prerelease, non-draft
# release — which is exactly what `apache/superset:latest`
# should reflect.
TAG=$(gh api "repos/${REPOSITORY}/releases/latest" --jq .tag_name)
if [ -z "$TAG" ] || [ "$TAG" = "null" ]; then
echo "::error::Could not determine latest release tag"
exit 1
fi
echo "Latest release: $TAG"
echo "tag=$TAG" >> "$GITHUB_OUTPUT"
# Only move `:latest` when the release flagged "latest" is also the
# highest semver release. This guards against a mis-click leaving an
# older maintenance release (e.g. a 5.x patch shipped after 6.0 GA)
# marked latest, which would otherwise roll `:latest` back a major
# version on the next cron run. If it isn't the newest, we still
# refresh that release's own version tag but leave `:latest` alone.
HIGHEST=$(gh api --paginate "repos/${REPOSITORY}/releases" \
--jq '.[] | select(.draft|not) | select(.prerelease|not) | .tag_name' \
| sed 's/^v//' | sort -V | tail -n1)
if [ "${TAG#v}" = "$HIGHEST" ]; then
echo "force-latest=1" >> "$GITHUB_OUTPUT"
else
echo "::warning::Latest-flagged release $TAG is not the highest semver ($HIGHEST); refreshing its version tag but leaving :latest untouched"
fi
docker-rebuild:
needs: config
if: needs.config.outputs.has-secrets == '1'
name: docker-rebuild
runs-on: ubuntu-26.04
strategy:
# Mirror the same matrix the release publisher uses so every variant
# operators consume from Docker Hub gets the refreshed base.
matrix:
build_preset: ["dev", "lean", "websocket", "dockerize", "py311", "py312"]
fail-fast: false
steps:
- name: "Checkout release tag: ${{ needs.config.outputs.latest-release }}"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ needs.config.outputs.latest-release }}
fetch-depth: 0
persist-credentials: false
# Keep workflow tooling on the triggering revision. Release tags can
# contain action pins that no longer satisfy the repository allowlist.
- name: Checkout workflow actions
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.sha }}
path: workflow-source
persist-credentials: false
sparse-checkout: .github/actions
- name: Setup Docker Environment
uses: ./workflow-source/.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
install-docker-compose: "false"
build: "true"
- name: Use Node.js 20
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version: 20
- name: Setup supersetbot
uses: ./workflow-source/.github/actions/setup-supersetbot/
- name: Rebuild and push
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_PRESET: ${{ matrix.build_preset }}
LATEST_RELEASE: ${{ needs.config.outputs.latest-release }}
FORCE_LATEST_FLAG: ${{ needs.config.outputs.force-latest == '1' && '--force-latest' || '' }}
run: |
# Reuses the same supersetbot invocation as the release
# publisher (`tag-release.yml`), so the resulting tags are
# identical to what a manual release dispatch would produce —
# just with a freshly-pulled base image layer underneath.
# `--force-latest` is only passed when the config job confirmed the
# fetched release is the newest one (see FORCE_LATEST_FLAG above).
supersetbot docker \
--push \
--preset "$BUILD_PRESET" \
--context release \
--context-ref "$LATEST_RELEASE" \
$FORCE_LATEST_FLAG \
--platform "linux/arm64" \
--platform "linux/amd64"
# The whole point of this cron is catching base-image CVEs, so a silent
# failure is the expensive case — a red X in the Actions tab nobody is
# watching on a Monday. File a tracked issue when any rebuild leg fails so
# a missed security refresh surfaces instead of sitting unnoticed.
notify-on-failure:
needs: [config, docker-rebuild]
if: failure() && needs.config.outputs.has-secrets == '1'
runs-on: ubuntu-26.04
permissions:
contents: read
issues: write
steps:
- name: Open a tracking issue
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
REPOSITORY: ${{ github.repository }}
LATEST_RELEASE: ${{ needs.config.outputs.latest-release }}
RUN_URL: ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}
run: |
gh issue create \
--repo "$REPOSITORY" \
--title "Scheduled Docker image refresh failed for ${LATEST_RELEASE}" \
--label "infra:container" \
--label "#bug" \
--body "The weekly Docker base-image refresh failed for release \`${LATEST_RELEASE}\`. Published images may be missing upstream base-layer security patches until this is resolved.
Failed run: ${RUN_URL}"
@@ -1,49 +0,0 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Shared freshness gate used by the Docs Deployment workflow
# (superset-docs-deploy.yml) both up front (check-freshness) and again right
# before the deploy step (recheck-freshness). Writes an output declaring
# whether BUILD_SHA is still master's current tip, so a superseded run can
# skip cleanly instead of racing (and clobbering, or being force-cancelled
# by) a fresher run.
#
# Required env vars:
# BUILD_SHA - the commit SHA this run is building
# REPO - "owner/repo" to query, e.g. github.repository
# OUTPUT_NAME - the GITHUB_OUTPUT key to write, e.g. "is-current"
# GITHUB_OUTPUT - path to append outputs to (set by the Actions runner)
# Optional env vars:
# EVENT_NAME - if "workflow_dispatch", bypasses the check and always
# reports current, since a manual dispatch is a deliberate,
# one-off action rather than something racing other triggers
# GH_TOKEN - passed through to `gh`, needed to call the GitHub API
set -euo pipefail
if [ "${EVENT_NAME:-}" = "workflow_dispatch" ]; then
echo "${OUTPUT_NAME}=true" >>"$GITHUB_OUTPUT"
exit 0
fi
latest_sha="$(gh api "repos/${REPO}/commits/master" --jq .sha)"
if [ "${latest_sha}" = "${BUILD_SHA}" ]; then
echo "${OUTPUT_NAME}=true" >>"$GITHUB_OUTPUT"
else
echo "${OUTPUT_NAME}=false" >>"$GITHUB_OUTPUT"
echo "::notice::master has moved on to ${latest_sha} since ${BUILD_SHA} was triggered — skipping this stale run."
fi
@@ -1,100 +0,0 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# Exercises check-docs-deploy-freshness.sh against a stubbed `gh`, covering
# the dispatch-bypass, current-tip and stale-tip branches so the output
# contract (is-current / still-current) can't silently regress. Run
# directly, no extra tooling required:
# bash .github/workflows/scripts/check-docs-deploy-freshness.test.sh
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
script_under_test="${script_dir}/check-docs-deploy-freshness.sh"
failures=0
# Runs the script under test with a stubbed `gh` reporting $1 as master's
# latest sha, asserting that GITHUB_OUTPUT ends up containing exactly $4.
run_case() {
local case_name="$1"
local latest_sha="$2"
local build_sha="$3"
local event_name="$4"
local expected_line="$5"
local workdir
workdir="$(mktemp -d)"
trap 'rm -rf "${workdir}"' RETURN
# Fake `gh` that just echoes back the requested "latest" sha regardless of
# arguments, so the script under test never touches the network.
cat >"${workdir}/gh" <<EOF
#!/bin/bash
echo '${latest_sha}'
EOF
chmod +x "${workdir}/gh"
local output_file="${workdir}/github_output"
: >"${output_file}"
if PATH="${workdir}:${PATH}" \
GITHUB_OUTPUT="${output_file}" \
OUTPUT_NAME="is-current" \
REPO="apache/superset" \
BUILD_SHA="${build_sha}" \
EVENT_NAME="${event_name}" \
GH_TOKEN="fake-token" \
bash "${script_under_test}"; then
:
else
echo "FAIL (${case_name}): script exited non-zero"
failures=$((failures + 1))
return
fi
local actual
actual="$(cat "${output_file}")"
if [ "${actual}" = "${expected_line}" ]; then
echo "PASS (${case_name})"
else
echo "FAIL (${case_name}): expected '${expected_line}', got '${actual}'"
failures=$((failures + 1))
fi
}
# `gh` prints "should-not-be-called" for the dispatch case above the trick:
# it's never actually invoked since the bypass short-circuits before the
# `gh api` call, but the fake still needs a body.
run_case "workflow_dispatch bypasses the check" \
"unused" "abc123" "workflow_dispatch" \
"is-current=true"
run_case "build sha matches master's tip" \
"abc123" "abc123" "push" \
"is-current=true"
run_case "build sha is stale" \
"def456" "abc123" "push" \
"is-current=false"
if [ "${failures}" -gt 0 ]; then
echo "${failures} case(s) failed"
exit 1
fi
echo "All cases passed"
-37
View File
@@ -1,37 +0,0 @@
name: 🎪 Showtime Cleanup
# Scheduled cleanup of expired environments
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
# Manual trigger for testing
workflow_dispatch:
# Common environment variables
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ vars.AWS_REGION || 'us-west-2' }}
GITHUB_ORG: ${{ github.repository_owner }}
GITHUB_REPO: ${{ github.event.repository.name }}
jobs:
cleanup-expired:
name: Clean up expired showtime environments
runs-on: ubuntu-latest
permissions:
contents: read
issues: write # delete orphaned showtime label definitions (label CRUD is the issues API)
pull-requests: write
steps:
- name: Install Superset Showtime
run: pip install superset-showtime
- name: Cleanup expired environments and orphaned labels
run: |
echo "Cleaning up environments respecting TTL labels, and pruning orphaned 🎪 labels"
python -m showtime cleanup --respect-ttl --force
-191
View File
@@ -1,191 +0,0 @@
name: 🎪 Superset Showtime
# Ultra-simple: just sync on any PR state change
on:
# zizmor: ignore[dangerous-triggers] - required to react to PR label changes; PR code is
# only checked out and built after the maintainer-authorization gate (write/admin actors)
pull_request_target:
types: [labeled, unlabeled, synchronize, closed]
# Manual testing
workflow_dispatch:
inputs:
pr_number:
description: "PR number to sync"
required: true
type: number
sha:
description: "Specific SHA to deploy (optional, defaults to latest)"
required: false
type: string
# Common environment variables for all jobs (non-sensitive only)
env:
AWS_REGION: us-west-2
GITHUB_ORG: ${{ github.repository_owner }}
GITHUB_REPO: ${{ github.event.repository.name }}
GITHUB_ACTOR: ${{ github.actor }}
jobs:
sync:
name: 🎪 Sync PR to desired state
runs-on: ubuntu-latest
timeout-minutes: 90
permissions:
contents: read
pull-requests: write
steps:
- name: Security Check - Authorize Maintainers Only
id: auth
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
script: |
const actor = context.actor;
console.log(`🔍 Checking authorization for ${actor}`);
// Early exit for workflow_dispatch - assume authorized since it's manually triggered
if (context.eventName === 'workflow_dispatch') {
console.log(`✅ Workflow dispatch event - assuming authorized for ${actor}`);
core.setOutput('authorized', 'true');
return;
}
const { data: permission } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: actor
});
console.log(`📊 Permission level for ${actor}: ${permission.permission}`);
const authorized = ['write', 'admin'].includes(permission.permission);
// If this is a synchronize event from unauthorized user, check if Showtime is active and set blocked label
if (!authorized && context.eventName === 'pull_request_target' && context.payload.action === 'synchronize') {
console.log(`🔒 Synchronize event detected - checking if Showtime is active`);
// Check if PR has any circus tent labels (Showtime is in use)
const { data: issue } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number
});
const hasCircusLabels = issue.labels.some(label => label.name.startsWith('🎪 '));
if (hasCircusLabels) {
console.log(`🎪 Circus labels found - setting blocked label to prevent auto-deployment`);
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
labels: ['🎪 🔒 showtime-blocked']
});
console.log(`✅ Blocked label set - Showtime will detect and skip operations`);
} else {
console.log(`️ No circus labels found - Showtime not in use, skipping block`);
}
}
if (!authorized) {
console.log(`🚨 Unauthorized user ${actor} - skipping all operations`);
core.setOutput('authorized', 'false');
return;
}
console.log(`✅ Authorized maintainer: ${actor}`);
core.setOutput('authorized', 'true');
- name: Install Superset Showtime
if: steps.auth.outputs.authorized == 'true'
run: |
echo "::notice::Maintainer $GITHUB_ACTOR triggered deploy for PR ${PULL_REQUEST_NUMBER}"
pip install --upgrade superset-showtime
showtime version
env:
PULL_REQUEST_NUMBER: ${{ github.event.pull_request.number || github.event.inputs.pr_number }}
- name: Check what actions are needed
if: steps.auth.outputs.authorized == 'true'
id: check
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
INPUT_PR_NUMBER: ${{ github.event.inputs.pr_number }}
INPUT_SHA: ${{ github.event.inputs.sha }}
run: |
# Bulletproof PR number extraction
if [[ -n "${{ github.event.pull_request.number }}" ]]; then
PR_NUM="${{ github.event.pull_request.number }}"
elif [[ -n "${INPUT_PR_NUMBER}" ]]; then
PR_NUM="${INPUT_PR_NUMBER}"
else
echo "❌ No PR number found in event or inputs"
exit 1
fi
echo "Using PR number: $PR_NUM"
# Run sync check-only with optional SHA override
if [[ -n "${INPUT_SHA}" ]]; then
OUTPUT=$(python -m showtime sync $PR_NUM --check-only --sha "${INPUT_SHA}")
else
OUTPUT=$(python -m showtime sync $PR_NUM --check-only)
fi
echo "$OUTPUT"
# Extract the outputs we need for conditional steps
BUILD=$(echo "$OUTPUT" | grep "build_needed=" | cut -d'=' -f2)
SYNC=$(echo "$OUTPUT" | grep "sync_needed=" | cut -d'=' -f2)
PR_NUM_OUT=$(echo "$OUTPUT" | grep "pr_number=" | cut -d'=' -f2)
TARGET_SHA=$(echo "$OUTPUT" | grep "target_sha=" | cut -d'=' -f2)
echo "build_needed=$BUILD" >> $GITHUB_OUTPUT
echo "sync_needed=$SYNC" >> $GITHUB_OUTPUT
echo "pr_number=$PR_NUM_OUT" >> $GITHUB_OUTPUT
echo "target_sha=$TARGET_SHA" >> $GITHUB_OUTPUT
- name: Checkout PR code (only if build needed)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.build_needed == 'true'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ steps.check.outputs.target_sha }}
persist-credentials: false
# Building fork PR code is Showtime's purpose: deploys are gated on the
# maintainer-authorization step above (write/admin actors only), so this
# checkout is an explicit, authorized opt-in rather than an automatic one.
allow-unsafe-pr-checkout: true
- name: Setup Docker Environment (only if build needed)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.build_needed == 'true'
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
build: "true"
install-docker-compose: "false"
- name: Execute sync (handles everything)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.sync_needed == 'true'
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
CHECK_PR_NUMBER: ${{ steps.check.outputs.pr_number }}
CHECK_TARGET_SHA: ${{ steps.check.outputs.target_sha }}
run: |
PR_NUM="$CHECK_PR_NUMBER"
TARGET_SHA="$CHECK_TARGET_SHA"
if [[ -n "$TARGET_SHA" ]]; then
python -m showtime sync $PR_NUM --sha "$TARGET_SHA"
else
python -m showtime sync $PR_NUM
fi
-76
View File
@@ -1,76 +0,0 @@
name: Superset App CLI tests
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
pull-requests: read
actions: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
test-load-examples:
runs-on: ubuntu-26.04
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
if: steps.check.outputs.python
uses: ./.github/actions/setup-backend/
- name: Setup Postgres
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: superset init
if: steps.check.outputs.python
run: |
uv pip install --system -e .
superset db upgrade
superset load_test_users
- name: superset load_examples
if: steps.check.outputs.python
run: |
# load examples without test data
superset load_examples --load-big-data
@@ -0,0 +1,103 @@
name: Applitools Cypress
on:
schedule:
- cron: "0 1 * * *"
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.APPLITOOLS_API_KEY != '' && secrets.APPLITOOLS_API_KEY != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
cypress-applitools:
needs: config
if: needs.config.outputs.has-secrets
runs-on: ubuntu-20.04
strategy:
fail-fast: false
matrix:
browser: ["chrome"]
node: [16]
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
APPLITOOLS_APP_NAME: Superset
APPLITOOLS_API_KEY: ${{ secrets.APPLITOOLS_API_KEY }}
APPLITOOLS_BATCH_ID: ${{ github.sha }}
APPLITOOLS_BATCH_NAME: Superset Cypress
services:
postgres:
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
ref: master
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: OS dependencies
uses: ./.github/actions/cached-dependencies
with:
run: apt-get-install
- name: Install python dependencies
uses: ./.github/actions/cached-dependencies
with:
run: |
pip-upgrade
pip install -r requirements/testing.txt
- name: Setup postgres
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
uses: ./.github/actions/cached-dependencies
with:
run: testdata
- name: Setup Node.js
uses: actions/setup-node@v3
with:
node-version: ${{ matrix.node }}
- name: Install npm dependencies
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Install cypress
uses: ./.github/actions/cached-dependencies
with:
run: cypress-install
- name: Run Cypress
uses: ./.github/actions/cached-dependencies
env:
CYPRESS_BROWSER: ${{ matrix.browser }}
with:
run: cypress-run-applitools
@@ -0,0 +1,55 @@
name: Applitools Storybook
on:
schedule:
- cron: "0 0 * * *"
env:
APPLITOOLS_APP_NAME: Superset
APPLITOOLS_API_KEY: ${{ secrets.APPLITOOLS_API_KEY }}
APPLITOOLS_BATCH_ID: ${{ github.sha }}
APPLITOOLS_BATCH_NAME: Superset Storybook
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.APPLITOOLS_API_KEY != '' && secrets.APPLITOOLS_API_KEY != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
cron:
needs: config
if: needs.config.outputs.has-secrets
runs-on: ubuntu-20.04
strategy:
matrix:
node: [16]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
ref: master
- name: Set up Node.js
uses: actions/setup-node@v3.1.1
with:
node-version: ${{ matrix.node }}
- name: Install eyes-storybook dependencies
uses: ./.github/actions/cached-dependencies
with:
run: eyes-storybook-dependencies
- name: Install NPM dependencies
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Run Applitools Eyes-Storybook
working-directory: ./superset-frontend
run: npx eyes-storybook -u https://superset-storybook.netlify.app/
+76
View File
@@ -0,0 +1,76 @@
name: Superset CLI tests
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
jobs:
test-load-examples:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
services:
postgres:
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
# Use custom ports for services to avoid accidentally connecting to
# GitHub action runner's default installations
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: superset init
if: steps.check.outcome == 'failure'
run: |
pip install -e .
superset db upgrade
superset load_test_users
- name: superset load_examples
if: steps.check.outcome == 'failure'
run: |
# load examples without test data
superset load_examples --load-big-data
-202
View File
@@ -1,202 +0,0 @@
name: Docs Deployment
on:
# Deploy after integration tests complete on master
# zizmor: ignore[dangerous-triggers] - runs in base-branch context after a trusted upstream workflow; scoped to master
workflow_run:
workflows: ["Python-Integration"]
types: [completed]
branches: [master]
# Also allow manual trigger and direct pushes to docs
push:
paths:
- "docs/**"
- "README.md"
branches:
- "master"
workflow_dispatch: {}
permissions:
contents: read
actions: read
jobs:
config:
runs-on: ubuntu-26.04
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${SUPERSET_SITE_BUILD}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
SUPERSET_SITE_BUILD: ${{ (secrets.SUPERSET_SITE_BUILD != '' && secrets.SUPERSET_SITE_BUILD != '') || '' }}
# Master gets frequent, sometimes bursty pushes, and each one can trigger a
# deploy attempt. Rather than let every superseded attempt get force-killed
# by the build-deploy concurrency group below (which shows up as a
# `cancelled` — i.e. red/failing-looking — check on that commit), have each
# run check up front whether it's still building master's current tip and,
# if not, skip cleanly. Deliberately outside the docs-deploy-asf-site
# concurrency group so it runs immediately for every trigger without
# blocking or being blocked by anything.
check-freshness:
runs-on: ubuntu-26.04
outputs:
is-current: ${{ steps.check.outputs.is-current }}
steps:
# Sparse checkout: this job's only job is to be fast, so it fetches
# nothing but the freshness-check script itself.
- name: Checkout freshness-check script
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
sparse-checkout: |
.github/workflows/scripts
sparse-checkout-cone-mode: false
- name: "Check whether this is still master's current commit"
id: check
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_SHA: ${{ github.event.workflow_run.head_sha || github.sha }}
EVENT_NAME: ${{ github.event_name }}
REPO: ${{ github.repository }}
OUTPUT_NAME: is-current
run: .github/workflows/scripts/check-docs-deploy-freshness.sh
build-deploy:
needs: [config, check-freshness]
# Only the run for master's current tip proceeds; anything superseded
# already skipped at check-freshness above instead of landing here.
# For workflow_run triggers, only deploy when the triggering run originated
# from this repository (not a fork), ensuring the checked-out code and any
# local actions executed with deploy credentials are trusted.
if: >-
needs.config.outputs.has-secrets &&
needs.check-freshness.outputs.is-current == 'true' &&
(github.event_name != 'workflow_run' ||
github.event.workflow_run.head_repository.full_name == github.repository)
name: Build & Deploy
runs-on: ubuntu-26.04
# Serialize deploys: the action pushes to apache/superset-site without
# rebasing, so concurrent runs race on the final push and the loser fails
# with `! [rejected] asf-site -> asf-site (fetch first)`. Queue instead of
# canceling: a run that already passed check-freshness can still be
# sitting in the queue for a runner when a newer run starts and finishes
# first. cancel-in-progress would let that stale, queued run kill the
# newer run's in-progress deploy the moment it's finally scheduled, and
# then skip itself at the re-check below — losing the deploy entirely.
# Queuing means the stale run just waits its turn and then no-ops at the
# re-check, so the fresher content that already deployed is never
# clobbered or lost. The check-freshness gate above means it should be
# rare for more than one run to reach this point, so the queue stays
# short in practice.
concurrency:
group: docs-deploy-asf-site
cancel-in-progress: false
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: "Checkout ${{ github.event.workflow_run.head_sha || github.sha }}"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./docs/.nvmrc"
- name: Setup Python
uses: ./.github/actions/setup-backend/
- uses: actions/setup-java@b6effb05e454b25005698d916606bdc6ffcbf961 # v5.7.0
with:
distribution: "zulu"
java-version: "21"
- name: Install Graphviz
uses: awalsh128/cache-apt-pkgs-action@553a35bb8ebd9fcabcb1c9451aa4c98e1b4ca8a9 # v1.6.3
with:
packages: graphviz
version: 1.0
execute_install_scripts: true
- name: Compute Entity Relationship diagram (ERD)
env:
SUPERSET_SECRET_KEY: not-a-secret
run: |
python scripts/erd/erd.py
curl -L http://sourceforge.net/projects/plantuml/files/1.2023.7/plantuml.1.2023.7.jar/download > ~/plantuml.jar
java -jar ~/plantuml.jar -v -tsvg -r -o "${{ github.workspace }}/docs/static/img/" "${{ github.workspace }}/scripts/erd/erd.puml"
- name: yarn install
working-directory: docs
run: |
yarn install --check-cache
- name: Download database diagnostics (if triggered by integration tests)
if: github.event_name == 'workflow_run' && github.event.workflow_run.conclusion == 'success'
uses: dawidd6/action-download-artifact@b6e2e70617bc3265edd6dab6c906732b2f1ae151 # v21
continue-on-error: true
with:
workflow: superset-python-integrationtest.yml
run_id: ${{ github.event.workflow_run.id }}
name: database-diagnostics
path: docs/src/data/
- name: Try to download latest diagnostics (for push/dispatch triggers)
if: github.event_name != 'workflow_run'
uses: dawidd6/action-download-artifact@b6e2e70617bc3265edd6dab6c906732b2f1ae151 # v21
continue-on-error: true
with:
workflow: superset-python-integrationtest.yml
name: database-diagnostics
path: docs/src/data/
branch: master
search_artifacts: true
if_no_artifact_found: warn
- name: Use diagnostics artifact if available
working-directory: docs
run: |
if [ -f "src/data/databases-diagnostics.json" ]; then
echo "Using fresh diagnostics from integration tests"
mv src/data/databases-diagnostics.json src/data/databases.json
else
echo "Using committed databases.json (no artifact found)"
fi
- name: yarn build
working-directory: docs
run: |
yarn build
# The check-freshness job above narrows the window but doesn't close it: an
# older run can observe is-current=true, then sit through this build while a
# newer run's own freshness check also passes and it deploys and finishes
# first. If this (stale) run then wins entry into the concurrency group, it
# would overwrite the newer content that already deployed. Re-check right
# before the one step that actually mutates superset-site, so a stale run
# skips deploying instead of clobbering a fresher one that already ran.
- name: "Re-check freshness immediately before deploying"
id: recheck-freshness
if: github.event_name != 'workflow_dispatch'
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_SHA: ${{ github.event.workflow_run.head_sha || github.sha }}
REPO: ${{ github.repository }}
OUTPUT_NAME: still-current
run: .github/workflows/scripts/check-docs-deploy-freshness.sh
- name: deploy docs
if: github.event_name == 'workflow_dispatch' || steps.recheck-freshness.outputs.still-current == 'true'
uses: ./.github/actions/github-action-push-to-another-repository
env:
API_TOKEN_GITHUB: ${{ secrets.SUPERSET_SITE_BUILD }}
with:
source-directory: "./docs/build"
destination-github-username: "apache"
destination-repository-name: "superset-site"
target-branch: "asf-site"
commit-message: "deploying docs: ${{ github.event.head_commit.message || 'triggered by integration tests' }} (apache/superset@${{ github.event.workflow_run.head_sha || github.sha }})"
user-email: dev@superset.apache.org
-163
View File
@@ -1,163 +0,0 @@
name: Docs Testing
on:
pull_request:
paths:
- "docs/**"
- "superset/db_engine_specs/**"
- ".github/workflows/superset-docs-verify.yml"
types: [synchronize, opened, reopened, ready_for_review]
# zizmor: ignore[dangerous-triggers] - runs in base-branch context and only consumes artifacts from the trusted upstream workflow
workflow_run:
workflows: ["Python-Integration"]
types: [completed]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.workflow_run.head_sha || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
jobs:
lychee:
# See docs here: https://github.com/lycheeverse/lychee-action
# Only run on pull_request, not workflow_run
if: github.event_name == 'pull_request'
name: Link Checking
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
# Restores the on-disk request cache (.lycheecache) lychee writes when
# `--cache` is set, so re-runs don't re-check every external link from
# scratch. Key never matches exactly (github.run_id is unique per run),
# so actions/cache always saves a fresh copy afterwards; restore-keys
# falls back to the most recent prior cache on lookup.
- uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: .lycheecache
key: lychee-cache-${{ github.run_id }}
restore-keys: lychee-cache-
# v2.8.0 and v2.9.0 are both on ASF Infra's action allowlist
# (apache/infrastructure-actions approved_patterns.yml) as of this
# writing. Do not bump past v2.9.0 without opening an Infra ticket
# to allow the new SHA first!
- uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2.9.0
continue-on-error: true # This will make the job advisory (non-blocking, no red X)
with:
fail: false
args: >-
--verbose --no-progress
--cache --max-cache-age 7d
--exclude '^https://github.com/apache/(superset|incubator-superset)/(pull|issues)/\d+'
--exclude '^https://github.com/apache/(superset|incubator-superset)/commit/[a-f0-9]+'
--exclude 'superset-frontend/.*CHANGELOG\.md'
--exclude 'http://localhost:8088/'
--exclude 'http://127.0.0.1:3000/'
--exclude 'http://localhost:9001/'
--exclude 'https://charts.bitnami.com/bitnami'
--exclude 'https://www.li.me/'
--exclude 'https://www.fanatics.com/'
--exclude 'https://tails.com/gb/'
--exclude 'https://www.techaudit.info/'
--exclude 'https://avetilearning.com/'
--exclude 'https://www.udemy.com/'
--exclude 'https://trustmedis.com/'
--exclude 'http://theiconic.com.au/'
--exclude 'https://dev.mysql.com/doc/refman/5.7/en/innodb-limits.html'
--exclude '^https://img\.shields\.io/.*'
--exclude 'https://vkusvill.ru/'
--exclude 'https://www.linkedin.com/in/mark-thomas-b16751158/'
--exclude 'https://theiconic.com.au/'
--exclude 'https://wattbewerb.de/'
--exclude 'https://timbr.ai/'
--exclude 'https://opensource.org/license/apache-2-0'
--exclude 'https://www.plaidcloud.com/'
'./**/*.md' './**/*.mdx'
build-on-pr:
# Build docs when PR changes docs/** (uses committed databases.json)
if: github.event_name == 'pull_request'
name: Build (PR trigger)
runs-on: ubuntu-26.04
defaults:
run:
working-directory: docs
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./docs/.nvmrc"
- name: yarn install
run: |
yarn install --check-cache
- name: Lint docs links
# Fast source-level check for bare relative internal links
# like `[Foo](../foo)` that Docusaurus's onBrokenLinks
# setting can't catch. Runs in seconds; fails fast before
# the expensive build step.
run: |
yarn lint:docs-links
- name: yarn typecheck
run: |
yarn typecheck
- name: yarn build
run: |
yarn build
build-after-tests:
# Build docs after integration tests complete (uses fresh diagnostics)
# Only runs if integration tests succeeded
if: >
github.event_name == 'workflow_run' &&
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.head_repository.full_name == github.repository
name: Build (after integration tests)
runs-on: ubuntu-26.04
defaults:
run:
working-directory: docs
steps:
- name: "Checkout PR head: ${{ github.event.workflow_run.head_sha }}"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.workflow_run.head_sha }}
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./docs/.nvmrc"
- name: yarn install
run: |
yarn install --check-cache
- name: Download database diagnostics from integration tests
uses: dawidd6/action-download-artifact@b6e2e70617bc3265edd6dab6c906732b2f1ae151 # v21
with:
workflow: superset-python-integrationtest.yml
run_id: ${{ github.event.workflow_run.id }}
name: database-diagnostics
path: docs/src/data/
if_no_artifact_found: "warning"
- name: Use fresh diagnostics
run: |
if [ -f "src/data/databases-diagnostics.json" ]; then
echo "Using fresh diagnostics from integration tests"
mv src/data/databases-diagnostics.json src/data/databases.json
else
echo "Warning: No diagnostics artifact found, using committed data"
fi
- name: yarn typecheck
run: |
yarn typecheck
- name: yarn build
run: |
yarn build
+56
View File
@@ -0,0 +1,56 @@
name: Docs
on:
push:
paths:
- "docs/**"
pull_request:
paths:
- "docs/**"
jobs:
config:
runs-on: "ubuntu-latest"
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${{ (secrets.SUPERSET_SITE_BUILD != '' && secrets.SUPERSET_SITE_BUILD != '') || '' }}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
build-deploy:
needs: config
if: needs.config.outputs.has-secrets
name: Build & Deploy
runs-on: ubuntu-20.04
defaults:
run:
working-directory: docs
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: yarn install
run: |
yarn install --check-cache
- name: yarn build
run: |
yarn build
- name: deploy docs
if: github.ref == 'refs/heads/master'
uses: ./.github/actions/github-action-push-to-another-repository
env:
API_TOKEN_GITHUB: ${{ secrets.SUPERSET_SITE_BUILD }}
with:
source-directory: './docs/build'
destination-github-username: 'apache'
destination-repository-name: 'superset-site'
target-branch: 'asf-site'
commit-message: "deploying docs: ${{ github.event.head_commit.message }} (apache/superset@${{ github.sha }})"
user-email: dev@superset.apache.org
+51 -284
View File
@@ -2,68 +2,29 @@ name: E2E
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/**/docs/**"
paths-ignore:
- "docs/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
workflow_dispatch:
inputs:
use_dashboard:
description: "Use Cypress Dashboard (true/false) [paid service - trigger manually when needed]. You MUST provide a branch and/or PR number below for this to work."
required: false
default: "false"
ref:
description: "The branch or tag to checkout"
required: false
default: ""
pr_id:
description: "The pull request ID to checkout"
required: false
default: ""
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
cypress-matrix:
needs: changes
if: (needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true') && github.event.pull_request.draft == false
runs-on: ubuntu-26.04
timeout-minutes: 30
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
permissions:
contents: read
pull-requests: read
strategy:
# when one test fails, DO NOT cancel the other
# app_root variant, because this will kill Cypress processes
# containers, because this will kill Cypress processes
# leaving the Dashboard hanging ...
# https://github.com/cypress-io/github-action/issues/48
fail-fast: false
matrix:
containers: [1, 2, 3]
browser: ["chrome"]
app_root: ${{ github.event_name == 'push' && fromJSON('["", "/app/prefix"]') || fromJSON('[""]') }}
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -71,11 +32,9 @@ jobs:
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
# Only use dashboard when explicitly requested via workflow_dispatch
USE_DASHBOARD: ${{ github.event.inputs.use_dashboard == 'true' || 'false' }}
services:
postgres:
image: postgres:17-alpine
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -86,275 +45,83 @@ jobs:
ports:
- 16379:6379
steps:
# -------------------------------------------------------
# Conditional checkout based on context
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: "Checkout (pull) ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
if: github.event_name == 'push'
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: "Checkout (pull_request) ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
if: github.event_name == 'pull_request' || github.event_name == 'pull_request_target'
with:
ref: "refs/pull/${{ github.event.number }}/merge"
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check if python or frontend changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python frontend
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: OS dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: apt-get-install
- name: Install python dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
pip-upgrade
pip install -r requirements/testing.txt
- name: Setup postgres
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: testdata
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
uses: actions/setup-node@v3
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version: "16"
- name: Install npm dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Install cypress
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: cypress-install
- name: Run Cypress
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
env:
CYPRESS_BROWSER: ${{ matrix.browser }}
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
NODE_OPTIONS: "--max-old-space-size=4096"
CYPRESS_KEY: YjljODE2MzAtODcwOC00NTA3LWE4NmMtMTU3YmFmMjIzOTRhCg==
with:
run: cypress-run-all ${{ env.USE_DASHBOARD }} ${{ matrix.app_root }}
- name: Set safe app root
if: failure()
id: set-safe-app-root
env:
APP_ROOT: ${{ matrix.app_root }}
run: |
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
run: cypress-run-all
- name: Upload Artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
uses: actions/upload-artifact@v3
if: failure()
with:
name: screenshots
path: ${{ github.workspace }}/superset-frontend/cypress-base/cypress/screenshots
name: cypress-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
playwright-tests:
needs: changes
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 30
permissions:
contents: read
pull-requests: read
strategy:
fail-fast: false
matrix:
browser: ["chromium"]
app_root: ${{ github.event_name == 'push' && fromJSON('["", "/app/prefix"]') || fromJSON('[""]') }}
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
# -------------------------------------------------------
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Setup Python
uses: ./.github/actions/setup-backend/
- name: Setup postgres
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
uses: ./.github/actions/cached-dependencies
with:
run: playwright_testdata
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
- name: Install npm dependencies
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Build embedded SDK
uses: ./.github/actions/cached-dependencies
with:
run: build-embedded-sdk
- name: Install Playwright
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Required Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
with:
run: playwright-run "${{ matrix.app_root }}"
- name: Run Playwright (Soft-delete Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
# Scoped to this step: each playwright-run boots its own gunicorn
# with the step's env, so the Required Tests server above keeps
# master's Flask configuration while this one runs with SOFT_DELETE
# on — the same isolation pattern as the Embedded step in
# superset-playwright.yml. Without a flag-on server the
# recently-archived specs skip themselves everywhere and ship zero
# executed E2E coverage; in the Required run above they are
# collected and skipped, which is expected.
SUPERSET_FEATURE_SOFT_DELETE: "true"
with:
run: playwright-run "${{ matrix.app_root }}" recently-archived/
- name: Set safe app root
if: failure()
id: set-safe-app-root
env:
APP_ROOT: ${{ matrix.app_root }}
run: |
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Playwright Artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: failure()
with:
path: |
${{ github.workspace }}/superset-frontend/playwright-results/
${{ github.workspace }}/superset-frontend/test-results/
name: playwright-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
# Stable required-status-check anchors. cypress-matrix and playwright-tests
# are matrix jobs gated on change detection (python || frontend). On a PR
# that touches neither — e.g. a docs-only PR — they are skipped at the job
# level, which happens before matrix expansion, so the per-combination
# contexts (`cypress-matrix (0, chrome)`, `playwright-tests (chromium)`) are
# never produced and branch protection waits on them forever. These
# always-running jobs report a single stable context that passes when the
# underlying matrix job succeeded or was skipped, and fails only on a real
# failure. Require these in .asf.yaml instead of the matrix-expanded names.
#
# A matrix job reads as "skipped" in two distinct cases, and only the first
# is a legitimate pass: (a) change detection succeeded and gated the job off
# (docs-only PR); (b) the `changes` job itself failed or was cancelled, in
# which case GHA skips its dependents too. Accepting (b) would let a broken
# change-detector report a false green, so each anchor first requires
# `changes` to have succeeded before honouring a skip.
cypress-matrix-required:
needs: [changes, cypress-matrix]
if: always()
runs-on: ubuntu-26.04
timeout-minutes: 5
permissions: {}
steps:
- name: Check cypress-matrix result
env:
CHANGES: ${{ needs.changes.result }}
RESULT: ${{ needs.cypress-matrix.result }}
run: |
if [ "$CHANGES" != "success" ]; then
echo "change detection did not succeed (result: $CHANGES); refusing to pass on a skipped matrix"
exit 1
fi
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "cypress-matrix did not pass (result: $RESULT)"
exit 1
fi
echo "cypress-matrix result: $RESULT (changes: $CHANGES)"
playwright-tests-required:
needs: [changes, playwright-tests]
if: always()
runs-on: ubuntu-26.04
timeout-minutes: 5
permissions: {}
steps:
- name: Check playwright-tests result
env:
CHANGES: ${{ needs.changes.result }}
RESULT: ${{ needs.playwright-tests.result }}
run: |
if [ "$CHANGES" != "success" ]; then
echo "change detection did not succeed (result: $CHANGES); refusing to pass on a skipped matrix"
exit 1
fi
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "playwright-tests did not pass (result: $RESULT)"
exit 1
fi
echo "playwright-tests result: $RESULT (changes: $CHANGES)"
actions-timeline:
needs: [cypress-matrix, playwright-tests, cypress-matrix-required, playwright-tests-required]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
@@ -1,81 +0,0 @@
name: Superset Extensions CLI Package Tests
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
pull-requests: read
actions: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
test-superset-extensions-cli-package:
runs-on: ubuntu-26.04
timeout-minutes: 30
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "next"]') }}
defaults:
run:
working-directory: superset-extensions-cli
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
if: steps.check.outputs.superset-extensions-cli
uses: ./.github/actions/setup-backend/
with:
python-version: ${{ matrix.python-version }}
requirements-type: dev
- name: Run pytest with coverage
if: steps.check.outputs.superset-extensions-cli
run: |
pytest --cov=superset_extensions_cli --cov-report=xml --cov-report=term-missing --cov-report=html -v --tb=short
- name: Upload coverage reports to Codecov
if: steps.check.outputs.superset-extensions-cli
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
file: ./coverage.xml
flags: superset-extensions-cli
name: superset-extensions-cli-coverage
fail_ci_if_error: false
- name: Upload HTML coverage report
if: steps.check.outputs.superset-extensions-cli
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: superset-extensions-cli-coverage-html
path: htmlcov/
actions-timeline:
needs: test-superset-extensions-cli-package
if: always()
runs-on: ubuntu-26.04
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
+53 -292
View File
@@ -1,311 +1,72 @@
name: "Frontend Build CI (unit tests, linting & sanity checks)"
name: Frontend
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/**/docs/**"
- "dependabot/**/cypress-base/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
env:
TAG: apache/superset:GHA-${{ github.run_id }}
permissions:
contents: read
jobs:
frontend-build:
runs-on: ubuntu-26.04
timeout-minutes: 30
# The change detector reads the PR's file list, which needs
# `pull-requests: read`. Public repos serve that endpoint without it;
# private forks return 403.
permissions:
contents: read
pull-requests: read
outputs:
should-run: ${{ steps.check.outputs.frontend }}
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
steps:
- name: Checkout Code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Check for File Changes
submodules: recursive
- name: Check npm lock file version
run: ./scripts/ci_check_npm_lock_version.sh ./superset-frontend/package-lock.json
- name: Check if frontend changes are present
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Build Docker Image
if: steps.check.outputs.frontend
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo "git rev-parse --short HEAD"
git rev-parse --short HEAD
echo "git show -s --format=raw HEAD"
git show -s --format=raw HEAD
docker buildx build \
-t $TAG \
--cache-from=type=registry,ref=apache/superset-cache:3.11-slim-trixie \
--target superset-node-ci \
.
- name: Save Docker Image as Artifact
if: steps.check.outputs.frontend
run: |
docker save $TAG | zstd -3 --threads=0 > docker-image.tar.zst
- name: Upload Docker Image Artifact
if: steps.check.outputs.frontend
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh frontend
- name: Setup Node.js
if: steps.check.outcome == 'failure'
uses: actions/setup-node@v3
with:
name: docker-image
path: docker-image.tar.zst
sharded-jest-tests:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
strategy:
matrix:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
fail-fast: false
runs-on: ubuntu-26.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
node-version: "16"
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
name: docker-image
- name: Load Docker Image
run: |
zstd -d < docker-image.tar.zst | docker load
- name: npm run test with coverage
run: |
mkdir -p ${{ github.workspace }}/superset-frontend/coverage
docker run \
-v ${{ github.workspace }}/superset-frontend/coverage:/app/superset-frontend/coverage \
--rm $TAG \
bash -c \
"npm run test -- --coverage --shard=${{ matrix.shard }}/8 --coverageReporters=json"
- name: Upload Coverage Artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: coverage-artifacts-${{ matrix.shard }}
path: superset-frontend/coverage
report-coverage:
needs: [sharded-jest-tests]
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 15
permissions:
id-token: write
steps:
- name: Checkout Code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
fetch-depth: 0
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Download Coverage Artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
pattern: coverage-artifacts-*
path: coverage/
- name: Upload Code Coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: javascript
use_oidc: true
verbose: true
directory: coverage
slug: apache/superset
lint-frontend:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: docker-image
- name: Load Docker Image
run: |
zstd -d < docker-image.tar.zst | docker load
run: npm-install
- name: lint
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: |
docker run --rm $TAG bash -c \
"npm i && npm run lint"
- name: tsc
npm run lint -- --quiet
npm run prettier-check
- name: Build plugins packages
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: npm run plugins:build
- name: Build plugins Storybook
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: npm run plugins:build-storybook
- name: superset-ui/core coverage
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: |
docker run --rm $TAG bash -c \
"npm i && npm run plugins:build && npm run type"
validate-frontend:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: docker-image
- name: Load Docker Image
npm run core:cover
- name: unit tests
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: |
zstd -d < docker-image.tar.zst | docker load
- name: Validate dependency tree
run: |
docker run --rm $TAG bash -c \
"npm ls --all --package-lock-only --depth=0 --json > /dev/null"
test-storybook:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 25
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: docker-image
- name: Load Docker Image
run: |
zstd -d < docker-image.tar.zst | docker load
- name: Build Storybook and Run Tests
run: |
docker run --rm $TAG bash -c \
"npm run build-storybook && npx playwright install-deps && npx playwright install chromium && npm run test-storybook:ci"
actions-timeline:
needs: [report-coverage, lint-frontend, validate-frontend, test-storybook]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
# Compares a PR's own bundle size against the last nightly-recorded
# baseline (see frontend-bundle-size-nightly.yml, which owns actually
# persisting new baselines). PR-only: a push to master doesn't need this
# check re-run against itself, and re-persisting the baseline on every
# push to master -- which happens many times a day -- would burn a full
# production build for no benefit nightly refresh doesn't already cover.
bundle-size:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true' && github.event_name == 'pull_request'
runs-on: ubuntu-26.04
timeout-minutes: 15
permissions:
contents: read
pull-requests: write
steps:
- name: Checkout Code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: docker-image
- name: Load Docker Image
run: |
zstd -d < docker-image.tar.zst | docker load
# webpack's persistent filesystem cache (superset-frontend/webpack.config.js)
# turns a warm production build into ~20s instead of several minutes,
# but GH-hosted runners are fresh VMs with nothing carried over between
# jobs -- without restoring it explicitly, every single PR would pay
# the full cold-build cost. Keyed on the same files webpack's own
# `buildDependencies` invalidates on, so a stale cache is never used.
- name: Restore webpack build cache
uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: superset-frontend/.temp_cache
key: >-
webpack-prod-cache-${{ hashFiles('superset-frontend/package-lock.json',
'superset-frontend/babel.config.js', 'superset-frontend/tsconfig.json',
'superset-frontend/webpack.config.js') }}
# Only ever pull the last recorded data point off the cache, keyed by
# run ID -- `restore-keys` prefix-matches the most recently created
# entry, which is always the latest nightly run. Absent before the
# first nightly run ever happens; benchmark-action starts a fresh
# history in that case.
- name: Restore bundle size history
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: bundle-size-history.json
key: bundle-size-history-${{ github.run_id }}
restore-keys: |
bundle-size-history-
- name: Build production bundle with stats
run: |
mkdir -p ${{ github.workspace }}/superset-frontend/bundle-stats
mkdir -p ${{ github.workspace }}/superset-frontend/.temp_cache
docker run \
-v ${{ github.workspace }}/superset-frontend/bundle-stats:/app/superset-frontend/bundle-stats \
-v ${{ github.workspace }}/superset-frontend/.temp_cache:/app/superset-frontend/.temp_cache \
--rm $TAG \
bash -c \
"npm i && BUNDLE_SIZE_STATS=true npm run build -- --json=bundle-stats/stats.json"
- name: Summarize bundle size
run: |
node superset-frontend/scripts/bundle-size-summary.js \
superset-frontend/bundle-stats/stats.json > bundle-size-summary.json
rm -rf superset-frontend/bundle-stats
# Comparison + alert only -- this job never persists. See
# frontend-bundle-size-nightly.yml for why.
#
# comment-on-alert is gated to same-repo PRs: on a fork PR,
# GITHUB_TOKEN is forced read-only regardless of the `permissions`
# block above, so once the alert threshold is crossed the action's
# `pulls.createReview` call 403s. That error isn't gated by
# fail-on-alert (which only governs the deliberate alert-threshold
# failure) -- it propagates and fails the job outright. Fork PRs
# still get the comparison via the job summary (summary-always).
- name: Compare bundle size against nightly baseline
uses: benchmark-action/github-action-benchmark@52576c92bccf6ac60c8223ec7eb2565637cae9ba # v1.22.1
with:
tool: customSmallerIsBetter
output-file-path: bundle-size-summary.json
external-data-json-path: bundle-size-history.json
github-token: ${{ secrets.GITHUB_TOKEN }}
comment-on-alert: ${{ github.event.pull_request.head.repo.full_name == github.repository }}
alert-threshold: "110%"
fail-on-alert: false
summary-always: true
npm run test -- --coverage --silent
# todo: remove this step when fix generator as a project in root jest.config.js
- name: generator-superset unit tests
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend/packages/generator-superset
run: npx jest
- name: Upload code coverage
if: steps.check.outcome == 'failure'
working-directory: ./superset-frontend
run: ../.github/workflows/codecov.sh -c -F javascript
@@ -1,77 +0,0 @@
name: "Helm: lint and test charts"
on:
pull_request:
types: [opened, reopened, synchronize]
paths:
- "helm/**"
permissions:
contents: read
actions: read
# Serialize runs per PR without cancelling: when a first-time contributor's
# queued runs are approved together, cancel-in-progress lets an older run
# cancel a newer one, leaving a permanently-cancelled required check on the
# head SHA. Queued runs are cheap here, so let them all complete.
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: false
jobs:
lint-test:
runs-on: ubuntu-slim
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
fetch-depth: 0
- name: Set up Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
with:
version: v3.21.3
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
install-superset: "false"
# Still vendored (not de-vendored like chart-releaser-action below): the
# allowlisted helm/chart-testing-action@v2.8.0 depends internally on
# astral-sh/setup-uv@v7.0.0, which isn't itself on the ASF Actions
# allowlist (only v8.1.0+ are, at apache/infrastructure-actions'
# actions.yml). Needs an INFRA request before this can de-vendor too.
- name: Set up chart-testing
uses: ./.github/actions/chart-testing-action
- name: Run chart-testing (list-changed)
id: list-changed
run: |
changed=$(ct list-changed --print-config)
if [[ -n "$changed" ]]; then
echo "changed=true" >> $GITHUB_OUTPUT
fi
env:
CT_CHART_DIRS: helm
CT_SINCE: HEAD
- name: Run chart-testing (lint)
run: ct lint --print-config
env:
CT_CHART_DIRS: helm
CT_LINT_CONF: lintconf.yaml
CT_SINCE: HEAD
CT_CHART_REPOS: bitnami=https://charts.bitnami.com/bitnami
- name: Set up helm-unittest
run: helm plugin install https://github.com/helm-unittest/helm-unittest --version v1.1.1
- name: Run helm-unittest
run: helm unittest helm/superset
+47
View File
@@ -0,0 +1,47 @@
name: Lint and Test Charts
on:
pull_request:
types: [opened, edited, reopened, synchronize]
jobs:
lint-test:
runs-on: ubuntu-latest
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
fetch-depth: 0
- name: Set up Helm
uses: azure/setup-helm@v1
with:
version: v3.5.4
- uses: actions/setup-python@v4
with:
python-version: "3.9"
- name: Set up chart-testing
uses: ./.github/actions/chart-testing-action
- name: Run chart-testing (list-changed)
id: list-changed
run: |
changed=$(ct list-changed --print-config)
if [[ -n "$changed" ]]; then
echo "::set-output name=changed::true"
fi
env:
CT_CHART_DIRS: helm
CT_SINCE: HEAD
- name: Run chart-testing (lint)
run: ct lint --print-config
env:
CT_CHART_DIRS: helm
CT_LINT_CONF: lintconf.yaml
CT_SINCE: HEAD
CT_CHART_REPOS: bitnami=https://charts.bitnami.com/bitnami
+7 -157
View File
@@ -1,47 +1,23 @@
# This workflow automates the release process for Helm charts.
# Each run force-recreates a single 'helm-publish' branch from the tip of 'gh-pages' and
# opens (or reuses) one pull request against 'gh-pages', allowing the changes to be
# reviewed and merged manually. Because chart-releaser rebuilds index.yaml from all
# published GitHub releases, the branch always contains every chart released since the
# last merge, and the PR can never go stale or conflict with gh-pages.
name: "Helm: release charts"
name: Release Charts
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
paths:
- "helm/**"
workflow_dispatch:
inputs:
ref:
description: "The branch, tag, or commit SHA to check out"
required: false
default: "master"
# Serialize runs: concurrent runs would race on force-pushing the shared
# helm-publish branch while chart-releaser is mid-release.
concurrency:
group: helm-release
cancel-in-progress: false
jobs:
release:
runs-on: ubuntu-26.04
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
steps:
- name: Checkout code
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
ref: ${{ inputs.ref || github.ref_name }}
persist-credentials: true
persist-credentials: false
submodules: recursive
fetch-depth: 0
@@ -51,143 +27,17 @@ jobs:
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Install Helm
uses: azure/setup-helm@9bc31f4ebc9c6b171d7bfbaa5d006ae7abdb4310 # v5.0.1
uses: azure/setup-helm@v1
with:
version: v3.5.4
- name: Add bitnami repo dependency
run: helm repo add bitnami https://charts.bitnami.com/bitnami
- name: Fetch/list all tags
run: |
# Debugging tags
git fetch --tags --force
git tag -d superset-helm-chart-0.13.4 || true
echo "DEBUG TAGS"
git show-ref --tags
- name: Set pages branch name
id: vars
run: echo "branch_name=helm-publish" >> $GITHUB_ENV
- name: Force recreate branch from gh-pages
env:
BRANCH_NAME: ${{ env.branch_name }}
run: |
# Ensure a clean working directory
git reset --hard
git clean -fdx
git checkout -b local_gha_temp
git submodule update
# Fetch the latest gh-pages branch
git fetch origin gh-pages
# Check out and reset the target branch based on gh-pages
git checkout -B "$BRANCH_NAME" origin/gh-pages
# Remove submodules from the branch
git submodule deinit -f --all
# Force push to the remote branch
git push origin "$BRANCH_NAME" --force
# Return to the original branch
git checkout local_gha_temp
- name: Run chart-releaser
uses: helm/chart-releaser-action@cae68fefc6b5f367a0275617c9f83181ba54714f # v1.7.0
uses: ./.github/actions/chart-releaser-action
with:
version: v1.6.0
charts_dir: helm
mark_as_latest: false
# A helm/** change without a Chart.yaml version bump repackages the
# already-released version; without this, cr aborts on the existing
# release tag (422 already_exists) instead of proceeding to rebuild
# the index, and the whole run fails.
skip_existing: true
pages_branch: ${{ env.branch_name }}
env:
CR_TOKEN: "${{ github.token }}"
CR_RELEASE_NAME_TEMPLATE: "superset-helm-chart-{{ .Version }}"
- name: Open or reuse Pull Request
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const branchName = process.env.BRANCH_NAME;
const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
if (!branchName) {
throw new Error("Branch name is not defined.");
}
// The branch is force-recreated from gh-pages on every run, so an
// already-open PR for it now reflects this run's charts; opening
// another would both fail (422) and recreate the stale-PR pileup
// this fixed branch exists to avoid.
const { data: existing } = await github.rest.pulls.list({
owner,
repo,
state: "open",
head: `${owner}:${branchName}`,
base: "gh-pages",
});
let current;
if (existing.length > 0) {
current = existing[0];
core.info(`Reusing existing pull request: ${current.html_url}`);
} else {
const { data: pr } = await github.rest.pulls.create({
owner,
repo,
title: "Helm chart release",
head: branchName,
base: "gh-pages", // Adjust if the target branch is different
body: [
"This PR releases Helm charts to the gh-pages branch.",
"",
"It is force-updated from the tip of `gh-pages` by every release run,",
"so it always contains every chart released since the last merge and",
"never needs to be closed as superseded.",
].join("\n"),
});
current = pr;
core.info(`Pull request created: ${current.html_url}`);
}
// Sweep release PRs left open by older runs (per-SHA helm-publish-*
// branches from the previous scheme). Their content is a subset of
// the evergreen PR, so close them with a pointer to it.
const { data: openPrs } = await github.rest.pulls.list({
owner,
repo,
state: "open",
base: "gh-pages",
per_page: 100,
});
for (const stale of openPrs) {
if (
stale.number !== current.number &&
stale.head.repo?.full_name === process.env.GITHUB_REPOSITORY &&
stale.head.ref.startsWith("helm-publish")
) {
await github.rest.issues.createComment({
owner,
repo,
issue_number: stale.number,
body: `Superseded by #${current.number}, which now carries all unreleased charts. Closing.`,
});
await github.rest.pulls.update({
owner,
repo,
pull_number: stale.number,
state: "closed",
});
core.info(`Closed superseded release PR #${stale.number}`);
}
}
env:
BRANCH_NAME: ${{ env.branch_name }}
-194
View File
@@ -1,194 +0,0 @@
name: Playwright Experimental Tests
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
workflow_dispatch:
inputs:
ref:
description: "The branch or tag to checkout"
required: false
default: ""
pr_id:
description: "The pull request ID to checkout"
required: false
default: ""
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
# NOTE: Required Playwright tests are in superset-e2e.yml (E2E / playwright-tests)
# This workflow contains only experimental tests that run in shadow mode
playwright-tests-experimental:
needs: changes
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 30
continue-on-error: true
permissions:
contents: read
pull-requests: read
strategy:
fail-fast: false
matrix:
browser: ["chromium"]
app_root: ["", "/app/prefix"]
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
services:
postgres:
image: postgres:17-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
# -------------------------------------------------------
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Setup Python
uses: ./.github/actions/setup-backend/
- name: Setup postgres
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
uses: ./.github/actions/cached-dependencies
with:
run: playwright_testdata
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
- name: Install npm dependencies
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Build embedded SDK
uses: ./.github/actions/cached-dependencies
with:
run: build-embedded-sdk
- name: Install Playwright
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Experimental Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
with:
run: playwright-run "${{ matrix.app_root }}" experimental/
- name: Run Playwright (Embedded Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
# Scope embedded-only env vars to this step. Setting them at the job
# level enabled the EMBEDDED_SUPERSET feature flag inside Flask for
# the preceding "Required Tests" and "Experimental Tests" steps too,
# which loads extra handlers and destabilizes the werkzeug dev
# server under the 2-worker Playwright load. Required Tests should
# match master's Flask configuration.
SUPERSET_FEATURE_EMBEDDED_SUPERSET: "true"
INCLUDE_EMBEDDED: "true"
with:
run: playwright-run "${{ matrix.app_root }}" embedded
- name: Run Playwright (Mobile Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
# Scoped to this step for the same reason as the embedded flags
# above: the mobile consumption mode should not alter Flask's
# configuration for the required desktop test steps.
SUPERSET_FEATURE_MOBILE_CONSUMPTION_MODE: "true"
INCLUDE_MOBILE: "true"
with:
run: playwright-run "${{ matrix.app_root }}" mobile/
- name: Set safe app root
if: failure()
id: set-safe-app-root
run: |
APP_ROOT="${{ matrix.app_root }}"
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Playwright Artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
if: failure()
with:
path: |
${{ github.workspace }}/superset-frontend/playwright-results/
${{ github.workspace }}/superset-frontend/test-results/
name: playwright-experimental-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
actions-timeline:
needs: playwright-tests-experimental
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
@@ -1,50 +1,20 @@
# Python integration tests
name: Python-Integration
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
test-mysql:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 45
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -53,18 +23,11 @@ jobs:
mysql+mysqldb://superset:superset@127.0.0.1:13306/superset?charset=utf8mb4&binary_prefix=true
services:
mysql:
image: mysql:8.0
# Authenticated pulls use our higher Docker Hub rate limit. Empty on
# fork PRs (secrets unavailable) -> runner falls back to anonymous.
image: mysql:5.7
env:
MYSQL_ROOT_PASSWORD: root
ports:
- 13306:3306
options: >-
--health-cmd="mysqladmin ping --silent"
--health-interval=10s
--health-timeout=5s
--health-retries=5
redis:
image: redis:7-alpine
options: --entrypoint redis-server
@@ -72,70 +35,52 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
uses: ./.github/actions/setup-backend/
- name: Setup MySQL
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: setup-mysql
- name: Start Celery worker
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-mysql
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (MySQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,mysql
verbose: true
use_oidc: true
slug: apache/superset
- name: Generate database diagnostics for docs
env:
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: |
mysql+mysqldb://superset:superset@127.0.0.1:13306/superset?charset=utf8mb4&binary_prefix=true
if: steps.check.outcome == 'failure'
run: |
python -c "
import json
from superset.app import create_app
from superset.db_engine_specs.lib import generate_yaml_docs
app = create_app()
with app.app_context():
docs = generate_yaml_docs()
# Wrap in the expected format
output = {
'generated': '$(date -Iseconds)',
'databases': docs
}
with open('databases-diagnostics.json', 'w') as f:
json.dump(output, f, indent=2, default=str)
print(f'Generated diagnostics for {len(docs)} databases')
"
- name: Upload database diagnostics artifact
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: database-diagnostics
path: databases-diagnostics.json
retention-days: 7
bash .github/workflows/codecov.sh -c -F python -F mysql
test-postgres:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 45
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "next"]') }}
python-version: ["3.9", "3.10"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -143,7 +88,7 @@ jobs:
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
services:
postgres:
image: postgres:17-alpine
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -157,49 +102,58 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
- name: Setup Postgres
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Start Celery worker
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,postgres
verbose: true
use_oidc: true
slug: apache/superset
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F postgres
test-sqlite:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 45
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
REDIS_PORT: 16379
SUPERSET__SQLALCHEMY_DATABASE_URI: |
sqlite:///${{ github.workspace }}/.temp/superset.db?check_same_thread=true
SUPERSET__SQLALCHEMY_EXAMPLES_URI: |
sqlite:///${{ github.workspace }}/.temp/examples.db?check_same_thread=true
sqlite:///${{ github.workspace }}/.temp/unittest.db
services:
redis:
image: redis:7-alpine
@@ -207,62 +161,42 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
# sqlite needs this working directory
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
mkdir ${{ github.workspace }}/.temp
- name: Start Celery worker
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python integration tests (SQLite)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,sqlite
verbose: true
use_oidc: true
slug: apache/superset
# Stable required-status-check anchor for the matrix-based test-postgres job.
# It is gated on change detection, so on non-Python PRs it is skipped and
# never produces its `test-postgres (current)` context (a job-level skip
# happens before matrix expansion). This always-running job reports a single
# context branch protection can require: it passes when test-postgres
# succeeded or was skipped, and fails only on a real failure.
test-postgres-required:
needs: [changes, test-postgres]
if: always()
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- name: Check test-postgres result
env:
RESULT: ${{ needs.test-postgres.result }}
if: steps.check.outcome == 'failure'
run: |
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "test-postgres did not pass (result: $RESULT)"
exit 1
fi
echo "test-postgres result: $RESULT"
actions-timeline:
needs: [test-mysql, test-postgres, test-sqlite, test-postgres-required]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
bash .github/workflows/codecov.sh -c -F python -F sqlite
+120
View File
@@ -0,0 +1,120 @@
# Python Misc unit tests
name: Python Misc
on:
push:
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
jobs:
python-lint:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
cache-dependency-path: "requirements/testing.txt"
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
- name: pylint
if: steps.check.outcome == 'failure'
# `-j 0` run Pylint in parallel
run: pylint -j 0 superset
pre-commit:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
cache-dependency-path: |
requirements/base.txt
requirements/integration.txt
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/base.txt
pip install -r requirements/integration.txt
# Add brew to the path - see https://github.com/actions/runner-images/issues/6283
- name: Enable brew and helm-docs
run: |
echo "/home/linuxbrew/.linuxbrew/bin:/home/linuxbrew/.linuxbrew/sbin" >> $GITHUB_PATH
eval "$(/home/linuxbrew/.linuxbrew/bin/brew shellenv)"
echo "HOMEBREW_PREFIX=$HOMEBREW_PREFIX" >>"${GITHUB_ENV}"
echo "HOMEBREW_CELLAR=$HOMEBREW_CELLAR" >>"${GITHUB_ENV}"
echo "HOMEBREW_REPOSITORY=$HOMEBREW_REPOSITORY" >>"${GITHUB_ENV}"
brew install norwoodj/tap/helm-docs
- name: pre-commit
run: pre-commit run --all-files
babel-extract:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Setup Python
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: "pip"
cache-dependency-path: "requirements/base.txt"
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/base.txt
- name: Test babel extraction
run: flask fab babel-extract --target superset/translations --output superset/translations/messages.pot --config superset/translations/babel.cfg -k _,__,t,tn,tct
@@ -3,44 +3,18 @@ name: Python Presto/Hive
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
test-postgres-presto:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 45
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -49,7 +23,7 @@ jobs:
SUPERSET__SQLALCHEMY_EXAMPLES_URI: presto://localhost:15433/memory/default
services:
postgres:
image: postgres:17-alpine
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -72,38 +46,52 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
uses: ./.github/actions/setup-backend/
- name: Setup Postgres
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Start Celery worker
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,presto
verbose: true
use_oidc: true
slug: apache/superset
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F presto
test-postgres-hive:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 45
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -113,7 +101,7 @@ jobs:
UPLOAD_FOLDER: /tmp/.superset/uploads/
services:
postgres:
image: postgres:17-alpine
image: postgres:14-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
@@ -127,45 +115,51 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Create csv upload directory
if: steps.check.outcome == 'failure'
run: sudo mkdir -p /tmp/.superset/uploads
- name: Give write access to the csv upload directory
if: steps.check.outcome == 'failure'
run: sudo chown -R $USER:$USER /tmp/.superset
- name: Start hadoop and hive
run: docker compose -f scripts/databases/hive/docker-compose.yml up -d
if: steps.check.outcome == 'failure'
run: docker-compose -f scripts/databases/hive/docker-compose.yml up -d
- name: Setup Python
uses: ./.github/actions/setup-backend/
- name: Setup Postgres
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Start Celery worker
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
setup-postgres
- name: Run celery
if: steps.check.outcome == 'failure'
run: celery --app=superset.tasks.celery_app:app worker -Ofair -c 2 &
- name: Python unit tests (PostgreSQL)
if: steps.check.outcome == 'failure'
run: |
uv pip install --system -e .[hive]
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,hive
verbose: true
use_oidc: true
slug: apache/superset
actions-timeline:
needs: [test-postgres-presto, test-postgres-hive]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
if: steps.check.outcome == 'failure'
run: |
bash .github/workflows/codecov.sh -c -F python -F hive
@@ -1,70 +0,0 @@
name: Python Unit Test Results
on:
# zizmor: ignore[dangerous-triggers] - runs in base-branch context and only consumes artifacts uploaded by Python-Unit; never checks out PR code (see note below)
workflow_run:
workflows: ["Python-Unit"]
types: [completed]
# This workflow publishes a check run annotating failing Python unit tests
# inline on the PR diff, using JUnit XML uploaded by the Python-Unit workflow.
# It uses the workflow_run trigger so that it always runs in the base-branch
# context and can safely be granted write permissions, even for PRs from
# forks or Dependabot.
#
# IMPORTANT: This workflow must NEVER check out code from the PR branch. All
# data comes from artifacts uploaded by the Python-Unit workflow.
permissions:
contents: read
checks: write
issues: read
actions: read
jobs:
report:
runs-on: ubuntu-26.04
timeout-minutes: 10
if: >
github.event.workflow_run.conclusion == 'success' ||
github.event.workflow_run.conclusion == 'failure'
steps:
# Fails soft (continue-on-error) because the source unit-tests job is
# itself gated on change detection: a docs-only PR skips it entirely,
# so there is nothing to download or report on.
- name: Download JUnit results
id: download
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
# merge-multiple is intentionally omitted: each matrix leg's
# artifact (junit-results-current, junit-results-next) uses the
# same XML filenames, so merging them into one directory would let
# one Python version's results overwrite the other's. Downloading
# into per-artifact subdirectories keeps both, and the glob below
# is recursive so it still picks up every XML file.
pattern: "junit-results-*"
path: artifacts
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Download event file
id: download-event
if: steps.download.outcome == 'success'
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: "Event File"
path: event
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
- name: Publish test results
if: steps.download.outcome == 'success' && steps.download-event.outcome == 'success'
uses: EnricoMi/publish-unit-test-result-action@d0a4676d0e0b938bc201470d88276b7c74c712b3 # v2.24.0
with:
commit: ${{ github.event.workflow_run.head_sha }}
event_file: event/event.json
event_name: ${{ github.event.workflow_run.event }}
files: "artifacts/**/*.xml"
check_name: "Python Unit Test Results"
comment_mode: "off"
+32 -119
View File
@@ -1,147 +1,60 @@
# Python unit tests
name: Python-Unit
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-26.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
unit-tests:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-26.04
timeout-minutes: 30
permissions:
id-token: write
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "next"]') }}
python-version: ["3.9", "3.10"]
env:
PYTHONPATH: ${{ github.workspace }}
# Promotes the SQLAlchemy 2.0 deprecation warnings already locked in as
# errors via pytest.ini's `filterwarnings` to actually run in CI, so a
# regression on those fails the build instead of relying on a
# contributor remembering to set this locally. See the migration
# battleplan: https://github.com/apache/superset/discussions/40273
SQLALCHEMY_WARN_20: "1"
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check if python changes are present
id: check
env:
GITHUB_REPO: ${{ github.repository }}
PR_NUMBER: ${{ github.event.pull_request.number }}
continue-on-error: true
run: ./scripts/ci_check_no_file_changes.sh python
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outcome == 'failure'
uses: actions/setup-python@v4
with:
python-version: ${{ matrix.python-version }}
cache: 'pip'
cache-dependency-path: 'requirements/testing.txt'
# TODO: separated requirements.txt file just for unit tests
- name: Install dependencies
if: steps.check.outcome == 'failure'
uses: ./.github/actions/cached-dependencies
with:
run: |
apt-get-install
pip-upgrade
pip install wheel
pip install -r requirements/testing.txt
mkdir ${{ github.workspace }}/.temp
- name: Python unit tests
if: steps.check.outcome == 'failure'
env:
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
run: |
pytest --durations-min=0.5 --cov-report= --cov=superset ./tests/common ./tests/unit_tests --cache-clear --maxfail=50 --junit-xml=test-results/junit-unit.xml
- name: Python 100% coverage unit tests
env:
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
run: |
pytest --durations-min=0.5 --cov=superset/sql/ ./tests/unit_tests/sql/ --cache-clear --cov-fail-under=100 --junit-xml=test-results/junit-sql-coverage.xml
pytest --durations-min=0.5 --cov=superset/semantic_layers/ ./tests/unit_tests/semantic_layers/ --cache-clear --cov-fail-under=100 --junit-xml=test-results/junit-semantic-layers-coverage.xml
pytest --durations-min=0.5 --cov-report= --cov=superset ./tests/common ./tests/unit_tests --cache-clear
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
with:
flags: python,unit
verbose: true
use_oidc: true
slug: apache/superset
# Uploaded even when a pytest step above fails, since that is exactly
# when the JUnit results are needed downstream, to annotate the PR with
# the failing tests. Consumed by the "Python Unit Test Results" workflow
# via workflow_run (see that workflow for why it can't just be a step
# here: it needs to run with write permissions, which this PR-triggered
# job can't safely have on a fork PR).
- name: Upload JUnit test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: junit-results-${{ matrix.python-version }}
path: test-results/
retention-days: 7
# Uploads the raw pull_request event payload so the "Python Unit Test
# Results" workflow (running via workflow_run, in base-branch context) can
# look up which PR/commit to annotate without checking out untrusted code.
event-file:
runs-on: ubuntu-26.04
timeout-minutes: 5
steps:
- name: Upload event file
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: Event File
path: ${{ github.event_path }}
retention-days: 7
# Stable required-status-check anchor. `unit-tests` is a matrix job gated on
# change detection, so on non-Python PRs it is skipped and never produces its
# `unit-tests (current)` context (a job-level skip happens before matrix
# expansion). This always-running job reports a single context that branch
# protection can require: it passes when unit-tests succeeded or was skipped,
# and fails only on a real failure.
unit-tests-required:
needs: [changes, unit-tests]
if: always()
runs-on: ubuntu-26.04
timeout-minutes: 5
permissions:
contents: read
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
- name: Check unit-tests result
env:
RESULT: ${{ needs.unit-tests.result }}
if: steps.check.outcome == 'failure'
run: |
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "unit-tests did not pass (result: $RESULT)"
exit 1
fi
echo "unit-tests result: $RESULT"
bash .github/workflows/codecov.sh -c -F python -F unit
@@ -1,88 +0,0 @@
name: Translation Regression Comment
on:
# zizmor: ignore[dangerous-triggers] - runs in base-branch context and only consumes the uploaded artifact; never checks out PR code (see note below)
workflow_run:
workflows: ["Translations"]
types: [completed]
# This workflow posts a PR comment when the Translations workflow detects a
# regression. It uses the workflow_run trigger so that it always runs in the
# base-branch context and can safely be granted write permissions, even for
# PRs from forks.
#
# IMPORTANT: This workflow must NEVER check out code from the PR branch.
# All data comes from the artifact uploaded by the Translations workflow.
permissions:
pull-requests: write
actions: read
jobs:
post-comment:
runs-on: ubuntu-26.04
# Only act when the Translations workflow failed (which means a regression
# was detected — the workflow exits 1 on regression).
if: github.event.workflow_run.conclusion == 'failure'
steps:
- name: Download regression artifact
id: download
continue-on-error: true
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
with:
name: translation-regression
run-id: ${{ github.event.workflow_run.id }}
github-token: ${{ secrets.GITHUB_TOKEN }}
path: /tmp/translation-regression
- name: Post or update PR comment
if: steps.download.outcome == 'success'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const fs = require('fs');
const prNumberFile = '/tmp/translation-regression/pr-number.txt';
const reportFile = '/tmp/translation-regression/regression-report.md';
if (!fs.existsSync(prNumberFile) || !fs.existsSync(reportFile)) {
console.log('Artifact files not found, skipping comment.');
return;
}
const prNumber = parseInt(fs.readFileSync(prNumberFile, 'utf8').trim(), 10);
if (!prNumber) {
console.log('Could not parse PR number, skipping comment.');
return;
}
const report = fs.readFileSync(reportFile, 'utf8');
const marker = '<!-- translation-regression-bot -->';
const body = `${marker}\n${report}`;
const { data: comments } = await github.rest.issues.listComments({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
});
const existing = comments.find(c => c.body && c.body.includes(marker));
if (existing) {
await github.rest.issues.updateComment({
owner: context.repo.owner,
repo: context.repo.repo,
comment_id: existing.id,
body,
});
console.log(`Updated existing comment ${existing.id} on PR #${prNumber}`);
} else {
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: prNumber,
body,
});
console.log(`Created new comment on PR #${prNumber}`);
}
+25 -137
View File
@@ -2,168 +2,56 @@ name: Translations
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
branches-ignore:
- "dependabot/npm_and_yarn/**"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
pull-requests: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
frontend-check-translations:
runs-on: ubuntu-26.04
permissions:
contents: read
pull-requests: read
frontend-check:
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Node.js
if: steps.check.outputs.frontend
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
uses: actions/setup-node@v3
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version: '16'
- name: Install dependencies
if: steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: lint
if: steps.check.outputs.frontend
working-directory: ./superset-frontend
run: |
npm run build-translation
npm run check-translation
babel-extract:
runs-on: ubuntu-26.04
permissions:
contents: read
pull-requests: read
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
strategy:
matrix:
python-version: ["3.9"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
uses: ./.github/actions/setup-backend/
- name: Install gettext tools
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
uses: awalsh128/cache-apt-pkgs-action@553a35bb8ebd9fcabcb1c9451aa4c98e1b4ca8a9 # v1.6.3
uses: actions/setup-python@v4
with:
packages: gettext
version: 1.0
# Fetch the base ref so we can compare PR-introduced regressions
# against a fair baseline (also runs babel_update against the base
# source) — this isolates the PR's contribution from any pre-existing
# drift on the base branch.
- name: Fetch base ref and create comparison worktree
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
env:
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
# For PRs use the base branch; for direct pushes compare against the previous commit.
BASE_REF="$PR_BASE_REF"
if [ -n "$BASE_REF" ]; then
git fetch --depth=1 origin "$BASE_REF"
else
git fetch --depth=2 origin "$GITHUB_REF"
fi
git worktree add /tmp/base-worktree FETCH_HEAD
# Run babel_update against BASE source + BASE translations. Any drift
# already present on the base branch (source strings that have changed
# without .po updates) shows up here as fuzzies — and will also show
# up in the PR run, so it cancels out in the comparison.
- name: Baseline — run babel_update against BASE source
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
working-directory: /tmp/base-worktree
run: ./scripts/translations/babel_update.sh
- name: Record baseline translation counts
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
run: |
python scripts/translations/check_translation_regression.py \
--count \
--translations-dir /tmp/base-worktree/superset/translations \
> /tmp/before.json
# Run babel_update against the PR source and PR translations. This keeps
# committed .po fixes in play while the base babel_update above still
# cancels out translation drift already present on the base branch.
- name: Run babel_update against PR source
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
run: ./scripts/translations/babel_update.sh
- name: Check for translation regression
id: regression
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
continue-on-error: true
run: |
python scripts/translations/check_translation_regression.py \
--compare /tmp/before.json \
--report /tmp/regression-report.md
# Save the PR number so the comment workflow can post the report without
# needing write permissions on this pull_request-triggered job.
- name: Save PR number for comment workflow
if: >-
github.event_name == 'pull_request' &&
steps.regression.outcome == 'failure'
run: echo "${{ github.event.pull_request.number }}" > /tmp/pr-number.txt
- name: Upload regression artifact
if: >-
github.event_name == 'pull_request' &&
steps.regression.outcome == 'failure'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
python-version: ${{ matrix.python-version }}
- name: Install dependencies
uses: ./.github/actions/cached-dependencies
with:
name: translation-regression
path: |
/tmp/regression-report.md
/tmp/pr-number.txt
- name: Fail if regression detected
if: steps.regression.outcome == 'failure'
run: exit 1
actions-timeline:
needs: [frontend-check-translations, babel-extract]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
with:
expand-composite-actions: true
run: |
apt-get-install
pip-upgrade
pip install -r requirements/base.txt
- name: Test babel extraction
run: ./scripts/babel_update.sh
+7 -26
View File
@@ -1,49 +1,30 @@
name: WebSocket server
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
paths:
- "superset-websocket/**"
pull_request:
paths:
- "superset-websocket/**"
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
app-checks:
runs-on: ubuntu-slim
timeout-minutes: 20
if: github.event.pull_request.draft == false
runs-on: ubuntu-20.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
uses: actions/checkout@v3
with:
persist-credentials: false
- name: Setup Node.js
uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0
with:
node-version-file: './superset-websocket/.nvmrc'
- name: Install dependencies
working-directory: ./superset-websocket
run: npm ci
- name: eslint
- name: lint
working-directory: ./superset-websocket
run: npm run eslint -- . --quiet
- name: typescript checks
run: npm run lint
- name: prettier
working-directory: ./superset-websocket
run: npm run type
- name: code format check
working-directory: ./superset-websocket
run: npm run format-check
run: npm run prettier-check
- name: unit tests
working-directory: ./superset-websocket
run: npm run test

Some files were not shown because too many files have changed in this diff Show More