Compare commits

..
Author SHA1 Message Date
Mike BridgeandClaude Opus 4.7 1c8d414494 debug(activity-view): drop @has_access on debug shell route
The previous @has_access decorator on ActivityDebugView.show was
redirecting authenticated-but-not-authorized requests to the home
page (FAB's access-denied default), which is what the user was seeing
instead of the React app. Then @login_required was tried briefly but
crashed because flask-login's redirect endpoint isn't wired up under
this app's auth backend.

Simplest fix: no decorator on the shell route. The shell page exposes
no data of its own — the React component fires API calls to
/api/v1/{resource}/{uuid}/activity/ which gate access via
raise_for_ownership on the path entity. Anonymous users would see
the React UI with 401 errors inline, which is correct UX for a debug
tool.

Verified: GET /activity-debug/dashboard/<uuid>/ → 200 + SPA shell HTML.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 466cd5ffae debug(activity-view): register Flask route to serve React shell
The React route at /activity-debug/{resource}/{uuid} was returning
a JSON 404 on fresh page-load because Superset's SPA model doesn't
use a Flask catch-all — every React-router path needs a corresponding
Flask view that calls render_app_template(). Adding the missing
piece.

* superset/views/activity_debug.py — new ActivityDebugView (uses
  BaseSupersetView). Single @expose("/<resource>/<uuid>/") returning
  the React shell; the in-app router (superset-frontend/src/views/
  routes.tsx) handles the actual page rendering. @has_access keeps
  it gated by login.
* superset/initialization/__init__.py — appbuilder.add_view_no_menu
  registration alongside the other SPA shell views.

Throwaway by design; both edits tagged with "Throwaway: sc-107283"
comments for easy removal when the activity-view feature ships.

Verified: GET /activity-debug/dashboard/<uuid>/ returns 302 to
/login when unauthenticated (correct), 200 with the SPA shell when
logged in.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 69e782a601 debug(activity-view): throwaway React debug UI for activity timelines
Adds /activity-debug/{dashboard,chart,dataset}/{uuid} for visual
verification of the sc-107283 activity-view endpoints. Lets you eyeball
the response shape without building a curl/jq habit:

* Controls for include / page / page_size / since / until
* Per-record cards showing entity_kind chip, source badge, kind tag,
  tombstone affordance, issued_at, changed_by, summary, entity_uuid,
  version_uuid, path, from/to values, impact
* Deleted entities render with strikethrough + "deleted" tag
* Soft-deleted state surfaces as an orange tag (currently always null
  until sc-103157 lands deleted_at)

**Throwaway by design**: when the activity-view feature ships, delete
the lazy import + route entry from superset-frontend/src/views/routes.tsx
and remove the superset-frontend/src/pages/ActivityDebug/ directory.
Both edits are tagged with "Throwaway: sc-107283" comments so they're
easy to find.

The component uses @superset-ui/core/components (Card, Tag, Radio, Input,
Space, Typography, Loading, Empty) per CLAUDE.md conventions. TypeScript
clean, ruff/eslint/oxlint clean, no any types. Uses a native HTML
<select> for page_size — the @superset-ui Select wrapper's stricter prop
types weren't worth fighting for a debug tool.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 ec851849fb refactor(activity-view): JSON entity_kind uses user-facing labels
Breaking change to the (still-unmerged) activity-view JSON contract:
``entity_kind`` now emits ``"dashboard"`` / ``"chart"`` / ``"dataset"``
instead of the Python class names ``"Dashboard"`` / ``"Slice"`` /
``"SqlaTable"``. The class names are developer-facing artifacts that
leaked the model layer (e.g. ``"Slice"`` predates the UI rename to
"chart"; ``"SqlaTable"`` predates "dataset"). User-facing JSON should
speak user language.

Implementation: a new ``_USER_FACING_KIND`` map translates at JSON
serialization time only (in ``_decorate_records``). Internal code keeps
the Python class-name form (``model_cls.__name__``) for dispatch — the
existing ``_NAME_COLUMN``, ``_NOT_FOUND_EXC``, ``_API_KIND_LABEL``,
``_can_read``, ``_compute_impact``, etc. all key off class names and
are unchanged. The translation happens at the single ``record["entity_kind"]
= ...`` assignment.

Schema validator ``ACTIVITY_ENTITY_KINDS`` updated to the new tuple.
Integration tests' response-shape assertions renamed via bulk sed; unit
tests testing internal helpers are unchanged (they operate on internal
api_kind / class names).

UPDATING.md example payload updated. Spec updates (spec.md, data-model.md,
contracts/activity-view.yaml) committed separately to the spec repo.

Full suite: 66 unit + 35 integration + 1 xfailed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 36203af96d feat(activity-view): T037/T038 observability + T050 cross-coupling check
T037 — Instrument the five phases of get_activity with stats_logger
timing metrics under the `superset.activity_view.<kind>.<phase>_ms`
key convention (plan §D-17). Wrap each phase with a stats_timing
context manager:

* superset.activity_view.<kind>.relationship_resolution_ms (_resolve_scope)
* superset.activity_view.<kind>.fetch_ms (_fetch_change_records)
* superset.activity_view.<kind>.visibility_filter_ms
* superset.activity_view.<kind>.denormalize_ms
* superset.activity_view.<kind>.decorate_ms

`<kind>` is the lowercased path-entity model class name (dashboard /
slice / sqlatable), enabling per-endpoint-family Grafana panels.

T038 — Add request- and response-shape attributes via the existing
counter/gauge interface:

* superset.activity_view.<kind>.requests.include_<value>  (incr)
* superset.activity_view.<kind>.requests.has_since_filter_<true|false>  (incr)
* superset.activity_view.<kind>.page_size  (gauge)
* superset.activity_view.<kind>.record_count  (gauge — post-visibility-filter)
* superset.activity_view.<kind>.related_entity_count.charts  (gauge)
* superset.activity_view.<kind>.related_entity_count.datasets  (gauge)

Confirmed no PII: entity names, diff content, user identifiers — none
flow into the metric layer. Only counts and shape tags.

T050 — Cross-coupling sanity test (unit-scope): asserts
_METRIC_PREFIX == "superset.activity_view" so a code review catches
accidental drift from sc-103156's eventual "superset.versioning.*"
sibling namespace. Both endpoint families belong to one Grafana panel
under "superset.versioning.* OR superset.activity_view.*".

Full suite: 66 unit + 35 integration + 1 xfailed (T044 sc-103156
restore-kind dependency, unchanged).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 3a1667522a test(activity-view): four more polish tasks — T039/T042/T044/T051
* T042 test_activity_marks_hard_deleted_chart_with_tombstone (D-15)
  Edit a chart, hard-delete it via db.session.delete(chart) + commit,
  hit dashboard activity. Asserts tombstoned record has
  entity_deleted=True, entity_uuid=None, entity_name preserved from
  the last shadow row. The fixture's _cleanup tolerates missing slices
  (uses Slice.id.in_(slice_ids) which silently skips).

* T044 test_activity_surfaces_dashboard_restore_event (AV-015) — xfail
  AV-015 requires sc-103156's restore code to emit a synthetic
  kind="restore" change record with to_value carrying the source
  version_uuid + label. sc-103156's restore_version() currently does
  not — the diff capture surfaces the field changes as kind="field".
  The activity layer correctly passes through whatever kind sc-103156
  emits; the test will start passing once the upstream emission lands.
  Marked strict=True so the day sc-103156 emits, this fails-to-fail
  and we know to revisit.

* T051 test_activity_excludes_records_after_retention_prune (AV-010)
  Edit a chart (creates a new version_transaction), backdate that
  transaction's issued_at past the 30-day retention cutoff, run
  _prune_old_versions_impl(retention_days=30). Assert the prune
  removed ≥1 transaction AND the activity endpoint's filtered count
  decreased.

* T039 test_activity_pagination_is_deterministic_and_disjoint
  (SC-AV-002 pragmatic interpretation). The spec's stricter "no
  skip/duplicate under concurrent writes" is unprovable with offset
  pagination — new top-inserted records shift every later page by
  one. Cursor pagination would solve this (deferred per plan §D-10).
  Under offset, the testable guarantees are: (a) same request fired
  twice produces identical pages; (b) page N and page N+1 are
  disjoint under one request round. Both come from the
  (issued_at DESC, transaction_id DESC, sequence DESC) sort.

Full activity-view suite: 35 passed, 1 xfailed in 46s. The xfail is
T044 with the documented sc-103156 dependency.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 fa73b00da6 test(activity-view): T040/T041 polish — ordering + page-size cap
Three of the Phase 6 polish tasks landed together:

* T040 test_activity_ordering_is_stable_by_issued_at_then_transaction_id
  — AV-006 deterministic-ordering contract. Iterates adjacent pairs in
  the response and asserts (issued_at, transaction_id) is monotonically
  non-increasing. Catches any sort-stability regression — random
  ordering would fail the pairwise check almost certainly.
* T041 test_activity_page_size_caps_returned_records_at_200 — pairs
  with the existing pagination_clamps test. The former confirms
  ?page_size=500 doesn't 400; this one confirms the response is
  bounded to ≤200 records as the OpenAPI contract guarantees.
* T053 verification: every fixture-mutating test in
  activity_view_tests.py was already following the
  try/finally + rollback convention (sweep verified 30 tests; zero
  non-conformant). No remediation needed; documenting the
  sweep result in the commit message.

Full activity-view suite: 32/32 integration + 65/65 unit tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 3a9f863194 refactor(activity-view): SQL review tidies — mappings + perf-seed docstring
Two structural changes from the second SQLAlchemy review. No behaviour
change; full activity-view suite (70 unit + 30 integration) still
green.

* Tidy 1 (Warning #2): switch _batch_chart_counts and
  _batch_datasets_used_by_charts from positional row-unpack to
  .mappings()-keyed access. Column-rename-safe — a future edit that
  reorders columns in the SELECT can't silently misassign values
  during the Python loop. Matches the existing convention in
  _select_change_rows_for_kinds.

* Tidy 2 (Suggestion #1): docstring on _seed_activity_history
  explaining why it intentionally commits without rollback (unlike
  the T053 convention in activity_view_tests.py). The seed IS the
  setup, not the unit under test — the endpoint reads a realistic
  post-commit state of the shadow tables.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 440e298c4f docs(activity-view): UPDATING.md note + T049 OpenAPI verification
Two doc-shaped tasks landed together because they're both about making
the activity-view endpoints discoverable by external consumers.

T047 — UPDATING.md: new section under the existing entity-version-history
entry documenting the three activity-view endpoints (dashboard / chart /
dataset), their query params (since / until / include / page / page_size),
the response shape with all DTO fields, the silent permission filter
(AV-008), tombstone behaviour (D-15), and the no-feature-flag / no-
new-tables impact statement. Mirrors the depth of the sc-103156
versioning section above it.

T048 satisfied by the UPDATING.md entry: the activity-view feature
adds no new config keys (no SUPERSET_* env vars, no feature flag), and
the per-endpoint API reference is auto-generated from the YAML
docstrings via FAB's OpenAPI integration. The `/swagger/v1` page picks
up the activity endpoints automatically — verified by the new tests
below. sc-103156 followed the same pattern (UPDATING.md only, no
standalone config doc).

T049 — Three new tests in TestActivityOpenApiSpec verify FAB's OpenAPI
generation includes the activity endpoints with the right shape:

* test_three_activity_paths_appear_in_openapi — the three
  /<uuid_str>/activity/ paths are surfaced in /api/v1/_openapi.
* test_activity_endpoints_document_query_params — since / until /
  include / page / page_size are all declared, and include's enum is
  exactly {"self", "related", "all"}.
* test_activity_endpoints_declare_200_response — 200 + 400/401/403/404
  are all declared response codes.

base_api_tests.py::TestOpenApiSpec::test_open_api_spec already
validates the full spec's structural correctness on every CI run, so
malformed YAML in the activity-view docstrings would have been caught
upstream. The new tests add activity-specific assertions about the
endpoints' presence and parameter shape.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 a2d5bd05d0 perf(activity-view): batch _datasets_used_by_chart for dashboard scope
Surfaced by T045's perf test (which failed at p95 4228ms vs 1500ms
budget) and an ad-hoc per-phase profile that pinpointed _resolve_scope
as accounting for ~1926ms of the wall clock.

The previous _resolve_dashboard_scope called _datasets_used_by_chart
once per slice on the dashboard. On a fixture with ~12 charts that's
12 sequential SQL round-trips for the same shape of query, each
returning a small result set. The new _batch_datasets_used_by_charts
takes the full set of slice_ids and returns
{slice_id: [(dataset_id, window), ...]} in one query; the singular
form _datasets_used_by_chart now forwards to the batch (used only by
_resolve_chart_scope which has a single slice).

Impact (measured on the same fixture):
* Wall-clock per request: 3955ms → 2257ms (43% faster)
* Total query count per request: 18,376 → 29 (driven by the
  identity-map populating during the warmup; the genuine count for
  the activity layer alone is ~13 SQL queries)
* T045 p95 over 50 runs: 4228ms → 2429ms

The remaining gap to the 1500ms budget is dominated by Python work
inside _fetch_change_records: post-filter and sort over ~26K
accumulated test-environment records. The spec's target scale is
"25 charts × 3 dataset windows" — the SQLite test environment has
accumulated far more state from prior test runs. T046 (index
migration) is deferred because the remaining bottleneck is Python-
side, not SQL-side; production data on Postgres with retention
pruning will not have this shape. Document and revisit when a
production-representative perf measurement is available.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 52a3ab0d71 test(activity-view): add T045 perf-validation tests
Two opt-in tests (SUPERSET_PERF_VALIDATION=1 to run, skipped otherwise)
under the existing PerfValidationTests class:

* test_av_sc001_activity_endpoint_p95_under_1500ms — seeds dense
  history on the birth_names dashboard (~12 chart edits, 5 dataset
  edits, 1 dashboard edit), then hits /api/v1/dashboard/<uuid>/activity/
  50 times after warmup and asserts p95 < 1500ms per SC-AV-001.

* test_av_sc003_save_path_p95_unaffected_by_activity_view — under the
  same fixture, runs 50 dashboard saves and re-measures the SC-004
  version-capture overhead (50ms p95 budget). Confirms the activity-
  view branch hasn't added a new save-path coupling that regresses
  sc-103156's save-path budget. PASSES on the current branch — read-
  only feature, no new save-path code paths.

Seed fixture differs from the spec's "25 charts × 3 dataset windows
each" target — the birth_names fixture has ~12 charts on a single
dataset, so we approximate by editing many charts plus the dataset
multiple times. A bespoke multi-dataset fixture would let us measure
against the spec's exact scale; that's future work.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 4ead3f405f perf(activity-view): batch impact-count query to remove N+1
Surfaced by the SQLAlchemy review (Warning #1) and required before
T045's p95 perf budget can be met on dashboards with many historical
dataset edits.

Before: _decorate_records fired one COUNT(DISTINCT slice_id) query
per related dataset record via _compute_impact → _count_dashboard_
charts_pointing_at_dataset_at_tx. For a dashboard activity stream
with N dataset-edit records, that's N round-trips to compute impacts.

After: _decorate_records collects the distinct (dataset_id, target_tx)
pairs once via _collect_impact_pairs, fires a single batched query
via _batch_chart_counts that pulls the (slice, dataset, validity-
window) state for every relevant slice, and filters by validity in
Python per pair. The result is a {(dataset_id, target_tx): count}
mapping consumed by the new pure helper _impact_for_record per row.

Round-trip count drops from O(N records) to O(1) for the impact
calculation. The SQL stays small and dialect-portable — same per-kind
IN-clause + Python validity filter pattern as _fetch_change_records.

Trade-off documented in _batch_chart_counts' docstring: the SELECT
pulls (slice_id, datasource_id, two validity-window pairs) for every
slice ever on the dashboard whose dataset matches one of the
requested dataset_ids. For a busy dashboard with 100 slices and 50
versions each, that's ~5000 rows into Python vs N small COUNT scans.
For N > ~5 (which is typical) the batch wins.

Removed:
* _compute_impact (replaced by _impact_for_record + _collect_impact_pairs)
* _count_dashboard_charts_pointing_at_dataset_at_tx (replaced by
  _batch_chart_counts)

Test changes: the three _compute_impact unit tests (no-impact paths)
become six _impact_for_record tests (positive count + four no-impact
paths + zero-count → None). Five new _collect_impact_pairs tests
cover dashboard/chart/dataset path branching plus dedupe and empty.

Full suite: 27/27 integration + 65/65 unit (was 57; +8 from the
restructure). No semantic regression on either side of the cut.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 b96cb01670 docs(activity-view): document per-kind query rationale in _select_change_rows_for_kinds
No-op docstring change. The function fires one SELECT per entity_kind
with entity_id IN (...) instead of one query with tuple_(entity_kind,
entity_id).in_(...). The latter is supported by Postgres but its SQL
emission across MySQL and SQLite is uneven (some dialects emit literal
ROW expressions, some emit a transformed multi-column form, some don't
support it at all in older versions). Three round-trips per request is
the correct trade-off given Superset's multi-dialect requirement.

Documenting now so a future reader doesn't reach for the tuple-IN
"optimisation" and silently break MySQL or SQLite. Surfaced by the
SQLAlchemy review.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 9d8a318f34 feat(activity-view): Phase 5 US3 — dataset /activity/ endpoint (T033-T036)
Implements the dataset activity stream. Per AV-004 datasets have no
transitive layer in V2: dataset activity = dataset's own edits only,
regardless of whether charts use the dataset. The chart/dashboard
endpoints surface dataset edits in their *related* streams (already
shipped); the dataset endpoint stays narrowly self-only.

* T033 DatasetRestApi.activity — third application of the shared
  resolve_endpoint_path_entity + parse_activity_query_params pattern.
  Endpoint body is ~10 lines of real logic, same shape as T016 and
  T028 but with SqlaTable as the path entity. The activity orchestrator
  already short-circuits the related-entity resolution for datasets
  (_resolve_scope returns [] for related when path_kind="SqlaTable"),
  so the dataset endpoint inherits the V2 semantics for free.

* T034 TestDatasetActivityView class added to activity_view_tests.py.

* T035 test_dataset_activity_excludes_chart_edits — AS-1 of US3 /
  AV-004 verbatim. Edit a chart that uses the dataset; assert the
  chart edit does NOT appear in the dataset's activity stream
  (datasets are read-only upstream of charts in V2).

* T036 test_dataset_activity_includes_dataset_self_edits — confirms
  the positive path: editing the dataset's own description surfaces
  a SqlaTable/self record.

* New test_dataset_activity_related_only_returns_empty — AV-004 in
  contract form: ?include=related on a dataset returns an empty
  result list and count=0 because there are no related entities to
  draw from.

Plus the standard boundary tests (404 for unknown UUID, 400 for
malformed UUID / invalid include, 403 for non-owner, 200 envelope
shape).

Full activity-view suite: 27/27 integration tests + 57/57 unit tests
green. All three endpoint families live.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 6b2d75ca0b feat(activity-view): Phase 4 US2 — chart /activity/ endpoint (T028-T032)
Implements the chart cross-entity activity stream (US2). Builds on the
resolve_endpoint_path_entity helper from the prior tidy commit
(1d60513079), so the new endpoint is a small composition of: resolve
path entity → parse params → get_activity(Slice, ...) → serialize.

* T028 ChartRestApi.activity — @expose("/<uuid_str>/activity/") on the
  chart blueprint. Decorator stack mirrors list_versions; registered
  in include_route_methods. The MODEL_API_RW_METHOD_PERMISSION_MAP
  entry "activity": "write" added in Phase 3 already covers this
  endpoint family (the map is class-permission-agnostic).

* T029 TestChartActivityView class added to activity_view_tests.py,
  following the same patterns as TestDashboardActivityView.

* T030 test_chart_activity_includes_dataset_edit_as_related — Given
  a chart pointing at the birth_names dataset, When the dataset's
  description is edited, Then the chart activity stream surfaces a
  SqlaTable/related record (AS-1 of US2).

* T032 test_chart_activity_excludes_sibling_dashboards — Given the
  chart is on a dashboard, When the dashboard's title is edited,
  Then NO Dashboard records appear in the chart's activity. Per the
  spec's Relationship Traversal section: charts don't see sideways
  to dashboards they happen to be on.

* T031 (datasource-switch attribution) deferred — needs a second
  dataset fixture which the birth_names environment doesn't provide
  cleanly. Will land with a multi-dataset test harness in a follow-up.

Additional coverage: 404 for unknown UUID, 400 for malformed UUID,
400 for invalid include, 403 for non-owner, 200 envelope smoke,
chart-self-edit appears as source=self, include=self filter test.

Full suite: 19/19 integration tests + 57/57 unit tests green.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 a412f1d9e6 refactor(activity-view): extract resolve_endpoint_path_entity helper
Make the change easy, then make the easy change. Phase 4 (T028 chart
/activity/ endpoint, T033 dataset endpoint) duplicates the UUID-parse
+ find_active_by_uuid + raise_for_ownership dance from the existing
DashboardRestApi.activity. Extracting now keeps the next two endpoints
single-line affairs.

* New PathEntityResponseError carries a pre-built error Response.
  Subclassing Exception (not ValueError) so the endpoint's catch is
  precise and the standard hierarchy stays unpolluted.
* New resolve_endpoint_path_entity(api, model_cls, uuid_str) runs the
  three-step preflight and raises PathEntityResponseError with the
  right 400/404/403 response carried inside.
* DashboardRestApi.activity refactored to use the helper. 12 lines of
  imports + try/except boilerplate collapse to a single try/except
  block that maps every preflight failure to its response.

No behaviour change: 10/10 dashboard activity integration tests still
green. Pure structural change, per Beck's tidy-first discipline (the
T028 endpoint code follows in the next commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:25 -06:00
Mike BridgeandClaude Opus 4.7 f840ce9813 refactor(activity-view): Phase 3 tidies from clean-code review
Five structural changes — no behaviour change — applied as one commit
per the tidy-first discipline. All from the clean-code review of
e4070a4716.

* Tidy 1 — Fix _fetch_change_records sort key. Was calling .timestamp()
  on issued_at with an `else 0` fallback; the fallback was dead defense
  (column is non-null per sc-103156 schema) and .timestamp() introduces
  non-determinism on tz-naive datetimes. Sort on the datetimes directly.

* Tidy 2 — parse_activity_query_params now raises ActivityParamsError
  (subclass of ValueError) instead of returning (Optional[dict],
  Optional[str]). The tuple was forcing every caller into a defensive
  `if error or params is None: return self.response_400(message=error
  or "Invalid query parameters")`. The new shape — `try: params =
  parse(...); except ActivityParamsError as exc: return
  response_400(str(exc))` — is shorter, type-safe, and the contract
  is enforced at the boundary.

* Tidy 3 — Test helper now uses Flask client's query_string= parameter
  instead of f-string concatenation. Handles URL-encoding correctly
  for the day a test passes a value containing & / = / + / etc.

* Tidy 4 — get_activity pipeline collapses to a single rolling
  `records` variable instead of the mid-stream `raw / visible_raw /
  enriched / visible` naming. Each function call's name documents
  what the step does; no intermediate variable names needed.

* Tidy 5 — Extracted four per-parameter parsers: _parse_optional_iso,
  _parse_include, _parse_page, _parse_page_size. The "parse one
  parameter" concept now has a name. Cost: four small helpers (each
  10-15 lines, one job). Benefit: parse_activity_query_params is a
  10-line table-driven dispatcher.

Test changes: parser unit tests now use pytest.raises(ActivityParamsError)
instead of unpacking the (params, error) tuple. Added one test
confirming ActivityParamsError subclasses ValueError so the standard
library exception hierarchy still catches it. Total unit tests: 57.
Integration tests still 10/10 green.

Deferred per the review: the UUID-parse + entity-find + ownership-
check dance is duplicated between activity / list_versions / get_version
and will grow to T028 / T033. Refactor when T028 lands — three real
callers > one prospective one.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 4360ad7101 feat(activity-view): Phase 3 US1 — dashboard /activity/ endpoint
Implements the dashboard cross-entity activity stream (US1 MVP).
Closes T015-T018, T023-T024, T026 from sc-107283 tasks.md. T019-T022
(complex fixture choreography) and T025 (RBAC fixture for restricted
chart access) deferred to a follow-up; T025's logical coverage is
provided by the new unit tests for _can_read kind dispatch.

* T015 Rename _dashboard_related_scope → _resolve_dashboard_scope and
  _chart_related_scope → _resolve_chart_scope, parallel to _resolve_scope
  / _resolve_path_entity. Aligns the codebase with the task spec names.
* T016 New activity(uuid_str) method on DashboardRestApi:
  @expose("/<uuid_str>/activity/"), @protect + @safe + @statsd_metrics
  + @event_logger. Same row-level ownership check pattern as
  /versions/. Registered in include_route_methods and mapped to "write"
  in MODEL_API_RW_METHOD_PERMISSION_MAP so the can_write Dashboard
  permission gates access (Alpha-non-owner gets 403 from
  raise_for_ownership, not 404 from missing permission).
* New parse_activity_query_params helper in activity.py — shared parser
  for since/until/include/page/page_size used by all three endpoint
  families. Tolerates the Z suffix Python <3.11 fromisoformat rejects.
  Silently clamps page_size to the contract max instead of rejecting.

Two correctness/scale bugs found and fixed under integration-test load:

* SQLite SQLITE_MAX_EXPR_DEPTH (1000) was tripping on dashboards with
  many slices × many historical attachment windows. _fetch_change_records
  used to emit one OR-clause per (entity_kind, entity_id, window) tuple;
  now it issues one SELECT per kind with entity_id IN (...) and filters
  by exact windows in Python via _row_within_any_window. SQL shape is
  proportional to the number of kinds (≤3); the per-entity window
  precision is preserved.

* _merge_entity_windows now unions overlapping/touching windows within
  each entity via _union_windows. Sequential fixture loads create many
  redundant Continuum shadow rows; without merging, the unfiltered
  windows still produce many OR branches downstream.

* Pipeline-ordering fix in get_activity: visibility filter runs BEFORE
  decoration. Decoration strips entity_id (not in API contract) and the
  filter needs it — and dropping invisible records early also avoids
  paying for name lookup + tombstone probes on records the requester
  can't see (AV-008's silent-filter contract).

Tests:

* tests/integration_tests/versioning/activity_view_tests.py — 10 tests
  for TestDashboardActivityView covering: 404 for unknown UUID, 400 for
  malformed UUID / invalid include / invalid since, 403 for non-owner,
  200 envelope smoke, chart-edit-appears-as-related, include=self
  filter, include=related filter, page_size clamping.
* tests/unit_tests/versioning/test_activity.py — grew from 30 to 56
  tests. New coverage: parse_activity_query_params (7 cases), _can_read
  per-kind dispatch (4 cases — covers T025 at unit scope), _union_windows
  (9 parametrized cases), _merge_entity_windows window-union case,
  _row_within_any_window (6 cases).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 9f9dee0bec refactor(activity-view): Phase 2 tidies + unit tests for pure helpers
Five structural changes — no behaviour change — applied as one commit
per the tidy-first discipline (separate from the prior feat commits
that landed the primitives and orchestrator):

* Add tests/unit_tests/versioning/test_activity.py — 30 tests for the
  pure-function helpers: _intersect_windows (11 boundary cases),
  _resolve_scope branching (per kind + per include mode), entity-
  window merging, AV-012 summary headlines, changed_by projection,
  kind-translation round-trip, _can_read fall-through, _compute_impact
  no-impact paths. Runs in <500ms, no DB, no Flask. The DB-touching
  helpers wait for the integration suite (Phase 3+).

* Hoist _datasets_used_by_chart(slice_id) out of the inner attachment-
  window loop in _dashboard_related_scope. Previously a chart with N
  attachment windows fired N queries for identical data; now once per
  slice. Removes a perf cliff at the historical-rich case the spec
  explicitly calls out (US1 AS-3).

* Delete the unused requesting_user parameter from
  _filter_records_by_visibility and from get_activity. It was never
  threaded through to the security manager calls (which read the user
  from Flask-Login implicitly) — speculative future-shape that the
  reader had to mentally trace through. If CLI/Celery bypass becomes
  necessary, add it then with a real call site.

* Delete the unused module-level logger. No observability lands here
  until T037/T038; reintroduce when those tasks add real logger calls.

* Fix the file docstring — it claimed _resolve_version_tables was
  reused; it isn't. Trimmed to (find_active_by_uuid,
  derive_version_uuid).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 fa54708276 feat(activity-view): Phase 2 orchestrator + get_activity (T011-T013)
Completes Phase 2 of sc-107283. The activity-view query layer now has
one public entry point — get_activity(model_cls, entity_uuid, ...) —
plus the supporting visibility filter and DTO decorator.

* T011 _filter_records_by_visibility — silent permission filter per
  AV-008. Batch-resolves live entities per kind, dispatches
  can_access_dashboard / can_access_chart / can_access_datasource.
  Tombstoned entities pass through (the decorator handles deleted
  messaging; no navigable entity_uuid is exposed). _can_read helper
  isolates the per-kind predicate dispatch.

* T012 _decorate_records — synthesizes the full ActivityRecord shape:
  entity_kind (API form), entity_uuid (live lookup; null for
  tombstones), entity_deleted, entity_deletion_state, source ("self"
  vs "related"), summary (AV-012 headline), impact (via T010),
  version_uuid (derive_version_uuid), changed_by (User projection).
  Strips internal-only columns (entity_id, sequence, raw user cols)
  from the final shape.

* T013 get_activity — top-level orchestrator. Branches by include
  ("self"/"related"/"all") via _resolve_scope; for related,
  _dashboard_related_scope walks charts-on-dashboard then datasets-
  used-by-chart and clips chart-on-dataset windows by attachment
  windows. _chart_related_scope skips the dashboard hop. Datasets
  return empty related scope per AV-004. _merge_entity_windows
  collapses repeated (kind, id) entries to keep the OR-clause in
  _fetch_change_records compact. Pagination is post-visibility-filter
  per AV-008; defaults page_size=25, clamps to 200.

Tasks T004-T014 in tasks.md updated to [X]. The next phase opens user
story implementation (T015 dashboard scope helper + T016 endpoint).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 5750cd4fef feat(activity-view): Phase 2 primitives (T004-T010, T014)
Adds the foundational query layer for the cross-entity activity view
in superset/versioning/activity.py. Pure helpers + single-purpose
shadow-table queries; no public entry point yet (T011-T013 follow).

* T004 _resolve_path_entity — UUID lookup via find_active_by_uuid;
  typed 404 dispatch by model class (DashboardNotFoundError /
  ChartNotFoundError / DatasetNotFoundError).
* T005 _charts_attached_to_dashboard — (slice_id, window) tuples from
  dashboard_slices_version, op=2 rows excluded.
* T006 _datasets_used_by_chart — (datasource_id, window) tuples from
  slices_version, filtered to datasource_type='table'.
* T007 _intersect_windows — pure half-open window intersection;
  end=None acts as positive infinity. Unit-verified.
* T008 _fetch_change_records — workhorse query joining version_changes
  + version_transaction + ab_user, OR-clause over (kind, id, [windows]),
  ordered by (issued_at DESC, transaction_id DESC, sequence DESC) per
  AV-006. No DB-level pagination here — AV-008's silent permission
  filter means count is computed after the visibility filter, so
  pagination happens in T013.
* T009 _denormalize_entity_names — batch shadow lookup keyed by
  (api_kind, entity_id, transaction_id); validity-strategy predicate.
  Helper _resolve_names_for_kind extracted to keep cyclomatic
  complexity under ruff's C901 threshold.
* T010 _compute_impact — dashboard+dataset path yields
  {"charts": N}; all other path/related shapes yield None per
  data-model.md §"impact computation".
* T014 _check_entity_tombstones — live-row existence + soft-delete
  state probe; tolerates the pre-sc-103157 absence of a deleted_at
  column via hasattr-style introspection on Table.c.

Kind translation isolated at the module boundary: version_changes
stores lowercase ("chart"/"dashboard"/"dataset"); the ActivityRecord
DTO returns class names ("Slice"/"Dashboard"/"SqlaTable"). The
_TABLE_KIND_TO_API / _API_KIND_TO_TABLE mappings translate at the
two crossings.

Phase 2 orchestration layer (T011-T013) lands in the next commit.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 47b804a7dd refactor(activity-view): enforce schema contracts (review polish)
Apply clean-code review polish on top of the Phase 1 scaffold before
Phase 2 builds on this surface:

* Replace magic-string field descriptions with ``validate.OneOf`` on
  ``entity_kind``, ``source``, ``entity_deletion_state``, and ``kind``.
  Canonical lists hoisted to module-level tuples (``ACTIVITY_*``).
* Type ``path`` as ``fields.List(fields.String())`` and ``impact`` as
  a new ``ActivityImpactSchema`` — replaces two ``fields.Raw`` ``Any``s
  whose shape was previously documented only in prose.
* Resolve the ``ActivityChangedBySchema`` docstring contradiction —
  the schema deliberately omits ``username`` so it is no longer
  honest to call it "identical shape to VersionChangedBySchema".
* Fix the ``activity.py`` module docstring: one polymorphic
  ``get_activity`` function, not three entry points.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike BridgeandClaude Opus 4.7 8f1ec17150 feat(activity-view): scaffold Phase 1 setup (T001/T002/T003)
* T001 — new ``superset/versioning/activity.py`` module with ASF
  header and docstring describing the cross-entity activity-view query
  layer (companion to ``queries.py``). Empty otherwise; Phase 2 fills
  it with the shared query helpers.
* T002 — empty test file
  ``tests/integration_tests/versioning/activity_view_tests.py`` with
  ASF header, module docstring naming the three test classes that will
  follow, and a ``SupersetTestCase`` import. Phase 3+ fills it with
  test classes.
* T003 — three Marshmallow schemas added to
  ``superset/versioning/schemas.py``:
  * ``ActivityChangedBySchema`` — User subset on each record.
  * ``ActivityRecordSchema`` — one change record (per data-model.md).
  * ``ActivityResponseSchema`` — envelope ``{result, count}``.
  Fields mirror data-model.md §"``ActivityRecord`` DTO" line-for-line;
  metadata descriptions are FAB/Swagger-ready.

No new endpoints, no listeners, no DB writes — pure scaffolding. The
three schemas are unused until Phase 3 (T016) wires the first endpoint.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:24 -06:00
Mike Bridge d333510e83 refactor(versioning): consolidate three iterative migrations into one
Squashes the three migrations that captured the versioning schema as
the feature was developed:

* ``56cd24c07170 add_entity_version_history_tables`` — version_transaction
  + dashboards_version / slices_version / tables_version parent shadows.
* ``e1f3c5a7b9d0 add_version_changes_table`` — the field-level diff log.
* ``f7a2b3c4d5e6 spike_add_child_continuum_shadow_tables`` —
  table_columns_version / sql_metrics_version / dashboard_slices_version
  child shadows.

Combined into a single migration ``2026-05-28_19-50_56cd24c07170_
add_versioning_tables.py`` that creates the same 8 tables in dependency
order (sequence + version_transaction first, parent shadows, version_
changes, child shadows last) and drops them in reverse on downgrade.

Why:
* All three migrations represent one logical feature ("enable
  versioning") — applying any subset leaves a broken intermediate
  state.
* The third migration was literally named "spike_..."; the iterative
  shape was a development artifact, not a design intent.
* Downstream operators get one migration to apply / reverse and one
  file to review.
* The dev DB was rolled back through the three downgrades, the new
  migration was applied via ``superset db upgrade``, and column-count
  spot checks confirmed byte-identical schema (4/9/21/26/35/19/15/5).

The ``revision`` hash is reused from the original first migration
(``56cd24c07170``) for chain continuity. ``down_revision`` stays
``2bee73611e32`` (sc-105349 composite-PK tip), unchanged from the
original chain's head.

Net: -70 lines (605 deleted, 535 added), same schema, one Alembic step
instead of three.
2026-05-28 15:37:20 -06:00
Mike Bridge b16ac7c21e refactor(versioning): leaf-level diff for JSON-blob fields (Shape B)
The diff engine now recurses into nested dicts inside JSON-blob fields
(``json_metadata``, ``Slice.params`` non-list sub-keys, layout node
``meta``) via a shared ``_recursive_leaf_diff`` helper, emitting one
``ChangeRecord`` per atomic leaf change rather than one record carrying
the whole top-level sub-tree on both sides.

Concretely, a dashboard header text change from "VERSION 2!" to
"HEADER!" used to emit a single record at path ``["edit", "header",
"HEADER-id-1"]`` with the entire layout node duplicated on both sides
of the diff. It now emits a single record at path ``["edit", "header",
"HEADER-id-1", "text"]`` with just the changed string as ``from_value``
/ ``to_value``. Multi-leaf edits produce one record per leaf.

Each call site passes its own depth cap via a named constant:

* ``_LAYOUT_META_DIFF_DEPTH = 3`` — layout meta is presentation state
  (text, sizes, colors), shallow by nature.
* ``_JSON_METADATA_DIFF_DEPTH = 6`` — native filter configuration can
  go five levels deep (``defaultDataMask.filterState.value``).
* ``_SLICE_PARAMS_DIFF_DEPTH = 6`` — adhoc filter sub-queries and pivot
  options can be similarly deep.

A cap is a usefulness bound (granularity that's meaningful in a
timeline), not a safety bound. Cap-on-dict-vs-dict emits a debug log so
production tuning can see when a cap is too tight for typical data.

Lists are treated as opaque leaves: positional paths break under
reorder. Lists with stable identity (adhoc filters, metrics, dataset
columns) already have natural-key walkers (``_diff_list_by_natural_key``)
that emit per-element records with the right identity; those are
unchanged.

Backward compatibility:
* Top-level scalar fields, child-collection records (column/metric),
  M2M slice membership, and layout add/remove/move records all keep
  their existing path/from/to shape. The change is scoped to JSON-blob
  recursion.
* All 56 existing diff unit tests pass without modification. 13 new
  tests cover the Shape B behaviour (helper directly, type mismatches,
  opaque list policy, depth cap, nested ``json_metadata``, layout
  edit, ``Slice.params`` deep unknown key, cross-node aggregation).
* Existing integration tests in ``change_records_tests.py`` assert on
  scalar-field changes only — unaffected.

Restoration is unaffected: the restore path reads from Continuum
shadow tables, not from ``version_changes``. This change is purely
about audit-log granularity.

Spec coverage: tasks.md T047d (added in spec repo). Rationale: plan.md
"Key Design Decision: Leaf-level recursive diff for JSON-blob fields
(Shape B)". Conventions: data-model.md "Leaf-level recursion".
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 863c73ec53 test(versioning): lock in flag_modified-suppresses-onupdate invariant (W2)
The ``_pin_audit_columns`` helper added in 1b0083bc03 relies on a
SA-version-dependent behavior: calling ``attributes.flag_modified(parent,
"changed_by_fk")`` causes the next UPDATE statement to use the
in-memory value rather than invoking the column's ``onupdate=callable``.
This is the mechanism that prevents a stale ``g.user.id`` from being
written into the parent's ``changed_by_fk`` when the synthetic
flag-flush triggers an UPDATE during an autoflush at a time when the
test user has already been deleted from ``ab_user`` — the cascade that
broke CI in sc-103156 T062.

Four unit tests, no app context required (uses a minimal in-memory
SQLite engine + declarative_base):

* ``test_flag_modified_suppresses_onupdate_callable``: the positive
  invariant — the in-memory value lands in the DB and ``onupdate`` is
  NOT called. If SA ever changes this semantic, the test fails.
* ``test_onupdate_does_fire_without_flag_modified``: negative sanity
  check — confirms the test setup is realistic. Without
  ``flag_modified``, ``onupdate`` fires as expected.
* ``test_pin_audit_columns_skips_missing_attribute``: the helper
  tolerates parents without audit columns (e.g., non-AuditMixin
  models). No raise.
* ``test_pin_audit_columns_tolerates_invalid_request_error``: SA
  raises ``InvalidRequestError`` from ``flag_modified`` when the
  attribute is unloaded in instance state (the freshly-constructed
  ``session.new`` case). The helper must catch and skip — this is the
  production-path safety net mentioned in the docstring.

Uses ``expire_on_commit=False`` for the positive invariant test so the
column stays loaded; the production-path case where the attribute is
expired is the ``InvalidRequestError`` branch covered separately.

Closes review item W2 from the sqlalchemy-review pass on
sc-103156-versioning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 9c9cb3a5f0 test(models): lock in UUIDMixin._coerce_uuid contract (S3)
Four unit tests pin the behavior the validator added in 46f138326b:

* Valid UUID-format strings get coerced to ``uuid.UUID`` instances —
  the primary contract that makes importers, test fixtures, and ad-hoc
  ``Model(uuid=...)`` construction see a consistent type.
* Already-``UUID`` values pass through unchanged (the validator must
  not re-wrap; ``is`` identity check pins this).
* Non-UUID-shaped strings pass through unchanged. This keeps test
  mocks that use placeholder strings (e.g.
  ``tests/unit_tests/mcp_service/dashboard/test_dashboard_schemas.py::test_dashboard_schema``
  with ``"dashboard-uuid-7"``) working. Without this contract, those
  tests would break under the validator and need to migrate to
  ``uuid.uuid4()``. If the contract is ever tightened to raise, this
  test catches the regression and signals the migration cost.
* ``None`` passes through unchanged.

Closes review item S3 from the sqlalchemy-review pass on
sc-103156-versioning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 1d78dd1ad0 refactor(versioning): thread pre-resolved entity through list/get_version (W3)
``ChartRestApi.list_versions`` (and the dashboard / dataset siblings)
resolves the entity via ``VersionDAO.find_active_by_uuid`` to run the
``raise_for_ownership`` check from T056, then calls
``VersionDAO.list_versions(model_cls, entity_uuid)`` — which internally
runs the SAME ``find_active_by_uuid`` query a second time. The lookup
filters on ``uuid`` (a unique non-PK column), so it isn't served from
SQLAlchemy's identity map — every call is a real SELECT.

Add an optional ``entity=`` keyword to ``list_versions``,
``get_version``, and ``resolve_version_uuid`` so callers that have
already resolved the path entity can skip the redundant lookup. The
public signature stays backward-compatible (kwarg-only).

Threading ``entity`` through ``get_version`` also avoids a third
redundant lookup inside the ``resolve_version_uuid`` call it makes.

Net: 6 fewer SELECT queries on the ``/versions/`` and
``/versions/<uuid>/`` hot paths (1 per call × 6 endpoints). All
``find_active_by_uuid`` callsites in the API layer were already
present from T056; this commit just plumbs them through.

Closes review item W3 from the sqlalchemy-review pass on
sc-103156-versioning.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 fb78c8670d test(versioning): defensive setUp/tearDown for TestDatasetRestoreApi
Reset session state (``rollback`` + ``expire_all``) before and after each
test in this class. Defensive hygiene against the Postgres-only multi-
test cascade documented in spec task T062 — a previous test in the
full-suite ordering can leave Continuum's shadow-table session attributes
in a state where the restore command's ``@transaction`` boundary raises
mid-flush and the API returns 422 "Dataset could not be updated."

Note: this *doesn't* fix the cascade itself (the polluter is writing to
the DB, not just the session; reproducer remains 3 failures on Postgres
full-suite ordering). The root cause is queued as T062. This change just
guarantees that whatever state this class' tests inherit, they start
from a clean session view.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 4a2ab1704d test(versioning): convert version-history test cleanup to try/finally
The version-history integration tests follow the codebase's dominant
pattern: a trailing ``# Cleanup`` block at the end of each test that
restores the fixture entity's mutated scalar fields (``description``,
``slice_name``, ``dashboard_title``). The pattern is brittle — when any
intermediate assertion fails, the cleanup is never reached and the
fixture stays polluted with the test's last edit, cascading state into
later tests in the same suite run.

This bit hard in CI's full-suite Postgres ordering: a failure in
``test_restore_applies_scalar_field`` left ``description="restore-test
v2"`` on the birth_names dataset, polluting downstream tests including
the RLS virtual-dataset cascade. The same shape is latent in the chart
and dashboard files.

Convert every mutating test in the three version-history files to wrap
the mutation block in ``try:`` and the restoration in ``finally:``,
with a ``db.session.rollback()`` at the top of the finally to clear any
half-flushed state left by a 422 response. Re-query the entity by id
inside the finally so the restore writes against a live object rather
than a stale handle the failing path may have expired or detached.

Pure hygiene improvement — doesn't fix the underlying Postgres failure
chain (still triggered by ``model_tests.py``'s interaction with later
tests in the full-suite ordering), but prevents this set of tests from
amplifying any future cascade.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 3314c2cd4b fix(versioning): align uuid-as-string assertions with UUIDMixin coercion
The ``@validates("uuid")`` coercion added in 46f138326b now ensures
``UUIDMixin.uuid`` always returns a ``uuid.UUID`` instance after
assignment, regardless of input type. This is the right semantic
(matches what ``UUIDType`` would emit on a fresh DB read) but it
broke three pre-existing tests that compared the attribute against
string literals — those assertions only passed before because the
post-INSERT refresh path either didn't run or ran inconsistently for
the model under test.

Update those three assertions to compare ``UUID`` to ``UUID`` so the
test contract matches the now-consistent attribute shape:

* ``test_import_database`` and ``test_import_database_no_creds`` —
  ``database.uuid`` now matches the coerced ``UUID`` literal.
* ``test_load_parquet_table_sets_uuid_on_new_table`` — ``tbl.uuid``
  same treatment.

Also harden the validator to pass non-UUID-shaped strings through
unchanged (instead of raising), so test mocks that use placeholder
strings like ``"dashboard-uuid-7"`` keep working — the SQL bind layer
will surface a clearer error if such a value is ever written to the
DB.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 279d6a4484 fix(versioning): coerce string uuid values in UUIDMixin.uuid setter
``sqlalchemy_utils.UUIDType`` only coerces in ``process_bind_param`` /
``process_result_value`` — string-to-UUID conversion happens on SQL
write / read, not on Python attribute assignment. Importers and ad-hoc
construction (e.g., ``SqlMetric(uuid="00000000-...")`` in the
``test_import_dataset`` unit test) leave the in-memory attribute as a
``str`` until the next DB round-trip refreshes it.

For non-versioned mappers a post-INSERT attribute expiration usually
refreshes the value before the caller reads it. With SQLAlchemy-
Continuum versioning on a child mapper (``TableColumn``, ``SqlMetric``),
the expire-on-INSERT behaviour shifts enough that the assertion
``metric.uuid == uuid.UUID(...)`` fails with a string-vs-UUID inequality.
Bisection confirmed: removing ``__versioned__`` from either child
restores the implicit coercion path; adding it reproduces the failure.

Rather than chase the Continuum/SA flush interaction, add a defensive
``@validates("uuid")`` to ``UUIDMixin`` so the attribute is always a
``UUID`` at runtime regardless of where the value came from. This
aligns the in-memory shape with what ``UUIDType`` would emit on a
fresh DB read, and the coercion is the same one the SQL layer would
apply on the next bind anyway.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 2685b15d5c fix(versioning): pin audit columns when force-flagging parent dirty
``_force_parent_dirty_on_child_change`` calls ``flag_modified`` on a
versioned parent so Continuum records a parent shadow row whenever a
versioned child changes. The resulting UPDATE on ``tables`` /
``dashboards`` / ``slices`` inherits ``AuditMixin``'s
``onupdate=get_user_id`` on ``changed_by_fk`` and ``onupdate=datetime.now``
on ``changed_on`` — SQLAlchemy calls ``get_user_id()`` at flush time
and stamps whoever ``g.user`` resolves to.

When the flush is autoflush-triggered during a *previous* test's
teardown (or the fixture cleanup between two tests), ``g.user`` can
still point at a user that's already been deleted from ``ab_user``.
The parent UPDATE then fails the FK to ``ab_user``, surfacing as
``IntegrityError: FOREIGN KEY constraint failed`` /
``ForeignKeyViolation``. In CI's full-suite ordering this fires from
two distinct places — ``test_rls_filter_alters_no_role_user_birth_names_query``
teardown and ``TestDatasetRestoreApi::test_restore_applies_scalar_field``
mid-restore (the latter returning 422 "Dataset could not be updated.").

Fix: also ``flag_modified`` ``changed_by_fk`` and ``changed_on`` on
the parent. The columns now have dirty attribute history, so
SQLAlchemy uses the in-memory (previously-committed, valid) values
instead of invoking ``onupdate``. The synthetic parent UPDATE carries
the existing audit values; the FK violation goes away. The behaviour
on a real user-driven save is unchanged — those code paths go through
the normal write path with a live ``g.user`` and the audit columns
update as before.

Extracted ``_pin_audit_columns`` helper to keep the caller under
ruff's ``C901`` complexity ceiling.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:20 -06:00
Mike BridgeandClaude Opus 4.7 eba4363eeb chore(versioning): ruff-format fix on dataset restore-test target filter
CI's ruff-format would collapse the two-clause if expression onto a
single line — local ruff at v0.9.7 reformats the same way. Pure
formatting, no behavior change. Counterpart of ``6de8811873``.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 3138526816 test(versioning): skip DELETE rows when picking restore target in dataset test
Same fix as ``7caf9862be`` for the dashboard restore test. CI's
integration DB accumulates Continuum shadow rows across fixture teardown
cycles, and ``TestDatasetRestoreApi::test_restore_applies_scalar_field``
picks the first row whose ``description`` matches the original. In CI
that match can land on an ``operation_type=2`` (DELETE) row from a prior
test run; Continuum's ``revert()`` then restores the dataset to its
deleted state and the restore API returns 422 "Dataset could not be
updated."

Filter ``operation_type != 2`` so the test only ever targets INSERT or
UPDATE shadow rows.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 b7c6181e38 feat(versioning): row-level ownership on list_versions and get_version (T056)
The SIP's *Authorization layering* commitment promises that all nine
version endpoints enforce both model-level access (``@protect()``) AND
row-level access (``security_manager.raise_for_ownership(entity)``).
Restore endpoints already do this via ``BaseRestoreVersionCommand.validate``,
but ``list_versions`` and ``get_version`` (six endpoints across chart,
dashboard, and dataset) skipped the row-level check — a non-owning user
with ``can_write`` on the model could list or read the shadow rows of
any entity they could reach by UUID.

Each affected handler now looks up the entity via
``VersionDAO.find_active_by_uuid`` (which already runs inside the DAO),
returns 404 if it's gone, then calls ``raise_for_ownership`` and maps a
``SupersetSecurityException`` to 403. Admins bypass automatically via
the existing FAB mechanism. The redundant lookup hits the session
identity map on the second call, so the cost is one PK read.

Tests:

- Add ``test_list_versions_denies_non_owner`` (chart, dashboard, dataset)
  and ``test_restore_denies_non_owner`` (chart) — Alpha has ``can_write``
  on each model but isn't an owner of the admin-owned fixture, so the
  row-level branch is the only thing that can reject. The dataset row-
  level test is alongside the existing model-level Gamma test, which
  only exercises the ``@protect()`` denial path.
- Unskip the three placeholder tests on chart and dashboard that were
  blocked waiting for this gap to close (``@pytest.mark.skip(reason="...
  no built-in no-write user to exercise the 403 branch...")``).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 442c87f30f temp(versioning): update DashboardList snapshot for added `uuid` column
Pairs with ``25f1922b2d temp(versioning): demo version-history dropdowns``,
which adds ``uuid`` to the dashboard-list page's ``select_columns`` so the
per-row VersionHistoryDropdown can call ``/api/v1/dashboard/<uuid>/versions/``.
``DashboardList.test.tsx`` asserts the list-fetch URL via
``toMatchInlineSnapshot``, so the snapshot needs the new column. Revert
this entry along with the dropdown component (same lifecycle as the temp
parent commit).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 f730e968b9 test(versioning): skip DELETE rows when picking restore target in dashboard test
The integration SQLite DB accumulates Continuum shadow rows across
fixture teardown cycles. ``TestDashboardRestoreApi::test_restore_applies_scalar_field``
picks the first version row whose snapshot matches the original title,
and that match can land on an ``operation_type=2`` (DELETE) row left
over from a prior test run. Continuum's ``revert()`` then dutifully
restores the entity to its deleted state — the dashboard row is gone
after the restore returns 200, and the post-restore re-query fails
with ``NoResultFound``.

Filter ``operation_type != 2`` so the test only ever targets
INSERT or UPDATE shadow rows, which is also what a real user would
pick from the version-history dropdown.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike Bridge b449889e69 fix(versioning): narrow phantom-dirty filter to session.dirty only
Previous fix (9c2391d1b4) gated the force-parent-dirty hook on
is_modified(child) for ALL session collections (dirty/new/deleted).
That was over-restrictive: is_modified checks attribute history,
and deletion is a state transition with no attribute history —
so deleted children evaluated as not-modified and the parent
wasn't flagged. The change-records listener then didn't see the
deletion and no removal record was emitted.

Symptom: test_restore_emits_full_child_diff_in_one_transaction
failed expecting a column-removed change record after a restore
that removed the column; instead only the parent's scalar fields
appeared in observed paths.

Refine: apply the is_modified filter ONLY to persistent rows in
session.dirty. session.new (creation) and session.deleted
(removal) are always real content changes by virtue of their
session-collection membership — no is_modified check needed (and
in deletion's case, the check returns the wrong answer).
2026-05-28 15:37:19 -06:00
Mike Bridge ed4c1792ad chore(versioning): import-sort autofix on daos/version.py (I001)
Pre-commit (previous) flagged I001 unsorted-imports on the
backward-compat façade. Two queries imports merged into one
block (the aliased ``derive_version_uuid as _derive_version_uuid``
moves inline rather than living in its own block), and the
restore-side names sorted: ``_RESTORE_RELATIONS``,
``_stamp_audit_fields_for_restore``, ``restore_version``.

Pure mechanical reformatting; no behaviour change.
2026-05-28 15:37:19 -06:00
Mike Bridge 2c080ab6a5 fix(versioning): force-parent-dirty only when child has real column changes
_force_parent_dirty_on_child_change was firing whenever ANY
TableColumn or SqlMetric of the parent appeared in
session.dirty / new / deleted — even when the child was there for
non-content reasons:

- Lazy-load side effects when a relationship is touched
- M2M relationship-cascade artifacts (e.g. RLS setUp doing
  rls_entry.tables.extend([dataset]) triggers cascade behavior
  that pulls children into the session)
- AuditMixin auto-bumps from earlier code paths
- Reverter side passes during restore

Force-touching the parent in those cases produced an incidental
UPDATE tables SET description=…, changed_on=…, changed_by_fk=…
whose changed_by_fk value or autoflush ordering tripped FK
integrity on some dialects. Symptoms:

- test_rls_filter_alters_no_role_user_birth_names_query → FK
  IntegrityError on autoflush during a query
- test_restore_applies_scalar_field → 422 "Dataset could not be
  updated" during restore

Fix: gate on Continuum's is_modified(child), which returns True
only when a non-excluded versioned column on the child has
SQLAlchemy attribute-history changes. New objects (session.new)
and genuinely-modified rows still flag the parent; phantom-dirty
rows do not.

The intended hook semantics — "child edit forces a parent shadow
row" — are preserved: a column-description edit through the
dataset API still triggers is_modified True, still flags the
parent. See test_dataset_column_edit_creates_parent_version.
2026-05-28 15:37:19 -06:00
Mike Bridge 9ff18b3e48 fix(versioning): flag `description instead of uuid` in force-parent-dirty
flag_modified(parent, "uuid") was producing FK integrity failures via
the column's BLOB/BINARY round-trip: SQLAlchemy logs the param as
``<memory at 0x…>`` and the UUID round-trip doesn't always match the
in-memory value byte-for-byte. Symptom: in scenarios where the parent
is already going to flush (Reverter applying historical state during
restore, RLS test triggering autoflush during a query), our added
``uuid`` UPDATE column tripped the FK check.

Pick ``description`` instead — plain Text column on all three
versioned parent classes (Dashboard, Slice, SqlaTable), no
TypeDecorator, no marshaling layer. Flagging it round-trips its
current value safely. Fallback chain ``description → uuid → col_keys[0]``
keeps the original deterministic-pick property for forks/subclasses
that excluded ``description``.

Should unblock test_restore_applies_scalar_field and the
test_rls_filter_alters_no_role_user_birth_names_query autoflush
error.
2026-05-28 15:37:19 -06:00
Mike Bridge 2731b099f1 chore(versioning): pre-commit autofixes (ruff 0.9.7) — round 2
- factory.py: TID251 banned ``import json``; switch to
  ``from superset.utils import json`` (project convention).
- factory.py: ruff format reflow on _matches_previous_version.
- version_restore.py: ruff format collapse on restore_version call.

CI was pinning a different ruff version than my local uvx default;
re-ran against ruff==0.9.7 (the version in requirements/development.txt)
which surfaced these.
2026-05-28 15:37:19 -06:00
Mike Bridge 771dd4bcd9 fix(versioning): replace session-state guard with InvalidRequestError catch
The previous attempt (d0520f6766) was too aggressive: skipping when
parent is in session.dirty/new/deleted bypassed the
persistent-and-clean case the hook EXISTS for. Some upstream code
paths put the dataset in session.dirty *before* this listener fires
(API controllers touching audit fields, etc.), so the
session-membership pre-check made us silently no-op on the very
scenario the hook needs to handle. CI symptom:
test_dataset_column_edit_creates_parent_version showed before=317,
after=317 (parent shadow not written).

Restore the unconditional flag_modified and catch the specific
InvalidRequestError that fires only for the session.new case
(uuid default callable hasn't populated state yet). Other states
fall through to the original behavior:
- persistent + clean → flag_modified succeeds, parent goes dirty,
  Continuum picks it up, SkipUnmodifiedPlugin keeps the row via
  _has_dirty_versioned_children. ✓
- persistent + dirty → flag_modified is harmless (already dirty).
- session.new → InvalidRequestError, skip (parent INSERTs anyway).
- session.deleted → flag_modified may or may not raise; if it does,
  we skip; if not, the delete dominates.

Should unblock test_dataset_column_edit_creates_parent_version,
test_get_version_returns_historical_snapshot_with_children, and
test_restore_with_column_edits_reverts_columns.
2026-05-28 15:37:19 -06:00
Mike Bridge 4af88e65c0 chore(versioning): apply pre-commit autofixes (ruff + auto-walrus)
- ruff: import sort + E501 reflow on the parent-state guard in
  baseline.py
- ruff format: function-signature collapse and join-chain reflow in
  queries.py
- auto-walrus: two ``entity_kind = …; if … is not None:`` patterns
  in queries.py converted to assignment-expressions
2026-05-28 15:37:19 -06:00
Mike Bridge 560567b64c fix(versioning): retention task FK violation on cross-entity transactions
When one ORM flush touches multiple versioned entities (dashboard +
slice + dataset all save at tx=X), each gets a shadow row sharing
that tx. If only the dashboard is later edited at tx=Y, the
dashboard row at tx=X is closed (end_tx=Y) while slice/dataset rows
stay live at tx=X. Retention then preserves tx=X (slice/dataset are
live there) and prunes tx=Y. The dashboard's closed row at tx=X
survives step 1, then its end_transaction_id=Y trips the FK when
step 2 deletes version_transaction row Y.

Fix: extend the shadow-row delete to also match end_transaction_id
IN tx_ids. Live rows have end_tx=NULL so they're never matched by
either predicate. Closed rows that touch a pruned tx at either
endpoint are pruned together — consistent with retention semantics
(any tx in the row's lifespan is gone, so the row's chain is broken
anyway).

Unblocks test_retention_prunes_old_rows on sqlite, mysql, postgres.
2026-05-28 15:37:19 -06:00
Mike Bridge cbb4e8c788 fix(versioning): skip non-clean parents in force-parent-dirty hook
The force-parent-dirty listener was calling attributes.flag_modified
on every parent reachable from a dirty child — including parents
themselves in session.new (e.g. brand-new SqlaTable + brand-new
TableColumns from POST /api/v1/dataset/). flag_modified rejects
unloaded attributes, and a session.new SqlaTable's uuid (default=uuid4
fires at flush time) is unloaded until then. CI caught this with
InvalidRequestError cascading into 422s across dataset creation /
upload / Playwright dataset specs.

The hook is only needed for the persistent-and-clean case (child
edited, parent's own scalars untouched, dropdown otherwise empty).
Anything in session.new will flush anyway; anything in session.dirty
is already flagged; session.deleted shouldn't be touched. Short-
circuit before the flag_modified call.

Unblocks test-sqlite, test-mysql, test-postgres (previous), and
playwright dataset specs.
2026-05-28 15:37:19 -06:00
Mike Bridge 43abeb4447 refactor(versioning): apply cross-PR review feedback (#39977 H1/M3/M5)
Three small follow-ups surfaced by aminghadersohi's review of the
SoftDeleteMixin PR (#39977) that apply equally here:

- H1: cache _child_to_parent_registry() with functools.cache. Called
  twice per save flush; mapping depends only on import-time model
  classes, so unbounded cache is the right shape (no invalidation).
- M5: tighten _CHILD_BASELINE_HANDLERS type from dict[str, Any] to
  dict[str, Callable[[Session, Any, int], None]] via a named alias.
  Mypy now catches a future broken handler signature.
- M3/M4: explain the inline-import pattern once in the module
  docstrings of baseline.py and changes.py. Both modules use
  pylint disable=import-outside-toplevel uniformly because they
  load during init_versioning() before mappers are configured;
  the per-callsite "why" comments would just repeat the same
  reason. Module-level explanation + a hint to comment unusual
  cases is the cleaner shape.

M6 (listener placement) doesn't apply — init_versioning() already
runs inside init_app_in_ctx(). M8 (loose OpenAPI schema in
*/api.py docstrings) is real but its own change.
2026-05-28 15:37:19 -06:00
Mike Bridge 6e1b7896e7 docs(versioning): record why SkipUnmodifiedPlugin doesn't clean up orphan version_transaction rows inline
Extends the existing docstring note ("the orphan is swept by retention")
with the reasoning behind not cleaning it up in the same flush. The
inline-delete is appealing in principle but would couple this plugin
to the change-records listener's buffer state via the ON DELETE
CASCADE on ``version_changes.transaction_id``: both listeners would
have to agree that the flush produced nothing before the version_transaction
row could be dropped safely. The orphan's ~40-byte storage cost +
retention's correct-by-construction handling (orphans have no parent
shadow, so they're never in the "preserve" set) make the coordination
overhead not worth it.

Captures the design decision in the file where the next reader will
look for it.
2026-05-28 15:37:19 -06:00
Mike Bridge 18d134605d tidy(versioning): reading-order shuffle in baseline.py (newspaper-article order)
Pure file shuffle, zero behaviour change. Reorders ``baseline.py`` so it
reads top-down by level of abstraction (newspaper-article rule): the
public entry point at the top, supporting helpers descending below.

Before: 14 private helpers, then ``register_baseline_listener`` at the
bottom. A reader opening the file met the leaf builders first and had
to accumulate context before finding the call site.

After (top-down):

  - Entry point: ``register_baseline_listener`` + inner ``capture_baseline``
  - High-level helpers used by ``capture_baseline``:
      ``_force_parent_dirty_on_child_change``,
      ``_collect_parents_to_baseline``,
      ``_child_to_parent_registry``,
      ``_version_table_for``,
      ``_shadow_row_count``,
      ``_insert_baseline_and_children``
  - Mid-level builders:
      ``_insert_baseline_row``,
      ``_baseline_children_for_parent``
  - Per-entity child handlers + their dispatch table:
      ``_baseline_dataset_children``,
      ``_baseline_dashboard_children``,
      ``_CHILD_BASELINE_HANDLERS``
  - Leaf builders:
      ``_insert_child_baseline_rows``,
      ``_baseline_attached_slices``,
      ``_insert_synthetic_slice_baseline``

Three section-divider comments mark the abstraction levels. The
``_CHILD_BASELINE_HANDLERS`` dict literal stays after its referenced
handlers (module-level literals evaluate at import time and need names
already bound); a comment now flags this constraint.

Function bodies are byte-for-byte unchanged; ``git log -L`` on any
function shows only its relocation. 96 unit tests pass.
2026-05-28 15:37:19 -06:00
Mike Bridge 6fcd762cd3 tidy(versioning): extract read_row_outside_flush helper
baseline.py:_insert_baseline_row and changes.py:_read_pre_state both
issued the same "read a single row through ``session.connection()``
inside ``with session.no_autoflush:``" pattern. Same five-line block,
same intent ("read the pre-flush state without triggering the in-flight
edit's flush").

Promoted to ``superset.versioning.utils.read_row_outside_flush(session,
table, entity_id)``. Companion to ``single_flush_scope`` — they sit
next to each other in utils.py and frame the two directions of the
"don't autoflush mid-listener" pattern.

Returns ``dict[str, Any]`` (or ``None``) so callers can't accidentally
hold a cursor-bound ``RowMapping`` past the listener boundary. Both
call sites get shorter by ~5 lines.

Also picks up Decimal stringification in the changes.py docstring
update (was listed in the W4 commit but the docstring still said
"(datetime, UUID, bytes)" — now matches the implementation).

Behaviour unchanged. 96 unit tests pass.
2026-05-28 15:37:19 -06:00
Mike Bridge 6fa48f8bc2 tidy(versioning): extract shared helpers between list_versions and get_version
After the SRP split (8c9cf36) put both functions in the same module
~150 lines apart, their overlap became visible: same JOIN of
version_table → version_transaction → ab_user, same baseline-first
ordering, same user-row → ``changed_by`` projection, same lookup
``_ENTITY_KIND_BY_CLASS_NAME.get(model_cls.__name__)``. About 30 lines
of duplication.

Five small helpers extracted at the module top:

- ``_resolve_version_tables(model_cls)`` returns ``(ver_tbl, tx_tbl, user_tbl)``
- ``_version_with_tx_user_join(ver_tbl, tx_tbl, user_tbl)`` builds the join
- ``_baseline_first_ordering(ver_tbl)`` returns the order-by tuple
- ``_user_select_cols(user_tbl)`` returns the user-column list with
  ``user_id`` as the stable label (normalises the prior asymmetry
  where ``list_versions`` labelled it ``user_id`` and ``get_version``
  labelled it ``_user_id`` to dodge a column-name collision — the
  ``user_id`` label collides with neither)
- ``_changed_by_from_row(row)`` projects user columns onto the API shape
- ``_entity_kind_for(model_cls)`` resolves the change-records taxonomy lookup

Both call sites get shorter and read what they do (build query / project
user / build row) rather than how. Behavior unchanged; no test changes.

Also two small inline tidyings while in the file:

- Replace the ternary
  ``changes_by_tx = list_change_records_batch(...) if entity_kind else {}``
  with an explicit two-line if-statement in both functions. The ternary
  buries the decision; the if-statement reads as one thought.
- Inline the one-shot ``meta_cols`` set declaration in ``get_version``
  into the ``if col.name in {...}`` check that uses it three lines later.

Net: about 110 lines → about 80 lines across the two functions, plus
a small helper section at the top.
2026-05-28 15:37:19 -06:00
Mike Bridge 5d19a34b2d refactor(versioning): sqlalchemy-review follow-ups (W1–W8)
Cleanup pass from the SQLAlchemy + migration code review. Eight items,
all in the "warnings / suggestions" tier — no behaviour change visible
to the API, but each closes a real correctness, perf, or maintainability
concern surfaced in review.

baseline.py
- Delete unused ``_get_user_id`` (W1). The function wrapped a broad
  ``except Exception:  # noqa: S110`` swallow that hid bugs; grep
  confirmed no callers anywhere. The legitimate audit-field paths
  (``row.get("changed_by_fk")`` etc.) already drive the
  ``version_transaction.user_id`` write.
- Batch ``_baseline_attached_slices`` from O(N) round-trips to
  three queries (W2): one membership SELECT, one existing-shadow
  SELECT, one bulk live-row SELECT for the missing ids. The previous
  per-slice ``COUNT(*)`` + ``SELECT`` was a measurable first-save
  hotspot on dashboards with many charts. Drops the now-unused
  ``_slice_has_shadow`` helper.
- Pick a stable column name for ``flag_modified`` in
  ``_force_parent_dirty_on_child_change`` (W3). ``uuid`` is on all
  three versioned parent classes and excluded by none, so the
  flagged attribute is deterministic across SQLAlchemy versions /
  mapper-config orders instead of depending on
  ``versioned_column_properties(parent)[0]``. Falls back to the
  first available column for forks that exclude ``uuid``.

changes.py
- Add ``Decimal`` handling to ``_jsonable`` (W4) — ``json.dumps``
  rejects ``Decimal``, so any numeric column (e.g. ``SqlMetric.currency``
  contents, or fork/plugin Decimal columns) would crash the bulk
  insert. Stringify rather than ``float()`` to preserve precision;
  the diff engine compares ``from_value`` / ``to_value`` by string
  equality after this coercion so both sides round-trip identically.

queries.py
- Promote the inline ``{0: "baseline", 1: "update", 2: "delete"}``
  dict to module-level ``_OP_TYPE_LABELS`` (W7). The literal was
  duplicated across ``list_versions`` and ``get_version``; the third
  caller is one bug fix away.
- Comment on ``resolve_version_uuid``'s Python-side ``derive_version_uuid``
  loop (W8) — no portable SQL form for UUIDv5 across PostgreSQL /
  MySQL / SQLite, iteration count is bounded by the retention
  window. Flags the place to revisit if retention is ever disabled
  (``=0``) on a heavily-edited entity.

migrations/2026-05-01_23-36 (composite-PK)
- Belt-and-braces guard in ``_downgrade_mysql_table`` (W6): asserts
  ``t.name in AFFECTED_TABLES`` before interpolating into the
  backtick-quoted ALTER statements. The invariant was already
  structurally implied (callers iterate ``AFFECTED_TABLES``), but
  making it load-bearing means a future refactor can't slip an
  arbitrary table name through.

(W5 was verified-no-change: grepped ``tests/`` for ``metadata.create_all``
callers that exercise versioning tables; none. The cascade-FK
gap on ``version_changes.transaction_id`` is already documented
in ``tests/integration_tests/versioning/change_records_tests.py:27-32``.)

62 versioning unit tests pass.
2026-05-28 15:37:19 -06:00
Mike Bridge 00604b92d9 refactor(versioning): split VersionDAO into queries + restore modules
VersionDAO carried five distinct concerns under one class — UUID
derivation, version metadata queries, change-record loading,
single-version snapshot retrieval, and restore orchestration. Bob's
"and" test (the clean-code review flagged this as the next structural
fix after the dead-code purge) gives ~600 lines of "queries about
versioned state of one entity AND the workflow that mutates it."

Splits the read and write sides into purpose-built modules:

- ``superset/versioning/queries.py`` — UUID derivation
  (``VERSION_UUID_NAMESPACE``, ``derive_version_uuid``) + read-side
  helpers (``find_active_by_uuid``, ``current_version_number``,
  ``current_live_transaction_id``, ``current_live_version_uuid``,
  ``list_versions``, ``resolve_version_uuid``, ``get_version``,
  ``list_change_records_batch``). ~475 lines.

- ``superset/versioning/restore.py`` — write-side (``restore_version``,
  ``_stamp_audit_fields_for_restore``, ``_RESTORE_RELATIONS``).
  ~140 lines. Depends only on ``queries.find_active_by_uuid`` and
  ``utils.single_flush_scope``.

- ``superset/daos/version.py`` — collapsed to an ~85-line backward-compat
  façade that re-exports both modules under a single ``VersionDAO``
  class via ``staticmethod`` aliases. The module also re-exports
  ``VERSION_UUID_NAMESPACE`` and ``derive_version_uuid`` at module level
  so the ~10 existing callers (api.py handlers, command classes, the
  ETag emitter, integration tests) don't have to change their imports.
  New code is encouraged to import from the sub-modules directly.

The functions themselves are unchanged byte-for-byte aside from
internal call sites being rewritten from ``VersionDAO.foo`` to the bare
function name (since they now live as module-level functions, not
class methods).

One unit-test mock target moved: ``test_restore_version_returns_none_for_unknown_entity``
now patches ``superset.versioning.restore.find_active_by_uuid`` (the
actual call site) instead of ``VersionDAO.find_active_by_uuid`` (which
is now just an alias).

Each of the three modules now has one reason to change. When the
sc-103157 soft-delete pass adds the ``deleted_at IS NULL`` filter to
``find_active_by_uuid``, it touches only ``queries.py``. When a
per-entity-type restore Strategy replaces the string-keyed
``_RESTORE_RELATIONS`` dispatch, it touches only ``restore.py``.
2026-05-28 15:37:19 -06:00
Mike Bridge cb13498f6a temp(versioning): strip URL params from dashboard restore navigation; regen lockfile
DashboardList demo dropdown previously instructed the user to "Reload
the page to see the change" after a restore. The URL the user
returns to may still carry ``?native_filters_key=…`` /
``permalink_key`` / ``form_data_key`` from a prior session — those
point at server-cached snapshots (in ``key_value`` and the
filter-state cache) captured before the restore. On rehydration the
cached state is merged on top of the restored ``json_metadata``,
masking the rollback (e.g. dashboard-level colour-scheme restore
appears not to take effect).

Replaces the alert + manual reload with a direct ``window.location.href``
navigation to ``/superset/dashboard/<uuid>/`` — drops all URL params,
forcing hydration from the freshly restored DB state.

Also regenerates ``package-lock.json`` to pick up the ``zod 4.4.1 →
4.4.3`` bump that master's ``package.json`` already reflects.

(``temp(versioning)`` prefix per the demo dropdown's status — this
file is not part of V1 scope per ADR-005; the V2 UI SIP owns the
actual restore UI surface.)
2026-05-28 15:37:19 -06:00
Mike Bridge 15abdbc678 refactor(versioning): rename find_active_by_uuid public + collapse restore commands onto BaseRestoreVersionCommand
Two coupled clean-code review fixes:

(1) Rename ``VersionDAO._find_active_entity_by_uuid`` →
``find_active_by_uuid``. The leading-underscore + three
``# pylint: disable=protected-access`` suppressions in the restore
commands were the smell of a wrongly-private API. The method is a
perfectly reasonable public DAO operation; dropping the underscore
removes the suppressions.

(2) Collapse ``RestoreChartVersionCommand``, ``RestoreDashboardVersionCommand``,
``RestoreDatasetVersionCommand`` onto a shared
``BaseRestoreVersionCommand`` (``superset/commands/version_restore.py``).
The three classes were textbook copy-paste — identical except for
the model class and three exception types. Each subclass now declares
``model_cls`` + ``not_found_exc`` + ``forbidden_exc`` and overrides
``run()`` with one ``@transaction(reraise=<failed_exc>)``-decorated
line delegating to ``self._do_restore()``. ~80 lines per file →
~45 lines per file; one shared workflow instead of three drift sources.

The api.py imports of ``RestoreChartVersionCommand`` /
``RestoreDashboardVersionCommand`` / ``RestoreDatasetVersionCommand`` are
unchanged — public class names preserved.
2026-05-28 15:37:19 -06:00
Mike Bridge 769d1670e2 refactor(versioning): purge dataset_snapshots dead code + fix get_version bug
The full-Continuum spike (ADR-004 revised) replaced the JSON-snapshot
restore path with Continuum's native Reverter and removed the
``dataset_snapshots`` / ``dashboard_snapshots`` tables from the
migration chain. Seven VersionDAO methods and two module-level
helpers that read/wrote those tables stayed in the code anyway and
went unused — dead code that looked live.

Worse, ``VersionDAO.get_version`` still read from
``dataset_snapshots`` in its SqlaTable branch. On any environment
where the snapshot tables don't exist (current production behavior),
``GET /api/v1/dataset/<uuid>/versions/<version_uuid>/`` raised
``OperationalError``. The branch is rewritten to read column and
metric state from Continuum's child shadow tables
(``table_columns_version`` / ``sql_metrics_version``) via the
existing ``_shadow_rows_valid_at`` helper.

Deleted:
- ``_deserialize_snapshot_value`` (module helper)
- ``_coerce_snapshot_list`` (module helper)
- ``RESTORE_EXCLUDE_FIELDS`` (constant — only referenced by deleted code
  and a docstring)
- ``VersionDAO._restore_dataset_children``
- ``VersionDAO._parse_slice_ids_json``
- ``VersionDAO._apply_dashboard_slices``
- ``VersionDAO._restore_dashboard_children``
- ``VersionDAO._apply_snapshot_children``

The corresponding ~17 unit tests in
``tests/unit_tests/daos/test_version_dao.py`` are removed alongside.

Stale docstring references in ``versioning/changes.py`` and
``versioning/diff.py`` that pointed at the retired snapshot tables are
also cleaned up.

Also strips an 8-line comment block in ``restore_version`` that
duplicated the docstring of ``_stamp_audit_fields_for_restore``.

Net: −290 lines from ``daos/version.py``; a production-shape bug
fixed; dead code that looked live is gone.
2026-05-28 15:37:19 -06:00
Mike Bridge 2c7ba5c3d5 refactor(versioning): single_flush_scope context manager + single-revert restore
VersionDAO.restore_version previously called Continuum's Reverter
once per relation in a split-revert loop with flush + expire between
calls. That closed an autoflush race in the Reverter when multiple
relations were reverted at once, but split one logical restore across
multiple Continuum transactions — and once the change-records listener
was wired up, the listener's tx-dedup guard skipped the second pass,
silently dropping child-addition records from version_changes. A
restore that re-added a calculated column would render as an empty
"Baseline" entry in the dropdown.

Replaces the split-revert with a single ``target_version.revert(relations=relations)``
call wrapped in a new ``single_flush_scope(db.session)`` context
manager (``superset/versioning/utils.py``). The context manager
suppresses autoflush inside the block and issues one trailing flush
on clean exit; on exception, the trailing flush is skipped so the
session's normal rollback path handles cleanup. Same autoflush window
closed, one Continuum transaction instead of N, the change-records
listener sees the complete shadow state in one after_flush pass.

The wrapper carries the full autoflush-race / cascade-add rationale
in its docstring so the restore_version call site can be a short
6-line block referencing it.

Integration coverage: ``test_restore_emits_full_child_diff_in_one_transaction``.
2026-05-28 15:37:19 -06:00
Mike Bridge eb3bcf0d3d feat(versioning): force-parent-dirty on versioned-child change
SQLAlchemy doesn't mark a parent as dirty when only its children
(``TableColumn`` / ``SqlMetric`` on ``SqlaTable``) are modified.
Continuum's UnitOfWork only creates operations for entities in
``session.dirty``, so a column-only edit produces shadow rows in
``table_columns_version`` but no parent shadow row in
``tables_version``. ``VersionDAO.list_versions`` queries the parent
shadow, so the version dropdown is empty for child-only saves —
exactly the failure mode reported when "I edited a column description
but no version appeared."

Extends ``register_baseline_listener`` with a new before-flush hook
``_force_parent_dirty_on_child_change`` that walks the existing
``_child_to_parent_registry`` and ``attributes.flag_modified(parent,
<first non-excluded versioned column>)`` whenever a versioned child
is dirty / new / deleted but the parent's own scalars haven't been
touched. The flag puts the parent in ``session.dirty`` so Continuum's
UoW creates a parent UPDATE operation; the resulting shadow row's
scalar columns mirror the previous version (only the children
actually changed), and the row exists to anchor the transaction in
the parent's version chain.

``SkipUnmodifiedPlugin._is_no_op_update`` is updated in this commit's
predecessor to recognize the "scalars match but children dirty" case
via ``_has_dirty_versioned_children`` so the forced parent UPDATE
isn't skipped.

Integration coverage: ``test_dataset_column_edit_creates_parent_version``.
2026-05-28 15:37:19 -06:00
Mike Bridge 0850679edd feat(versioning): SkipUnmodifiedPlugin audit-key normalize for Dashboard.json_metadata
Continuum's no-op suppression compared post-flush column values
byte-for-byte against the previous live shadow row. For
``Dashboard.json_metadata`` that produced false-positive version rows
on saves where the user authored nothing — the frontend re-stamps
``map_label_colors`` (regenerated from the ``LabelsColorMap``
singleton) on every save, plus ``chart_configuration`` /
``global_chart_configuration`` / ``show_chart_timestamps`` /
``color_namespace`` (derived from the current chart set), so two
consecutive identical saves produce different bytes for the column.
The diff engine already excluded those keys via
``DASHBOARD_JSON_METADATA_AUDIT_KEYS`` when computing change records;
the skip-plugin diverged.

Adds a ``_COLUMN_NORMALIZERS`` registry keyed on
``(class_name, column_name)`` that maps to a per-column normalizer
applied to both pre- and post-image before equating. The first
entry parses ``Dashboard.json_metadata`` as JSON and drops the
audit-key set before comparing. The same registry is the extension
point for analogous transient fields on charts and datasets.

Promotes ``_DASHBOARD_JSON_METADATA_AUDIT_KEYS`` to a public name
(``DASHBOARD_JSON_METADATA_AUDIT_KEYS``) so the skip-plugin can import
it from ``superset.versioning.diff`` without reaching across a
leading-underscore boundary.

Integration coverage: ``test_map_label_colors_only_change_does_not_create_version``.
2026-05-28 15:37:19 -06:00
Mike Bridge 6cc0853952 fix(importer): use ORM relationship assignment for dashboard_slices
The v1 import pipeline previously wrote dashboard ↔ chart membership
via raw Core DML (``db.session.execute(delete(dashboard_slices)…)`` +
``db.session.execute(insert(dashboard_slices)…)``). With Continuum's
M2M tracker enabled by the versioning feature, those Core writes
emit malformed shadow INSERTs into ``dashboard_slices_version`` —
the tracker can't see the composite-PK columns through the Core
layer and produces rows with only ``(transaction_id, operation_type)``
populated, triggering a ``NOT NULL`` violation on
``(dashboard_id, slice_id)``.

Rewrites both import paths (``ImportAssetsCommand._import`` in
``commands/importers/v1/assets.py`` and ``ImportDashboardsCommand._import``
in ``commands/dashboard/importers/v1/__init__.py``) to use ORM-level
``dashboard.slices = [...]`` reassignment followed by an explicit
``db.session.flush()``. The explicit flush is necessary to land the
M2M rows before any subsequent autoflush fires an inner-flush event
handler that would reset the relationship change (cf. the SAWarning
``Attribute history events accumulated on N previously clean instances
within inner-flush event handlers have been reset``).

The unit tests previously called ``_import`` directly twice in the same
session — production wraps ``run()`` in ``@transaction`` so each invocation
gets its own DB+Continuum transaction. Added ``db.session.commit()`` between
calls in ``test_import_adds_dashboard_charts``,
``test_import_removes_dashboard_charts``, and
``test_dashboard_import_with_overwrite_replaces_charts`` so the tests
mirror production semantics; otherwise the second call's M2M shadow
inserts conflict with the first call's on
``UNIQUE (dashboard_id, slice_id, transaction_id)``.
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 96018dab71 temp(versioning): demo version-history dropdowns + French i18n
Adds debug-only ``VersionHistoryDropdown`` widgets to the chart,
dashboard, and dataset list pages so the version surface can be
exercised from the UI during the spike. Each row's actions column
gets a clock-icon dropdown that fetches ``/api/v1/{resource}/<uuid>/
versions/`` on click, lists the ten most recent versions with a
formatted change-log summary, and offers per-version restore via
``POST .../versions/<uuid>/restore``.

Strings are wrapped in ``t('...')`` with placeholder formatting
(e.g. ``t('Added %(kind)s "%(name)s"', { kind, name })``) so
translators can reorder verbs and nouns rather than concatenating
fragments. ``KIND_LABELS`` is a static map keying English layout
kinds (``chart``, ``row``, ``column``, ``tab``, ``markdown``, etc.)
to ``t(...)``-extractable labels. Empty change lists render as
"Baseline" rather than "No changes recorded" since the empty case
is overwhelmingly the ``operation_type=0`` baseline row.

Locale-aware date rendering: ``new Date(iso).toLocaleString(lang)``
where ``lang`` comes from ``document.documentElement.lang`` (set
by ``src/views/App.tsx`` from the bootstrap ``locale``), so dates
follow the user's chosen Superset locale rather than the browser's.

French translations for the new strings are appended to
``superset/translations/fr/LC_MESSAGES/messages.po`` (Ajouté,
Supprimé, Modifié, Version initiale, kind labels, …). Run
``npm run build-translation`` and ``pybabel compile -l fr`` to
regenerate the JSON / MO packs.

This commit is **demo-only** per ADR-005 (V1 is backend-only). It
is intentionally marked ``temp`` so it can be reverted before the
PR splits — the production V1 ships without UI.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 39cfda37e7 test(versioning): integration tests for SkipUnmodifiedPlugin (FR-026)
Locks in the no-op-suppression behavior implemented by
``SkipUnmodifiedPlugin`` (which lives in ``superset/versioning/factory.py``
shipping with the foundation commit). Five integration tests:

1. Owners-only edit doesn't mint a version row — exercises the
   case where every dirty column is an excluded relationship.
2. Re-save with identical scalar values doesn't mint a row —
   exercises the json_metadata re-serialise path where
   ``set_dash_metadata`` rewrites the column to a different byte
   sequence with identical parsed content; the plugin must compare
   post-flush values against the prior shadow row to detect this.
3. Real scalar change DOES mint a row — guards against the plugin
   over-suppressing.
4. Same assertion on a Slice (covers the ``String`` column path on
   a different entity type).
5. ``json_metadata`` sub-key edit DOES mint a row — covers the
   ``MediumText`` column path past the plugin's content-equality
   check.

Tests are designed so a column-type change in the parent entities
(e.g. flipping ``json_metadata`` from ``MediumText`` to ``JSON``)
will fail one of these if the plugin's Python ``!=`` comparison
breaks for the new type.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 47a1c9fa95 feat(versioning): ETag helper module + integration tests (T055)
Helper module that derives the strong-validator ``ETag`` value from
an entity's current live ``version_uuid`` and attaches it to a
Flask response. Two functions:

- ``set_version_etag(response, version_uuid)`` — direct path used by
  PUT handlers that already compute ``new_version_uuid`` (see the
  REST API commit two prior). Cheap; no extra query.
- ``set_version_etag_by_uuid(response, model_cls, entity_uuid)`` —
  used by version endpoints that operate on ``entity_uuid``; looks
  up ``entity_id`` then derives ``version_uuid`` via ``VersionDAO``.
  Costs one extra ``SELECT id WHERE uuid = ?``; documented in the
  docstring so callers prefer the cheap variant when they have the
  id already.

Integration tests cover all three entity types and four endpoint
shapes (entity GET, save PUT, version-list GET, single-version GET)
plus the entity-with-no-versions edge case (header is correctly
absent).

The ETag is wired into the API endpoints in the REST-API commit
(group 3) and the CORS ``expose_headers: ["ETag"]`` ships with the
retention commit (group 4) since both touch ``superset/config.py``.
Locking enforcement (``If-Match`` → 412) is explicitly NOT in this
change — deferred to the follow-up UI SIP per Open Question §7.
``ETag`` is informational in v1.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 963f059df3 feat(versioning): time-based retention via Celery beat (FR-007)
Adds a scheduled Celery task that prunes version history older than
``SUPERSET_VERSION_HISTORY_RETENTION_DAYS`` (default 30; settable
via env var; ``0`` disables retention entirely).

**Task** — ``superset.tasks.version_history_retention.prune_old_versions``:

1. Computes ``cutoff = utcnow() - timedelta(days=N)``.
2. Selects ``version_transaction.id`` rows with ``issued_at <
   cutoff`` and filters out any tx whose parent shadow includes a
   live row (``end_transaction_id IS NULL``). The live row is the
   only preservation rule — closed historical rows including the
   baseline (``operation_type=0``) age out. Per-entity minimum-history
   floor is an open question tracked in ``future-work.md``.
3. Deletes rows owned by surviving txs in each parent shadow
   table (``dashboards_version`` / ``slices_version`` /
   ``tables_version``).
4. Deletes child-shadow rows for the same transactions
   (``table_columns_version`` / ``sql_metrics_version`` /
   ``dashboard_slices_version``).
5. Drops the surviving ``version_transaction`` rows. The
   ``version_changes`` rows cascade via the FK from the previous
   commit.

Idempotent and safely retried on partial failure.

**Schedule** — ``superset/config.py`` adds the task to the default
``CeleryConfig.beat_schedule`` (nightly at 03:00). Operators who
override ``CeleryConfig`` in their ``superset_config.py`` need to
merge this entry — see UPDATING.md.

Also adds ``"expose_headers": ["ETag"]`` to the default
``CORS_OPTIONS`` so cross-origin browser clients can read the
``ETag`` header introduced in the next commit. (Co-located here
because both touch ``superset/config.py``; the ETag mechanism
itself ships in the next commit.)

**Auto-discovery** — ``superset/tasks/celery_app.py`` adds
``version_history_retention`` to its late-imports so Celery's
auto-discovery picks up the task.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 24eef8c984 feat(versioning): REST API endpoints + restore commands
Exposes the version surface as three new endpoints per entity type
(chart, dashboard, dataset), each carrying the standard Superset
decorator stack (``@protect()``, ``@safe``, ``@statsd_metrics``,
``@event_logger.log_this_with_context``) so they appear in FAB's
``action_log`` alongside other audited operations.

| Method | Path | Purpose |
|---|---|---|
| GET  | ``/api/v1/{resource}/<uuid>/versions/`` | List version history (oldest-first; per entry: ``version_uuid``, ``version_number``, ``transaction_id``, ``operation_type``, ``issued_at``, ``changed_by``, ``changes`` array) |
| GET  | ``/api/v1/{resource}/<uuid>/versions/<version_uuid>/`` | Read-only snapshot of the entity at the requested version (scalar fields plus ``columns`` / ``metrics`` for datasets) |
| POST | ``/api/v1/{resource}/<uuid>/versions/<version_uuid>/restore`` | Replay the snapshot onto the live entity via Continuum's ``Reverter`` (non-destructive — produces a new version row stamping the restoring user via the standard save path) |

``<version_uuid>`` is a deterministic ``UUIDv5(entity_uuid,
transaction_id)`` so it's stable across replicas and retention
pruning. Authorisation reuses the resource's existing ``can_write``
permission; workspace admins can list / restore any entity.

**Restore commands** — ``superset/commands/{chart,dashboard,dataset}/
restore_version.py`` wrap ``VersionDAO.restore_version`` in the
standard ``@transaction()`` boundary. The command resolves the
``Reverter`` once per related collection (split-revert pattern, with
``flush + expire`` between calls) so a multi-relation restore
doesn't trip Continuum's autoflush race that would otherwise mark
half the collection as ``state.deleted=True`` mid-revert.

**Save responses** — ``PUT /api/v1/{resource}/<pk>`` is updated to
include ``old_version`` / ``new_version`` (0-based numbers),
``old_transaction_id`` / ``new_transaction_id`` (stable across
pruning), and ``old_version_uuid`` / ``new_version_uuid`` body
fields so callers can correlate a save with its resulting version
row. The ``ETag`` response header in the next commit is built on
top of this, but the body fields stay — they predate the header
and remain useful for clients that don't read response headers.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 7ec3091d55 feat(versioning): change records + diff engine
Adds a structured per-field change log alongside the foundational
shadow tables. Each save flush emits zero or more ``version_changes``
rows describing what changed relative to the previous version, with
shape ``[{kind, path, from_value, to_value, sequence}]`` keyed to
``version_transaction.id`` (FR-016 .. FR-021).

**Schema** — ``version_changes`` table, FK to ``version_transaction``
with ``ON DELETE CASCADE`` so retention drops dependent records
without explicit cleanup. Composite unique index on
``(transaction_id, entity_kind, entity_id, sequence)`` so the
listener can write monotonically and downstream readers see a
deterministic order.

**Diff engine** (``superset/versioning/diff.py``) — pure-function
diffing of pre-/post-state pairs:

- ``diff_scalar_fields`` for ordinary columns; emits one record per
  changed field with JSON-safe ``from_value`` / ``to_value``.
- ``diff_json_field`` for ``json_metadata`` and ``params``, walking
  the parsed structure and emitting per-sub-key records. Honours
  an ``exclude_keys`` set
  (``_DASHBOARD_JSON_METADATA_AUDIT_KEYS``: ``chart_configuration``,
  ``global_chart_configuration``, ``map_label_colors``,
  ``show_chart_timestamps``, ``color_namespace``;
  ``_CHART_PARAMS_AUDIT_KEYS``) so frontend-stamped sub-keys that
  mutate on every save don't dominate the change log (FR-022).
- ``diff_dashboard_layout`` walks ``position_json`` structurally
  and emits ``[verb, kind, id]`` records (verbs ``add``, ``remove``,
  ``move``, ``edit``; kinds from a ``CHART``/``ROW``/``COLUMN``/etc.
  → english map) so a UI can render "Added chart 'Foo'" without
  re-parsing JSON. ``HEADER_ID`` is suppressed because it duplicates
  the ``dashboard_title`` scalar record.
- ``fold_dashboard_layout_with_chart_changes`` deduplicates layout
  records against M2M / chart-membership records by UUID so an
  add-and-attach doesn't appear twice.
- ``_values_equivalent`` treats ``None`` and ``""`` as equal; this
  matches the save path's habit of normalising nullable strings to
  the empty string.

**Listener** — ``superset/versioning/changes.py`` registers a
``before_flush`` listener that captures pre-state for each dirty
entity and an ``after_flush`` listener that runs the diff engine
against the post-state and writes ``version_changes`` rows under
the resolved ``transaction_id``. Tracks processed transaction ids
on ``session.info`` so re-firings within a single transaction
(autoflush triggered by mid-commit queries) don't double-insert and
trip the unique constraint. Reads child rows via raw SELECT against
``table_columns`` / ``sql_metrics`` rather than ``dataset.columns``
because the live collection is stale during the restore path's raw
DELETE+INSERT cycle.

**Endpoint surface** — ``VersionDAO.list_change_records_batch``
batches the lookup across multiple transactions with a single
``WHERE transaction_id IN (...)`` query so the version-list
endpoint avoids N+1 round-trips. ``list_versions`` / ``get_version``
return entries with a populated ``changes`` array (empty for
``operation_type=0`` baseline rows).

**Tests** — ``test_diff.py`` covers the diff engine shape (39
unit cases across scalar, JSON, layout, child-collection, and
fold paths). ``change_records_tests.py`` exercises the listener
end-to-end with realistic save flows. ``perf_validation_tests.py``
is the T044 harness for SC-002/3/4 (list endpoint p95 < 1s,
restore < 3s, save overhead < 50ms).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 b68fea17af feat(versioning): foundation — Continuum capture + parent/child shadow tables + VersionDAO
Adds SQLAlchemy-Continuum as a dependency and wires it as the
canonical capture mechanism for chart, dashboard, and dataset edits.

**Schema** — three Alembic migrations, leaving the chain at one
foundation revision plus one child-shadow revision:

- ``version_transaction`` (renamed from Continuum's default
  ``transaction``; SQL-reserved-word workaround) carries the per-save
  ``user_id`` / ``issued_at`` and is the join target for all shadow
  rows. Auto-incrementing PK; user_id has no FK so import / Celery /
  CLI saves can write rows without an active Flask user.
- Parent shadow tables for the three entity types:
  ``dashboards_version``, ``slices_version``, ``tables_version``.
- Child shadow tables for dataset children + dashboard M2M:
  ``table_columns_version``, ``sql_metrics_version``,
  ``dashboard_slices_version`` (composite PK on the M2M shadow,
  matching the live ``dashboard_slices`` reshape from
  sc-105349-composite-association-pks).

**Models** — ``Dashboard``, ``Slice``, ``SqlaTable`` (and dataset
children ``TableColumn`` / ``SqlMetric``) gain ``__versioned__``
class attributes. The exclude lists carry both M2M relationships
(``owners``, ``roles``, ``dashboards``) and the ``AuditMixin``
columns (``changed_on`` / ``created_on`` / ``changed_by_fk`` /
``created_by_fk`` plus ``last_saved_at`` / ``last_saved_by_fk``
on ``Slice``) so auto-bumped audit fields cannot trigger a
version row on their own (FR-025).

**Plugins** — ``superset/versioning/factory.py`` ships three
Continuum plugins:

- ``VersionTransactionFactory`` renames the transaction table and
  appends the unconditional ``user_id`` column.
- ``VersioningFlaskPlugin`` sources the acting user from Superset's
  ``g.user`` rather than ``flask_login.current_user`` (Superset's
  JWT auth populates ``g.user`` but leaves ``current_user``
  anonymous on API routes).
- ``SkipUnmodifiedPlugin`` filters Continuum's UPDATE operations,
  marking content-equivalent re-saves as ``processed=True`` so they
  don't mint no-op shadow rows (FR-026; see follow-up commits for
  the test). Lives in this commit because it shares the file with
  the other plugins.

**Save-path glue** — a ``before_flush`` baseline listener
(``superset/versioning/baseline.py``) inserts an ``operation_type=0``
shadow row the first time a pre-existing entity is saved, including
the slice-baseline-under-dashboard pattern that gives the dashboard
M2M shadow a row to join against. ``UpdateDashboardCommand`` wraps
its body in ``no_autoflush`` so ``process_tab_diff`` /
``process_native_filter_diff`` don't fire intermediate flushes that
would mint extra version rows. ``DatasetDAO.update_columns`` is
rewritten as a natural-key upsert keyed on ``column_name`` so child
edits flow through ORM events Continuum sees.

**DAO** — ``superset/daos/version.py`` exposes the read API used by
the version endpoints in the next commits:
``current_version_number`` (0-based index, unstable under retention
pruning), ``current_live_transaction_id`` (stable across pruning),
``current_live_version_uuid`` (deterministic UUIDv5), plus
``list_versions`` / ``get_version`` / ``restore_version`` and a
batch ``list_change_records_batch`` for N+1 avoidance.

**Initialization** — ``superset/initialization/__init__.py`` wires
``init_versioning()`` after ``make_versioned()`` runs and the
versioned mappers are configured. Registers the baseline listener
plus the change-record listener (the latter's body lives in the
next commit but the registration site is here because it shares
the init function).

**Tests** — version-capture and version-list integration tests for
each entity type, plus a ``VersionDAO`` unit test suite. Retention
test uses a backdated ``issued_at`` so it can drive
``_prune_old_versions_impl`` synchronously.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:19 -06:00
Mike BridgeandClaude Opus 4.7 887d49cbac fix(migration): address aminghadersohi review feedback (sc-105349)
Three improvements from @aminghadersohi's review on
apache/superset#39859:

1. **`fk["name"]` unguarded in ``_downgrade_mysql_table`` re-add loop**

   The drop loop gates on ``if fk_name := fk.get("name"):`` but the
   re-add loop accessed ``fk["name"]`` unconditionally in an f-string.
   MySQL/InnoDB always assigns FK names, so this branch was defensive,
   but the asymmetry was confusing. Symmetrized via ``continue`` at the
   top of the re-add loop.

2. **``ondelete`` whitelist before raw-SQL interpolation**

   The value comes from MySQL's ``information_schema`` (not user
   input), but interpolating a reflected string into raw SQL without a
   guard left a "what if an unexpected value appears" footgun. Added
   ``_VALID_ONDELETE_ACTIONS`` (the four SQL-standard actions) and a
   ``RuntimeError`` when an unexpected value is reflected.

3. **Direct ALTER on PostgreSQL for tables with pre-existing UNIQUE**

   ``recreate="always"`` is dialect-agnostic — on PostgreSQL it
   triggers ``CREATE TABLE AS SELECT → DROP → RENAME`` holding
   ``ACCESS EXCLUSIVE`` for the full table-copy duration. For a
   multi-million-row ``dashboard_slices``, that lock window can be
   noticeable. The reflected UNIQUE constraint has a stable name on
   PostgreSQL (default ``<table>_<cols>_key`` convention), so dropping
   it directly and then running structural change as direct ALTER
   avoids the copy entirely.

   The reflected UNIQUE name is wrapped in a new
   ``_drop_redundant_unique_by_name()`` helper. Postgres takes the
   direct path; MySQL keeps ``recreate="always"`` because InnoDB binds
   FKs to the UNIQUE's underlying index for back-reference (``DROP
   CONSTRAINT`` on the UNIQUE there raises ``ERROR 1553``); SQLite
   keeps ``recreate="always"`` because unnamed UNIQUEs reflect with
   ``name=None`` and can't be dropped by name.

Verified end-to-end: downgrade-then-upgrade against MySQL with
~12M total junction rows seeded completes in ~1m 41s (within the
range of the prior measurements).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:37:02 -06:00
Mike BridgeandClaude Opus 4.7 8f5a230c84 fix(migration): skip alter_column nullable=False on non-SQLite (sc-105349)
Justin Park (@justinpark) reported on apache/superset#39859:

  MySQLdb.OperationalError: (1832, "Cannot change column 'dashboard_id':
  used in a foreign key constraint 'fk_dashboard_roles_dashboard_id_dashboards'")

Root cause: ``batch_op.alter_column(fk1, nullable=False)`` for the six
non-UNIQUE association tables emits ``ALTER COLUMN`` on a column that
participates in an FK constraint. MySQL 8 rejects this with ERROR 1832
when the table has data — even when the change is just ``NULL`` →
``NOT NULL`` and the column is already part of a freshly-added
composite primary key (which InnoDB has just made implicitly NOT NULL
anyway). The error fires on populated tables only; CI's ``test-mysql``
shard runs against empty tables and so didn't catch this, while a
real production-shaped install does.

The ``alter_column`` was only ever needed for SQLite, where composite
``PRIMARY KEY`` does not promote constituent columns to ``NOT NULL``
(a long-standing SQLite quirk — only ``INTEGER PRIMARY KEY`` does).
PostgreSQL and MySQL implicitly promote PK columns to ``NOT NULL`` as
part of ``ADD PRIMARY KEY``, so the explicit step is unnecessary on
both — and on MySQL it's actively broken on populated tables.

Fix: extract the ``alter_column`` pair into a helper
``_enforce_not_null_for_sqlite()`` that no-ops on Postgres and MySQL.
Both branches of the per-table upgrade (the ``recreate="always"`` path
for the two UNIQUE-bearing tables, and the direct-ALTER path for the
other six) now call the helper instead of inlining the
``alter_column``.

Verified end-to-end: downgrade-then-upgrade against MySQL with
~12M total junction rows (10M dashboard_slices + 1M each
slice_user/dashboard_user + 100K dashboard_roles) completes in
1m 39s with no ERROR 1832. The 44 in-memory SQLite tests still pass.

Considered Justin's alternative (drop FKs on MySQL across all eight
tables, unifying the two branches) but rejected as more invasive —
it would require capturing FK metadata and explicitly re-creating
the FKs for the six non-recreate tables, since they don't go through
the ``copy_from`` path that re-creates FKs automatically. The
SQLite-only approach is more targeted: it removes the operation that
MySQL rejects rather than working around the rejection.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 15:36:58 -06:00
Mike BridgeandClaude Opus 4.7 aaa40761b1 feat(scripts): add --dirty-duplicates-pct to seed_junction_load.py
Extends the stress-test seed script with an optional duplicate-row
injection step, used to measure the empirical cost of the migration's
``_dedupe_by_min_id`` phase.

Usage: after running the normal seed at a given scale, add
``--dirty-duplicates-pct 5`` (or any non-zero value) to inject that
percentage of duplicate ``(fk1, fk2)`` rows into each non-UNIQUE
junction (slice_user, dashboard_user, dashboard_roles —
dashboard_slices is skipped because its UNIQUE constraint, present
both pre- and post-migration, rejects duplicates).

Pre-condition: requires the DB to be at the pre-migration revision
(33d7e0e21daa). The post-migration composite PK rejects duplicates,
so attempting to inject on the upgraded schema errors out.

Empirical result on MySQL @ 10M dashboard_slices + ~2.1M other
junction rows + 105K injected duplicates (5% on the 3 non-UNIQUE
tables):
  Upgrade time: 1m 36s vs clean baseline 1m 37s
  → dedupe cost is within measurement noise; the table-scan that
    the migration already performs dominates whether or not
    duplicates exist.

This empirically confirms what the cost-model predicted: the
``_dedupe_by_min_id`` GROUP BY scan is the dominant cost of that
phase, and the actual per-duplicate DELETE is negligible.

NULL-FK injection deliberately skipped — would require altering the
six non-UNIQUE FK columns from NOT NULL back to nullable (the
migration's downgrade keeps them NOT NULL by design), which adds
per-backend ALTER complexity for a code path that's structurally
identical in cost shape (DELETE WHERE col IS NULL is the same scan
shape as the dedupe scan).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 c2a761e44d build(scripts): add stress-test data generator for migration timing
Add ``scripts/seed_junction_load.py``, a backend-agnostic script that
bulk-inserts synthetic parent rows (dashboards, slices, users, roles,
tables, dbs) and many-to-many junction rows for the four largest
association tables targeted by the composite-PK migration:
``dashboard_slices``, ``slice_user``, ``dashboard_user``,
``dashboard_roles``.

Designed for measuring migration runtime at varying scales — run with
a series of size flags (100K / 1M / 5M / 10M for the target table)
and time the migration at each scale to verify the predicted
``O(N log N)`` extrapolation against real numbers.

Properties:
- **Reproducible**: deterministic cross-product walk through parent IDs
  produces a stable pair sequence; re-running is replayable.
- **Idempotent**: re-running with the same target is a no-op; with a
  higher target, only new rows are added.
- **Backend-agnostic**: connects via Superset's standard ``DATABASE_*``
  env vars (or ``SUPERSET__SQLALCHEMY_DATABASE_URI``). Branches on
  dialect for ``BINARY(16)`` vs ``UUID`` vs TEXT/BLOB UUID columns.
- **Batched**: bulk INSERT 10K rows per statement.
- **Per-phase timing**: logs elapsed wall time for the parents phase,
  the junctions phase as a whole, and per junction-table.
- **Avoidance set**: loads existing junction pairs into a Python set
  so re-runs on top of pre-existing data don't collide on the
  uniqueness constraint.

Usage (inside the Superset container):

    docker exec superset-superset-1 \\
        /app/.venv/bin/python /app/scripts/seed_junction_load.py \\
        --dashboard-slices 1000000

Defaults target a "large multi-team install" shape: 1M
``dashboard_slices``, 100K each ``slice_user`` / ``dashboard_user``,
10K ``dashboard_roles``. Override per-table via flags.

Tested locally on MySQL (the user's current eval stack):
- 200/100/100/50 row mini-run produced expected counts.
- Re-running at the same target is a no-op (idempotent).
- ``--dry-run`` plans without writing.

Junction tables not yet covered (``sqlatable_user``, ``rls_filter_*``,
``report_schedule_user``) are typically small in production and
require additional parent seeding (RLS filters, report schedules)
that wasn't worth the scope here. Adding them is straightforward by
extending ``JUNCTIONS`` and writing the corresponding parent seeder.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 acfdd22c21 fix(docker): MySQL examples DB + EXAMPLES_PORT override (sc-105349)
Fix two follow-on issues reported when starting the dev stack with
docker-compose-mysql.yml:

1. ``superset-init`` step 4 (load-examples) fails with
   ``MySQLdb.OperationalError: (2002, "Can't connect to server on 'db'")``
   because the analytics-examples DB connection inherits ``EXAMPLES_PORT=5432``
   (Postgres port) from ``docker/.env``. The override flipped
   ``DATABASE_DIALECT`` to ``mysql+mysqldb`` but left the EXAMPLES_*
   group on Postgres defaults, so the URI became
   ``mysql+mysqldb://examples:examples@db:5432/examples`` — MySQL
   container has no listener on 5432.

   Fix: add ``EXAMPLES_HOST/PORT/DB/USER/PASSWORD`` and a complete
   ``SUPERSET__SQLALCHEMY_EXAMPLES_URI`` to the ``mysql-env`` anchor.

2. The Postgres init scripts under
   ``docker/docker-entrypoint-initdb.d/`` (``cypress-init.sh``,
   ``examples-init.sh``) get mounted on the MySQL container too —
   compose merges volume lists. They invoke ``psql`` which doesn't
   exist in the MySQL image, abort with ``psql: command not found``,
   and prevent the ``examples`` DB from being created.

   Fix: add a MySQL-specific init script
   ``docker/mysql-init/examples-init.sql`` that creates the
   ``examples`` database and user, and mount it at
   ``/docker-entrypoint-initdb.d`` in the override. Compose's
   later-takes-precedence rule on duplicate volume targets displaces
   the Postgres init dir, so the MySQL container only sees the
   MySQL-compatible script.

   (Used a plain duplicate-target mount rather than the ``!override``
   tag because pre-commit's ``check-yaml`` doesn't recognize Compose's
   custom YAML tags.)

Recovery for an existing failed MySQL stack: ``docker compose -f
docker-compose.yml -f docker-compose-mysql.yml down``, then
``docker volume rm superset_db_home_mysql`` (so the new init script
runs on the next fresh boot), then ``up`` again.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 1fc7b17dfa build(docker): add MySQL compose override for dialect-swap evaluation
Adds ``docker-compose-mysql.yml``, a compose-override file that swaps
the default Postgres metadata DB for MySQL 8 with one extra ``-f``
flag:

  docker compose -f docker-compose.yml -f docker-compose-mysql.yml up

Useful for evaluating dialect-specific behaviour (e.g., the runtime
cost of DDL migrations on a deployment whose production metadata DB
is MySQL — the question raised by review feedback on this PR).

Mirrors the connection settings used by CI's ``test-mysql`` shard:
``mysql+mysqldb`` dialect, charset ``utf8mb4`` with binary_prefix.
Host port defaults to 13306 (configurable via ``DATABASE_PORT_MYSQL``)
to avoid colliding with a native MySQL install on 3306.

A separate volume (``db_home_mysql``) keeps MySQL data isolated from
the Postgres ``db_home`` volume, so switching between the two with
``-f`` flag toggles doesn't corrupt either side.

The Postgres-specific init scripts under
``docker/docker-entrypoint-initdb.d/`` are not mounted on the MySQL
service (they are postgres-only). Examples / cypress fixtures still
load via ``superset-init``'s post-startup steps, which run
``superset load-examples`` against whichever metadata DB is in use.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 08cfcf8fd4 docs(UPDATING): add MySQL-targeted maintenance-window queries (sc-105349)
Mirror of the PostgreSQL diagnostic queries added in 11148779ed,
adapted for MySQL/InnoDB. One important difference: InnoDB rebuilds
the clustered index on every PK change, so all eight tables undergo
a full table rebuild on MySQL — not just the two that go through
the explicit ``recreate="always"`` path. The lock-window estimate
query is updated to cover all eight rather than just two, and the
"migration_path" column makes the rebuild expectation explicit
("direct ALTER (still rebuilds InnoDB clustered index)").

Other notes:
- ``information_schema.TABLES.TABLE_ROWS`` is an InnoDB estimate,
  analogous to PostgreSQL's ``reltuples``; documented inline.
- ``KEY_COLUMN_USAGE`` carries both sides of the FK in a single
  row on MySQL, so the external-FK pre-flight check is simpler
  than the PostgreSQL version (no joins between three views).
- The aggregated dedupe query is portable standard SQL; included
  verbatim for copy-paste convenience.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 4eb42d7bdf docs(UPDATING): add Postgres-targeted maintenance-window queries (sc-105349)
Add a "Sizing the maintenance window on PostgreSQL" sub-section to the
operator runbook. The simple per-table COUNT/duplicate/NULL queries
that were already there are dialect-portable but only count rows;
operators on PostgreSQL with large deployments need to characterize
the migration's runtime cost before scheduling it.

Adds four diagnostic queries:

- Per-table size, row count (from pg_class.reltuples), and which
  migration path each table will take (recreate-rewrite vs direct
  ALTER). Sizes the work concretely.
- Aggregated duplicate-row roll-up: dup_groups + total rows_dropped
  per table. Replaces eight separate per-table queries with one
  consolidated result for audit/dump-before-apply decisions.
- External-FK pre-flight check (the same one the migration runs at
  upgrade time and aborts on). Lets operators surface any blocking
  external reference ahead of the maintenance window. Should be
  empty on a stock install.
- Lock-window estimate for the two full-rewrite tables, using
  pg_relation_size and a conservative 100 MB/s rewrite throughput
  assumption. The other six use direct ALTER and are dominated by
  composite-index build time (seconds for low-millions-of-rows
  tables).

Prompted by reviewer feedback on apache/superset#39859 from a large
deployment asking how to size the maintenance window. The original
pre-flight queries are kept for cross-dialect operators (MySQL,
SQLite) since the new queries use PostgreSQL-specific catalog views.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 9f03047913 fix(migration): rebase down_revision onto 33d7e0e21daa (sc-105349)
CI cypress + playwright shards were red with:

  ERROR [flask_migrate] Error: Multiple head revisions are present
  for given argument 'head'

The recent rebase onto master pulled in
``33d7e0e21daa_add_semantic_layers_and_views.py`` (from PR #37815,
"semantic layer extension"), which had been authored against
``ce6bd21901ab`` as its parent — the same parent our migration
referenced. After the rebase both migrations point at
``ce6bd21901ab``, producing two heads and breaking ``flask db
upgrade head`` for any downstream consumer (CI's Cypress / Playwright
shards spin up a real Superset instance via ``superset db upgrade``,
which is why those shards failed first; the integration shards run
against a precomputed schema and didn't surface this).

Fix: chain our migration after the semantic-layer migration by
pointing ``down_revision`` at ``33d7e0e21daa``. The chain is now
linear:

    ... → ce6bd21901ab → 33d7e0e21daa (semantic layers)
                          → 2bee73611e32 (composite PK, this PR)

Verified with ``superset db heads`` (returns single head
``2bee73611e32``) and the local migration test suite (44 passed,
1 skipped).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 7fe649ef68 fix(migration): explicit NOT NULL on FK columns for SQLite (sc-105349)
Found by running fresh-install + round-trip against a real SQLite DB:
6 of the 8 affected tables had FK columns that were originally
declared nullable. PostgreSQL and MySQL implicitly promote the
constituent columns of an ``ALTER TABLE ... ADD PRIMARY KEY`` to
``NOT NULL``; SQLite does not (it's a long-standing SQLite quirk —
only ``INTEGER PRIMARY KEY`` enforces NOT NULL on a composite-PK
column). Result: a fresh SQLite install would accept
``INSERT INTO dashboard_slices (NULL, 5)`` despite both columns
being part of the composite PK.

Our integration tests previously masked this: the test fixture seeds
columns with ``nullable=False``, so the post-upgrade NOT NULL
assertion passed regardless of whether the migration enforced it.

Fix: add explicit ``batch_op.alter_column(fk, nullable=False)`` for
both FK columns inside the per-table batch_alter_table block. On
PostgreSQL and MySQL this is a no-op (PK already implies NOT NULL);
on SQLite it adds the missing NOT NULL declaration so a fresh
install matches the data-model.md "After" contract.

Verified end-to-end:
- Postgres + MySQL: column shape unchanged (still NOT NULL)
- SQLite fresh install + round-trip: all 8 tables now have NOT NULL
  on FK columns, ``INSERT (NULL, 5)`` correctly rejected with
  IntegrityError on dashboard_slices, dashboard_user, sqlatable_user

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 e4e0b4b144 fix(migration): MySQL downgrade FK + AUTO_INCREMENT (sc-105349)
Two MySQL-only failures in the downgrade path, found by running the
full migration history against a fresh MySQL 8 container:

1. ``MySQLdb.OperationalError: (1553, "Cannot drop index 'PRIMARY':
   needed in a foreign key constraint")``. InnoDB uses the composite
   PK index to back the FK on the leftmost column. The downgrade
   tried to drop the composite PK before dropping the FKs, orphaning
   the FK's backing index. PostgreSQL and SQLite create separate
   indexes for FK columns and don't trip on this.

2. ``Field 'id' doesn't have a default value`` on subsequent INSERT.
   ``sa.Identity(always=False)`` only emits ``AUTO_INCREMENT`` on
   MySQL when the column is created with ``primary_key=True`` — our
   portable path adds the column first then creates the PK separately,
   so MySQL leaves the column without auto-generation. Existing rows
   would all collide on id=0; future inserts fail because no default.
   Postgres' ``GENERATED BY DEFAULT AS IDENTITY`` and SQLite's
   ``INTEGER PRIMARY KEY`` rowid alias don't have this gap.

Fix: extract ``_downgrade_mysql_table()`` that emits the canonical
MySQL idiom — drop FKs, then a single ALTER combining
``DROP PRIMARY KEY, ADD COLUMN id INT NOT NULL AUTO_INCREMENT,
ADD PRIMARY KEY (id)`` (which backfills existing rows with sequential
ids and preserves AUTO_INCREMENT), restore the redundant UNIQUE on
the 2 tables that originally had it, and re-add the FKs with their
original names. Postgres and SQLite keep the existing portable
``batch_alter_table`` path.

Raw SQL is unavoidable for the combined-ALTER form; per the
constitution it's allowed for dialect-specific DDL with no SQLA
equivalent, with triple-quoted strings for legibility.

Verified end-to-end: upgrade → downgrade → upgrade against a fresh
MySQL 8 container with INSERT-without-id sanity check showing the
restored ``id`` column auto-increments correctly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 bb996a578e fix(migration): drop FKs before recreate on MySQL (sc-105349)
CI test-mysql failed with:

  MySQLdb.OperationalError: (1826, "Duplicate foreign key constraint
  name 'fk_dashboard_slices_slice_id_slices'")

Root cause: MySQL scopes foreign-key constraint names per-database,
not per-table (PostgreSQL and SQLite scope per-table). The
``batch_alter_table(... recreate="always", copy_from=...)`` path
used for ``dashboard_slices`` and ``report_schedule_user`` builds
``_alembic_tmp_<table>`` carrying the original FK names from
``copy_from`` while the original table still holds those names — MySQL
rejects the temp-table creation with ERROR 1826.

Fix: on MySQL only, drop the original FK constraints by name before
the ``batch_alter_table`` runs. The ``copy_from`` re-creates them on
the rebuilt table with their original names, so the post-migration
shape is unchanged. On PostgreSQL and SQLite the original code path
still runs unchanged.

Local SQLite tests (44 passed, 1 skipped) still pass; CI will validate
on MySQL.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 828fd40eec refactor(migration): build pre-flight SQL via SQLAlchemy core (review)
Address Beto's review comments on apache/superset#39859: replace
``sa.text(f"...")`` SQL construction in the three pre-flight helpers
(``_delete_null_fk_rows``, ``_dedupe_by_min_id``, ``_assert_no_duplicates``)
with SQLAlchemy core constructs (``sa.delete``, ``sa.select``,
``sa.func``, ``.subquery()``, ``.notin_()``).

A small ``_table_clause()`` helper builds a lightweight ``TableClause``
exposing the columns the queries reference; the three helpers consume
it. Removes all ``# noqa: S608`` comments — they are no longer needed
because there is no string-interpolated SQL.

Verified the compiled SQL is identical on Postgres, MySQL, and SQLite,
including the MySQL ERROR 1093 workaround (the inner aggregation is
wrapped in a derived table via ``.subquery()``, producing
``... NOT IN (SELECT keep_id FROM (SELECT min(id) ...) AS keep_min)``).

Also drops the redundant ``f`` prefix on the two non-interpolating
lines of the ``_check_no_external_fks_to_id`` error message.

44 migration tests still pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 229ed98a1b docs(migration): address SQLAlchemy review follow-ups
Four operator-experience improvements from the second review pass:

1. ``TABLES_WITH_NULLABLE_FKS`` is now explicitly documented as an
   informational set that is not consulted at runtime; the comment
   explains the previous ``dashboard_roles`` omission was the bug
   that motivated the always-run cleanup.
2. ``_delete_null_fk_rows`` docstring updated to match the
   "always run" semantics (was still claiming "called only on tables
   in TABLES_WITH_NULLABLE_FKS").
3. ``_check_no_external_fks_to_id`` now documents its scope
   limitation: ``Inspector.get_table_names()`` returns the default
   schema only, so cross-schema FKs in non-standard multi-schema
   PostgreSQL deployments would not be caught. The single-schema
   case (Superset's documented deployment) is fully covered.
4. ``_dedupe_by_min_id`` now logs a sample of up to 10 discarded
   ``(fk1, fk2, id)`` tuples at WARN before deletion, so operators
   can audit which rows the ``MIN(id)`` policy drops. The keep-
   original policy is correct in practice but discards later
   re-grants on ownership tables; the sample makes that visible.
5. ``UPDATING.md`` documents the upgrade/downgrade primary-key
   name divergence (``pk_<table>`` vs ``<table>_pkey``) so
   operators using schema-comparison tools don't mistake it for
   migration drift.

No schema or runtime-behaviour changes. All 44 migration tests pass.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 7b2c2d3b00 fix(migration): always run NULL-FK cleanup; correct RLS test parent name
Two cleanups from PR review:

1. ``dashboard_roles.dashboard_id`` was created nullable in revision
   e11ccdd12658 but was missing from ``TABLES_WITH_NULLABLE_FKS``. A
   production database with a stray NULL ``dashboard_id`` row would have
   failed the PK-add with a cryptic constraint violation. Fix by running
   the NULL-FK cleanup on every affected table — it is a no-op DELETE on
   tables whose FK columns are already NOT NULL, and it eliminates the
   risk of further drift in the hardcoded set. ``dashboard_roles`` is
   added to the documentation set; the runtime now does not consult it.

2. The unit-test parent-table name for ``rls_filter_roles`` and
   ``rls_filter_tables`` was ``rls_filter`` (does not exist) instead of
   the real parent ``row_level_security_filters``. Test passes either
   way (the in-memory FK is self-consistent), but the parameter is now
   accurate.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
Mike BridgeandClaude Opus 4.7 f537472291 refactor(db): composite PK on M2M association tables (sc-105349)
Replace synthetic id INTEGER PRIMARY KEY with composite PRIMARY KEY (fk1, fk2)
on the eight pure-junction tables: dashboard_roles, dashboard_slices,
dashboard_user, report_schedule_user, rls_filter_roles, rls_filter_tables,
slice_user, sqlatable_user. The redundant UNIQUE(fk1, fk2) on dashboard_slices
and report_schedule_user is dropped (subsumed by the new PK).

Migration handles dialect quirks: copy_from for tables with pre-existing
UNIQUE (so SQLite's anonymous-constraint reflection doesn't matter), wrapped-
subquery dedupe for MySQL (ERROR 1093), sa.Identity(always=False) on downgrade
to backfill the restored id column without NOT NULL violations, and distinct
PK names per direction (pk_<table> on upgrade, <table>_pkey on downgrade) to
avoid round-trip index-name collisions on Postgres.

ORM Table() definitions updated to match. UPDATING.md entry added with
operator runbook (BI-tool impact, pre-flight inventory queries, dedupe-row-
loss notice, pg_dump workaround, FK-NOT-NULL downgrade asymmetry note).

Tests: 8 schema-shape assertions (post-upgrade), 8 duplicate-rejection unit
tests, 8 distinct-pair sanity tests, 1 round-trip + idempotency test
(in-memory SQLite via Alembic MigrationContext).

Continuum-restore verification against the new shape is out of scope for this
PR; it is the responsibility of the versioning epic (sc-103156).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
2026-05-28 13:39:09 -06:00
1344 changed files with 62122 additions and 202822 deletions
+10 -4
View File
@@ -77,17 +77,23 @@ github:
# combination here.
contexts:
- lint-check
- cypress-matrix-required
- cypress-matrix (0, chrome)
- cypress-matrix (1, chrome)
- cypress-matrix (2, chrome)
- cypress-matrix (3, chrome)
- cypress-matrix (4, chrome)
- cypress-matrix (5, chrome)
- dependency-review
- frontend-build
- playwright-tests-required
- playwright-tests (chromium)
- pre-commit (current)
- pre-commit (previous)
- test-mysql
- test-postgres-required
- test-postgres (current)
- test-postgres-hive
- test-postgres-presto
- test-sqlite
- unit-tests-required
- unit-tests (current)
required_pull_request_reviews:
dismiss_stale_reviews: false
+1 -1
View File
@@ -41,8 +41,8 @@ body:
label: Superset version
options:
- master / latest-dev
- "6.1.0"
- "6.0.0"
- "5.0.0"
validations:
required: true
- type: dropdown
+75
View File
@@ -0,0 +1,75 @@
# Security Policy
This is a project of the [Apache Software Foundation](https://apache.org) and follows the
ASF [vulnerability handling process](https://apache.org/security/#vulnerability-handling).
## Reporting Vulnerabilities
**⚠️ Please do not file GitHub issues for security vulnerabilities as they are public! ⚠️**
Apache Software Foundation takes a rigorous standpoint in annihilating the security issues
in its software projects. Apache Superset is highly sensitive and forthcoming to issues
pertaining to its features and functionality.
If you have any concern or believe you have found a vulnerability in Apache Superset,
please get in touch with the Apache Superset Security Team privately at
e-mail address [security@superset.apache.org](mailto:security@superset.apache.org).
More details can be found on the ASF website at
[ASF vulnerability reporting process](https://apache.org/security/#reporting-a-vulnerability)
**Submission Standards & AI Policy**
To ensure engineering focus remains on verified risks and to manage high reporting volumes, all reports must meet the following criteria:
- Plain Text Format: In accordance with Apache guidelines, please provide all details in plain text within the email body. Avoid sending PDFs, Word documents, or password-protected archives.
- Mandatory AI Disclosure: If you utilized Large Language Models (LLMs) or AI tools to identify a flaw or assist in writing a report, you must disclose this in your submission so our triage team can contextualize the findings.
- Human-Verified PoC: All submissions must include a manual, step-by-step Proof of Concept (PoC) performed on a supported release. Raw AI outputs, hypothetical chat transcripts, or unverified scanner logs will be closed as Invalid.
We kindly ask you to include the following information in your report to assist our developers in triaging and remediating issues efficiently:
- Version/Commit: The specific version of Apache Superset or the Git commit hash you are using.
- Configuration: A sanitized copy of your `superset_config.py` file or any config overrides.
- Environment: Your deployment method (e.g., Docker Compose, Helm, or source) and relevant OS/Browser details.
- Impacted Component: Identification of the affected area (e.g., Python backend, React frontend, or a specific database connector).
- Expected vs. Actual Behavior: A clear description of the intended system behavior versus the observed vulnerability.
- Detailed Reproduction Steps: Clear, manual steps to reproduce the vulnerability.
**Vulnerability Definition**
Apache Superset considers a security vulnerability to be a demonstrable issue that has meaningful impact on confidentiality, integrity, or availability beyond the intended security model. Low-impact boundary variations or technical edge cases in existing access controls may be classified as hardening improvements rather than vulnerabilities, even if exploitable.
**Out of Scope Vulnerabilities**
To prioritize engineering efforts on genuine architectural risks, the following scenarios are explicitly out of scope and will not be issued a CVE:
- **Attacks requiring Admin privileges**: (e.g., CSS injection, template manipulation, dashboard ownership overrides, or modifying global system settings). Per the CVE vulnerability definition in CNA Operational Rules 4.1, a qualifying vulnerability must allow violation of a security policy. The Admin role is a fully trusted operational boundary defined by Apache Superset's security policy; actions within this boundary do not violate that policy and are therefore considered intended capabilities 'by design,' not vulnerabilities.
- **Brute Force and Rate Limiting**: Reports targeting a lack of resource exhaustion protections, generic rate-limiting, or volumetric Denial of Service (DoS) attempts.
- **Theoretical attack vectors**: Issues without a demonstrable, reproducible exploit path.
- **Non-Exploitable Findings**: Missing security headers, generic banner disclosures, or descriptive error messages that do not lead to a direct, documented exploit.
- **User enumeration**: API responses, timing differences, or error messages that reveal whether user accounts, IDs, dashboards, or datasets exist.
- **Information disclosure (low impact)**: Software version disclosure, generic error messages, stack traces without sensitive data exposure, or system configuration details that don't enable further exploitation.
- **Resource exhaustion requiring authentication**: Denial of Service attacks that require valid user credentials and don't bypass rate limiting or resource controls.
- **Missing security headers**: Without demonstration of a concrete exploit scenario that leverages the missing header.
**Outcome of Reports**
Reports that are deemed out-of-scope for a CVE but represent valid security best practices or hardening opportunities may be converted into public GitHub issues. This allows the community to contribute to the general hardening of the platform even when a specific vulnerability threshold is not met.
Note that Apache Superset is not responsible for any third-party dependencies that may
have security issues. Any vulnerabilities found in third-party dependencies should be
reported to the maintainers of those projects. Results from security scans of Apache
Superset dependencies found on its official Docker image can be remediated at release time
by extending the image itself.
**Vulnerability Aggregation & CVE Attribution**
In accordance with MITRE CNA Operational Rules (4.1.10, 4.1.11, and 4.2.13), Apache Superset issues CVEs based on the underlying architectural root cause rather than the number of affected endpoints or exploit payloads.
- Aggregation: If multiple exploit vectors stem from the same programmatic failure or shared vulnerable code, they must be aggregated into a single, comprehensive report.
- Independent Fixes: Separate CVEs will only be assigned if the vulnerabilities reside in decoupled architectural modules and can be fixed independently of one another.
Reports that fail to aggregate related findings will be merged during triage to ensure an accurate and defensible CVE record.
**Your responsible disclosure and collaboration are invaluable.**
## Extra Information
- [Apache Superset documentation](https://superset.apache.org/docs/security)
- [Common Vulnerabilities and Exposures by release](https://superset.apache.org/docs/security/cves)
- [How Security Vulnerabilities are Reported & Handled in Apache Superset (Blog)](https://preset.io/blog/how-security-vulnerabilities-are-reported-and-handled-in-apache-superset/)
+13 -22
View File
@@ -24,41 +24,32 @@ runs:
- name: Interpret Python Version
id: set-python-version
shell: bash
env:
INPUT_PYTHON_VERSION: ${{ inputs.python-version }}
run: |
if [ "$INPUT_PYTHON_VERSION" = "current" ]; then
RESOLVED_VERSION="3.11"
elif [ "$INPUT_PYTHON_VERSION" = "next" ]; then
if [ "${{ inputs.python-version }}" = "current" ]; then
echo "PYTHON_VERSION=3.11" >> $GITHUB_ENV
elif [ "${{ inputs.python-version }}" = "next" ]; then
# currently disabled in GHA matrixes because of library compatibility issues
RESOLVED_VERSION="3.12"
elif [ "$INPUT_PYTHON_VERSION" = "previous" ]; then
RESOLVED_VERSION="3.10"
elif printf '%s' "$INPUT_PYTHON_VERSION" | grep -Eq '^[0-9]+\.[0-9]+(\.[0-9]+)?$'; then
RESOLVED_VERSION="$INPUT_PYTHON_VERSION"
echo "PYTHON_VERSION=3.12" >> $GITHUB_ENV
elif [ "${{ inputs.python-version }}" = "previous" ]; then
echo "PYTHON_VERSION=3.10" >> $GITHUB_ENV
else
echo "Invalid python-version: '$INPUT_PYTHON_VERSION'" >&2
exit 1
echo "PYTHON_VERSION=${{ inputs.python-version }}" >> $GITHUB_ENV
fi
echo "python-version=$RESOLVED_VERSION" >> "$GITHUB_OUTPUT"
- name: Set up Python ${{ steps.set-python-version.outputs.python-version }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
- name: Set up Python ${{ env.PYTHON_VERSION }}
uses: actions/setup-python@v5
with:
python-version: ${{ steps.set-python-version.outputs.python-version }}
python-version: ${{ env.PYTHON_VERSION }}
cache: ${{ inputs.cache }}
- name: Install dependencies
env:
INPUT_INSTALL_SUPERSET: ${{ inputs.install-superset }}
INPUT_REQUIREMENTS_TYPE: ${{ inputs.requirements-type }}
run: |
if [ "$INPUT_INSTALL_SUPERSET" = "true" ]; then
if [ "${{ inputs.install-superset }}" = "true" ]; then
sudo apt-get update && sudo apt-get -y install libldap2-dev libsasl2-dev
pip install --upgrade pip setuptools wheel uv
if [ "$INPUT_REQUIREMENTS_TYPE" = "dev" ]; then
if [ "${{ inputs.requirements-type }}" = "dev" ]; then
uv pip install --system -r requirements/development.txt
elif [ "$INPUT_REQUIREMENTS_TYPE" = "base" ]; then
elif [ "${{ inputs.requirements-type }}" = "base" ]; then
uv pip install --system -r requirements/base.txt
fi
+3 -3
View File
@@ -26,7 +26,7 @@ runs:
- name: Set up QEMU
if: ${{ inputs.build == 'true' }}
uses: docker/setup-qemu-action@06116385d9baf250c9f4dcb4858b16962ea869c3 # v4.1.0
uses: docker/setup-qemu-action@29109295f81e9208d7d86ff1c6c12d2833863392 # v3.6.0
with:
# Pin the binfmt image to a specific QEMU release. The default
# (`tonistiigi/binfmt:latest`) is a moving target, and drift across
@@ -39,12 +39,12 @@ runs:
- name: Set up Docker Buildx
if: ${{ inputs.build == 'true' }}
uses: docker/setup-buildx-action@d7f5e7f509e45cec5c76c4d5afdd7de93d0b3df5 # v4.1.0
uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3.12.0
- name: Try to login to DockerHub
if: ${{ inputs.login-to-dockerhub == 'true' }}
continue-on-error: true
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3.7.0
with:
username: ${{ inputs.dockerhub-user }}
password: ${{ inputs.dockerhub-token }}
+2 -3
View File
@@ -10,7 +10,7 @@ runs:
steps:
- name: Setup Node Env
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
uses: actions/setup-node@v4
with:
node-version: '20'
@@ -21,9 +21,8 @@ runs:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: ${{ inputs.from-npm == 'false' }}
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
uses: actions/checkout@v4
with:
persist-credentials: false
repository: apache-superset/supersetbot
path: supersetbot
+62 -30
View File
@@ -3,13 +3,23 @@ enable-beta-ecosystems: true
updates:
- package-ecosystem: "github-actions"
directory: "/"
ignore:
# Ignore temporarily as release schedule is too mentally taxing for dep-handling maintainers
# Additionally, very few PRs are reviewed by this action.
- dependency-name: anthropics/claude-code-action
schedule:
interval: "daily"
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
ignore:
# TODO: remove below entries until React >= 18.0.0
- dependency-name: "storybook"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- dependency-name: "@storybook*"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- dependency-name: "eslint-plugin-storybook"
- dependency-name: "react-error-boundary"
- dependency-name: "@rjsf/*"
# remark-gfm v4+ requires react-markdown v9+, which needs React 18
@@ -32,6 +42,14 @@ updates:
# and confirm the issue https://github.com/apache/superset/issues/39600 is fixed
- dependency-name: "react-checkbox-tree"
update-types: ["version-update:semver-major"]
groups:
storybook:
applies-to: version-updates
patterns:
- "@storybook*"
- "storybook"
update-types:
- "patch"
directory: "/superset-frontend/"
schedule:
interval: "daily"
@@ -41,7 +59,7 @@ updates:
open-pull-requests-limit: 30
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "pip"
@@ -58,7 +76,7 @@ updates:
- pip
- dependabot
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: ".github/actions"
@@ -67,18 +85,32 @@ updates:
open-pull-requests-limit: 10
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/docs/"
ignore:
# TODO: remove below entries until React >= 18.0.0 in superset-frontend
- dependency-name: "storybook"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- dependency-name: "@storybook*"
update-types: ["version-update:semver-major", "version-update:semver-minor"]
- dependency-name: "eslint-plugin-storybook"
- dependency-name: "react-error-boundary"
groups:
storybook:
applies-to: version-updates
patterns:
- "@storybook*"
- "storybook"
update-types:
- "patch"
schedule:
interval: "daily"
open-pull-requests-limit: 10
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-websocket/"
@@ -89,7 +121,7 @@ updates:
- dependabot
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-websocket/utils/client-ws-app/"
@@ -101,7 +133,7 @@ updates:
open-pull-requests-limit: 10
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
# Now for all of our plugins and packages!
@@ -115,7 +147,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-partition/"
@@ -127,7 +159,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-world-map/"
@@ -139,7 +171,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-pivot-table/"
@@ -154,7 +186,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-chord/"
@@ -166,7 +198,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-horizon/"
@@ -178,7 +210,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-rose/"
@@ -190,7 +222,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-preset-chart-deckgl/"
@@ -202,7 +234,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-table/"
@@ -217,7 +249,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-country-map/"
@@ -229,7 +261,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-map-box/"
@@ -241,7 +273,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-preset-chart-nvd3/"
@@ -253,7 +285,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-word-cloud/"
@@ -265,7 +297,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-paired-t-test/"
@@ -277,7 +309,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-echarts/"
@@ -289,7 +321,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-ag-grid-table/"
@@ -301,7 +333,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-cartodiagram/"
@@ -313,7 +345,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-parallel-coordinates/"
@@ -325,7 +357,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-handlebars/"
@@ -341,7 +373,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/generator-superset/"
@@ -353,7 +385,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-chart-controls/"
@@ -365,7 +397,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-core/"
@@ -382,7 +414,7 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-switchboard/"
@@ -394,4 +426,4 @@ updates:
open-pull-requests-limit: 5
versioning-strategy: increase
cooldown:
default-days: 7
default-days: 5
+7 -17
View File
@@ -20,6 +20,10 @@ set -e
GITHUB_WORKSPACE=${GITHUB_WORKSPACE:-.}
ASSETS_MANIFEST="$GITHUB_WORKSPACE/superset/static/assets/manifest.json"
# Rounded job start time, used to create a unique Cypress build id for
# parallelization so we can manually rerun a job after 20 minutes
NONCE=$(echo "$(date "+%Y%m%d%H%M") - ($(date +%M)%20)" | bc)
# Echo only when not in parallel mode
say() {
if [[ $(echo "$INPUT_PARALLEL" | tr '[:lower:]' '[:upper:]') != 'TRUE' ]]; then
@@ -55,15 +59,6 @@ build-assets() {
say "::endgroup::"
}
build-embedded-sdk() {
cd "$GITHUB_WORKSPACE/superset-embedded-sdk"
say "::group::Build embedded SDK bundle for E2E tests"
npm ci
npm run build
say "::endgroup::"
}
build-instrumented-assets() {
cd "$GITHUB_WORKSPACE/superset-frontend"
@@ -114,7 +109,7 @@ testdata() {
say "::group::Load test data"
# must specify PYTHONPATH to make `tests.superset_test_config` importable
export PYTHONPATH="$GITHUB_WORKSPACE"
uv pip install --system -e .
pip install -e .
superset db upgrade
superset load_test_users
superset load_examples --load-test-data
@@ -127,7 +122,7 @@ playwright_testdata() {
say "::group::Load all examples for Playwright tests"
# must specify PYTHONPATH to make `tests.superset_test_config` importable
export PYTHONPATH="$GITHUB_WORKSPACE"
uv pip install --system -e .
pip install -e .
superset db upgrade
superset load_test_users
superset load_examples
@@ -281,12 +276,7 @@ playwright-run() {
cd "$GITHUB_WORKSPACE"
local serverlog="${HOME}/superset-playwright.log"
local port=8081
# Use 127.0.0.1 explicitly: `flask run` binds IPv4 only, and Node's DNS
# resolution for `localhost` can return `::1` first (IPv6), which then
# refuses against the IPv4 listener and surfaces as
# `connect ECONNREFUSED ::1:<port>` in API helpers driven from Node
# (e.g., the embedded test app's exposed token fetcher).
PLAYWRIGHT_BASE_URL="http://127.0.0.1:${port}"
PLAYWRIGHT_BASE_URL="http://localhost:${port}"
if [ -n "$APP_ROOT" ]; then
export SUPERSET_APP_ROOT=$APP_ROOT
PLAYWRIGHT_BASE_URL=${PLAYWRIGHT_BASE_URL}${APP_ROOT}/
+3 -2
View File
@@ -30,8 +30,9 @@ jobs:
pull-requests: write
checks: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: true
ref: master
@@ -40,7 +41,7 @@ jobs:
uses: ./.github/actions/setup-supersetbot/
- name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6.2.0
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: "3.10"
+43
View File
@@ -0,0 +1,43 @@
name: Cancel Duplicates
on:
workflow_run:
workflows:
- "Miscellaneous"
types:
- requested
jobs:
cancel-duplicate-runs:
name: Cancel duplicate workflow runs
runs-on: ubuntu-24.04
permissions:
actions: write
contents: read
steps:
- name: Check number of queued tasks
id: check_queued
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPO: ${{ github.repository }}
run: |
get_count() {
echo $(curl -s -H "Authorization: token $GITHUB_TOKEN" \
"https://api.github.com/repos/$GITHUB_REPO/actions/runs?status=$1" | \
jq ".total_count")
}
count=$(( `get_count queued` + `get_count in_progress` ))
echo "Found $count unfinished jobs."
echo "count=$count" >> $GITHUB_OUTPUT
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: steps.check_queued.outputs.count >= 20
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Cancel duplicate workflow runs
if: steps.check_queued.outputs.count >= 20
env:
GITHUB_TOKEN: ${{ github.token }}
GITHUB_REPOSITORY: ${{ github.repository }}
run: |
pip install click requests typing_extensions python-dateutil
python ./scripts/cancel_github_workflows.py
+1 -14
View File
@@ -22,7 +22,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -38,19 +38,6 @@ jobs:
if: steps.check.outputs.python
uses: ./.github/actions/setup-backend/
# Authenticate the Docker daemon so the python:slim pull in
# uv-pip-compile.sh uses our (much higher) authenticated rate limit
# instead of the shared-runner anonymous one. Best-effort: on fork PRs the
# secrets are unavailable, so this no-ops and the pull falls back to
# anonymous (covered by the retry loop in the script).
- name: Login to Docker Hub
if: steps.check.outputs.python
continue-on-error: true
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Run uv
if: steps.check.outputs.python
run: ./scripts/uv-pip-compile.sh
@@ -25,9 +25,7 @@ jobs:
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Check and notify
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
+84
View File
@@ -0,0 +1,84 @@
name: Claude PR Assistant
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
jobs:
check-permissions:
if: |
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude'))
runs-on: ubuntu-latest
outputs:
allowed: ${{ steps.check.outputs.allowed }}
steps:
- name: Check if user is allowed
id: check
env:
COMMENTER: ${{ github.event.comment.user.login }}
run: |
# List of allowed users
ALLOWED_USERS="mistercrunch,rusackas"
echo "Checking permissions for user: $COMMENTER"
# Check if user is in allowed list
if [[ ",$ALLOWED_USERS," == *",$COMMENTER,"* ]]; then
echo "allowed=true" >> $GITHUB_OUTPUT
echo "✅ User $COMMENTER is allowed to use Claude"
else
echo "allowed=false" >> $GITHUB_OUTPUT
echo "❌ User $COMMENTER is not allowed to use Claude"
fi
deny-access:
needs: check-permissions
if: needs.check-permissions.outputs.allowed == 'false'
runs-on: ubuntu-latest
permissions:
issues: write
pull-requests: write
steps:
- name: Comment access denied
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
env:
COMMENTER_LOGIN: ${{ github.event.comment.user.login || github.event.review.user.login || github.event.issue.user.login }}
with:
script: |
const commenter = process.env.COMMENTER_LOGIN;
const message = `👋 Hi @${commenter}!
Thanks for trying to use Claude Code, but currently only certain team members have access to this feature.
If you believe you should have access, please contact a project maintainer.`;
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: message
});
claude-code-action:
needs: check-permissions
if: needs.check-permissions.outputs.allowed == 'true'
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
issues: write
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
fetch-depth: 1
- name: Run Claude PR Action
uses: anthropics/claude-code-action@5fb899572b81d2bb648d4d187173a2f423a9677c # beta
with:
anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
timeout_minutes: "60"
+9 -31
View File
@@ -15,39 +15,12 @@ concurrency:
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
analyze:
name: Analyze
needs: changes
# Skip on PRs that touch neither code group (e.g. docs-only) so the
# analysis runners don't spin up. push/schedule runs always proceed:
# the change-detector returns "all changed" for non-PR events.
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
actions: read
contents: read
pull-requests: read
security-events: write
strategy:
@@ -58,13 +31,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
persist-credentials: false
token: ${{ secrets.GITHUB_TOKEN }}
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
uses: github/codeql-action/init@7211b7c8077ea37d8641b6271f6a365a22a5fbfa # v4
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -75,6 +52,7 @@ jobs:
# queries: security-extended,security-and-quality
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@8aad20d150bbac5944a9f9d289da16a4b0d87c1e # v4
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: github/codeql-action/analyze@7211b7c8077ea37d8641b6271f6a365a22a5fbfa # v4
with:
category: "/language:${{matrix.language}}"
+3 -7
View File
@@ -27,9 +27,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout Repository"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: "Dependency Review"
uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5.0.0
continue-on-error: true
@@ -43,7 +41,7 @@ jobs:
# the latest version. It's MIT: https://github.com/nbubna/store/blob/master/LICENSE-MIT
# pkg:npm/node-forge@1.3.1
# selecting BSD-3-Clause licensing terms for node-forge to ensure compatibility with Apache
allow-dependencies-licenses: pkg:npm/rgbcolor, pkg:npm/jszip@3.10.1
allow-dependencies-licenses: pkg:npm/store2@2.14.2, pkg:npm/node-forge@1.3.1, pkg:npm/rgbcolor, pkg:npm/jszip@3.10.1
python-dependency-liccheck:
# NOTE: Configuration for liccheck lives in our pyproject.yml.
@@ -51,9 +49,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: "Checkout Repository"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Setup Python
uses: ./.github/actions/setup-backend/
+34 -98
View File
@@ -18,30 +18,9 @@ concurrency:
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
docker: ${{ steps.check.outputs.docker }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
setup_matrix:
runs-on: ubuntu-24.04
timeout-minutes: 5
outputs:
matrix_config: ${{ steps.set_matrix.outputs.matrix_config }}
steps:
@@ -53,13 +32,8 @@ jobs:
docker-build:
name: docker-build
needs: [setup_matrix, changes]
if: >-
needs.changes.outputs.python == 'true' ||
needs.changes.outputs.frontend == 'true' ||
needs.changes.outputs.docker == 'true'
needs: setup_matrix
runs-on: ubuntu-24.04
timeout-minutes: 60
strategy:
matrix:
build_preset: ${{fromJson(needs.setup_matrix.outputs.matrix_config)}}
@@ -70,30 +44,20 @@ jobs:
IMAGE_TAG: apache/superset:GHA-${{ matrix.build_preset }}-${{ github.run_id }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- name: Free up disk space
shell: bash
run: |
# Reclaim large preinstalled toolchains we don't use. The image
# build, and especially the docker-compose sanity check (which
# rebuilds from scratch whenever the registry cache image
# apache/superset-cache is unavailable), can otherwise exhaust the
# runner's root disk and fail with "no space left on device".
echo "Disk before cleanup:"; df -h /
sudo rm -rf \
/usr/share/dotnet \
/usr/local/lib/android \
/opt/ghc \
/usr/local/.ghcup \
/opt/hostedtoolcache/CodeQL \
/usr/local/share/boost || true
echo "Disk after cleanup:"; df -h /
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Docker Environment
if: steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
@@ -101,56 +65,40 @@ jobs:
build: "true"
- name: Setup supersetbot
if: steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker
uses: ./.github/actions/setup-supersetbot/
- name: Build Docker Image
if: steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker
shell: bash
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
BUILD_PRESET: ${{ matrix.build_preset }}
run: |
# Single platform builds in pull_request context to speed things up
if [ "$GITHUB_EVENT_NAME" = "push" ]; then
if [ "${{ github.event_name }}" = "push" ]; then
PLATFORM_ARG="--platform linux/arm64 --platform linux/amd64"
# can only --load images in single-platform builds
PUSH_OR_LOAD="--push"
elif [ "$GITHUB_EVENT_NAME" = "pull_request" ]; then
elif [ "${{ github.event_name }}" = "pull_request" ]; then
PLATFORM_ARG="--platform linux/amd64"
PUSH_OR_LOAD="--load"
fi
# Retry to absorb transient Docker Hub registry errors (base-image
# pull timeouts, 504/401 on push, ECONNRESET) that otherwise fail
# the whole job. buildx reuses the buildkit layer cache from the
# failed attempt, so a retry mostly re-does just the failed push.
for attempt in 1 2 3; do
if supersetbot docker \
$PUSH_OR_LOAD \
--preset "$BUILD_PRESET" \
--context "$EVENT" \
--context-ref "$RELEASE" $FORCE_LATEST \
--extra-flags "--build-arg INCLUDE_CHROMIUM=false --tag $IMAGE_TAG" \
$PLATFORM_ARG; then
break
fi
if [ "$attempt" -eq 3 ]; then
echo "::error::supersetbot docker build failed after 3 attempts"
exit 1
fi
echo "::warning::Build attempt ${attempt} failed; retrying in 30s..."
sleep 30
done
supersetbot docker \
$PUSH_OR_LOAD \
--preset ${{ matrix.build_preset }} \
--context "$EVENT" \
--context-ref "$RELEASE" $FORCE_LATEST \
--extra-flags "--build-arg INCLUDE_CHROMIUM=false --tag $IMAGE_TAG" \
$PLATFORM_ARG
# in the context of push (using multi-platform build), we need to pull the image locally
- name: Docker pull
if: github.event_name == 'push'
run: |
for i in 1 2 3; do
docker pull $IMAGE_TAG && break
[ $i -lt 3 ] && sleep 30
done
if: github.event_name == 'push' && (steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker)
run: docker pull $IMAGE_TAG
- name: Print docker stats
if: steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker
run: |
echo "SHA: ${{ github.sha }}"
echo "IMAGE: $IMAGE_TAG"
@@ -158,12 +106,10 @@ jobs:
docker history $IMAGE_TAG
- name: docker-compose sanity check
if: matrix.build_preset == 'dev'
if: (steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker) && matrix.build_preset == 'dev'
shell: bash
env:
BUILD_PRESET: ${{ matrix.build_preset }}
run: |
export SUPERSET_BUILD_TARGET=$BUILD_PRESET
export SUPERSET_BUILD_TARGET=${{ matrix.build_preset }}
# This should reuse the CACHED image built in the previous steps
docker compose build superset-init --build-arg DEV_MODE=false --build-arg INCLUDE_CHROMIUM=false
docker compose up superset-init --exit-code-from superset-init
@@ -171,31 +117,20 @@ jobs:
docker-compose-image-tag:
# Run this job only on pushes to master (not for PRs)
# goal is to check that building the latest image works, not required for all PR pushes
needs: changes
if: github.event_name == 'push' && github.ref == 'refs/heads/master' && needs.changes.outputs.docker == 'true'
if: github.event_name == 'push' && github.ref == 'refs/heads/master'
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- name: Free up disk space
shell: bash
run: |
# The sanity check rebuilds the image from scratch whenever the
# registry cache image apache/superset-cache is unavailable, which
# can exhaust the runner's root disk ("no space left on device").
echo "Disk before cleanup:"; df -h /
sudo rm -rf \
/usr/share/dotnet \
/usr/local/lib/android \
/opt/ghc \
/usr/local/.ghcup \
/opt/hostedtoolcache/CodeQL \
/usr/local/share/boost || true
echo "Disk after cleanup:"; df -h /
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Docker Environment
if: steps.check.outputs.docker
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
@@ -203,6 +138,7 @@ jobs:
build: "false"
install-docker-compose: "true"
- name: docker-compose sanity check
if: steps.check.outputs.docker
shell: bash
run: |
docker compose -f docker-compose-image-tag.yml up superset-init --exit-code-from superset-init
+22 -16
View File
@@ -10,29 +10,35 @@ permissions:
contents: read
jobs:
config:
runs-on: ubuntu-24.04
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${NPM_TOKEN}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
NPM_TOKEN: ${{ (secrets.NPM_TOKEN != '') || '' }}
build:
# Publishing uses npm trusted publishing (OIDC), so there is no NPM_TOKEN to
# gate on. Restrict to the canonical repo: forks cannot mint a valid OIDC
# token for this package and must not publish.
if: github.repository == 'apache/superset'
needs: config
if: needs.config.outputs.has-secrets
runs-on: ubuntu-24.04
permissions:
contents: read
id-token: write # required for npm trusted publishing (OIDC)
defaults:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
# Note: registry-url is intentionally omitted. When set, actions/setup-node
# writes an .npmrc with `_authToken=${NODE_AUTH_TOKEN}` and a placeholder
# token, which makes npm attempt token auth and skip the OIDC
# trusted-publishing exchange. With no .npmrc auth line, npm authenticates
# via OIDC against the default registry (registry.npmjs.org).
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-embedded-sdk/.nvmrc"
node-version-file: './superset-embedded-sdk/.nvmrc'
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm run ci:release
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+3 -5
View File
@@ -21,13 +21,11 @@ jobs:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-embedded-sdk/.nvmrc"
registry-url: "https://registry.npmjs.org"
node-version-file: './superset-embedded-sdk/.nvmrc'
registry-url: 'https://registry.npmjs.org'
- run: npm ci
- run: npm test
- run: npm run build
@@ -0,0 +1,83 @@
name: Cleanup ephemeral envs (PR close) [DEPRECATED]
# ⚠️ DEPRECATION NOTICE ⚠️
# This workflow is deprecated and will be removed in a future version.
# The new Superset Showtime workflow handles cleanup automatically.
# See .github/workflows/showtime.yml and showtime-cleanup.yml for replacements.
# Migration guide: https://github.com/mistercrunch/superset-showtime
on:
pull_request_target:
types: [closed]
jobs:
config:
runs-on: ubuntu-24.04
outputs:
has-secrets: ${{ steps.check.outputs.has-secrets }}
steps:
- name: "Check for secrets"
id: check
shell: bash
run: |
if [ -n "${AWS_ACCESS_KEY_ID}" ]; then
echo "has-secrets=1" >> "$GITHUB_OUTPUT"
fi
env:
AWS_ACCESS_KEY_ID: ${{ (secrets.AWS_ACCESS_KEY_ID != '' && secrets.AWS_SECRET_ACCESS_KEY != '') || '' }}
ephemeral-env-cleanup:
needs: config
if: needs.config.outputs.has-secrets
name: Cleanup ephemeral envs
runs-on: ubuntu-24.04
permissions:
pull-requests: write
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # v6
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Describe ECS service
id: describe-services
run: |
echo "active=$(aws ecs describe-services --cluster superset-ci --services pr-${{ github.event.number }}-service | jq '.services[] | select(.status == "ACTIVE") | any')" >> $GITHUB_OUTPUT
- name: Delete ECS service
if: steps.describe-services.outputs.active == 'true'
id: delete-service
run: |
aws ecs delete-service \
--cluster superset-ci \
--service pr-${{ github.event.number }}-service \
--force
- name: Login to Amazon ECR
if: steps.describe-services.outputs.active == 'true'
id: login-ecr
uses: aws-actions/amazon-ecr-login@fa648b43de3d4d023bcb3f89ed6940096949c419 # v2
- name: Delete ECR image tag
if: steps.describe-services.outputs.active == 'true'
id: delete-image-tag
run: |
aws ecr batch-delete-image \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-${{ github.event.number }}
- name: Comment (success)
if: steps.describe-services.outputs.active == 'true'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{github.token}}
script: |
github.rest.issues.createComment({
issue_number: ${{ github.event.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '⚠️ **DEPRECATED WORKFLOW** - Ephemeral environment shutdown and build artifacts deleted. Please migrate to the new Superset Showtime system for future PRs.'
})
+350
View File
@@ -0,0 +1,350 @@
name: Ephemeral env workflow [DEPRECATED]
# ⚠️ DEPRECATION NOTICE ⚠️
# This workflow is deprecated and will be removed in a future version.
# Please use the new Superset Showtime workflow instead:
# - Use label "🎪 trigger-start" instead of "testenv-up"
# - Showtime provides better reliability and easier management
# - See .github/workflows/showtime.yml for the replacement
# - Migration guide: https://github.com/mistercrunch/superset-showtime
# Example manual trigger:
# gh workflow run ephemeral-env.yml --ref fix_ephemerals --field label_name="testenv-up" --field issue_number=666
on:
pull_request_target:
types:
- labeled
workflow_dispatch:
inputs:
label_name:
description: 'Label name to simulate label-based /testenv trigger'
required: true
default: 'testenv-up'
issue_number:
description: 'Issue or PR number'
required: true
jobs:
ephemeral-env-label:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}-label
cancel-in-progress: true
name: Evaluate ephemeral env label trigger
runs-on: ubuntu-24.04
permissions:
pull-requests: write
outputs:
slash-command: ${{ steps.eval-label.outputs.result }}
feature-flags: ${{ steps.eval-feature-flags.outputs.result }}
sha: ${{ steps.get-sha.outputs.sha }}
env:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
steps:
- name: Check for the "testenv-up" label
id: eval-label
run: |
if [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
LABEL_NAME="${INPUT_LABEL_NAME}"
else
LABEL_NAME="${{ github.event.label.name }}"
fi
echo "Evaluating label: $LABEL_NAME"
if [[ "$LABEL_NAME" == "testenv-up" ]]; then
echo "result=up" >> $GITHUB_OUTPUT
else
echo "result=noop" >> $GITHUB_OUTPUT
fi
env:
INPUT_LABEL_NAME: ${{ github.event.inputs.label_name }}
- name: Get event SHA
id: get-sha
if: steps.eval-label.outputs.result == 'up'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
let prSha;
// If event is workflow_dispatch, use the issue_number from inputs
if (context.eventName === "workflow_dispatch") {
const prNumber = "${{ github.event.inputs.issue_number }}";
if (!prNumber) {
console.log("No PR number found.");
return;
}
// Fetch PR details using the provided issue_number
const { data: pr } = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber
});
prSha = pr.head.sha;
} else {
// If it's not workflow_dispatch, use the PR head sha from the event
prSha = context.payload.pull_request.head.sha;
}
console.log(`PR SHA: ${prSha}`);
core.setOutput("sha", prSha);
- name: Looking for feature flags in PR description
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
id: eval-feature-flags
if: steps.eval-label.outputs.result == 'up'
with:
script: |
const description = context.payload.pull_request
? context.payload.pull_request.body || ''
: context.payload.inputs.pr_description || '';
const pattern = /FEATURE_(\w+)=(\w+)/g;
let results = [];
[...description.matchAll(pattern)].forEach(match => {
const config = {
name: `SUPERSET_FEATURE_${match[1]}`,
value: match[2],
};
results.push(config);
});
return results;
- name: Reply with confirmation comment
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
if: steps.eval-label.outputs.result == 'up'
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
const action = '${{ steps.eval-label.outputs.result }}';
const user = context.actor;
const runId = context.runId;
const workflowUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${runId}`;
const issueNumber = context.payload.pull_request
? context.payload.pull_request.number
: context.payload.inputs.issue_number;
if (!issueNumber) {
throw new Error("Issue number is not available.");
}
const body = `⚠️ **DEPRECATED WORKFLOW** ⚠️\n\n@${user} This workflow is deprecated! Please use the new **Superset Showtime** system instead:\n\n` +
`- Replace "testenv-up" label with "🎪 trigger-start"\n` +
`- Better reliability and easier management\n` +
`- See https://github.com/mistercrunch/superset-showtime for details\n\n` +
`Processing your ephemeral environment request [here](${workflowUrl}). Action: **${action}**.` +
` More information on [how to use or configure ephemeral environments]` +
`(https://superset.apache.org/docs/contributing/howtos/#github-ephemeral-environments)`;
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: issueNumber,
body,
});
ephemeral-docker-build:
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}-build
cancel-in-progress: true
needs: ephemeral-env-label
if: needs.ephemeral-env-label.outputs.slash-command == 'up'
name: ephemeral-docker-build
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ needs.ephemeral-env-label.outputs.sha }} : ${{steps.get-sha.outputs.sha}} )"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ needs.ephemeral-env-label.outputs.sha }}
persist-credentials: false
- name: Setup Docker Environment
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
build: "true"
install-docker-compose: "false"
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
- name: Build ephemeral env image
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
supersetbot docker \
--push \
--load \
--preset ci \
--platform linux/amd64 \
--context-ref "$RELEASE" \
--extra-flags "--build-arg INCLUDE_CHROMIUM=false"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # v6
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@fa648b43de3d4d023bcb3f89ed6940096949c419 # v2
- name: Load, tag and push image to ECR
id: push-image
env:
ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
ECR_REPOSITORY: superset-ci
IMAGE_TAG: apache/superset:${{ needs.ephemeral-env-label.outputs.sha }}-ci
PR_NUMBER: ${{ github.event.inputs.issue_number || github.event.pull_request.number }}
run: |
docker tag $IMAGE_TAG $ECR_REGISTRY/$ECR_REPOSITORY:pr-$PR_NUMBER-ci
docker push -a $ECR_REGISTRY/$ECR_REPOSITORY
ephemeral-env-up:
needs: [ephemeral-env-label, ephemeral-docker-build]
if: needs.ephemeral-env-label.outputs.slash-command == 'up'
name: Spin up an ephemeral environment
runs-on: ubuntu-24.04
permissions:
contents: read
pull-requests: write
steps:
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@8df5847569e6427dd6c4fb1cf565c83acfa8afa7 # v6
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
aws-region: us-west-2
- name: Login to Amazon ECR
id: login-ecr
uses: aws-actions/amazon-ecr-login@fa648b43de3d4d023bcb3f89ed6940096949c419 # v2
- name: Check target image exists in ECR
id: check-image
continue-on-error: true
env:
PR_NUMBER: ${{ github.event.inputs.issue_number || github.event.pull_request.number }}
run: |
aws ecr describe-images \
--registry-id $(echo "${{ steps.login-ecr.outputs.registry }}" | grep -Eo "^[0-9]+") \
--repository-name superset-ci \
--image-ids imageTag=pr-$PR_NUMBER-ci
- name: Fail on missing container image
if: steps.check-image.outcome == 'failure'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{ github.token }}
script: |
const errMsg = '@${{ github.event.comment.user.login }} Container image not yet published for this PR. Please try again when build is complete.';
github.rest.issues.createComment({
issue_number: ${{ github.event.inputs.issue_number || github.event.pull_request.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: errMsg
});
core.setFailed(errMsg);
- name: Fill in the new image ID in the Amazon ECS task definition
id: task-def
uses: aws-actions/amazon-ecs-render-task-definition@6853cfae8c3a7d978fbf68b5a55453395541dfbb # v1
with:
task-definition: .github/workflows/ecs-task-definition.json
container-name: superset-ci
image: ${{ steps.login-ecr.outputs.registry }}/superset-ci:pr-${{ github.event.inputs.issue_number || github.event.pull_request.number }}-ci
- name: Update env vars in the Amazon ECS task definition
run: |
cat <<< "$(jq '.containerDefinitions[0].environment += ${{ needs.ephemeral-env-label.outputs.feature-flags }}' < ${{ steps.task-def.outputs.task-definition }})" > ${{ steps.task-def.outputs.task-definition }}
- name: Describe ECS service
id: describe-services
run: |
echo "active=$(aws ecs describe-services --cluster superset-ci --services pr-${INPUT_ISSUE_NUMBER}-service | jq '.services[] | select(.status == "ACTIVE") | any')" >> $GITHUB_OUTPUT
env:
INPUT_ISSUE_NUMBER: ${{ github.event.inputs.issue_number || github.event.pull_request.number }}
- name: Create ECS service
id: create-service
if: steps.describe-services.outputs.active != 'true'
env:
ECR_SUBNETS: subnet-0e15a5034b4121710,subnet-0e8efef4a72224974
ECR_SECURITY_GROUP: sg-092ff3a6ae0574d91
PR_NUMBER: ${{ github.event.inputs.issue_number || github.event.pull_request.number }}
run: |
aws ecs create-service \
--cluster superset-ci \
--service-name pr-$PR_NUMBER-service \
--task-definition superset-ci \
--launch-type FARGATE \
--desired-count 1 \
--platform-version LATEST \
--network-configuration "awsvpcConfiguration={subnets=[$ECR_SUBNETS],securityGroups=[$ECR_SECURITY_GROUP],assignPublicIp=ENABLED}" \
--tags key=pr,value=$PR_NUMBER key=github_user,value=${{ github.actor }}
- name: Deploy Amazon ECS task definition
id: deploy-task
uses: aws-actions/amazon-ecs-deploy-task-definition@a310a830f5c14e583e35d84e4e1ec7dd177c3c9c # v2
with:
task-definition: ${{ steps.task-def.outputs.task-definition }}
service: pr-${{ github.event.inputs.issue_number || github.event.pull_request.number }}-service
cluster: superset-ci
wait-for-service-stability: true
wait-for-minutes: 10
- name: List tasks
id: list-tasks
run: |
echo "task=$(aws ecs list-tasks --cluster superset-ci --service-name pr-${INPUT_ISSUE_NUMBER}-service | jq '.taskArns | first')" >> $GITHUB_OUTPUT
env:
INPUT_ISSUE_NUMBER: ${{ github.event.inputs.issue_number || github.event.pull_request.number }}
- name: Get network interface
id: get-eni
run: |
echo "eni=$(aws ecs describe-tasks --cluster superset-ci --tasks ${{ steps.list-tasks.outputs.task }} | jq '.tasks[0].attachments[0].details | map(select(.name=="networkInterfaceId"))[0].value')" >> $GITHUB_OUTPUT
- name: Get public IP
id: get-ip
run: |
echo "ip=$(aws ec2 describe-network-interfaces --network-interface-ids ${{ steps.get-eni.outputs.eni }} | jq -r '.NetworkInterfaces | first | .Association.PublicIp')" >> $GITHUB_OUTPUT
- name: Comment (success)
if: ${{ success() }}
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{github.token}}
script: |
const issue_number = context.payload.inputs?.issue_number || context.issue.number;
github.rest.issues.createComment({
issue_number: issue_number,
owner: context.repo.owner,
repo: context.repo.repo,
body: `@${{ github.actor }} Ephemeral environment spinning up at http://${{ steps.get-ip.outputs.ip }}:8080. Credentials are 'admin'/'admin'. Please allow several minutes for bootstrapping and startup.`
});
- name: Comment (failure)
if: ${{ failure() }}
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
github-token: ${{github.token}}
script: |
const issue_number = context.payload.inputs?.issue_number || context.issue.number;
github.rest.issues.createComment({
issue_number: issue_number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '@${{ github.event.inputs.user_login || github.event.comment.user.login }} Ephemeral environment creation failed. Please check the Actions logs for details.'
})
+2 -2
View File
@@ -32,12 +32,12 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287 # v5.3.0
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
with:
distribution: "temurin"
java-version: "11"
+5 -20
View File
@@ -6,41 +6,26 @@ on:
- "master"
- "[0-9].[0-9]*"
pull_request:
branches:
- "**"
types: [synchronize, opened, reopened, ready_for_review]
permissions:
contents: read
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
validate-all-ghas:
runs-on: ubuntu-24.04
permissions:
contents: read
# Required for the zizmor action to upload its SARIF results to
# GitHub code scanning (advanced-security is enabled by default).
security-events: write
steps:
- name: Checkout Repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: "20"
node-version: '20'
- name: Install Dependencies
run: npm install -g @action-validator/core @action-validator/cli --save-dev
- name: Run Script
run: bash .github/workflows/github-action-validator.sh
- name: Check for security issues on GHA workflows
uses: zizmorcore/zizmor-action@5f14fd08f7cf1cb1609c1e344975f152c7ee938d # v0.5.6
+2 -1
View File
@@ -15,8 +15,9 @@ jobs:
pull-requests: write
issues: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
+1 -6
View File
@@ -2,11 +2,6 @@ name: "Pull Request Labeler"
on:
- pull_request_target
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
labeler:
permissions:
@@ -14,7 +9,7 @@ jobs:
pull-requests: write
runs-on: ubuntu-24.04
steps:
- uses: actions/labeler@f27b608878404679385c85cfa523b85ccb86e213 # v6.1.0
- uses: actions/labeler@v6
with:
sync-labels: true
+21 -23
View File
@@ -11,29 +11,27 @@ jobs:
contents: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
submodules: recursive
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for latest tag
id: latest-tag
env:
RELEASE_TAG_NAME: ${{ github.event.release.tag_name }}
run: |
source ./scripts/tag_latest_release.sh "$RELEASE_TAG_NAME" --dry-run
- name: Check for latest tag
id: latest-tag
run: |
source ./scripts/tag_latest_release.sh $(echo ${{ github.event.release.tag_name }}) --dry-run
- name: Configure Git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Configure Git
run: |
git config user.name "$GITHUB_ACTOR"
git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
- name: Run latest-tag
uses: ./.github/actions/latest-tag
if: steps.latest-tag.outputs.SKIP_TAG != 'true'
with:
description: Superset latest release
tag-name: latest
env:
GITHUB_TOKEN: ${{ github.token }}
- name: Run latest-tag
uses: ./.github/actions/latest-tag
if: steps.latest-tag.outputs.SKIP_TAG != 'true'
with:
description: Superset latest release
tag-name: latest
env:
GITHUB_TOKEN: ${{ github.token }}
+4 -4
View File
@@ -18,14 +18,14 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287 # v5.3.0
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
with:
distribution: "temurin"
java-version: "11"
distribution: 'temurin'
java-version: '11'
- name: Run license check
run: ./scripts/check_license.sh
+3 -7
View File
@@ -8,11 +8,6 @@ on:
# Possible values: https://help.github.com/en/actions/reference/events-that-trigger-workflows#pull-request-event-pull_request
types: [opened, edited, reopened, synchronize]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
lint-check:
runs-on: ubuntu-24.04
@@ -21,7 +16,7 @@ jobs:
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -31,5 +26,6 @@ jobs:
on-failed-regex-fail-action: true
on-failed-regex-request-changes: false
on-failed-regex-create-review: false
on-failed-regex-comment: "Please format your PR title to match: `%regex%`!"
on-failed-regex-comment:
"Please format your PR title to match: `%regex%`!"
repo-token: "${{ github.token }}"
+5 -13
View File
@@ -19,16 +19,12 @@ concurrency:
jobs:
pre-commit:
runs-on: ubuntu-24.04
timeout-minutes: 20
strategy:
matrix:
# Run the full version spread on push (master/release) and nightly,
# but only the current version on PRs — lint/format/type results
# rarely differ across patch versions, so 3x per PR is wasteful.
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "previous", "next"]') }}
python-version: ["current", "previous", "next"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -48,9 +44,7 @@ jobs:
- name: Setup Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version: '20'
- name: Install Frontend Dependencies
run: |
@@ -74,15 +68,13 @@ jobs:
id: changed_files
uses: ./.github/actions/file-changes-action
with:
output: " "
output: ' '
- name: pre-commit
env:
CHANGED_FILES: ${{ steps.changed_files.outputs.files }}
run: |
set +e # Don't exit immediately on failure
export SKIP=type-checking-frontend
pre-commit run --files $CHANGED_FILES
pre-commit run --files ${{ steps.changed_files.outputs.files }}
PRE_COMMIT_EXIT_CODE=$?
git diff --quiet --exit-code
GIT_DIFF_EXIT_CODE=$?
+2 -8
View File
@@ -6,9 +6,6 @@ on:
- "master"
- "[0-9].[0-9]*"
permissions:
contents: read
jobs:
config:
runs-on: ubuntu-24.04
@@ -30,12 +27,9 @@ jobs:
if: needs.config.outputs.has-secrets
name: Bump version and publish package(s)
runs-on: ubuntu-24.04
permissions:
contents: write
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
# pulls all commits (needed for lerna / semantic release to correctly version)
fetch-depth: 0
- name: Get tags and filter trigger tags
@@ -52,7 +46,7 @@ jobs:
if: env.HAS_TAGS
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
node-version-file: './superset-frontend/.nvmrc'
- name: Cache npm
if: env.HAS_TAGS
+6 -9
View File
@@ -2,7 +2,6 @@ name: 🎪 Superset Showtime
# Ultra-simple: just sync on any PR state change
on:
# zizmor: ignore[dangerous-triggers] - required to react to PR label changes; this workflow does not check out or execute PR-provided code
pull_request_target:
types: [labeled, unlabeled, synchronize, closed]
@@ -10,11 +9,11 @@ on:
workflow_dispatch:
inputs:
pr_number:
description: "PR number to sync"
description: 'PR number to sync'
required: true
type: number
sha:
description: "Specific SHA to deploy (optional, defaults to latest)"
description: 'Specific SHA to deploy (optional, defaults to latest)'
required: false
type: string
@@ -103,7 +102,7 @@ jobs:
- name: Install Superset Showtime
if: steps.auth.outputs.authorized == 'true'
run: |
echo "::notice::Maintainer $GITHUB_ACTOR triggered deploy for PR ${PULL_REQUEST_NUMBER}"
echo "::notice::Maintainer ${{ github.actor }} triggered deploy for PR ${PULL_REQUEST_NUMBER}"
pip install --upgrade superset-showtime
showtime version
@@ -152,7 +151,7 @@ jobs:
- name: Checkout PR code (only if build needed)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.build_needed == 'true'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ steps.check.outputs.target_sha }}
persist-credentials: false
@@ -174,11 +173,9 @@ jobs:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
CHECK_PR_NUMBER: ${{ steps.check.outputs.pr_number }}
CHECK_TARGET_SHA: ${{ steps.check.outputs.target_sha }}
run: |
PR_NUM="$CHECK_PR_NUMBER"
TARGET_SHA="$CHECK_TARGET_SHA"
PR_NUM="${{ steps.check.outputs.pr_number }}"
TARGET_SHA="${{ steps.check.outputs.target_sha }}"
if [[ -n "$TARGET_SHA" ]]; then
python -m showtime sync $PR_NUM --sha "$TARGET_SHA"
else
+2 -2
View File
@@ -41,7 +41,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -61,7 +61,7 @@ jobs:
- name: superset init
if: steps.check.outputs.python
run: |
uv pip install --system -e .
pip install -e .
superset db upgrade
superset load_test_users
- name: superset load_examples
+6 -16
View File
@@ -2,7 +2,6 @@ name: Docs Deployment
on:
# Deploy after integration tests complete on master
# zizmor: ignore[dangerous-triggers] - runs in base-branch context after a trusted upstream workflow; scoped to master
workflow_run:
workflows: ["Python-Integration"]
types: [completed]
@@ -28,9 +27,6 @@ concurrency:
group: docs-deploy-asf-site
cancel-in-progress: true
permissions:
contents: read
jobs:
config:
runs-on: ubuntu-24.04
@@ -49,18 +45,12 @@ jobs:
SUPERSET_SITE_BUILD: ${{ (secrets.SUPERSET_SITE_BUILD != '' && secrets.SUPERSET_SITE_BUILD != '') || '' }}
build-deploy:
needs: config
# For workflow_run triggers, only deploy when the triggering run originated
# from this repository (not a fork), ensuring the checked-out code and any
# local actions executed with deploy credentials are trusted.
if: >-
needs.config.outputs.has-secrets &&
(github.event_name != 'workflow_run' ||
github.event.workflow_run.head_repository.full_name == github.repository)
if: needs.config.outputs.has-secrets
name: Build & Deploy
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.event.workflow_run.head_sha || github.sha }}"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event.workflow_run.head_sha || github.sha }}
persist-credentials: false
@@ -68,13 +58,13 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./docs/.nvmrc"
node-version-file: './docs/.nvmrc'
- name: Setup Python
uses: ./.github/actions/setup-backend/
- uses: actions/setup-java@ad2b38190b15e4d6bdf0c97fb4fca8412226d287 # v5.3.0
- uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
with:
distribution: "zulu"
java-version: "21"
distribution: 'zulu'
java-version: '21'
- name: Install Graphviz
run: sudo apt-get install -y graphviz
- name: Compute Entity Relationship diagram (ERD)
+8 -15
View File
@@ -7,7 +7,6 @@ on:
- "superset/db_engine_specs/**"
- ".github/workflows/superset-docs-verify.yml"
types: [synchronize, opened, reopened, ready_for_review]
# zizmor: ignore[dangerous-triggers] - runs in base-branch context and only consumes artifacts from the trusted upstream workflow
workflow_run:
workflows: ["Python-Integration"]
types: [completed]
@@ -17,9 +16,6 @@ concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.workflow_run.head_sha || github.run_id }}
cancel-in-progress: true
permissions:
contents: read
jobs:
linkinator:
# See docs here: https://github.com/marketplace/actions/linkinator
@@ -28,12 +24,10 @@ jobs:
name: Link Checking
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
# Do not bump this linkinator-action version without opening
# an ASF Infra ticket to allow the new version first!
- uses: JustinBeckwith/linkinator-action@af984b9f30f63e796ae2ea5be5e07cb587f1bbd9 # v2.3
- uses: JustinBeckwith/linkinator-action@af984b9f30f63e796ae2ea5be5e07cb587f1bbd9 # v2.3
continue-on-error: true # This will make the job advisory (non-blocking, no red X)
with:
paths: "**/*.md, **/*.mdx"
@@ -73,14 +67,14 @@ jobs:
working-directory: docs
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./docs/.nvmrc"
node-version-file: './docs/.nvmrc'
- name: yarn install
run: |
yarn install --check-cache
@@ -103,8 +97,7 @@ jobs:
# Only runs if integration tests succeeded
if: >
github.event_name == 'workflow_run' &&
github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.head_repository.full_name == github.repository
github.event.workflow_run.conclusion == 'success'
name: Build (after integration tests)
runs-on: ubuntu-24.04
defaults:
@@ -112,7 +105,7 @@ jobs:
working-directory: docs
steps:
- name: "Checkout PR head: ${{ github.event.workflow_run.head_sha }}"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ github.event.workflow_run.head_sha }}
persist-credentials: false
@@ -120,7 +113,7 @@ jobs:
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./docs/.nvmrc"
node-version-file: './docs/.nvmrc'
- name: yarn install
run: |
yarn install --check-cache
@@ -131,7 +124,7 @@ jobs:
run_id: ${{ github.event.workflow_run.id }}
name: database-diagnostics
path: docs/src/data/
if_no_artifact_found: "warning"
if_no_artifact_found: 'warning'
- name: Use fresh diagnostics
run: |
if [ -f "src/data/databases-diagnostics.json" ]; then
+44 -119
View File
@@ -10,49 +10,26 @@ on:
workflow_dispatch:
inputs:
use_dashboard:
description: "Use Cypress Dashboard (true/false) [paid service - trigger manually when needed]. You MUST provide a branch and/or PR number below for this to work."
description: 'Use Cypress Dashboard (true/false) [paid service - trigger manually when needed]. You MUST provide a branch and/or PR number below for this to work.'
required: false
default: "false"
default: 'false'
ref:
description: "The branch or tag to checkout"
description: 'The branch or tag to checkout'
required: false
default: ""
default: ''
pr_id:
description: "The pull request ID to checkout"
description: 'The pull request ID to checkout'
required: false
default: ""
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
cypress-matrix:
needs: changes
if: (needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true') && github.event.pull_request.draft == false
# Somehow one test flakes on 24.04 for unknown reasons, this is the only GHA left on 22.04
runs-on: ubuntu-22.04
timeout-minutes: 30
permissions:
contents: read
pull-requests: read
@@ -63,14 +40,9 @@ jobs:
# https://github.com/cypress-io/github-action/issues/48
fail-fast: false
matrix:
parallel_id: [0, 1]
parallel_id: [0, 1, 2, 3, 4, 5]
browser: ["chrome"]
app_root: ${{ github.event_name == 'push' && fromJSON('["", "/app/prefix"]') || fromJSON('[""]') }}
# The /app/prefix variant (push events only) is smoke-tested on a single
# shard rather than the full matrix, so exclude it from the other shards.
exclude:
- parallel_id: 1
app_root: "/app/prefix"
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -97,60 +69,71 @@ jobs:
# Conditional checkout based on context
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python || steps.check.outputs.frontend
- name: Setup postgres
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Install cypress
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: cypress-install
- name: Run Cypress
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
env:
CYPRESS_BROWSER: ${{ matrix.browser }}
PARALLEL_ID: ${{ matrix.parallel_id }}
PARALLELISM: 2
PARALLELISM: 6
CYPRESS_RECORD_KEY: ${{ secrets.CYPRESS_RECORD_KEY }}
NODE_OPTIONS: "--max-old-space-size=4096"
with:
@@ -158,9 +141,8 @@ jobs:
- name: Set safe app root
if: failure()
id: set-safe-app-root
env:
APP_ROOT: ${{ matrix.app_root }}
run: |
APP_ROOT="${{ matrix.app_root }}"
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Artifacts
@@ -171,10 +153,7 @@ jobs:
name: cypress-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}-${{ matrix.parallel_id }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
playwright-tests:
needs: changes
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 30
permissions:
contents: read
pull-requests: read
@@ -207,59 +186,66 @@ jobs:
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python || steps.check.outputs.frontend
- name: Setup postgres
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright_testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Build embedded SDK
uses: ./.github/actions/cached-dependencies
with:
run: build-embedded-sdk
- name: Install Playwright
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Required Tests)
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
@@ -268,9 +254,8 @@ jobs:
- name: Set safe app root
if: failure()
id: set-safe-app-root
env:
APP_ROOT: ${{ matrix.app_root }}
run: |
APP_ROOT="${{ matrix.app_root }}"
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Playwright Artifacts
@@ -281,63 +266,3 @@ jobs:
${{ github.workspace }}/superset-frontend/playwright-results/
${{ github.workspace }}/superset-frontend/test-results/
name: playwright-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
# Stable required-status-check anchors. cypress-matrix and playwright-tests
# are matrix jobs gated on change detection (python || frontend). On a PR
# that touches neither — e.g. a docs-only PR — they are skipped at the job
# level, which happens before matrix expansion, so the per-combination
# contexts (`cypress-matrix (0, chrome)`, `playwright-tests (chromium)`) are
# never produced and branch protection waits on them forever. These
# always-running jobs report a single stable context that passes when the
# underlying matrix job succeeded or was skipped, and fails only on a real
# failure. Require these in .asf.yaml instead of the matrix-expanded names.
#
# A matrix job reads as "skipped" in two distinct cases, and only the first
# is a legitimate pass: (a) change detection succeeded and gated the job off
# (docs-only PR); (b) the `changes` job itself failed or was cancelled, in
# which case GHA skips its dependents too. Accepting (b) would let a broken
# change-detector report a false green, so each anchor first requires
# `changes` to have succeeded before honouring a skip.
cypress-matrix-required:
needs: [changes, cypress-matrix]
if: always()
runs-on: ubuntu-24.04
timeout-minutes: 5
permissions: {}
steps:
- name: Check cypress-matrix result
env:
CHANGES: ${{ needs.changes.result }}
RESULT: ${{ needs.cypress-matrix.result }}
run: |
if [ "$CHANGES" != "success" ]; then
echo "change detection did not succeed (result: $CHANGES); refusing to pass on a skipped matrix"
exit 1
fi
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "cypress-matrix did not pass (result: $RESULT)"
exit 1
fi
echo "cypress-matrix result: $RESULT (changes: $CHANGES)"
playwright-tests-required:
needs: [changes, playwright-tests]
if: always()
runs-on: ubuntu-24.04
timeout-minutes: 5
permissions: {}
steps:
- name: Check playwright-tests result
env:
CHANGES: ${{ needs.changes.result }}
RESULT: ${{ needs.playwright-tests.result }}
run: |
if [ "$CHANGES" != "success" ]; then
echo "change detection did not succeed (result: $CHANGES); refusing to pass on a skipped matrix"
exit 1
fi
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "playwright-tests did not pass (result: $RESULT)"
exit 1
fi
echo "playwright-tests result: $RESULT (changes: $CHANGES)"
@@ -20,18 +20,15 @@ concurrency:
jobs:
test-superset-extensions-cli-package:
runs-on: ubuntu-24.04
timeout-minutes: 30
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["previous", "current", "next"]') }}
python-version: ["previous", "current", "next"]
defaults:
run:
working-directory: superset-extensions-cli
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -56,7 +53,7 @@ jobs:
- name: Upload coverage reports to Codecov
if: steps.check.outputs.superset-extensions-cli
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
file: ./coverage.xml
flags: superset-extensions-cli
+3 -12
View File
@@ -16,18 +16,14 @@ concurrency:
env:
TAG: apache/superset:GHA-${{ github.run_id }}
permissions:
contents: read
jobs:
frontend-build:
runs-on: ubuntu-24.04
timeout-minutes: 30
outputs:
should-run: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout Code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
fetch-depth: 0
@@ -75,7 +71,6 @@ jobs:
shard: [1, 2, 3, 4, 5, 6, 7, 8]
fail-fast: false
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
@@ -105,12 +100,11 @@ jobs:
needs: [sharded-jest-tests]
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 15
permissions:
id-token: write
steps:
- name: Checkout Code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
fetch-depth: 0
@@ -134,7 +128,7 @@ jobs:
run: npx nyc merge coverage/ merged-output/coverage-summary.json
- name: Upload Code Coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: javascript
use_oidc: true
@@ -147,7 +141,6 @@ jobs:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
@@ -172,7 +165,6 @@ jobs:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
@@ -192,7 +184,6 @@ jobs:
needs: frontend-build
if: needs.frontend-build.outputs.should-run == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 25
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8
+2 -2
View File
@@ -19,7 +19,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -33,7 +33,7 @@ jobs:
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
install-superset: "false"
install-superset: 'false'
- name: Set up chart-testing
uses: ./.github/actions/chart-testing-action
+4 -6
View File
@@ -29,7 +29,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
ref: ${{ inputs.ref || github.ref_name }}
persist-credentials: true
@@ -62,8 +62,6 @@ jobs:
run: echo "branch_name=helm-publish-${GITHUB_SHA:0:7}" >> $GITHUB_ENV
- name: Force recreate branch from gh-pages
env:
BRANCH_NAME: ${{ env.branch_name }}
run: |
# Ensure a clean working directory
git reset --hard
@@ -75,13 +73,13 @@ jobs:
git fetch origin gh-pages
# Check out and reset the target branch based on gh-pages
git checkout -B "$BRANCH_NAME" origin/gh-pages
git checkout -B ${{ env.branch_name }} origin/gh-pages
# Remove submodules from the branch
git submodule deinit -f --all
# Force push to the remote branch
git push origin "$BRANCH_NAME" --force
git push origin ${{ env.branch_name }} --force
# Return to the original branch
git checkout local_gha_temp
@@ -106,7 +104,7 @@ jobs:
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
script: |
const branchName = process.env.BRANCH_NAME;
const branchName = '${{ env.branch_name }}';
const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/');
if (!branchName) {
+21 -51
View File
@@ -10,46 +10,23 @@ on:
workflow_dispatch:
inputs:
ref:
description: "The branch or tag to checkout"
description: 'The branch or tag to checkout'
required: false
default: ""
default: ''
pr_id:
description: "The pull request ID to checkout"
description: 'The pull request ID to checkout'
required: false
default: ""
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
frontend: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
# NOTE: Required Playwright tests are in superset-e2e.yml (E2E / playwright-tests)
# This workflow contains only experimental tests that run in shadow mode
playwright-tests-experimental:
needs: changes
if: needs.changes.outputs.python == 'true' || needs.changes.outputs.frontend == 'true'
runs-on: ubuntu-22.04
timeout-minutes: 30
continue-on-error: true
permissions:
contents: read
@@ -83,78 +60,71 @@ jobs:
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python || steps.check.outputs.frontend
- name: Setup postgres
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright_testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Build embedded SDK
uses: ./.github/actions/cached-dependencies
with:
run: build-embedded-sdk
- name: Install Playwright
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Experimental Tests)
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
with:
run: playwright-run "${{ matrix.app_root }}" experimental/
- name: Run Playwright (Embedded Tests)
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
# Scope embedded-only env vars to this step. Setting them at the job
# level enabled the EMBEDDED_SUPERSET feature flag inside Flask for
# the preceding "Required Tests" and "Experimental Tests" steps too,
# which loads extra handlers and destabilizes the werkzeug dev
# server under the 2-worker Playwright load. Required Tests should
# match master's Flask configuration.
SUPERSET_FEATURE_EMBEDDED_SUPERSET: "true"
INCLUDE_EMBEDDED: "true"
with:
run: playwright-run "${{ matrix.app_root }}" embedded
- name: Set safe app root
if: failure()
id: set-safe-app-root
@@ -1,11 +1,6 @@
# Python integration tests
name: Python-Integration
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:
@@ -19,30 +14,8 @@ concurrency:
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
test-mysql:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 45
permissions:
id-token: write
env:
@@ -54,8 +27,6 @@ jobs:
services:
mysql:
image: mysql:8.0
# Authenticated pulls use our higher Docker Hub rate limit. Empty on
# fork PRs (secrets unavailable) -> runner falls back to anonymous.
env:
MYSQL_ROOT_PASSWORD: root
ports:
@@ -72,31 +43,41 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python
- name: Setup MySQL
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: setup-mysql
- name: Start Celery worker
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Python integration tests (MySQL)
if: steps.check.outputs.python
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,mysql
verbose: true
use_oidc: true
slug: apache/superset
- name: Generate database diagnostics for docs
if: steps.check.outputs.python
env:
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: |
@@ -119,23 +100,19 @@ jobs:
print(f'Generated diagnostics for {len(docs)} databases')
"
- name: Upload database diagnostics artifact
if: steps.check.outputs.python
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: database-diagnostics
path: databases-diagnostics.json
retention-days: 7
test-postgres:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 45
permissions:
id-token: write
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "previous", "next"]') }}
python-version: ["current", "previous", "next"]
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
@@ -157,28 +134,37 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python
with:
python-version: ${{ matrix.python-version }}
- name: Setup Postgres
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: |
setup-postgres
- name: Start Celery worker
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Python integration tests (PostgreSQL)
if: steps.check.outputs.python
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,postgres
verbose: true
@@ -186,10 +172,7 @@ jobs:
slug: apache/superset
test-sqlite:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 45
permissions:
id-token: write
env:
@@ -207,51 +190,38 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python
- name: Install dependencies
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: |
# sqlite needs this working directory
mkdir ${{ github.workspace }}/.temp
- name: Start Celery worker
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Python integration tests (SQLite)
if: steps.check.outputs.python
run: |
./scripts/python_tests.sh
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,sqlite
verbose: true
use_oidc: true
slug: apache/superset
# Stable required-status-check anchor for the matrix-based test-postgres job.
# It is gated on change detection, so on non-Python PRs it is skipped and
# never produces its `test-postgres (current)` context (a job-level skip
# happens before matrix expansion). This always-running job reports a single
# context branch protection can require: it passes when test-postgres
# succeeded or was skipped, and fails only on a real failure.
test-postgres-required:
needs: [changes, test-postgres]
if: always()
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- name: Check test-postgres result
env:
RESULT: ${{ needs.test-postgres.result }}
run: |
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "test-postgres did not pass (result: $RESULT)"
exit 1
fi
echo "test-postgres result: $RESULT"
@@ -15,30 +15,8 @@ concurrency:
cancel-in-progress: true
jobs:
changes:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
test-postgres-presto:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 45
permissions:
id-token: write
env:
@@ -72,25 +50,36 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python == 'true'
- name: Setup Postgres
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
run: |
echo "${{ steps.check.outputs.python }}"
setup-postgres
- name: Start Celery worker
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Python unit tests (PostgreSQL)
if: steps.check.outputs.python
run: |
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,presto
verbose: true
@@ -98,10 +87,7 @@ jobs:
slug: apache/superset
test-postgres-hive:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 45
permissions:
id-token: write
env:
@@ -127,32 +113,44 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Create csv upload directory
if: steps.check.outputs.python
run: sudo mkdir -p /tmp/.superset/uploads
- name: Give write access to the csv upload directory
if: steps.check.outputs.python
run: sudo chown -R $USER:$USER /tmp/.superset
- name: Start hadoop and hive
if: steps.check.outputs.python
run: docker compose -f scripts/databases/hive/docker-compose.yml up -d
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python
- name: Setup Postgres
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Start Celery worker
if: steps.check.outputs.python
uses: ./.github/actions/cached-dependencies
with:
run: celery-worker
- name: Python unit tests (PostgreSQL)
if: steps.check.outputs.python
run: |
uv pip install --system -e .[hive]
pip install -e .[hive]
./scripts/python_tests.sh -m 'chart_data_flow or sql_json_flow'
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,hive
verbose: true
+14 -57
View File
@@ -1,11 +1,6 @@
# Python unit tests
name: Python-Unit
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:
@@ -20,56 +15,40 @@ concurrency:
cancel-in-progress: true
jobs:
changes:
unit-tests:
runs-on: ubuntu-24.04
timeout-minutes: 10
permissions:
contents: read
pull-requests: read
outputs:
python: ${{ steps.check.outputs.python }}
id-token: write
strategy:
matrix:
python-version: ["previous", "current", "next"]
env:
PYTHONPATH: ${{ github.workspace }}
steps:
- name: Checkout
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
unit-tests:
needs: changes
if: needs.changes.outputs.python == 'true'
runs-on: ubuntu-24.04
timeout-minutes: 30
permissions:
id-token: write
strategy:
matrix:
# Full version spread on push (master/release) + nightly; current only
# on PRs to cut runner cost (cross-version breaks are caught at merge).
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["previous", "current", "next"]') }}
env:
PYTHONPATH: ${{ github.workspace }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
submodules: recursive
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python
with:
python-version: ${{ matrix.python-version }}
- name: Python unit tests
if: steps.check.outputs.python
env:
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
run: |
pytest --durations-min=0.5 --cov-report= --cov=superset ./tests/common ./tests/unit_tests --cache-clear --maxfail=50
- name: Python 100% coverage unit tests
if: steps.check.outputs.python
env:
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
@@ -77,31 +56,9 @@ jobs:
pytest --durations-min=0.5 --cov=superset/sql/ ./tests/unit_tests/sql/ --cache-clear --cov-fail-under=100
pytest --durations-min=0.5 --cov=superset/semantic_layers/ ./tests/unit_tests/semantic_layers/ --cache-clear --cov-fail-under=100
- name: Upload code coverage
uses: codecov/codecov-action@fb8b3582c8e4def4969c97caa2f19720cb33a72f # v7.0.0
uses: codecov/codecov-action@e79a6962e0d4c0c17b229090214935d2e33f8354 # v5
with:
flags: python,unit
verbose: true
use_oidc: true
slug: apache/superset
# Stable required-status-check anchor. `unit-tests` is a matrix job gated on
# change detection, so on non-Python PRs it is skipped and never produces its
# `unit-tests (current)` context (a job-level skip happens before matrix
# expansion). This always-running job reports a single context that branch
# protection can require: it passes when unit-tests succeeded or was skipped,
# and fails only on a real failure.
unit-tests-required:
needs: [changes, unit-tests]
if: always()
runs-on: ubuntu-24.04
timeout-minutes: 5
steps:
- name: Check unit-tests result
env:
RESULT: ${{ needs.unit-tests.result }}
run: |
if [ "$RESULT" != "success" ] && [ "$RESULT" != "skipped" ]; then
echo "unit-tests did not pass (result: $RESULT)"
exit 1
fi
echo "unit-tests result: $RESULT"
@@ -1,7 +1,6 @@
name: Translation Regression Comment
on:
# zizmor: ignore[dangerous-triggers] - runs in base-branch context and only consumes the uploaded artifact; never checks out PR code (see note below)
workflow_run:
workflows: ["Translations"]
types: [completed]
+12 -12
View File
@@ -25,7 +25,7 @@ jobs:
pull-requests: read
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -40,9 +40,7 @@ jobs:
if: steps.check.outputs.frontend
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
cache: "npm"
cache-dependency-path: "superset-frontend/package-lock.json"
node-version-file: './superset-frontend/.nvmrc'
- name: Install dependencies
if: steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
@@ -61,7 +59,7 @@ jobs:
pull-requests: read
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
submodules: recursive
@@ -86,15 +84,13 @@ jobs:
# drift on the base branch.
- name: Fetch base ref and create comparison worktree
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
env:
PR_BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
# For PRs use the base branch; for direct pushes compare against the previous commit.
BASE_REF="$PR_BASE_REF"
BASE_REF="${{ github.event.pull_request.base.ref }}"
if [ -n "$BASE_REF" ]; then
git fetch --depth=1 origin "$BASE_REF"
else
git fetch --depth=2 origin "$GITHUB_REF"
git fetch --depth=2 origin "${{ github.ref }}"
fi
git worktree add /tmp/base-worktree FETCH_HEAD
@@ -115,9 +111,13 @@ jobs:
--translations-dir /tmp/base-worktree/superset/translations \
> /tmp/before.json
# Run babel_update against the PR source and PR translations. This keeps
# committed .po fixes in play while the base babel_update above still
# cancels out translation drift already present on the base branch.
# Reset the PR worktree's translations to the pristine BASE state so
# both babel_update runs start from the same .po files. The only
# difference between the runs is the source code.
- name: Reset PR worktree translations to pristine BASE
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
run: git checkout FETCH_HEAD -- superset/translations/
- name: Run babel_update against PR source
if: steps.check.outputs.python == 'true' || steps.check.outputs.frontend == 'true'
run: ./scripts/translations/babel_update.sh
+1 -2
View File
@@ -22,10 +22,9 @@ concurrency:
jobs:
app-checks:
runs-on: ubuntu-24.04
timeout-minutes: 20
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
- name: Install dependencies
+2 -2
View File
@@ -9,7 +9,7 @@ on:
workflow_dispatch:
inputs:
comment_body:
description: "Comment Body"
description: 'Comment Body'
required: true
type: string
@@ -38,7 +38,7 @@ jobs:
});
- name: "Checkout ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
@@ -1,60 +0,0 @@
name: Sync requirements for Python dependency PRs
on:
pull_request:
types: [opened, synchronize]
permissions:
contents: write
pull-requests: read
jobs:
sync-python-dep-requirements:
# This action is limited for (1) PRs authored by Dependabot and (2) upstream repo due to write back to remote
if: github.repository == 'apache/superset' && github.event.pull_request.user.login == 'dependabot[bot]' && github.event.pull_request.head.repo.fork == false
runs-on: ubuntu-26.04
steps:
- name: Fetch Dependabot metadata
id: dependabot-metadata
shell: bash
env:
BRANCH_NAME: ${{ github.head_ref }}
run: |
# Get current branch name, extract the package ecosystem and return as GHA step output
packageEcosystem=$(echo "$BRANCH_NAME" | cut -d'/' -f2)
echo "package-ecosystem=$packageEcosystem" >> $GITHUB_OUTPUT
# zizmor: ignore[artipacked] - required persisted credentials to push synced requirement changes back to remote
- name: Checkout source code
if: ${{ steps.dependabot-metadata.outputs.package-ecosystem == 'pip' }}
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
ref: ${{ github.event.pull_request.head.sha }}
persist-credentials: true
# Authenticate the Docker daemon so the python:slim pull in
# uv-pip-compile.sh uses our (much higher) authenticated rate limit
# instead of the shared-runner anonymous one.
- name: Login to Docker Hub
if: ${{ steps.dependabot-metadata.outputs.package-ecosystem == 'pip' }}
continue-on-error: true
uses: docker/login-action@650006c6eb7dba73a995cc03b0b2d7f5ca915bee # v4.2.0
with:
username: ${{ secrets.DOCKERHUB_USER }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
- name: Sync requirements in containerized environment
if: ${{ steps.dependabot-metadata.outputs.package-ecosystem == 'pip' }}
run: ./scripts/uv-pip-compile.sh
- name: Push changes to remote PRs
if: ${{ steps.dependabot-metadata.outputs.package-ecosystem == 'pip' }}
run: |
git config user.name 'github-actions[bot]'
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
git add requirements
git diff --cached --quiet && exit 0
git commit --signoff --message "build(deps): sync pinned requirements for Dependabot pip PRs"
git push origin "HEAD:refs/heads/${GITHUB_EVENT_PULL_REQUEST_HEAD_REF}"
env:
GITHUB_EVENT_PULL_REQUEST_HEAD_REF: ${{ github.event.pull_request.head.ref }}
+10 -22
View File
@@ -16,14 +16,11 @@ on:
force-latest:
required: true
type: choice
default: "false"
default: 'false'
description: Whether to force a latest tag on the release
options:
- "true"
- "false"
permissions:
contents: read
- 'true'
- 'false'
jobs:
config:
runs-on: ubuntu-24.04
@@ -45,18 +42,15 @@ jobs:
if: needs.config.outputs.has-secrets
name: docker-release
runs-on: ubuntu-24.04
permissions:
contents: write
strategy:
matrix:
build_preset:
["dev", "lean", "py310", "websocket", "dockerize", "py311", "py312"]
build_preset: ["dev", "lean", "py310", "websocket", "dockerize", "py311", "py312"]
fail-fast: false
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
fetch-depth: 0
- name: Setup Docker Environment
@@ -68,11 +62,9 @@ jobs:
build: "true"
- name: Use Node.js 20
# zizmor: ignore[cache-poisoning] - node only runs the supersetbot CLI; no dependency cache is enabled
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: 20
package-manager-cache: false
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
@@ -85,9 +77,8 @@ jobs:
INPUT_RELEASE: ${{ github.event.inputs.release }}
INPUT_FORCE_LATEST: ${{ github.event.inputs.force-latest }}
INPUT_GIT_REF: ${{ github.event.inputs.git-ref }}
GITHUB_EVENT_RELEASE_TAG_NAME: ${{ github.event.release.tag_name }}
run: |
RELEASE="${GITHUB_EVENT_RELEASE_TAG_NAME}"
RELEASE="${{ github.event.release.tag_name }}"
FORCE_LATEST=""
EVENT="${{github.event_name}}"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
@@ -119,18 +110,16 @@ jobs:
contents: read
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
persist-credentials: false
fetch-depth: 0
- name: Use Node.js 20
# zizmor: ignore[cache-poisoning] - node only runs the supersetbot CLI; no dependency cache is enabled
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version: 20
package-manager-cache: false
- name: Setup supersetbot
uses: ./.github/actions/setup-supersetbot/
@@ -139,12 +128,11 @@ jobs:
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
INPUT_RELEASE: ${{ github.event.inputs.release }}
GITHUB_EVENT_RELEASE_TAG_NAME: ${{ github.event.release.tag_name }}
run: |
export GITHUB_ACTOR=""
git fetch --all --tags
git checkout master
RELEASE="${GITHUB_EVENT_RELEASE_TAG_NAME}"
RELEASE="${{ github.event.release.tag_name }}"
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
# in the case of a manually-triggered run, read release from input
RELEASE="${INPUT_RELEASE}"
+2 -4
View File
@@ -32,14 +32,12 @@ jobs:
name: Generate Reports
steps:
- name: Checkout Repository
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7.0.0
with:
persist-credentials: false
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
- name: Set up Node.js
uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6
with:
node-version-file: "./superset-frontend/.nvmrc"
node-version-file: './superset-frontend/.nvmrc'
- name: Install Dependencies
run: npm ci
+4 -11
View File
@@ -1,29 +1,22 @@
name: Welcome New Contributor
on:
# zizmor: ignore[dangerous-triggers] - posts a welcome comment only; does not check out or execute PR-provided code
pull_request_target:
types: [opened]
jobs:
welcome:
runs-on: ubuntu-24.04
if: github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
permissions:
pull-requests: write
steps:
- name: Welcome Message
uses: actions/first-interaction@1c4688942c71f71d4f5502a26ea67c331730fa4d # v3.1.0
uses: actions/first-interaction@v3
continue-on-error: true
with:
repo_token: ${{ github.token }}
issue_message: |-
Congrats on opening your first issue and thank you for contributing to Superset! :tada: :heart:
Please read our [New Contributor Welcome & Expectations](https://github.com/apache/superset/wiki/New-Contributor-Welcome-&-Expectations) guide.
pr_message: |-
repo-token: ${{ github.token }}
pr-message: |-
Congrats on making your first PR and thank you for contributing to Superset! :tada: :heart:
Please read our [New Contributor Welcome & Expectations](https://github.com/apache/superset/wiki/New-Contributor-Welcome-&-Expectations) guide.
We hope to see you in our [Slack](https://apache-superset.slack.com/) community too! Not signed up? Use our [Slack App](http://bit.ly/join-superset-slack) to self-register.
-11
View File
@@ -158,14 +158,3 @@ repos:
language: system
files: ^superset/config\.py$
pass_filenames: false
- id: zizmor
name: zizmor (GHA security audit)
entry: zizmor
language: python
additional_dependencies: [zizmor==1.25.2]
files: ^\.github/
types: [yaml]
pass_filenames: false
# Advisory until pre-existing findings are resolved; remove
# --no-exit-codes to make this hook blocking.
args: [--no-exit-codes, .github/]
-38
View File
@@ -52,43 +52,6 @@ Common pre-commit failures:
- **External API exposure** - Use UUIDs in public APIs instead of internal integer IDs
- **Existing models** - Add UUID fields alongside integer IDs for gradual migration
## Security and Threat Model
Before evaluating any code path for security issues, read [`SECURITY.md`](SECURITY.md). It is the canonical, authoritative source for Apache Superset's security model and is referenced by both human reporters and automated scanners.
In short, the test for whether a finding is in scope is one question:
> *Does it let a principal perform an action the role and capability matrix in `SECURITY.md` does not entitle them to?*
If yes, it is in scope. If no, it is not.
The three trust boundaries are:
1. **The Admin role** is a fully trusted operational principal. Anything an Admin can do through documented configuration, API, or UI is an intended capability, not a vulnerability.
2. **The operator** owns deployment-time decisions (secrets, network exposure, feature-flag selection, connector and codec choices, notification destinations, third-party plugins). Misconfiguration at this layer is a deployment defect, not a Superset vulnerability.
3. **The codebase** is responsible for enforcing the role and capability matrix wherever it exposes functionality to a principal: API routes, command and DAO layers, UI handlers, background jobs, and any other entry point. A missing or incorrect enforcement check is in scope no matter where it lives.
The security model assumes that operator-controlled infrastructure, including the metadata database, cache backends, message brokers, secret stores, and deployment environment, remains within the operator's trust boundary. Vulnerabilities must demonstrate a security boundary violation by an attacker who does not already control those systems.
Route-level authorization in this codebase uses one of three Flask-AppBuilder decorators depending on the route type:
- `@protect()` for REST API routes (`ModelRestApi` / `BaseApi`)
- `@has_access_api` for legacy view routes
- `@has_access` for legacy HTML view routes
Object-level authorization via `security_manager.raise_for_access(...)` applies to data-bearing resources: dashboards, charts, datasets and datasources, queries, database and table access, and query contexts. Other resources (annotations, tags, CSS templates, reports, RLS rules, and similar) rely on the route-level decorator plus DAO `base_filters` for ownership scoping; the absence of `raise_for_access` on these resources is by design, not a finding. Code that omits the per-object gate on a route that returns or mutates a specific data-bearing object is in scope; code that follows the correct pattern for its resource class can still contain injection, SSRF, XSS, or other classes of finding unrelated to authorization, which are evaluated separately.
The full role and capability matrix, in-scope and out-of-scope class lists, and CVE aggregation rules are in [`SECURITY.md`](SECURITY.md). Defer to that document for any specifics.
**Requirements for findings filed by automated tooling**
Automated scanners (LLM-based code scanners, static analyzers, dependency tools) that file findings against this codebase must, in each finding, name:
1. The specific role and capability matrix row in [`SECURITY.md`](SECURITY.md) the finding believes is violated.
2. The principal the finding assumes the attacker holds (Public, Gamma, sql_lab, Alpha, Admin, Embedded guest token, or a custom role with explicit capability grants).
Findings that cannot identify both should be filed as questions, not vulnerabilities. This requirement exists to ensure every reported issue is testable against the published security model and to keep speculative or pattern-match-only reports out of the triage queue.
## Key Directories
```
@@ -165,7 +128,6 @@ The Developer Portal auto-generates MDX documentation from Storybook stories. **
## Architecture Patterns
### Security & Features
- **Security model**: see the top-level [Security and Threat Model](#security-and-threat-model) section and [`SECURITY.md`](SECURITY.md)
- **RBAC**: Role-based access via Flask-AppBuilder
- **Feature flags**: Control feature rollouts
- **Row-level security**: SQL-based data access control
-1
View File
@@ -50,4 +50,3 @@ under the License.
- [4.1.4](./CHANGELOG/4.1.4.md)
- [5.0.0](./CHANGELOG/5.0.0.md)
- [6.0.0](./CHANGELOG/6.0.0.md)
- [6.1.0](./CHANGELOG/6.1.0.md)
-1563
View File
File diff suppressed because it is too large Load Diff
+2 -9
View File
@@ -29,7 +29,7 @@ ARG BUILD_TRANSLATIONS="false"
######################################################################
# superset-node-ci used as a base for building frontend assets and CI
######################################################################
FROM --platform=${BUILDPLATFORM} node:24-trixie-slim AS superset-node-ci
FROM --platform=${BUILDPLATFORM} node:22-trixie-slim AS superset-node-ci
ARG BUILD_TRANSLATIONS
ENV BUILD_TRANSLATIONS=${BUILD_TRANSLATIONS}
ARG DEV_MODE="false" # Skip frontend build in dev mode
@@ -55,13 +55,6 @@ WORKDIR /app/superset-frontend
RUN mkdir -p /app/superset/static/assets \
/app/superset/translations
# Harden `npm ci` against transient npm-registry network blips (e.g. ECONNRESET),
# which otherwise fail the entire multi-platform image build with no retry.
ENV npm_config_fetch_retries=5 \
npm_config_fetch_retry_mintimeout=20000 \
npm_config_fetch_retry_maxtimeout=120000 \
npm_config_fetch_timeout=600000
# Mount package files and install dependencies if not in dev mode
# NOTE: we mount packages and plugins as they are referenced in package.json as workspaces
# ideally we'd COPY only their package.json. Here npm ci will be cached as long
@@ -120,7 +113,7 @@ RUN useradd --user-group -d ${SUPERSET_HOME} -m --no-log-init --shell /bin/bash
# Some bash scripts needed throughout the layers
COPY --chmod=755 docker/*.sh /app/docker/
COPY --from=ghcr.io/astral-sh/uv:latest /uv /usr/local/bin/uv
RUN pip install --no-cache-dir --upgrade uv
# Using uv as it's faster/simpler than pip
RUN uv venv /app/.venv
+5 -12
View File
@@ -23,14 +23,11 @@ PYTHON=`command -v python3.11 || command -v python3.10`
install: superset pre-commit
superset:
# Bootstrap uv (the project's installer) into the active environment
pip install uv
# Install external dependencies
uv pip install -r requirements/development.txt
pip install -r requirements/development.txt
# Install Superset in editable (development) mode
uv pip install -e .
pip install -e .
# Create an admin user in your metadata database
superset fab create-admin \
@@ -55,14 +52,11 @@ superset:
update: update-py update-js
update-py:
# Bootstrap uv (the project's installer) into the active environment
pip install uv
# Install external dependencies
uv pip install -r requirements/development.txt
pip install -r requirements/development.txt
# Install Superset in editable (development) mode
uv pip install -e .
pip install -e .
# Initialize the database
superset db upgrade
@@ -85,8 +79,7 @@ activate:
pre-commit:
# setup pre commit dependencies
pip install uv
uv pip install -r requirements/development.txt
pip3 install -r requirements/development.txt
pre-commit install
format: py-format js-format
-5
View File
@@ -189,11 +189,6 @@ Try out Superset's [quickstart](https://superset.apache.org/docs/quickstart/) gu
- [Join our community's Slack](http://bit.ly/join-superset-slack)
and please read our [Slack Community Guidelines](https://github.com/apache/superset/blob/master/CODE_OF_CONDUCT.md#slack-community-guidelines)
- [Join our dev@superset.apache.org Mailing list](https://lists.apache.org/list.html?dev@superset.apache.org). To join, simply send an email to [dev-subscribe@superset.apache.org](mailto:dev-subscribe@superset.apache.org)
- Follow us on social media:
[X](https://x.com/apachesuperset) |
[LinkedIn](https://www.linkedin.com/company/apache-superset) |
[Bluesky](https://bsky.app/profile/apachesuperset.bsky.social) |
[Reddit](https://reddit.com/r/apache-superset)
- If you want to help troubleshoot GitHub Issues involving the numerous database drivers that Superset supports, please consider adding your name and the databases you have access to on the [Superset Database Familiarity Rolodex](https://docs.google.com/spreadsheets/d/1U1qxiLvOX0kBTUGME1AHHi6Ywel6ECF8xk_Qy-V9R8c/edit#gid=0)
- Join Superset's Town Hall and [Operational Model](https://preset.io/blog/the-superset-operational-model-wants-you/) recurring meetings. Meeting info is available on the [Superset Community Calendar](https://superset.apache.org/community)
+26 -46
View File
@@ -83,9 +83,6 @@ categories:
- name: Clark.de
url: https://clark.de/
- name: Cover Genius
url: https://covergenius.com/
- name: EnquiryLabs
url: https://www.enquirylabs.co.uk
@@ -95,10 +92,6 @@ categories:
- name: KarrotPay
url: https://www.daangnpay.com/
- name: NICE Actimize
url: https://www.niceactimize.com/
contributors: ["@stevensuting"]
- name: Remita
url: https://remita.net
contributors: ["@mujibishola"]
@@ -119,6 +112,9 @@ categories:
url: https://xendit.co/
contributors: ["@LieAlbertTriAdrian"]
- name: Cover Genius
url: https://covergenius.com/
Gaming:
- name: Popoko VM Games Studio
url: https://popoko.live
@@ -300,6 +296,7 @@ categories:
logo: hifadih.png
contributors: ["@saintLaurent00"]
# Logo approved by @anmol-hpe on behalf of HPE
- name: HPE
url: https://www.hpe.com/in/en/home.html
logo: hpe.png
@@ -399,10 +396,6 @@ categories:
url: https://www.techaudit.info
contributors: ["@ETselikov"]
- name: Tech Solution
url: https://www.tech-solution.com.ar/
contributors: ["@danteGiuliano", "@LeandroVallejos", "@McJaben", "@xJeree", "@zeo-return-null"]
- name: Tenable
url: https://www.tenable.com
contributors: ["@dflionis"]
@@ -432,10 +425,6 @@ categories:
logo: userguiding.svg
contributors: ["@tzercin"]
- name: Value Ad
url: https://bestpair.info/
contributors: ["@stevensuting"]
- name: Virtuoso QA
url: https://www.virtuosoqa.com
@@ -520,6 +509,10 @@ categories:
url: https://www.sunbird.org/
contributors: ["@eksteporg"]
- name: The GRAPH Network
url: https://thegraphnetwork.org/
contributors: ["@fccoelho"]
- name: Udemy
url: https://www.udemy.com/
contributors: ["@sungjuly"]
@@ -528,24 +521,7 @@ categories:
url: https://www.vipkid.com.cn/
contributors: ["@illpanda"]
Social Organization:
- name: Living Goods
url: https://www.livinggoods.org
contributors: ["@chelule"]
- name: One Acre Fund
url: https://oneacrefund.org/
contributors: ["@stevensuting"]
- name: Quest Alliance
url: https://www.questalliance.net/
contributors: ["@stevensuting"]
- name: The GRAPH Network
url: https://thegraphnetwork.org/
contributors: ["@fccoelho"]
- name: Wikimedia Foundation
- name: WikiMedia Foundation
url: https://wikimediafoundation.org
contributors: ["@vg"]
@@ -558,10 +534,6 @@ categories:
url: https://www.douroeci.com/
contributors: ["@nunohelibeires"]
- name: Rogow
url: https://rogow.com.br/
contributors: ["@nilmonto"]
- name: Safaricom
url: https://www.safaricom.co.ke/
contributors: ["@mmutiso"]
@@ -574,10 +546,11 @@ categories:
url: https://wattbewerb.de/
contributors: ["@wattbewerb"]
Healthcare:
- name: 2070Health
url: https://2070health.com/
- name: Rogow
url: https://rogow.com.br/
contributors: ["@nilmonto"]
Healthcare:
- name: Amino
url: https://amino.com
contributors: ["@shkr"]
@@ -590,6 +563,10 @@ categories:
url: https://www.getcare.io/
contributors: ["@alandao2021"]
- name: Living Goods
url: https://www.livinggoods.org
contributors: ["@chelule"]
- name: Maieutical Labs
url: https://maieuticallabs.it
contributors: ["@xrmx"]
@@ -608,10 +585,10 @@ categories:
- name: WeSure
url: https://www.wesure.cn/
HR / Staffing:
- name: bluquist
url: https://bluquist.com/
- name: 2070Health
url: https://2070health.com/
HR / Staffing:
- name: Swile
url: https://www.swile.co/
contributors: ["@PaoloTerzi"]
@@ -619,18 +596,21 @@ categories:
- name: Symmetrics
url: https://www.symmetrics.fyi
- name: bluquist
url: https://bluquist.com/
Government:
- name: City of Ann Arbor, MI
url: https://www.a2gov.org/
contributors: ["@sfirke"]
- name: NRLM - Sarathi, India
url: https://pib.gov.in/PressReleasePage.aspx?PRID=1999586
- name: RIS3 Strategy of CZ, MIT CR
url: https://www.ris3.cz/
contributors: ["@RIS3CZ"]
- name: NRLM - Sarathi, India
url: https://pib.gov.in/PressReleasePage.aspx?PRID=1999586
Mobile Software:
- name: VLMedia
url: https://www.vlmedia.com.tr
-145
View File
@@ -1,145 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Security Policy
This is a project of the [Apache Software Foundation](https://apache.org) and follows the
ASF [vulnerability handling process](https://apache.org/security/#vulnerability-handling).
## Reporting Vulnerabilities
**⚠️ Please do not file GitHub issues for security vulnerabilities as they are public! ⚠️**
Apache Software Foundation takes a rigorous standpoint in resolving the security issues
in its software projects. Apache Superset is highly sensitive and forthcoming to issues
pertaining to its features and functionality.
If you have any concern or believe you have found a vulnerability in Apache Superset,
please get in touch with the Apache Superset Security Team privately at
e-mail address [security@superset.apache.org](mailto:security@superset.apache.org).
More details can be found on the ASF website at
[ASF vulnerability reporting process](https://apache.org/security/#reporting-a-vulnerability)
**Submission Standards & AI Policy**
To ensure engineering focus remains on verified risks and to manage high reporting volumes, all reports must meet the following criteria:
- Plain Text Format: In accordance with Apache guidelines, please provide all details in plain text within the email body. Avoid sending PDFs, Word documents, or password-protected archives.
- Mandatory AI Disclosure: If you utilized Large Language Models (LLMs) or AI tools to identify a flaw or assist in writing a report, you must disclose this in your submission so our triage team can contextualize the findings.
- Human-Verified PoC: All submissions must include a manual, step-by-step Proof of Concept (PoC) performed on a supported release. Raw AI outputs, hypothetical chat transcripts, or unverified scanner logs will be closed as Invalid.
We kindly ask you to include the following information in your report to assist our developers in triaging and remediating issues efficiently:
- Version/Commit: The specific version of Apache Superset or the Git commit hash you are using.
- Configuration: A sanitized copy of your `superset_config.py` file or any config overrides.
- Environment: Your deployment method (e.g., Docker Compose, Helm, or source) and relevant OS/Browser details.
- Impacted Component: Identification of the affected area (e.g., Python backend, React frontend, or a specific database connector).
- Expected vs. Actual Behavior: A clear description of the intended system behavior versus the observed vulnerability.
- Detailed Reproduction Steps: Clear, manual steps to reproduce the vulnerability.
## Security Model
This section defines what Apache Superset considers a security issue and what it does not. It is the canonical reference for reporters, the Apache Superset Security Team, and any automated tool (LLM-based scanner, static analyzer, dependency tool) that needs to constrain its hypotheses to behaviors that genuinely violate the project's security policy.
The model is intentionally written in terms of principals, trust boundaries, and capability surface rather than in terms of specific files, functions, or libraries. New code paths inherit the model automatically.
### Trust Boundaries
Apache Superset's threat model assumes three trust boundaries.
1. *The Admin role* is a fully trusted operational principal. Anything an Admin can do through the documented user interface, REST API, or configuration system is an intended capability, not a vulnerability, even if individually powerful or destructive. The Admin role is, by policy, equivalent to operating-system-level trust over the Apache Superset application. This is unavoidable rather than aspirational: an Admin can, for example, register new database connections of arbitrary type, execute arbitrary SQL through SQL Lab, render Jinja templates that resolve to SQL or rendered HTML, and override application configuration. Granting Admin is functionally equivalent to granting shell access on the host, which is the reasoning behind treating it as a trust boundary in the sense of MITRE CNA Operational Rules 4.1.
2. *The operator* is whoever deploys, configures, and runs Apache Superset. Behaviors that depend on deployment-time decisions are the operator's responsibility, not Apache Superset's. This includes the values of secrets, the network reachability of the application and its data sources, the choice of database connectors and cache backends, the selection of feature flags, the destinations of notifications, and the trust placed in third-party plugins. Defaults that fail closed are the responsibility of the Apache Superset codebase. Defaults that fail open must be accompanied by a documented hardening requirement; applying that hardening is the operator's responsibility, while shipping an undocumented or unflagged fail-open default is a codebase issue.
3. *The Apache Superset codebase* is responsible for enforcing the role and capability matrix below across its product surface. A failure to enforce, anywhere in that surface, is in scope. The codebase's commitments are limited to the role and capability matrix and to controls Apache Superset's own documentation (this file and the linked Security documentation) explicitly positions as security boundaries; configurable hardening that operators can layer on top is treated separately under *Vulnerability Scope* below.
### Roles and Capabilities
Apache Superset ships with the following first-class principals. Detailed permission definitions live in the [Security documentation](https://superset.apache.org/docs/security).
| Principal | Read data | Write objects | Execute SQL | Manage databases | Manage users, roles, RLS |
|---|---|---|---|---|---|
| Public (anonymous) | none by default | no | no | no | no |
| Gamma | only granted datasets | own charts and dashboards on granted datasets | no by default (requires the `sql_lab` role) | no | no |
| Alpha | all data sources | own charts, dashboards, and datasets | no by default (requires the `sql_lab` role) | data upload to existing databases only | no |
| Admin | all | all | yes | yes | yes |
| Embedded guest token | data sources reachable through the embedded dashboards the token authorizes | no | no | no | no |
The `sql_lab` role is *additive*: it grants the SQL Lab permission set on top of the base role above, and is the only path by which Gamma or Alpha gain SQL execution capability. Database access is still scoped per the base role's grants. Admin includes SQL Lab access by default.
Deployments may grant or revoke individual view-menu permissions, which shifts the boundary for that deployment but does not redefine the model. Any custom role created by an operator inherits the same principle: its capabilities are whatever the operator has explicitly granted it. The Public principal follows the same rule: operators may grant the Public role read access to specific datasets or dashboards (typically for anonymous reporting use cases), which shifts the boundary for that deployment without redefining the model.
### Vulnerability Scope
The test for whether a finding is in scope is a single question:
> *Does this finding let a principal perform an action the role and capability matrix above does not entitle them to?*
If yes, it is in scope. If no, it is out of scope. The lists below apply that test to the classes Apache Superset most commonly receives reports about; they are illustrative, not exhaustive.
*In Scope*
- A user, embedded guest, or anonymous visitor reads, modifies, or deletes data outside their granted set. Includes object-level access bypass on charts, dashboards, datasets, saved queries, tags, annotations, and similar per-object endpoints, and row-level-security rule bypass.
- A user supplies input that the codebase should sanitize or parameterize but does not, causing arbitrary SQL, template code, or scripts to execute. Includes injection through Jinja templates, SQL-construction paths, and any field the codebase passes to a query engine or template engine.
- A user bypasses authentication, fixates or reuses another user's session, or reaches an authenticated endpoint without logging in.
- An embedded guest token authorizes actions outside the dashboard it was issued for, or can be forged, replayed, or escalated to a higher principal.
- Apache Superset, acting on behalf of an unprivileged user, fetches an outbound URL the user controls in a feature where Apache Superset itself, not the operator, controls the outbound destination set (server-side request forgery).
- An Apache Superset default fails open without an accompanying documented hardening requirement. The codebase is responsible for shipping fail-closed defaults or for documenting the hardening required when a default fails open; failures of that responsibility are in scope (see *Trust Boundaries*).
- A user bypasses a control Apache Superset documents specifically as a security boundary. This includes row-level security, the access checks tied to the role and capability matrix above, and any feature whose documentation positions it as security-relevant. The codebase commits to enforcing those controls; bypasses are in scope regardless of which principal triggers them.
- A user causes a script to execute in another user's browser through a field the codebase renders to that other user (cross-site scripting), or causes cross-origin leakage of authenticated session state or data.
- A user reaches a route, page, or API endpoint that requires a role they do not have.
*Out of Scope*
- Any action an Admin role can perform through documented configuration, API, or UI. The Admin role is a trusted operational principal by policy. Per MITRE CNA Operational Rules 4.1, a qualifying vulnerability must violate a security policy; behavior within a documented trust boundary does not.
- Deployment or operator decisions: the values of secrets and tokens, whether internal networks are reachable from the server, which database connectors or cache backends are enabled, which feature flags are set, where notifications are delivered, and which third-party plugins are loaded.
- Compromise, modification, or malicious control of trusted backend infrastructure. Apache Superset assumes the integrity of its metastore, cache backends (for example Redis or Memcached), message brokers, secret stores, and other operator-managed infrastructure. Findings that require an attacker to read from, write to, or otherwise tamper with these systems, including injecting malicious state, serialized objects, cache entries, task metadata, configuration, or database records, are post-compromise scenarios and do not constitute vulnerabilities in Apache Superset itself. A finding remains in scope only if an unprivileged user can cause such modification through a vulnerability in Apache Superset.
- The continued presence of expired key-value or metastore-cache entries that have not yet been deleted from the metadata database. Such entries are excluded from reads once expired, are purged opportunistically on write, and are removed in bulk by the scheduled `prune_key_value` maintenance task; their lingering until purged is an eventual-cleanup property, not a security boundary, and does not constitute a vulnerability.
- How a downstream application (spreadsheet program, email client, browser handling user-downloaded files) interprets output Apache Superset produced for it.
- Findings without a reproducible proof of concept against a supported release. The burden of demonstrating exploitability rests with the reporter; findings closed for lack of a proof of concept may be refiled if one is later produced.
- Brute force, rate limiting, denial of service, or resource exhaustion that does not bypass a documented control.
- Missing security headers, banner or version disclosure, user or object enumeration through error messages or timing, and similar low-impact information disclosure that does not enable a further concrete exploit.
- Bypasses of configurable defense-in-depth hardening that Apache Superset does not document as a security boundary. Apache Superset is not a SQL or database firewall: operator-deployable filters such as SQL function or table denylists, URI restrictions on already-authorized database connectors, and similar belt-and-braces controls are provided to let operators layer hardening on top of the role and capability matrix, not as firewall-grade guarantees the codebase commits to. Findings against such hardening are improvements, not vulnerabilities, unless the documentation positions the specific control as security-relevant.
- Hardening suggestions that improve defense in depth but do not violate the security model.
Findings in third-party dependencies fall into two cases. A finding in a transitive dependency, or in an operator-selected dependency that Apache Superset does not ship, is out of scope and should be reported to the dependency's maintainers. A finding caused by Apache Superset pinning a known-vulnerable version of a direct dependency it ships, or using a dependency in a way that creates a vulnerability the dependency itself does not have, remains in scope. Dependency findings in the official Apache Superset Docker image that fall into the first case can be remediated by extending the image at release time.
When uncertain whether a finding falls in scope, please file it through the reporting process above. The triage team will classify it and explain the reasoning if it is closed as out of scope.
**Outcome of Reports**
Reports that are deemed out of scope for a CVE but represent valid security best practices or hardening opportunities are typically converted into public GitHub issues, where the community can contribute fixes alongside the maintainers. The triage decision and reasoning are communicated back to the reporter in either case.
**Vulnerability Aggregation & CVE Attribution**
In accordance with MITRE CNA Operational Rules (4.1.10, 4.1.11, and 4.2.13), Apache Superset issues CVEs based on the underlying architectural root cause rather than the number of affected endpoints or exploit payloads.
- Aggregation: If multiple exploit vectors stem from the same programmatic failure or shared vulnerable code, they must be aggregated into a single, comprehensive report.
- Independent Fixes: Separate CVEs will only be assigned if the vulnerabilities reside in decoupled architectural modules and can be fixed independently of one another.
Reports that fail to aggregate related findings will be merged during triage to ensure an accurate and defensible CVE record.
**Your responsible disclosure and collaboration are invaluable.**
## Extra Information
- [Apache Superset documentation](https://superset.apache.org/docs/security)
- [Common Vulnerabilities and Exposures by release](https://superset.apache.org/docs/security/cves)
- [How Security Vulnerabilities are Reported & Handled in Apache Superset (Blog)](https://preset.io/blog/how-security-vulnerabilities-are-reported-and-handled-in-apache-superset/)
**For LLM agents and automated scanners**
This file is the canonical source of truth for what Apache Superset considers a security vulnerability. Automated tooling (LLM-based code scanners, static analyzers, dependency tools) should treat the Security Model section as authoritative when classifying findings. The repository's [AGENTS.md](AGENTS.md) file contains a short pointer to this document for agents that read AGENTS.md as their entry point.
+324 -147
View File
@@ -24,179 +24,119 @@ assists people when migrating to a new version.
## Next
### Pivot table First/Last aggregations follow data order
### Entity version history for charts, dashboards, and datasets
The pivot table chart's `First` and `Last` aggregations now return the first and last value in data (query result) order, instead of effectively returning the minimum and maximum. Existing pivot tables that use these aggregations for totals/subtotals may show different values after upgrading. For deterministic results, ensure the underlying query has a stable sort order.
Saves of charts, dashboards, and datasets now automatically produce a version history — browsable and restorable via new API endpoints. No frontend UI in this release; the backend plumbing is the deliverable.
### `thumbnail_url` removed from dashboard list API response
**New endpoints** (per entity type — same pattern for `chart`, `dashboard`, and `dataset`):
The `thumbnail_url` field has been removed from `GET /api/v1/dashboard/` list responses. External consumers relying on this field must now construct the thumbnail URL client-side using `id` and `changed_on_utc`:
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/v1/{resource}/<uuid>/versions/` | List the entity's version history (0-based `version_number`, `version_uuid`, `issued_at`, `changed_by`) |
| `GET` | `/api/v1/{resource}/<uuid>/versions/<version_uuid>/` | Get a single version snapshot (scalar fields at that version; plus `columns` / `metrics` for datasets) |
| `POST` | `/api/v1/{resource}/<uuid>/versions/<version_uuid>/restore` | Restore the entity to the state captured by that version |
```
/api/v1/dashboard/{id}/thumbnail/{changed_on_utc}/
`<version_uuid>` is a deterministic `UUIDv5` derived from the entity's UUID and the Continuum transaction id — stable across replicas and retention pruning. Authorisation reuses the resource's existing `can_write` permission; workspace admins can list/restore any entity.
**Version response shape — `changes` array:**
Each entry returned by `GET /api/v1/{resource}/<uuid>/versions/` and `GET .../versions/<version_uuid>/` includes a `changes` array describing what changed relative to the previous version:
```json
"changes": [
{"kind": "field", "path": "slice_name", "from_value": "Old", "to_value": "New"}
]
```
The thumbnail endpoint redirects to the current digest URL regardless of whether the supplied digest is exact. If the image is not yet cached, that digest URL may return `202` and trigger async generation. Using `changed_on_utc` as the digest is sufficient for cache-busting purposes.
The array is empty for baseline (`operation_type=0`) transactions. `kind` enumerates structured record types (`field`, layout-walker records for dashboards, dataset child diffs for `TableColumn` / `SqlMetric`); `path` is a dotted JSON-pointer-style locator; `from_value` / `to_value` are JSON-safe scalars or compact records.
### Webhook alerts/reports block private/internal hosts by default
**Save-response and ETag headers:**
Webhook alert/report dispatch (`WebhookNotification.send`) now validates the target URL's host against the same private/internal-IP block applied to dataset import URLs. If the resolved host is in a loopback, link-local, private (RFC-1918), shared-CGNAT, or multicast range, the webhook is rejected with `NotificationParamException`.
- Save responses (`PUT /api/v1/{resource}/<pk>`) include `old_version_uuid` and `new_version_uuid` body fields so the client can correlate a save with its resulting version row.
- All entity GETs (`GET /api/v1/{chart,dashboard,dataset}/<pk>`), version-list GETs, single-version GETs, and save responses emit an `ETag: "<version_uuid>"` header reflecting the entity's current live version. The default `CORS_OPTIONS` now sets `expose_headers: ["ETag"]` so cross-origin browser clients can read the header. **No `If-Match` enforcement in v1**`ETag` is informational; concurrent-edit detection is deferred to a follow-up SIP.
- **Operators overriding `CORS_OPTIONS` in `superset_config.py` MUST include `"expose_headers": ["ETag"]`** (or merge with the default) for cross-origin clients to read the ETag. A bare `CORS_OPTIONS = {"origins": [...]}` will silently drop the expose-headers default.
Deployments that intentionally point webhooks at internal targets (chatops bridges, internal automation servers, on-premises Mattermost/Rocket.Chat, etc.) can opt out by setting `ALERT_REPORTS_WEBHOOK_ALLOW_INTERNAL_HOSTS = True` in `superset_config.py`. This mirrors the existing `DATASET_IMPORT_ALLOW_INTERNAL_DATA_URLS` opt-out for dataset imports.
**Behaviour changes on save:**
### Impala cancel_query blocks private/internal hosts by default
- Every save of a chart, dashboard, or dataset produces one new version row. Rows preserve the full post-save state (scalar fields for all three entity types; `TableColumn` / `SqlMetric` children for datasets; `dashboard_slices` chart membership for dashboards — children versioned via SQLAlchemy-Continuum shadow tables `table_columns_version`, `sql_metrics_version`, and `dashboard_slices_version`).
- First save after an entity already exists in the DB creates a retroactive baseline version so the UI can show "what this looked like before I edited it."
- Tags, owners, and roles are **not** versioned in v1 (ADR-005). A restore leaves those at their live values.
The Impala engine spec's `cancel_query` issues an HTTP request from the Superset backend to the host configured on the Impala database connection. That host is now validated before the request: if it resolves to a private/internal IP range, the cancel call is refused and a warning is logged. Operators whose Impala cluster runs on an internal network can opt out by setting `IMPALA_CANCEL_QUERY_ALLOW_INTERNAL_HOSTS = True` in `superset_config.py`. This mirrors the dataset-import and webhook opt-out flags.
### Map chart renderer and OpenStreetMap migration behavior
**New config key:**
The MapLibre migration for deck.gl charts preserves saved non-Mapbox styles on
the MapLibre-compatible path. Saved styles such as OpenStreetMap, `tile://`
tile templates, generic HTTPS style URLs, and charts without a saved style are
not reclassified as Mapbox during migration and do not require
`MAPBOX_API_KEY` only because of the migration.
| Key | Default | Purpose |
|---|---|---|
| `SUPERSET_VERSION_HISTORY_RETENTION_DAYS` | `30` | Versions older than this many days are pruned by a nightly Celery beat task (`superset.tasks.version_history_retention.prune_old_versions`). Each entity's live row (`end_transaction_id IS NULL`) is always preserved; closed historical rows including the baseline age out with the rest. Set to `0` to disable retention entirely. |
Saved true Mapbox styles whose value starts with `mapbox://` remain
Mapbox-backed. If a Superset deployment does not configure `MAPBOX_API_KEY`,
those saved Mapbox charts keep the existing missing-key message instead of
silently falling back to MapLibre or another provider. In Explore, deck.gl and
point-cluster renderer controls preserve saved Mapbox state, but the Mapbox
choice is not available as a new working renderer without a configured key.
**Impact on external integrations:**
The MapLibre style choices include `Streets (OSM)`, backed by
`https://tile.openstreetmap.org/{z}/{x}/{y}.png`. This OpenStreetMap tile
service requires visible `© OpenStreetMap contributors` attribution and should
be used through normal browser map tile requests and caching; it is not intended
for bulk prefetch or offline tile downloads.
- New tables populated on every save — `dashboards_version`, `slices_version`, `tables_version` (parent shadow tables for the three entity types), `table_columns_version`, `sql_metrics_version`, `dashboard_slices_version` (child shadow tables), plus the shared `version_transaction` and `version_changes` tables. External tooling that queries Superset's DB directly will see writes to these tables proportional to save traffic.
- Existing entity endpoints (`GET`/`PUT /api/v1/{chart,dashboard,dataset}/<pk>`) gain an `ETag` response header and the save response gains `old_version_uuid` / `new_version_uuid` body fields. No existing fields are removed or repurposed.
- Version capture is always active — no feature flag.
### Password complexity policy enabled by default
### Cross-entity activity stream for charts, dashboards, and datasets
Superset now ships a default password-complexity policy, enforced (via Flask-AppBuilder) across self-registration, the user create/edit/reset forms, and the User REST API. The policy requires a minimum password length of 8 characters and rejects a built-in blocklist of common/guessable passwords.
A read-only companion to the version-history endpoints (above). Each entity type gains an `/activity/` endpoint that returns a chronological stream of edits — the entity's own edits plus, for dashboards and charts, transitive edits to related entities during their association windows.
This is enabled by default (`FAB_PASSWORD_COMPLEXITY_ENABLED = True`), so new or reset passwords that are too short or appear in the blocklist will be rejected where they were previously accepted. Existing stored passwords are unaffected until they are next changed.
**New endpoints** (per entity type):
Operators can tune or disable the policy via config:
| Method | Path | Purpose |
|---|---|---|
| `GET` | `/api/v1/dashboard/<uuid>/activity/` | Dashboard own edits + edits to charts attached during their dashboard window + edits to datasets those charts pointed at during their chart window |
| `GET` | `/api/v1/chart/<uuid>/activity/` | Chart own edits + edits to datasets the chart pointed at during association |
| `GET` | `/api/v1/dataset/<uuid>/activity/` | Dataset own edits only (no transitive layer in V2) |
- `AUTH_PASSWORD_MIN_LENGTH` — minimum length (default `8`).
- `AUTH_PASSWORD_COMMON_BLOCKLIST` — extra passwords to reject, in addition to the built-in list.
- `FAB_PASSWORD_COMPLEXITY_VALIDATOR` — replace with your own callable for custom rules.
- `FAB_PASSWORD_COMPLEXITY_ENABLED = False` — disable enforcement entirely.
**Query parameters** (all optional):
### Data uploads bounded by UPLOAD_MAX_FILE_SIZE_BYTES
| Param | Type | Default | Purpose |
|---|---|---|---|
| `since` | ISO 8601 datetime | — | Lower bound on `issued_at` |
| `until` | ISO 8601 datetime | — | Upper bound on `issued_at` |
| `include` | `self` \| `related` \| `all` | `all` | Filter to only the entity's own edits, only related edits, or both |
| `page` | integer ≥ 0 | `0` | 0-based page index |
| `page_size` | integer in `[1, 200]` | `25` | Records per page (clamped silently to 200) |
Single data-file uploads (CSV, Excel, columnar) are now bounded by the `UPLOAD_MAX_FILE_SIZE_BYTES` config option, which defaults to `100 * 1024 * 1024` (100 MB). Files larger than this are rejected with a `413` before their contents are buffered into memory. Set `UPLOAD_MAX_FILE_SIZE_BYTES = None` to disable the check and restore unbounded uploads.
**Response shape:**
### Duration formatter precision
The `DURATION` number formatter now uses `Intl.DurationFormat` for locale-aware output. By default, sub-second fields are omitted, so values that previously displayed fractional seconds with `pretty-ms`, such as `10500` milliseconds rendering as `10.5s`, now render as `10s`.
To preserve sub-second precision in custom duration formatters, enable `formatSubMilliseconds`.
### Cache warmup authenticates via SUPERSET_CACHE_WARMUP_USER
The `cache-warmup` Celery task now drives a real WebDriver session for reliable authentication and reads the user to authenticate as from the new `SUPERSET_CACHE_WARMUP_USER` config option. It no longer consults `CACHE_WARMUP_EXECUTORS` for the warmup path. `SUPERSET_CACHE_WARMUP_USER` defaults to `None`, so the task fails fast with a clear message until you set it. Operators who previously relied on `CACHE_WARMUP_EXECUTORS` for cache warmup must set `SUPERSET_CACHE_WARMUP_USER` to a dedicated least-privilege user with access to the dashboards they want warmed up before the next warmup run.
### YDB now uses a native sqlglot dialect
YDB SQL parsing now relies on the dedicated [`ydb-sqlglot-plugin`](https://pypi.org/project/ydb-sqlglot-plugin/) dialect, which registers itself with sqlglot automatically. YDB users must install this plugin (e.g., via `pip install "apache-superset[ydb]"`) to avoid a `ValueError` when Superset parses YDB queries.
### Embedded dashboards enforce configured Allowed Domains for postMessage
The embedded dashboard page now validates the origin of incoming `postMessage` events against the dashboard's configured **Allowed Domains**. The server-rendered embedded page exposes the configured domains in its bootstrap payload, and the frontend rejects message events whose origin is not in that list.
Enforcement only applies when the Allowed Domains list is non-empty. If the list is empty (the default), any origin is accepted, so there is no behavior change for embeds that did not configure Allowed Domains.
### Default guest/async JWT secrets are rejected at startup
Superset already refuses to start in production (non-debug, non-testing) when `SECRET_KEY` is left at its built-in default, and when `GUEST_TOKEN_JWT_SECRET` is left at its default while `EMBEDDED_SUPERSET` is enabled. This behavior is extended to `GLOBAL_ASYNC_QUERIES_JWT_SECRET`: if the `GLOBAL_ASYNC_QUERIES` feature flag is enabled and the secret is still the publicly known default (`test-secret-change-me`), Superset logs a clear error and refuses to start.
As with the existing `SECRET_KEY` check, this only fails in production. In debug mode, testing mode, or under the test runner, a warning is logged instead of exiting, so local development is unaffected.
To resolve the error, set a strong random value in `superset_config.py`:
```python
GLOBAL_ASYNC_QUERIES_JWT_SECRET = "<output of: openssl rand -base64 42>"
```json
{
"result": [
{
"version_uuid": "...",
"entity_kind": "chart",
"entity_uuid": "...",
"entity_name": "Top 10 Girls",
"entity_deleted": false,
"entity_deletion_state": null,
"source": "related",
"transaction_id": 1234,
"issued_at": "2026-05-26T12:00:00",
"changed_by": {"id": 5, "first_name": "Mike", "last_name": "Bridge"},
"kind": "filter",
"path": ["params", "adhoc_filters", "country"],
"from_value": null,
"to_value": "US",
"summary": "Chart filter changed: Top 10 Girls",
"impact": null
}
],
"count": 47
}
```
The check is only active when the relevant feature is enabled, so deployments that do not use global async queries (or embedding) are not affected.
`count` is the total record count *after* the silent permission filter (see below), not the raw query size.
### Guest token revocation (opt-in)
**Authorisation:** reuses the resource's existing `can_write` permission. Workspace admins can read any entity's activity stream. The endpoint runs `raise_for_ownership` on the path entity — non-owners get `403`.
Embedded guest tokens can be coarsely revoked at runtime via a new opt-in mechanism. A new config flag `GUEST_TOKEN_REVOCATION_ENABLED` (default `False`) gates the feature. When enabled, every minted guest token carries a revocation version, and tokens whose version is below the current expected version (stored in the metadata database) are rejected at validation time.
**Silent permission filter (AV-008):** records whose source entity the requesting user can't read are silently dropped — no placeholder, no count contribution. The frontend cannot distinguish "no activity" from "you can't see this activity."
Bump the expected version with the new CLI command to invalidate all outstanding guest tokens:
**Tombstones (AV-009 / D-15):** when an activity record references a hard-deleted source entity, the record still appears with `entity_deleted: true`, `entity_uuid: null`, and `entity_name` recovered from the last shadow row.
```bash
superset revoke-guest-tokens
```
**Impact on external integrations:**
This change is backward compatible. The feature is off by default, and even when enabled nothing is revoked until an admin explicitly bumps the version: the expected version starts at `0`, and tokens minted before this change (which carry no version claim) are treated as version `0`. No database migration is required.
### Sessions are terminated when an account is disabled
Disabling a user account (setting `active` to `False`, via the admin UI, REST API, or CLI) now terminates that user's outstanding sessions on their next request, instead of relying on a passive check. This works for both client-side cookie sessions and server-side session stores via a per-user invalidation epoch (`user_attribute.sessions_invalidated_at`, added by a migration). The mechanism is inert for users that were never disabled (NULL epoch), so there is no behavior change for active users. Re-enabling an account and logging in again starts a fresh, valid session. The migration backfills the epoch for accounts that are already disabled at upgrade time, so re-enabling such an account does not revive a session that predates this feature.
### Opt-in SSH tunnel server host key verification
SSH tunnels can now optionally pin the expected SSH server host key as a defense-in-depth measure against man-in-the-middle attacks. paramiko's transport performs no known-hosts checking by default, so previously the SSH server's identity was not verified. This feature is opt-in and off by default; existing tunnels are unaffected.
- A new nullable `server_host_key` column on the `ssh_tunnels` table stores the expected host key in authorized-key form (e.g. `ssh-ed25519 AAAA...`). It is a public key and is stored in plaintext. It can be set via the SSH tunnel POST/PUT payloads (`ssh_tunnel.server_host_key`).
- When a tunnel has `server_host_key` set, Superset connects to the SSH server, reads the host key it presents, and rejects the tunnel if it does not match.
- A new config flag `SSH_TUNNEL_STRICT_HOST_KEY_CHECKING` (default `False`) controls fail-closed behavior. When `True`, every tunnel must declare a `server_host_key`; a tunnel without one is rejected.
Runbook to adopt:
1. Capture the SSH server's host key, e.g. `ssh-keyscan -t ed25519 ssh.example.com` (verify it out-of-band).
2. Set that value on the tunnel's `server_host_key` (via the database/SSH tunnel API or UI payload).
3. Optionally set `SSH_TUNNEL_STRICT_HOST_KEY_CHECKING = True` in `superset_config.py` to require host-key verification on all tunnels.
### Dataset import validates catalog against the target connection
Importing a dataset now validates the `catalog` field against the target database connection. When the connection has multi-catalog disabled (`allow_multi_catalog` off) and the dataset's catalog is not the connection's default catalog, the import fails instead of silently persisting the non-default catalog. This matches the validation already enforced on the dataset update path and prevents imported datasets from querying an unintended database.
If you relied on importing datasets with a non-default catalog, enable "Allow changing catalogs" on the target connection, or set the dataset's catalog to the connection's default before importing.
### Extension supply-chain controls (denylist + version policy)
Two opt-in static gates control which extensions are allowed to load:
- `EXTENSION_DENYLIST` refuses extensions matching an id (every version) or `id@version` (a single version), e.g. `["compromised-extension", "other-ext@1.2.3"]`.
- `EXTENSION_VERSION_POLICY` enforces a minimum version per extension id, e.g. `{"acme.widget": "1.2.0"}` (PEP 440 comparison); a release below the minimum is refused.
Both default to empty (no behavior change). They apply to both the `LOCAL_EXTENSIONS` and `EXTENSIONS_PATH` load paths.
### Dynamic Group By respects the sort toggle for display values
The Dynamic Group By chart customization now orders its display values according to the "Sort display control values" toggle: ascending (AZ), descending (ZA), or the dataset's source order when the toggle is unset. Previously the dropdown always sorted alphabetically. Existing dashboards where the toggle was never set will show options in source order instead of AZ; open the customization and enable the toggle to restore alphabetical ordering.
### Selectable encryption engine for app-encrypted fields (AES-GCM)
App-encrypted fields (database passwords, SSH tunnel credentials, OAuth tokens, etc.) can now use authenticated **AES-GCM** encryption instead of the historical unauthenticated **AES-CBC**. A new config selects the engine for the default adapter:
```python
# "aes" (AES-CBC, historical default) | "aes-gcm" (authenticated, recommended for new installs)
SQLALCHEMY_ENCRYPTED_FIELD_ENGINE = "aes"
```
**No action required / no behavior change:** the default remains `"aes"`, so existing installs are unaffected.
**Opting in on an existing install:** flipping the engine on a populated database without re-encrypting first will make stored secrets undecryptable, because the two ciphertext formats are not compatible. A migrator is provided. Recommended runbook:
1. Take a metadata-DB backup.
2. Re-encrypt existing secrets into the new engine (the `SECRET_KEY` is unchanged):
```bash
superset re-encrypt-secrets --engine aes-gcm
```
3. Set `SQLALCHEMY_ENCRYPTED_FIELD_ENGINE = "aes-gcm"` in your config.
4. Restart Superset.
5. Re-run the migrator once more after the restart:
```bash
superset re-encrypt-secrets --engine aes-gcm
```
A live instance keeps writing *new* secrets as AES-CBC during the window between step 2 and the restart in step 4; this second pass sweeps those up (it is idempotent, so already-migrated values are skipped).
Schedule the cutover in a quiet window. Runtime reads use only the single configured engine, so in a multi-worker deployment there is an unavoidable brief decrypt-outage between the migration commit and the last worker restarting with the new config — each migrator run is transactional, but the fleet-wide cutover is not zero-downtime.
The migration is transactional (all-or-nothing) and idempotent — it can be safely re-run or resumed. Note that AES-GCM, unlike AES-CBC, does not support querying directly over encrypted columns; audit any code that filters on an encrypted column before switching. See the SIP at `docs/sip/authenticated-encryption-at-rest.md` for details.
- Pure read-only. No new tables, no new columns, no migrations. Reads sc-103156's shadow tables and the `version_changes` table.
- No new save-path code paths — perf-validation gate confirms the activity-view branch does not regress sc-103156's SC-004 50ms-overhead budget.
- No feature flag; the endpoints are always available once sc-103156's version-history feature is enabled.
### Granular Export Controls
@@ -227,9 +167,6 @@ Added a new combined datasource list endpoint at `GET /api/v1/datasource/` to se
- The endpoint is available to users with at least one of `can_read` on `Dataset` or `SemanticView`.
- Semantic views are included only when the `SEMANTIC_LAYERS` feature flag is enabled.
- The endpoint enforces strict `order_column` validation and returns `400` for invalid sort columns.
## 6.1.0
### ClickHouse minimum driver version bump
The minimum required version of `clickhouse-connect` has been raised to `>=0.13.0`. If you are using the ClickHouse connector, please upgrade your `clickhouse-connect` package. The `_mutate_label` workaround that appended hash suffixes to column aliases has also been removed, as it is no longer needed with modern versions of the driver.
@@ -487,6 +424,246 @@ See `superset/mcp_service/PRODUCTION.md` for deployment guides.
}
```
### Composite primary keys on many-to-many association tables
The eight M:N association tables listed below have been changed from a synthetic surrogate `id INTEGER PRIMARY KEY` to a composite `PRIMARY KEY (fk1, fk2)` on the two foreign-key columns. The `id` column is dropped, and the two tables that previously carried a redundant `UNIQUE (fk1, fk2)` constraint have that constraint removed (it is now subsumed by the composite primary key).
**Affected tables and their composite-PK column pairs:**
| Table | Composite PK |
|---|---|
| `dashboard_roles` | `(dashboard_id, role_id)` |
| `dashboard_slices` | `(dashboard_id, slice_id)` |
| `dashboard_user` | `(user_id, dashboard_id)` |
| `report_schedule_user` | `(user_id, report_schedule_id)` |
| `rls_filter_roles` | `(role_id, rls_filter_id)` |
| `rls_filter_tables` | `(table_id, rls_filter_id)` |
| `slice_user` | `(user_id, slice_id)` |
| `sqlatable_user` | `(user_id, table_id)` |
**Impact on external readers:** Any BI tool, custom report, backup script, or external integration that references these tables by their old surrogate `id` column (e.g., `SELECT id FROM dashboard_slices WHERE …`, `WHERE dashboard_slices.id IN (…)`) will break. Update such queries to project or filter on the FK pair (`dashboard_id, slice_id`) instead. The FK columns themselves are unchanged.
**Pre-flight inventory queries.** Before applying the upgrade, operators are encouraged to run the queries below against their database to assess what the migration will change. Two classes of pre-existing data are not preserved by the migration: duplicate `(fk1, fk2)` rows (the migration keeps `MIN(id)` and deletes the rest) and rows with `NULL` in either FK column (the migration deletes them, since FK columns are promoted to `NOT NULL` for the composite PK). Compliance- or audit-sensitive operators should also `\copy` (Postgres) or `SELECT … INTO OUTFILE` (MySQL) the affected rows for their own records before upgrading.
```sql
-- Duplicate (fk1, fk2) pairs (the migration will keep MIN(id) per group, delete the rest)
SELECT dashboard_id, role_id, COUNT(*) FROM dashboard_roles GROUP BY dashboard_id, role_id HAVING COUNT(*) > 1;
SELECT dashboard_id, slice_id, COUNT(*) FROM dashboard_slices GROUP BY dashboard_id, slice_id HAVING COUNT(*) > 1;
SELECT user_id, dashboard_id, COUNT(*) FROM dashboard_user GROUP BY user_id, dashboard_id HAVING COUNT(*) > 1;
SELECT user_id, report_schedule_id, COUNT(*) FROM report_schedule_user GROUP BY user_id, report_schedule_id HAVING COUNT(*) > 1;
SELECT role_id, rls_filter_id, COUNT(*) FROM rls_filter_roles GROUP BY role_id, rls_filter_id HAVING COUNT(*) > 1;
SELECT table_id, rls_filter_id, COUNT(*) FROM rls_filter_tables GROUP BY table_id, rls_filter_id HAVING COUNT(*) > 1;
SELECT user_id, slice_id, COUNT(*) FROM slice_user GROUP BY user_id, slice_id HAVING COUNT(*) > 1;
SELECT user_id, table_id, COUNT(*) FROM sqlatable_user GROUP BY user_id, table_id HAVING COUNT(*) > 1;
-- Rows with a NULL in either FK (the migration will delete these)
SELECT COUNT(*) FROM dashboard_roles WHERE dashboard_id IS NULL OR role_id IS NULL;
SELECT COUNT(*) FROM dashboard_slices WHERE dashboard_id IS NULL OR slice_id IS NULL;
SELECT COUNT(*) FROM dashboard_user WHERE user_id IS NULL OR dashboard_id IS NULL;
SELECT COUNT(*) FROM report_schedule_user WHERE user_id IS NULL OR report_schedule_id IS NULL;
SELECT COUNT(*) FROM rls_filter_roles WHERE role_id IS NULL OR rls_filter_id IS NULL;
SELECT COUNT(*) FROM rls_filter_tables WHERE table_id IS NULL OR rls_filter_id IS NULL;
SELECT COUNT(*) FROM slice_user WHERE user_id IS NULL OR slice_id IS NULL;
SELECT COUNT(*) FROM sqlatable_user WHERE user_id IS NULL OR table_id IS NULL;
```
**Sizing the maintenance window on PostgreSQL.** The queries above are dialect-portable but only count rows. Operators on PostgreSQL can run the diagnostic queries below to characterize the migration's runtime cost ahead of time: per-table row count and on-disk size, an aggregated duplicate roll-up, the external-FK pre-flight check (the migration runs the same check and aborts if it returns rows), and a rewrite-time estimate for the two tables that go through the slower full-table-rebuild path.
```sql
-- Per-table size, row count, and which migration path each will take.
-- Two tables ("dashboard_slices", "report_schedule_user") have a
-- redundant UNIQUE constraint that the migration drops via a full
-- table rewrite (op.batch_alter_table(recreate="always")). The other
-- six use direct ALTER TABLE, which is much cheaper.
WITH affected(name, has_unique) AS (
VALUES
('dashboard_roles', false),
('dashboard_slices', true),
('dashboard_user', false),
('report_schedule_user', true),
('rls_filter_roles', false),
('rls_filter_tables', false),
('slice_user', false),
('sqlatable_user', false)
)
SELECT
a.name AS table_name,
CASE WHEN a.has_unique THEN 'recreate (full rewrite)'
ELSE 'direct ALTER' END AS migration_path,
c.reltuples::bigint AS estimated_rows,
pg_size_pretty(pg_total_relation_size(c.oid)) AS total_size,
pg_size_pretty(pg_relation_size(c.oid)) AS heap_size,
pg_size_pretty(pg_indexes_size(c.oid)) AS index_size
FROM affected a
JOIN pg_class c ON c.relname = a.name AND c.relkind = 'r'
ORDER BY pg_total_relation_size(c.oid) DESC;
```
```sql
-- Aggregated duplicate-row roll-up.
-- "dup_groups" is the number of (fk1, fk2) pairs that appear more
-- than once; "rows_dropped" is the total number of rows the
-- migration will delete during the dedupe pass (it keeps MIN(id) per
-- group and discards the rest).
SELECT 'dashboard_roles' AS t, COUNT(*) AS dup_groups, SUM(c) - COUNT(*) AS rows_dropped
FROM (SELECT COUNT(*) c FROM dashboard_roles GROUP BY dashboard_id, role_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'dashboard_slices', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM dashboard_slices GROUP BY dashboard_id, slice_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'dashboard_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM dashboard_user GROUP BY user_id, dashboard_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'report_schedule_user',COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM report_schedule_user GROUP BY user_id, report_schedule_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'rls_filter_roles', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM rls_filter_roles GROUP BY role_id, rls_filter_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'rls_filter_tables', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM rls_filter_tables GROUP BY table_id, rls_filter_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'slice_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM slice_user GROUP BY user_id, slice_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'sqlatable_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM sqlatable_user GROUP BY user_id, table_id HAVING COUNT(*) > 1) g
ORDER BY rows_dropped DESC NULLS LAST;
```
```sql
-- External-FK pre-flight check.
-- The migration runs the equivalent check at upgrade time and aborts
-- if any external FK references one of the soon-to-be-removed `id`
-- columns. Running it ahead of time lets you discover (and migrate)
-- any such reference before the maintenance window. On a stock
-- Superset install this should return zero rows. (Default schema
-- only; multi-schema deployments need to broaden the lookup.)
SELECT
rc.constraint_name,
kcu.table_schema || '.' || kcu.table_name AS referencing_table,
kcu.column_name AS referencing_column,
ccu.table_name AS referenced_table,
ccu.column_name AS referenced_column
FROM information_schema.referential_constraints rc
JOIN information_schema.key_column_usage kcu
ON kcu.constraint_name = rc.constraint_name
AND kcu.constraint_schema = rc.constraint_schema
JOIN information_schema.constraint_column_usage ccu
ON ccu.constraint_name = rc.constraint_name
AND ccu.constraint_schema = rc.constraint_schema
WHERE ccu.table_name IN (
'dashboard_roles','dashboard_slices','dashboard_user',
'report_schedule_user','rls_filter_roles','rls_filter_tables',
'slice_user','sqlatable_user')
AND ccu.column_name = 'id';
```
```sql
-- Lock-window estimate for the two full-rewrite tables.
-- recreate="always" takes ACCESS EXCLUSIVE on the table for the full
-- rewrite. Use heap size combined with your hardware's effective
-- write throughput (~100-200 MB/s on commodity SSD; faster on NVMe)
-- to size the maintenance window. The other six tables use direct
-- ALTER and are dominated by composite-index build time, typically
-- seconds for tables in the low millions of rows.
SELECT
c.relname AS table_name,
pg_size_pretty(pg_relation_size(c.oid)) AS heap_size,
pg_relation_size(c.oid) / 1024 / 1024 AS heap_size_mb,
ROUND(pg_relation_size(c.oid) / 1024 / 1024 / 100.0, 1) AS est_rewrite_seconds_at_100mbs
FROM pg_class c
WHERE c.relname IN ('dashboard_slices', 'report_schedule_user');
```
**Sizing the maintenance window on MySQL.** Equivalent diagnostic queries for MySQL/InnoDB. One important difference from PostgreSQL: InnoDB rebuilds the clustered index on every PK change, so *all eight* tables undergo a full table rebuild on MySQL — not just the two that go through the explicit `recreate="always"` path. The lock-window estimate query below therefore covers all eight tables.
```sql
-- Per-table size, row count, and which migration path each will take.
-- TABLE_ROWS is an InnoDB estimate (analogous to PostgreSQL's reltuples);
-- run SELECT COUNT(*) per table for an exact count if needed.
SELECT
TABLE_NAME AS table_name,
CASE WHEN TABLE_NAME IN ('dashboard_slices', 'report_schedule_user')
THEN 'recreate (explicit, drops UNIQUE)'
ELSE 'direct ALTER (still rebuilds InnoDB clustered index)'
END AS migration_path,
TABLE_ROWS AS estimated_rows,
CONCAT(ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024 / 1024, 1), ' MB') AS total_size,
CONCAT(ROUND(DATA_LENGTH / 1024 / 1024, 1), ' MB') AS heap_size,
CONCAT(ROUND(INDEX_LENGTH / 1024 / 1024, 1), ' MB') AS index_size
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME IN (
'dashboard_roles', 'dashboard_slices', 'dashboard_user',
'report_schedule_user', 'rls_filter_roles', 'rls_filter_tables',
'slice_user', 'sqlatable_user'
)
ORDER BY (DATA_LENGTH + INDEX_LENGTH) DESC;
```
```sql
-- Aggregated duplicate-row roll-up. Same SQL as the PostgreSQL version
-- (standard SQL); included here for copy-paste convenience.
SELECT 'dashboard_roles' AS t, COUNT(*) AS dup_groups, SUM(c) - COUNT(*) AS rows_dropped
FROM (SELECT COUNT(*) c FROM dashboard_roles GROUP BY dashboard_id, role_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'dashboard_slices', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM dashboard_slices GROUP BY dashboard_id, slice_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'dashboard_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM dashboard_user GROUP BY user_id, dashboard_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'report_schedule_user',COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM report_schedule_user GROUP BY user_id, report_schedule_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'rls_filter_roles', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM rls_filter_roles GROUP BY role_id, rls_filter_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'rls_filter_tables', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM rls_filter_tables GROUP BY table_id, rls_filter_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'slice_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM slice_user GROUP BY user_id, slice_id HAVING COUNT(*) > 1) g
UNION ALL SELECT 'sqlatable_user', COUNT(*), SUM(c) - COUNT(*)
FROM (SELECT COUNT(*) c FROM sqlatable_user GROUP BY user_id, table_id HAVING COUNT(*) > 1) g
ORDER BY rows_dropped DESC;
```
```sql
-- External-FK pre-flight check. KEY_COLUMN_USAGE on MySQL carries
-- both sides of the FK in a single row, so this is simpler than the
-- PostgreSQL version. Should return zero rows on a stock install.
SELECT
CONSTRAINT_NAME,
CONCAT(TABLE_SCHEMA, '.', TABLE_NAME) AS referencing_table,
COLUMN_NAME AS referencing_column,
REFERENCED_TABLE_NAME AS referenced_table,
REFERENCED_COLUMN_NAME AS referenced_column
FROM information_schema.KEY_COLUMN_USAGE
WHERE TABLE_SCHEMA = DATABASE()
AND REFERENCED_TABLE_NAME IN (
'dashboard_roles', 'dashboard_slices', 'dashboard_user',
'report_schedule_user', 'rls_filter_roles', 'rls_filter_tables',
'slice_user', 'sqlatable_user'
)
AND REFERENCED_COLUMN_NAME = 'id';
```
```sql
-- Lock-window estimate for ALL EIGHT tables (InnoDB rebuilds the
-- clustered index on PK change, so even "direct ALTER" is a rewrite).
-- ADD PRIMARY KEY is INPLACE but not LOCK=NONE — it allows concurrent
-- reads but blocks writes. Use heap size combined with your effective
-- rebuild throughput (~100-200 MB/s on commodity SSD; higher on NVMe).
SELECT
TABLE_NAME AS table_name,
CONCAT(ROUND(DATA_LENGTH / 1024 / 1024, 1), ' MB') AS heap_size,
ROUND(DATA_LENGTH / 1024 / 1024, 1) AS heap_size_mb,
ROUND(DATA_LENGTH / 1024 / 1024 / 100.0, 1) AS est_rewrite_seconds_at_100mbs
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = DATABASE()
AND TABLE_NAME IN (
'dashboard_roles', 'dashboard_slices', 'dashboard_user',
'report_schedule_user', 'rls_filter_roles', 'rls_filter_tables',
'slice_user', 'sqlatable_user'
)
ORDER BY DATA_LENGTH DESC;
```
**Restoring an old `pg_dump` (or equivalent) against the new schema.** A dump taken before the migration includes `INSERT` statements that populate the now-removed `id` column. Restoring such a dump against the post-migration schema will fail. The supported workaround is to dump only the schema and reference data, then re-create the M:N associations from application data after restore — for example with `pg_dump --exclude-table-data` (or per-table `--exclude-table-data=dashboard_slices` etc.) for the eight junction tables, restore the rest, then run a one-shot script that re-INSERTs `(fk1, fk2)` pairs derived from your application export. Operators who need to restore an old dump verbatim should restore against a pre-migration Superset and then re-run the upgrade.
**Intentional downgrade asymmetry.** The migration's `downgrade()` restores the surrogate `id` column and (for `dashboard_slices` and `report_schedule_user`) the original `UNIQUE (fk1, fk2)` constraint, but it does **not** restore the original `NULL`-allowed state on the FK columns — they remain `NOT NULL`. This is intentional: under SQLAlchemy's `secondary=` semantics, a `NULL` in either FK column of a junction table is meaningless (it cannot participate in either side of the relationship). Operators downgrading are not expected to need this restored. The asymmetry is documented for completeness so that round-trip schema diffs are not mistaken for migration bugs.
**Constraint-name divergence between upgrade and downgrade.** The composite primary key created on upgrade is named `pk_<table>` (Alembic's default for `op.create_primary_key("pk_<table>", ...)`), while the surrogate `id` primary key restored on downgrade is named `<table>_pkey` (PostgreSQL's default convention for `PrimaryKeyConstraint("id")`). The two names alternate so that a round-trip (upgrade → downgrade → upgrade) does not collide on a pre-existing constraint name. Operators using schema-comparison tools (e.g. `pg_diff`, `migra`) against a downgraded database may see this as drift versus a fresh-install schema. It is cosmetic — no application code references either constraint name.
## 6.0.0
- [33055](https://github.com/apache/superset/pull/33055): Upgrades Flask-AppBuilder to 5.0.0. The AUTH_OID authentication type has been deprecated and is no longer available as an option in Flask-AppBuilder. OpenID (OID) is considered a deprecated authentication protocol - if you are using AUTH_OID, you will need to migrate to an alternative authentication method such as OAuth, LDAP, or database authentication before upgrading.
- [34871](https://github.com/apache/superset/pull/34871): Fixed Jest test hanging issue from Ant Design v5 upgrade. MessageChannel is now mocked in test environment to prevent rc-overflow from causing Jest to hang. Test environment only - no production impact.
+117
View File
@@ -0,0 +1,117 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Compose override that swaps the default Postgres metadata DB for MySQL 8.
# Useful for evaluating dialect-specific behaviour (e.g., DDL-migration
# cost on a deployment whose production metadata DB is MySQL).
#
# Usage:
# docker compose -f docker-compose.yml -f docker-compose-mysql.yml up
# docker compose -f docker-compose.yml -f docker-compose-mysql.yml down
#
# To switch back to Postgres, just drop the second `-f` flag — the MySQL
# data lives in a separate volume (`db_home_mysql`) so neither side is
# corrupted by switching dialects.
#
# Notes:
# - Mirrors the connection settings used by CI's `test-mysql` shard:
# dialect ``mysql+mysqldb``, charset utf8mb4 with binary_prefix.
# - Host port 13306 (configurable via DATABASE_PORT_MYSQL) to avoid
# colliding with a native MySQL install on 3306.
# - The Postgres-specific init scripts under
# docker/docker-entrypoint-initdb.d/ are not mounted (they are
# postgres-only); examples / cypress fixtures still load via
# `superset-init`'s post-startup steps.
# Shared environment override applied to every Superset-side service that
# connects to the metadata DB. ``environment:`` takes precedence over the
# values inherited from the env_file in docker-compose.yml.
x-mysql-env: &mysql-env
DATABASE_DIALECT: mysql+mysqldb
DATABASE_HOST: db
DATABASE_PORT: "3306"
DATABASE_DB: superset
DATABASE_USER: superset
DATABASE_PASSWORD: superset
SQLALCHEMY_DATABASE_URI: "mysql+mysqldb://superset:superset@db:3306/superset?charset=utf8mb4&binary_prefix=true"
# Override the analytics-examples DB connection too. ``EXAMPLES_PORT``
# in docker/.env is hardcoded to 5432 (the Postgres port); without
# this override the examples connection would try MySQL on 5432 and
# fail. The examples user/DB are created by docker/mysql-init/
# examples-init.sql on first MySQL boot.
EXAMPLES_HOST: db
EXAMPLES_PORT: "3306"
EXAMPLES_DB: examples
EXAMPLES_USER: examples
EXAMPLES_PASSWORD: examples
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "mysql+mysqldb://examples:examples@db:3306/examples?charset=utf8mb4&binary_prefix=true"
services:
db:
image: mysql:8.0
environment:
MYSQL_DATABASE: superset
MYSQL_USER: superset
MYSQL_PASSWORD: superset
MYSQL_ROOT_PASSWORD: root
# The original 5432 port mapping is harmless on a MySQL container
# (nothing listens on 5432 inside it) but we add 13306->3306 so the
# MySQL port is reachable from the host without colliding with a
# native MySQL on 3306. Compose merges port lists.
ports:
- "127.0.0.1:${DATABASE_PORT_MYSQL:-13306}:3306"
# Override the init-scripts mount by re-binding the same target path
# to a MySQL-compatible directory. Compose merges volume lists by
# target path; later definitions win on conflict, so this displaces
# the Postgres-specific ``./docker/docker-entrypoint-initdb.d`` mount
# from docker-compose.yml. Without this, MySQL would try to run
# ``cypress-init.sh`` (which invokes ``psql``, not in the MySQL
# image), abort the init phase, and never create the ``examples``
# database. Add the MySQL data volume separately.
volumes:
- db_home_mysql:/var/lib/mysql
- ./docker/mysql-init:/docker-entrypoint-initdb.d
command:
- --default-authentication-plugin=caching_sha2_password
- --character-set-server=utf8mb4
- --collation-server=utf8mb4_0900_ai_ci
healthcheck:
test: ["CMD-SHELL", "mysqladmin ping -h localhost -uroot -proot --silent"]
interval: 5s
timeout: 5s
retries: 20
superset:
environment: *mysql-env
superset-init:
environment: *mysql-env
superset-worker:
environment: *mysql-env
superset-worker-beat:
environment: *mysql-env
superset-node:
environment: *mysql-env
superset-tests-worker:
environment: *mysql-env
volumes:
db_home_mysql:
-28
View File
@@ -61,34 +61,6 @@ services:
volumes:
- ./docker/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
- ./docker/nginx/templates:/etc/nginx/templates:ro
# Wait for the webpack dev server's manifest.json to be served before
# starting nginx. This prevents 404s on static assets at startup. The
# probe targets host.docker.internal so it works regardless of whether
# the dev server runs in the superset-node container
# (BUILD_SUPERSET_FRONTEND_IN_DOCKER=true, the default) or directly on
# the host (BUILD_SUPERSET_FRONTEND_IN_DOCKER=false).
command:
- /bin/bash
- -c
- |
url="http://host.docker.internal:9000/static/assets/manifest.json"
max_attempts=300 # ~10 minutes at 2s intervals; first build can be slow
echo "Waiting for webpack dev server at $$url..."
attempt=0
until curl -sf --max-time 5 -H "Host: localhost" -o /dev/null "$$url"; do
attempt=$$((attempt + 1))
if [ "$$attempt" -ge "$$max_attempts" ]; then
echo "ERROR: webpack dev server did not serve $$url after $$max_attempts attempts." >&2
echo "Is the dev server running? With BUILD_SUPERSET_FRONTEND_IN_DOCKER=false you must start it on the host (e.g. 'npm run dev' in superset-frontend)." >&2
exit 1
fi
if [ $$((attempt % 15)) -eq 0 ]; then
echo "Still waiting for webpack dev server... ($$attempt/$$max_attempts)"
fi
sleep 2
done
echo "Webpack dev server is ready; starting nginx."
exec /docker-entrypoint.sh nginx -g 'daemon off;'
redis:
image: redis:7
+3 -21
View File
@@ -71,34 +71,16 @@ case "${1}" in
worker)
echo "Starting Celery worker..."
# setting up only 2 workers by default to contain memory usage in dev environments
celery --app=superset.tasks.celery_app:app worker -O fair -l INFO --concurrency=${CELERYD_CONCURRENCY:-2} ${WORKER_LOG_FILE:+--logfile=$WORKER_LOG_FILE}
celery --app=superset.tasks.celery_app:app worker -O fair -l INFO --concurrency=${CELERYD_CONCURRENCY:-2}
;;
beat)
echo "Starting Celery beat..."
rm -f /tmp/celerybeat.pid
celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO -s "${SUPERSET_HOME}"/celerybeat-schedule ${BEAT_LOG_FILE:+--logfile=$BEAT_LOG_FILE}
celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid -l INFO -s "${SUPERSET_HOME}"/celerybeat-schedule
;;
app)
echo "Starting web app (using development server)..."
# Default to Flask debug mode in this dev compose entrypoint so the Talisman
# dev CSP (which permits 'unsafe-eval' required by React Refresh / HMR) is
# served. Operators can still set FLASK_DEBUG=false in docker/.env-local
# to exercise the production-like CSP and error handling.
: "${FLASK_DEBUG:=1}"
export FLASK_DEBUG
# Werkzeug's interactive debugger (/console) is a separate, security-sensitive
# feature and must be opted into explicitly via SUPERSET_DEBUG_ENABLED=true.
if [[ "${SUPERSET_DEBUG_ENABLED:-}" == "true" ]]; then
DEBUGGER_FLAG="--debugger"
echo " ⚠️ Werkzeug debugger enabled (requires PIN for /console access)"
else
DEBUGGER_FLAG="--no-debugger"
echo " 🔒 Werkzeug debugger disabled (set SUPERSET_DEBUG_ENABLED=true to enable)"
fi
flask run -p $PORT --reload $DEBUGGER_FLAG --host=0.0.0.0 --exclude-patterns "*/node_modules/*:*/.venv/*:*/build/*:*/__pycache__/*:*/superset-frontend/*"
flask run -p $PORT --reload --debugger --host=0.0.0.0 --exclude-patterns "*/node_modules/*:*/.venv/*:*/build/*:*/__pycache__/*:*/superset-frontend/*"
;;
app-gunicorn)
echo "Starting web app..."
+1 -1
View File
@@ -19,7 +19,7 @@
#
HYPHEN_SYMBOL='-'
exec gunicorn \
gunicorn \
--bind "${SUPERSET_BIND_ADDRESS:-0.0.0.0}:${SUPERSET_PORT:-8088}" \
--access-logfile "${ACCESS_LOG_FILE:-$HYPHEN_SYMBOL}" \
--error-logfile "${ERROR_LOG_FILE:-$HYPHEN_SYMBOL}" \
+32
View File
@@ -0,0 +1,32 @@
-- Licensed to the Apache Software Foundation (ASF) under one
-- or more contributor license agreements. See the NOTICE file
-- distributed with this work for additional information
-- regarding copyright ownership. The ASF licenses this file
-- to you under the Apache License, Version 2.0 (the
-- "License"); you may not use this file except in compliance
-- with the License. You may obtain a copy of the License at
--
-- http://www.apache.org/licenses/LICENSE-2.0
--
-- Unless required by applicable law or agreed to in writing,
-- software distributed under the License is distributed on an
-- "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-- KIND, either express or implied. See the License for the
-- specific language governing permissions and limitations
-- under the License.
-- MySQL counterpart to docker/docker-entrypoint-initdb.d/examples-init.sh.
-- Creates the analytics-examples database and user that Superset's
-- ``load-examples`` command writes to. Mounted by docker-compose-mysql.yml
-- at /docker-entrypoint-initdb.d/ so the MySQL image's first-boot
-- entrypoint runs it automatically. (The Postgres init scripts under
-- docker/docker-entrypoint-initdb.d/ are NOT mounted on the MySQL
-- service — they invoke psql, which doesn't exist in the MySQL image.)
CREATE DATABASE IF NOT EXISTS examples
CHARACTER SET utf8mb4
COLLATE utf8mb4_0900_ai_ci;
CREATE USER IF NOT EXISTS 'examples'@'%' IDENTIFIED BY 'examples';
GRANT ALL PRIVILEGES ON examples.* TO 'examples'@'%';
FLUSH PRIVILEGES;
@@ -18,8 +18,6 @@
# Configuration for docker-compose-light.yml - disables Redis and uses minimal services
# Import all settings from the main config first
import os
from flask_caching.backends.filesystemcache import FileSystemCache
from superset_config import * # noqa: F403
@@ -38,32 +36,3 @@ THUMBNAIL_CACHE_CONFIG = CACHE_CONFIG
# Disable Celery entirely for lightweight mode
CELERY_CONFIG = None # type: ignore[assignment,misc]
# Honor SUPERSET_FEATURE_<NAME> env vars on top of any flags inherited from
# superset_config. Lets local dev/e2e enable features (e.g. EMBEDDED_SUPERSET)
# without editing shipped config files. Only the literal string "true"
# (case-insensitive) is treated as enabled — "1"/"yes"/"on" are not, matching
# the strict-string convention used elsewhere in Superset's env parsing.
FEATURE_FLAGS = {
**FEATURE_FLAGS, # noqa: F405
**{
name[len("SUPERSET_FEATURE_") :]: value.strip().lower() == "true"
for name, value in os.environ.items()
if name.startswith("SUPERSET_FEATURE_")
},
}
if os.environ.get("SUPERSET_FEATURE_EMBEDDED_SUPERSET", "").strip().lower() == "true":
# Disable Talisman so /embedded/<uuid> doesn't return X-Frame-Options:SAMEORIGIN.
# Without this, browsers refuse to render Superset inside an iframe from a
# different origin (i.e. the embedded SDK use case). Production/CI configures
# Talisman with explicit `frame-ancestors`; for the lightweight local stack we
# just turn it off.
TALISMAN_ENABLED = False
# Guest tokens (used by the embedded SDK) inherit the "Public" role's perms.
# Out of the box Public has zero perms, so embedded dashboards immediately fail
# their first call (`/api/v1/me/roles/`) with 403. Mirror Public to Gamma —
# the standard read-only viewer role — so the embedded flow can authenticate
# and load dashboard data in local dev.
PUBLIC_ROLE_LIKE = "Gamma"
+1 -1
View File
@@ -1 +1 @@
v24.16.0
v22.22.0
-33
View File
@@ -86,39 +86,6 @@ instead requires a cachelib object.
See [Async Queries via Celery](/admin-docs/configuration/async-queries-celery) for details.
## Celery beat
Superset has a Celery task that will periodically warm up the cache based on different strategies.
To use it, add the following to your `superset_config.py`:
```python
from celery.schedules import crontab
from superset.config import CeleryConfig
# User that will be used to authenticate and render dashboards for cache warmup
SUPERSET_CACHE_WARMUP_USER = "user_with_permission_to_dashboards"
# Extend the default CeleryConfig to add cache warmup schedule
class CustomCeleryConfig(CeleryConfig):
beat_schedule = {
**CeleryConfig.beat_schedule,
'cache-warmup-hourly': {
'task': 'cache-warmup',
'schedule': crontab(minute=0, hour='*'), # hourly
'kwargs': {
'strategy_name': 'top_n_dashboards',
'top_n': 5,
'since': '7 days ago',
},
},
}
CELERY_CONFIG = CustomCeleryConfig
```
This will cache the top 5 most popular dashboards every hour. For other
strategies, check the `superset/tasks/cache.py` file.
## Caching Thumbnails
This is an optional feature that can be turned on by activating its [feature flag](/admin-docs/configuration/configuring-superset#feature-flags) on config:
@@ -455,51 +455,6 @@ def FLASK_APP_MUTATOR(app: Flask) -> None:
app.before_request_funcs.setdefault(None, []).append(make_session_permanent)
```
## Customizing the landing page (index view)
The page served at `/` is rendered by an index view. By default Superset registers
`SupersetIndexView`, which redirects to `/superset/welcome/` and also adds the
`/lang/<locale>` locale handler. You can replace it with your own view, for example
to send users straight to a specific dashboard or to a chart list.
Set `FAB_INDEX_VIEW` to the **importable dotted path** of your view class. Flask-AppBuilder
resolves this during app initialization and uses it in place of the default:
```python
# my_overrides.py — must be importable on the PYTHONPATH
from flask import redirect
from superset.initialization import SupersetIndexView
from superset.superset_typing import FlaskResponse
from flask_appbuilder import expose
class MyIndexView(SupersetIndexView):
@expose("/")
def index(self) -> FlaskResponse:
return redirect("/chart/list/")
```
```python
# superset_config.py
FAB_INDEX_VIEW = "my_overrides.MyIndexView"
```
A few things that commonly trip people up:
- **Subclass `SupersetIndexView`, not Flask-AppBuilder's bare `IndexView`.** Subclassing
keeps Superset's `/lang/<locale>` locale handling; replacing it with a bare `IndexView`
silently drops that behavior.
- **The class must be importable as a real module.** `FAB_INDEX_VIEW` is resolved by
importing the dotted path, which is independent of how `superset_config.py` itself is
loaded. Superset only copies **uppercase** names out of `superset_config.py` into its
runtime config, so a `FAB_INDEX_VIEW = "superset_config.MyIndexView"` reference only works
if `superset_config` is itself importable by that name on the `PYTHONPATH`. If you load
config via `SUPERSET_CONFIG_PATH` (an arbitrary file path), put the view in a separate
importable module instead and reference that module.
- **Don't set `appbuilder.indexview` from `FLASK_APP_MUTATOR`.** The mutator runs after
routes are already registered, so the assignment has no effect on the `/` route. Use
`FAB_INDEX_VIEW` instead.
## Feature Flags
To support a diverse set of users, Superset has some features that are not enabled by default. For
+17 -17
View File
@@ -22,24 +22,31 @@ level dependencies.
**Debian and Ubuntu**
The following command will ensure that the required dependencies are installed (tested on Ubuntu 20.04, 22.04, and 24.04):
Ubuntu **24.04** uses python 3.12 per default, which currently is not supported by Superset. You need to add a second python installation of 3.11 and install the required additional dependencies.
```bash
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev python3-pip python3-venv libsasl2-dev libldap2-dev libpq-dev default-libmysqlclient-dev pkg-config
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.11 python3.11-dev python3.11-venv build-essential libssl-dev libffi-dev libsasl2-dev libldap2-dev default-libmysqlclient-dev
```
Refer to the
[pyproject.toml](https://github.com/apache/superset/blob/master/pyproject.toml) file for the list of
Python versions officially supported by Superset, and install a matching `python3` interpreter for
your distribution. The `libpq-dev` package is only needed if you intend to connect to (or use) a
PostgreSQL database; you can omit it otherwise.
In Ubuntu **20.04 and 22.04** the following command will ensure that the required dependencies are installed:
```bash
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev python3-pip libsasl2-dev libldap2-dev default-libmysqlclient-dev
```
In Ubuntu **before 20.04** the following command will ensure that the required dependencies are installed:
```bash
sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-pip libsasl2-dev libldap2-dev default-libmysqlclient-dev
```
**Fedora and RHEL-derivative Linux distributions**
Install the following packages using the `yum` package manager:
```bash
sudo yum install gcc gcc-c++ libffi-devel python3-devel python3-pip python3-wheel openssl-devel cyrus-sasl-devel openldap-devel
sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel
```
In more recent versions of CentOS and Fedora, you may need to install a slightly different set of packages using `dnf`:
@@ -150,15 +157,8 @@ superset load_examples
superset init
# To start a development web server on port 8088, use -p to bind to another port
superset run -p 8088 --with-threads --reload
# For debugging with interactive console (⚠️ localhost only)
# superset run -p 8088 --with-threads --reload --debugger
superset run -p 8088 --with-threads --reload --debugger
```
:::warning Security Note
The `--debugger` flag enables Werkzeug's interactive console at `/console`. Only use this for local development and never bind to `0.0.0.0` or expose the server to networks when debugging is enabled.
:::
If everything worked, you should be able to navigate to `hostname:port` in your browser (e.g.
locally by default at `localhost:8088`) and login using the username and password you created.
@@ -157,15 +157,8 @@ superset load_examples
superset init
# To start a development web server on port 8088, use -p to bind to another port
superset run -p 8088 --with-threads --reload
# For debugging with interactive console (⚠️ localhost only)
# superset run -p 8088 --with-threads --reload --debugger
superset run -p 8088 --with-threads --reload --debugger
```
:::warning Security Note
The `--debugger` flag enables Werkzeug's interactive console at `/console`. Only use this for local development and never bind to `0.0.0.0` or expose the server to networks when debugging is enabled.
:::
If everything worked, you should be able to navigate to `hostname:port` in your browser (e.g.
locally by default at `localhost:8088`) and login using the username and password you created.
@@ -102,8 +102,6 @@ Affecting the Docker build process:
save some precious time on startup by `SUPERSET_LOAD_EXAMPLES=no docker compose up`
- **SUPERSET_LOG_LEVEL (default=info)**: Can be set to debug, info, warning, error, critical
for more verbose logging
- **SUPERSET_DEBUG_ENABLED (default=false)**: Enable Werkzeug debugger with interactive console.
Set to `true` for debugging: `SUPERSET_DEBUG_ENABLED=true docker compose up`
For more env vars that affect your configuration, see this
[superset_config.py](https://github.com/apache/superset/blob/master/docker/pythonpath_dev/superset_config.py)
-17
View File
@@ -917,23 +917,6 @@ const config: Config = {
footer: {
links: [],
copyright: `
<div class="footer__social-links">
<a href="https://bit.ly/join-superset-slack" target="_blank" rel="noopener noreferrer" title="Join us on Slack" aria-label="Slack">
<img src="/img/community/slack-symbol.svg" alt="Slack" />
</a>
<a href="https://x.com/apachesuperset" target="_blank" rel="noopener noreferrer" title="Follow us on X" aria-label="X">
<img src="/img/community/x-symbol.svg" alt="X" />
</a>
<a href="https://www.linkedin.com/company/apache-superset" target="_blank" rel="noopener noreferrer" title="Follow us on LinkedIn" aria-label="LinkedIn">
<img src="/img/community/linkedin-symbol.svg" alt="LinkedIn" />
</a>
<a href="https://bsky.app/profile/apachesuperset.bsky.social" target="_blank" rel="noopener noreferrer" title="Follow us on Bluesky" aria-label="Bluesky">
<img src="/img/community/bluesky-symbol.svg" alt="Bluesky" />
</a>
<a href="https://reddit.com/r/apache-superset" target="_blank" rel="noopener noreferrer" title="Follow us on Reddit" aria-label="Reddit">
<img src="/img/community/reddit-symbol.svg" alt="Reddit" />
</a>
</div>
<div class="footer__ci-services">
<span>CI powered by</span>
<a href="https://www.netlify.com/" target="_blank" rel="nofollow noopener noreferrer"><img src="/img/netlify.png" alt="Netlify" title="Netlify - Deploy Previews" /></a>
@@ -1,6 +1,6 @@
{
"copyright": {
"message": "\n <div class=\"footer__social-links\">\n <a href=\"https://bit.ly/join-superset-slack\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Join us on Slack\" aria-label=\"Slack\">\n <img src=\"/img/community/slack-symbol.svg\" alt=\"Slack\" />\n </a>\n <a href=\"https://x.com/apachesuperset\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Follow us on X\" aria-label=\"X\">\n <img src=\"/img/community/x-symbol.svg\" alt=\"X\" />\n </a>\n <a href=\"https://www.linkedin.com/company/apache-superset\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Follow us on LinkedIn\" aria-label=\"LinkedIn\">\n <img src=\"/img/community/linkedin-symbol.svg\" alt=\"LinkedIn\" />\n </a>\n <a href=\"https://bsky.app/profile/apachesuperset.bsky.social\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Follow us on Bluesky\" aria-label=\"Bluesky\">\n <img src=\"/img/community/bluesky-symbol.svg\" alt=\"Bluesky\" />\n </a>\n <a href=\"https://reddit.com/r/apache-superset\" target=\"_blank\" rel=\"noopener noreferrer\" title=\"Follow us on Reddit\" aria-label=\"Reddit\">\n <img src=\"/img/community/reddit-symbol.svg\" alt=\"Reddit\" />\n </a>\n </div>\n <div class=\"footer__ci-services\">\n <span>CI powered by</span>\n <a href=\"https://www.netlify.com/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img src=\"/img/netlify.png\" alt=\"Netlify\" title=\"Netlify - Deploy Previews\" /></a>\n </div>\n <p>Copyright © 2026,\n The <a href=\"https://www.apache.org/\" target=\"_blank\" rel=\"noreferrer\">Apache Software Foundation</a>,\n Licensed under the Apache <a href=\"https://apache.org/licenses/LICENSE-2.0\" target=\"_blank\" rel=\"noreferrer\">License</a>.</p>\n <p><small>Apache Superset, Apache, Superset, the Superset logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.\n <a href=\"https://www.apache.org/\" target=\"_blank\">Apache Software Foundation</a> resources</small></p>\n <img class=\"footer__divider\" src=\"/img/community/line.png\" alt=\"Divider\" />\n <p>\n <small>\n <a href=\"/admin-docs/security/\" target=\"_blank\" rel=\"noreferrer\">Security</a>&nbsp;|&nbsp;\n <a href=\"https://www.apache.org/foundation/sponsorship.html\" target=\"_blank\" rel=\"noreferrer\">Donate</a>&nbsp;|&nbsp;\n <a href=\"https://www.apache.org/foundation/thanks.html\" target=\"_blank\" rel=\"noreferrer\">Thanks</a>&nbsp;|&nbsp;\n <a href=\"https://apache.org/events/current-event\" target=\"_blank\" rel=\"noreferrer\">Events</a>&nbsp;|&nbsp;\n <a href=\"https://apache.org/licenses/\" target=\"_blank\" rel=\"noreferrer\">License</a>&nbsp;|&nbsp;\n <a href=\"https://privacy.apache.org/policies/privacy-policy-public.html\" target=\"_blank\" rel=\"noreferrer\">Privacy</a>\n </small>\n </p>\n <!-- telemetry/analytics pixel: -->\n <img referrerPolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=39ae6855-95fc-4566-86e5-360d542b0a68\" />\n ",
"message": "\n <div class=\"footer__ci-services\">\n <span>CI powered by</span>\n <a href=\"https://www.netlify.com/\" target=\"_blank\" rel=\"nofollow noopener noreferrer\"><img src=\"/img/netlify.png\" alt=\"Netlify\" title=\"Netlify - Deploy Previews\" /></a>\n </div>\n <p>Copyright © 2026,\n The <a href=\"https://www.apache.org/\" target=\"_blank\" rel=\"noreferrer\">Apache Software Foundation</a>,\n Licensed under the Apache <a href=\"https://apache.org/licenses/LICENSE-2.0\" target=\"_blank\" rel=\"noreferrer\">License</a>.</p>\n <p><small>Apache Superset, Apache, Superset, the Superset logo, and the Apache feather logo are either registered trademarks or trademarks of The Apache Software Foundation. All other products or name brands are trademarks of their respective holders, including The Apache Software Foundation.\n <a href=\"https://www.apache.org/\" target=\"_blank\">Apache Software Foundation</a> resources</small></p>\n <img class=\"footer__divider\" src=\"/img/community/line.png\" alt=\"Divider\" />\n <p>\n <small>\n <a href=\"/docs/security/\" target=\"_blank\" rel=\"noreferrer\">Security</a>&nbsp;|&nbsp;\n <a href=\"https://www.apache.org/foundation/sponsorship.html\" target=\"_blank\" rel=\"noreferrer\">Donate</a>&nbsp;|&nbsp;\n <a href=\"https://www.apache.org/foundation/thanks.html\" target=\"_blank\" rel=\"noreferrer\">Thanks</a>&nbsp;|&nbsp;\n <a href=\"https://apache.org/events/current-event\" target=\"_blank\" rel=\"noreferrer\">Events</a>&nbsp;|&nbsp;\n <a href=\"https://apache.org/licenses/\" target=\"_blank\" rel=\"noreferrer\">License</a>&nbsp;|&nbsp;\n <a href=\"https://privacy.apache.org/policies/privacy-policy-public.html\" target=\"_blank\" rel=\"noreferrer\">Privacy</a>\n </small>\n </p>\n <!-- telemetry/analytics pixel: -->\n <img referrerPolicy=\"no-referrer-when-downgrade\" src=\"https://static.scarf.sh/a.png?x-pxid=39ae6855-95fc-4566-86e5-360d542b0a68\" />\n ",
"description": "The footer copyright"
}
}
+3 -16
View File
@@ -25,22 +25,9 @@
command = "yarn install && yarn build"
# Output directory (relative to base)
publish = "build"
# Skip builds when no docs changes (exit 0 = skip, non-zero = build).
# Checks for changes in docs/ and README.md (which gets pulled into docs).
#
# $CACHED_COMMIT_REF is the last *deployed* commit; it is set on incremental
# builds (notably the master production deploy) and empty on a context's
# first build (every deploy preview). The production path diffs against it
# and skips correctly.
#
# Deploy previews need different handling: Netlify checks out a *merge*
# commit, so $COMMIT_REF (the PR head SHA) is frequently not resolvable in
# the clone, and on a shallow clone `git merge-base` can fail too -- so the
# previous logic fell through to a build on every PR, even non-docs ones.
# Instead, always diff the checked-out HEAD against its merge-base with
# master, deepening the shallow clone until that merge-base resolves. If it
# genuinely can't be determined, exit non-zero to build (fail safe).
ignore = 'if [ -n "$CACHED_COMMIT_REF" ]; then git diff --quiet "$CACHED_COMMIT_REF" HEAD -- . ../README.md; else git fetch --no-tags origin master >/dev/null 2>&1 || true; i=0; while [ "$i" -lt 10 ] && ! git merge-base origin/master HEAD >/dev/null 2>&1; do git fetch --deepen=200 origin master >/dev/null 2>&1 || break; i=$((i+1)); done; BASE="$(git merge-base origin/master HEAD 2>/dev/null || true)"; if [ -z "$BASE" ]; then exit 1; fi; git diff --quiet "$BASE" HEAD -- . ../README.md; fi'
# Skip builds when no docs changes (exit 0 = skip, exit 1 = build)
# Checks for changes in docs/ and README.md (which gets pulled into docs)
ignore = "git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF -- . ../README.md"
[build.environment]
# Node version matching docs/.nvmrc
+12 -18
View File
@@ -43,7 +43,7 @@
"version:remove:components": "node scripts/manage-versions.mjs remove components"
},
"dependencies": {
"@ant-design/icons": "^6.2.5",
"@ant-design/icons": "^6.2.3",
"@docusaurus/core": "^3.10.1",
"@docusaurus/faster": "^3.10.1",
"@docusaurus/plugin-client-redirects": "^3.10.1",
@@ -70,13 +70,13 @@
"@storybook/preview-api": "^8.6.18",
"@storybook/theming": "^8.6.15",
"@superset-ui/core": "^0.20.4",
"@swc/core": "^1.15.41",
"antd": "^6.4.4",
"baseline-browser-mapping": "^2.10.37",
"caniuse-lite": "^1.0.30001799",
"@swc/core": "^1.15.33",
"antd": "^6.4.3",
"baseline-browser-mapping": "^2.10.31",
"caniuse-lite": "^1.0.30001793",
"docusaurus-plugin-openapi-docs": "^5.0.2",
"docusaurus-theme-openapi-docs": "^5.0.2",
"js-yaml": "^4.2.0",
"js-yaml": "^4.1.1",
"js-yaml-loader": "^1.2.2",
"json-bigint": "^1.0.0",
"prism-react-renderer": "^2.4.1",
@@ -101,16 +101,16 @@
"@types/js-yaml": "^4.0.9",
"@types/react": "^19.1.8",
"@typescript-eslint/eslint-plugin": "^8.59.3",
"@typescript-eslint/parser": "^8.61.0",
"@typescript-eslint/parser": "^8.59.3",
"eslint": "^9.39.2",
"eslint-config-prettier": "^10.1.8",
"eslint-plugin-prettier": "^5.5.6",
"eslint-plugin-prettier": "^5.5.5",
"eslint-plugin-react": "^7.37.5",
"globals": "^17.6.0",
"prettier": "^3.8.4",
"prettier": "^3.8.3",
"typescript": "~6.0.3",
"typescript-eslint": "^8.61.1",
"webpack": "^5.107.2"
"typescript-eslint": "^8.59.4",
"webpack": "^5.107.1"
},
"browserslist": {
"production": [
@@ -128,13 +128,7 @@
"react-redux": "^9.2.0",
"@reduxjs/toolkit": "^2.5.0",
"baseline-browser-mapping": "^2.9.19",
"swagger-client": "3.37.3",
"lodash": "4.18.1",
"lodash-es": "4.18.1",
"yaml": "1.10.3",
"uuid": "11.1.1",
"serialize-javascript": "7.0.5",
"d3-color": "3.1.0"
"swagger-client": "3.37.3"
},
"packageManager": "yarn@1.22.22+sha1.ac34549e6aa8e7ead463a7407e1c7390f61a6610"
}
@@ -1,136 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# SIP: Authenticated encryption (AES-GCM) for app-encrypted fields
## [DRAFT — proposal for discussion]
This document is a draft proposal accompanying the code in this PR. It is
intended to seed the formal SIP discussion. The code here ships the
backward-compatible engine selection **and** the re-encryption migrator
(Phases 12 below); both are opt-in and change nothing for existing installs by
default. Flipping the default for fresh installs (Phase 3) remains future work.
## Motivation
Superset app-encrypts a number of sensitive fields before persisting them to
the metadata database, including:
- database connection passwords and `encrypted_extra` (`superset/models/core.py`),
- SSH tunnel credentials — password, private key, private-key password
(`superset/databases/ssh_tunnel/models.py`),
- OAuth2 tokens and other secrets stored via `EncryptedType`.
These fields are encrypted with `sqlalchemy_utils.EncryptedType`, which
**defaults to `AesEngine` (AES-CBC)**. AES-CBC provides confidentiality but is
**unauthenticated**: it has no integrity tag. An attacker with write access to
the ciphertext (e.g. direct metadata-DB access, a backup, or a compromised
replica) can perform **bit-flipping / chosen-ciphertext manipulation** to
silently alter the decrypted plaintext of a secret without detection.
`AesGcmEngine` (AES-GCM) is authenticated encryption: tampering causes
decryption to fail loudly rather than yielding attacker-influenced plaintext.
Using authenticated encryption for secrets at rest is an ASVS L1 expectation
(11.3.2 / cryptography best practice).
`config.py` already documents that operators *can* switch to GCM by writing a
custom `AbstractEncryptedFieldAdapter`, but:
1. it is opt-in, undocumented as a security recommendation, and easy to miss;
2. there is **no migration path** — flipping the engine on a populated database
makes every existing secret undecryptable, because GCM ciphertext is not
format-compatible with CBC.
## Proposed change
A three-part change, delivered incrementally so existing deployments are never
broken:
### Phase 1 — engine selection (this PR)
- Add a `SQLALCHEMY_ENCRYPTED_FIELD_ENGINE` config (`"aes"` | `"aes-gcm"`),
**defaulting to `"aes"`** (no behavior change for existing installs).
- Teach the default `SQLAlchemyUtilsAdapter` to honor it (an explicit `engine`
kwarg still wins, so the migrator can pin an engine).
- This lets **new** deployments choose AES-GCM from day one with a one-line
config, instead of writing a custom adapter.
### Phase 2 — CBC→GCM re-encryption migrator (this PR)
The existing `SecretsMigrator` (previously only used for `SECRET_KEY` rotation)
gains an **engine migration** mode that:
1. discovers every `EncryptedType` column (via `discover_encrypted_fields()`),
2. decrypts each value with the **source** engine (AES-CBC) under the current
`SECRET_KEY`,
3. re-encrypts with the **target** engine (AES-GCM),
4. runs transactionally per the existing all-or-nothing semantics, and is
idempotent per column (already-migrated values are skipped), so a run can be
safely repeated or resumed.
Exposed via a new `--engine` option on the existing CLI command:
`superset re-encrypt-secrets --engine aes-gcm`, runnable by operators with a DB
backup in hand. The `SECRET_KEY` is unchanged; an engine change and a key
rotation can also be combined (pass `--previous_secret_key` as well).
### Phase 3 — flip the default for new installs
Once the migrator and docs are in place, change the default to `"aes-gcm"` for
**fresh** installs only (e.g. keyed off an empty metadata DB / documented in
`UPDATING.md`), keeping existing installs on `"aes"` until they run Phase 2.
## New or changed public interfaces
- New config: `SQLALCHEMY_ENCRYPTED_FIELD_ENGINE: Literal["aes", "aes-gcm"]`.
- New (Phase 2) CLI: `superset re-encrypt-secrets --engine <name>`.
- No schema changes; ciphertext format changes per migrated column.
## Migration plan and compatibility
- **Backward compatible by default.** Phase 1 changes nothing unless the
operator opts in.
- Switching an existing deployment to `"aes-gcm"` **without** running the Phase
2 migrator will make existing secrets undecryptable — this is called out in
the config comment and must be in `UPDATING.md`.
- Recommended operator runbook: take a metadata-DB backup → run
`re-encrypt-secrets --engine aes-gcm` → set
`SQLALCHEMY_ENCRYPTED_FIELD_ENGINE = "aes-gcm"` → restart → re-run
`re-encrypt-secrets --engine aes-gcm` once more to sweep up any secrets a live
instance wrote as AES-CBC during the cutover window. The canonical, more
detailed version of this runbook lives in `UPDATING.md`; this is a summary.
- `AesEngine` allows queryability over encrypted fields; AES-GCM does not.
Any code that filters/queries on an encrypted column directly must be audited
before Phase 3 (none is expected, but it must be verified).
## Rejected alternatives
- **Flip the default immediately.** Rejected: bricks every existing
deployment's secrets with no migration path.
- **Document-only (custom adapter).** Status quo; high friction and no
migration tooling — most operators will never do it.
## Open questions
- GCM→CBC rollback (for operators who need queryability) already works via the
same command (`re-encrypt-secrets --engine aes`), since the migrator is
engine-symmetric. Should rollback be documented as a supported path or
discouraged?
- The migrator already supports a concurrent `SECRET_KEY` rotation + engine
change in a single pass (pass `--previous_secret_key` alongside `--engine`).
Is that combination worth calling out in the operator docs, or kept advanced?
+2 -2
View File
@@ -7235,10 +7235,10 @@
"pypi_packages": [
"oracledb"
],
"connection_string": "oracle+oracledb://{username}:{password}@{hostname}:{port}",
"connection_string": "oracle://{username}:{password}@{hostname}:{port}",
"default_port": 1521,
"notes": "Previously used cx_Oracle, now uses oracledb.",
"docs_url": "https://python-oracledb.readthedocs.io/en/latest/user_guide/installation.html",
"docs_url": "https://cx-oracle.readthedocs.io/en/latest/user_guide/installation.html",
"category": "Other Databases"
},
"engine": "oracle",
+1 -51
View File
@@ -260,45 +260,10 @@ a > span > svg {
.footer {
position: relative;
padding-top: 130px;
padding-top: 90px;
font-size: 15px;
}
.footer__social-links {
background-color: #173036;
position: absolute;
top: 52px;
left: 0;
width: 100%;
padding: 10px 0;
display: flex;
align-items: center;
justify-content: center;
gap: 24px;
}
.footer__social-links a {
display: inline-flex;
align-items: center;
transition: opacity 0.2s, transform 0.2s;
}
.footer__social-links a:hover {
opacity: 0.8;
transform: scale(1.1);
}
.footer__social-links img {
height: 24px;
width: 24px;
/* The brand SVGs ship in their native colors (e.g. Slack's dark aubergine,
X's near-black), which disappear on the dark footer. Render them all as
uniform white silhouettes. The icons are single-path glyphs whose
counters (the LinkedIn "in", Slack gaps, Reddit face) are transparent
cut-outs, so they stay legible against the footer background. */
filter: brightness(0) invert(1);
}
.footer__ci-services {
background-color: #0d3e49;
color: #e1e1e1;
@@ -344,21 +309,6 @@ a > span > svg {
}
@media only screen and (max-width: 996px) {
.footer {
padding-top: 120px;
}
.footer__social-links {
top: 44px;
gap: 20px;
padding: 8px 16px;
}
.footer__social-links img {
height: 20px;
width: 20px;
}
.footer__ci-services {
gap: 12px;
padding: 10px 16px;
-21
View File
@@ -1,21 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="40" height="40" fill="#FF4500">
<path d="M12 0A12 12 0 0 0 0 12a12 12 0 0 0 12 12 12 12 0 0 0 12-12A12 12 0 0 0 12 0zm5.01 4.744c.688 0 1.25.561 1.25 1.249a1.25 1.25 0 0 1-2.498.056l-2.597-.547-.8 3.747c1.824.07 3.48.632 4.674 1.488.308-.309.73-.491 1.207-.491.968 0 1.754.786 1.754 1.754 0 .716-.435 1.333-1.01 1.614a3.111 3.111 0 0 1 .042.52c0 2.694-3.13 4.87-7.004 4.87-3.874 0-7.004-2.176-7.004-4.87 0-.183.015-.366.043-.534A1.748 1.748 0 0 1 4.028 12c0-.968.786-1.754 1.754-1.754.463 0 .898.196 1.207.49 1.207-.883 2.878-1.43 4.744-1.487l.885-4.182a.342.342 0 0 1 .14-.197.35.35 0 0 1 .238-.042l2.906.617a1.214 1.214 0 0 1 1.108-.701zM9.25 12c-.688 0-1.25.561-1.25 1.25 0 .687.562 1.248 1.25 1.248.687 0 1.248-.561 1.248-1.249 0-.688-.561-1.249-1.249-1.249zm5.5 0c-.687 0-1.248.561-1.248 1.25 0 .687.561 1.248 1.249 1.248.688 0 1.249-.561 1.249-1.249 0-.687-.562-1.249-1.25-1.249zm-5.466 3.99a.327.327 0 0 0-.231.094.33.33 0 0 0 0 .463c.842.842 2.484.913 2.961.913.477 0 2.105-.056 2.961-.913a.361.361 0 0 0 .029-.463.33.33 0 0 0-.464 0c-.547.533-1.684.73-2.512.73-.828 0-1.979-.196-2.512-.73a.326.326 0 0 0-.232-.095z"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.9 KiB

-21
View File
@@ -1,21 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<svg xmlns="http://www.w3.org/2000/svg" viewBox="0 0 24 24" width="40" height="40" fill="#4A154B">
<path d="M5.042 15.165a2.528 2.528 0 0 1-2.52 2.523A2.528 2.528 0 0 1 0 15.165a2.527 2.527 0 0 1 2.522-2.52h2.52v2.52zm1.271 0a2.527 2.527 0 0 1 2.521-2.52 2.527 2.527 0 0 1 2.521 2.52v6.313A2.528 2.528 0 0 1 8.834 24a2.528 2.528 0 0 1-2.521-2.522v-6.313zM8.834 5.042a2.528 2.528 0 0 1-2.521-2.52A2.528 2.528 0 0 1 8.834 0a2.528 2.528 0 0 1 2.521 2.522v2.52H8.834zm0 1.271a2.528 2.528 0 0 1 2.521 2.521 2.528 2.528 0 0 1-2.521 2.521H2.522A2.528 2.528 0 0 1 0 8.834a2.528 2.528 0 0 1 2.522-2.521h6.312zm10.124 2.521a2.528 2.528 0 0 1 2.522-2.521A2.528 2.528 0 0 1 24 8.834a2.528 2.528 0 0 1-2.52 2.521h-2.522V8.834zm-1.271 0a2.528 2.528 0 0 1-2.521 2.521 2.528 2.528 0 0 1-2.521-2.521V2.522A2.528 2.528 0 0 1 15.166 0a2.528 2.528 0 0 1 2.521 2.522v6.312zm-2.521 10.124a2.528 2.528 0 0 1 2.521 2.522A2.528 2.528 0 0 1 15.166 24a2.528 2.528 0 0 1-2.521-2.52v-2.522h2.521zm0-1.271a2.528 2.528 0 0 1-2.521-2.521 2.528 2.528 0 0 1 2.521-2.521h6.312A2.528 2.528 0 0 1 24 15.165a2.528 2.528 0 0 1-2.52 2.521h-6.313z"/>
</svg>

Before

Width:  |  Height:  |  Size: 1.9 KiB

@@ -1808,10 +1808,6 @@ If you enable DML in the meta database users will be able to run DML queries on
Second, you might want to change the value of `SUPERSET_META_DB_LIMIT`. The default value is 1000, and defines how many are read from each database before any aggregations and joins are executed. You can also set this value `None` if you only have small tables.
:::warning
`SUPERSET_META_DB_LIMIT` is applied to **each** underlying table *before* the in-memory join runs, not to the final result. If any table involved in a join has more rows than the limit, the meta database will read only the first `SUPERSET_META_DB_LIMIT` rows of that table, which means matching rows can be silently dropped and the join can return **incomplete or even empty** results with no error. If you join tables larger than the limit, raise `SUPERSET_META_DB_LIMIT` to comfortably exceed your largest joined table, or set it to `None` when working only with small tables, to get correct results.
:::
Additionally, you might want to restrict the databases to with the meta database has access to. This can be done in the database configuration, under "Advanced" -> "Other" -> "ENGINE PARAMETERS" and adding:
```json
+370 -478
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -15,7 +15,7 @@
# limitations under the License.
#
apiVersion: v2
appVersion: "6.1.0"
appVersion: "5.0.0"
description: Apache Superset is a modern, enterprise-ready business intelligence web application
name: superset
icon: https://artifacthub.io/image/68c1d717-0e97-491f-b046-754e46f46922@2x
@@ -29,7 +29,7 @@ maintainers:
- name: craig-rueda
email: craig@craigrueda.com
url: https://github.com/craig-rueda
version: 0.17.2 # See [README](https://github.com/apache/superset/blob/master/helm/superset/README.md#versioning) for version details.
version: 0.15.5 # See [README](https://github.com/apache/superset/blob/master/helm/superset/README.md#versioning) for version details.
dependencies:
- name: postgresql
version: 16.7.27
+4 -5
View File
@@ -23,7 +23,7 @@ NOTE: This file is generated by helm-docs: https://github.com/norwoodj/helm-docs
# superset
![Version: 0.17.2](https://img.shields.io/badge/Version-0.17.2-informational?style=flat-square)
![Version: 0.15.5](https://img.shields.io/badge/Version-0.15.5-informational?style=flat-square)
Apache Superset is a modern, enterprise-ready business intelligence web application
@@ -111,6 +111,9 @@ On helm this can be set on `extraSecretEnv.SUPERSET_SECRET_KEY` or `configOverri
| init.resources | object | `{}` | |
| init.tolerations | list | `[]` | |
| init.topologySpreadConstraints | list | `[]` | TopologySpreadConstrains to be added to init job |
| initImage.pullPolicy | string | `"IfNotPresent"` | |
| initImage.repository | string | `"apache/superset"` | |
| initImage.tag | string | `"dockerize"` | |
| nameOverride | string | `nil` | Provide a name to override the name of the chart |
| nodeSelector | object | `{}` | |
| postgresql | object | see `values.yaml` | Configuration values for the postgresql dependency. ref: https://github.com/bitnami/charts/tree/main/bitnami/postgresql |
@@ -216,7 +219,6 @@ On helm this can be set on `extraSecretEnv.SUPERSET_SECRET_KEY` or `configOverri
| supersetNode.extraContainers | list | `[]` | Launch additional containers into supersetNode pod |
| supersetNode.forceReload | bool | `false` | If true, forces deployment to reload on each upgrade |
| supersetNode.initContainers | list | a container waiting for postgres | Init containers |
| supersetNode.lifecycle | object | `{}` | Container lifecycle hooks, e.g. a preStop sleep so the Service/Ingress stops routing to the pod before gunicorn receives SIGTERM |
| supersetNode.livenessProbe.failureThreshold | int | `3` | |
| supersetNode.livenessProbe.httpGet.path | string | `"/health"` | |
| supersetNode.livenessProbe.httpGet.port | string | `"http"` | |
@@ -249,7 +251,6 @@ On helm this can be set on `extraSecretEnv.SUPERSET_SECRET_KEY` or `configOverri
| supersetNode.startupProbe.successThreshold | int | `1` | |
| supersetNode.startupProbe.timeoutSeconds | int | `1` | |
| supersetNode.strategy | object | `{}` | |
| supersetNode.terminationGracePeriodSeconds | string | `nil` | Pod termination grace period (seconds). Set greater than GUNICORN_TIMEOUT so in-flight requests can drain before SIGKILL |
| supersetNode.topologySpreadConstraints | list | `[]` | TopologySpreadConstrains to be added to supersetNode deployments |
| supersetWebsockets.affinity | object | `{}` | Affinity to be added to supersetWebsockets deployment |
| supersetWebsockets.command | list | `[]` | |
@@ -313,7 +314,6 @@ On helm this can be set on `extraSecretEnv.SUPERSET_SECRET_KEY` or `configOverri
| supersetWorker.extraContainers | list | `[]` | Launch additional containers into supersetWorker pod |
| supersetWorker.forceReload | bool | `false` | If true, forces deployment to reload on each upgrade |
| supersetWorker.initContainers | list | a container waiting for postgres and redis | Init container |
| supersetWorker.lifecycle | object | `{}` | Container lifecycle hooks for the worker pod |
| supersetWorker.livenessProbe.exec.command | list | a `celery inspect ping` command | Liveness probe command |
| supersetWorker.livenessProbe.failureThreshold | int | `3` | |
| supersetWorker.livenessProbe.initialDelaySeconds | int | `120` | |
@@ -334,7 +334,6 @@ On helm this can be set on `extraSecretEnv.SUPERSET_SECRET_KEY` or `configOverri
| supersetWorker.resources | object | `{}` | Resource settings for the supersetWorker pods - these settings overwrite might existing values from the global resources object defined above. |
| supersetWorker.startupProbe | object | `{}` | No startup/readiness probes by default since we don't really care about its startup time (it doesn't serve traffic) |
| supersetWorker.strategy | object | `{}` | |
| supersetWorker.terminationGracePeriodSeconds | string | `nil` | Pod termination grace period (seconds) for the worker pod so in-flight tasks can drain before SIGKILL |
| supersetWorker.topologySpreadConstraints | list | `[]` | TopologySpreadConstrains to be added to supersetWorker deployments |
| tolerations | list | `[]` | |
| topologySpreadConstraints | list | `[]` | TopologySpreadConstrains to be added to all deployments |
+1 -1
View File
@@ -126,7 +126,7 @@ spec:
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
{{- if .Values.supersetCeleryBeat.extraContainers }}
{{- tpl (toYaml .Values.supersetCeleryBeat.extraContainers) . | nindent 8 }}
{{- toYaml .Values.supersetCeleryBeat.extraContainers | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
@@ -121,7 +121,7 @@ spec:
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
{{- if .Values.supersetCeleryFlower.extraContainers }}
{{- tpl (toYaml .Values.supersetCeleryFlower.extraContainers) . | nindent 8 }}
{{- toYaml .Values.supersetCeleryFlower.extraContainers | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
@@ -134,9 +134,6 @@ spec:
{{- if .Values.supersetWorker.livenessProbe }}
livenessProbe: {{- .Values.supersetWorker.livenessProbe | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.supersetWorker.lifecycle }}
lifecycle: {{- .Values.supersetWorker.lifecycle | toYaml | nindent 12 }}
{{- end }}
resources:
{{- if .Values.supersetWorker.resources }}
{{- toYaml .Values.supersetWorker.resources | nindent 12 }}
@@ -144,7 +141,7 @@ spec:
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
{{- if .Values.supersetWorker.extraContainers }}
{{- tpl (toYaml .Values.supersetWorker.extraContainers) . | nindent 8 }}
{{- toYaml .Values.supersetWorker.extraContainers | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
@@ -173,9 +170,6 @@ spec:
{{- with .Values.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.supersetWorker.terminationGracePeriodSeconds }}
terminationGracePeriodSeconds: {{ .Values.supersetWorker.terminationGracePeriodSeconds }}
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets: {{- toYaml .Values.imagePullSecrets | nindent 8 }}
{{- end }}
+1 -1
View File
@@ -120,7 +120,7 @@ spec:
livenessProbe: {{- .Values.supersetWebsockets.livenessProbe | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.supersetWebsockets.extraContainers }}
{{- tpl (toYaml .Values.supersetWebsockets.extraContainers) . | nindent 8 }}
{{- toYaml .Values.supersetWebsockets.extraContainers | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
+1 -7
View File
@@ -144,9 +144,6 @@ spec:
{{- if .Values.supersetNode.livenessProbe }}
livenessProbe: {{- .Values.supersetNode.livenessProbe | toYaml | nindent 12 }}
{{- end }}
{{- if .Values.supersetNode.lifecycle }}
lifecycle: {{- .Values.supersetNode.lifecycle | toYaml | nindent 12 }}
{{- end }}
resources:
{{- if .Values.supersetNode.resources }}
{{- toYaml .Values.supersetNode.resources | nindent 12 }}
@@ -154,7 +151,7 @@ spec:
{{- toYaml .Values.resources | nindent 12 }}
{{- end }}
{{- if .Values.supersetNode.extraContainers }}
{{- tpl (toYaml .Values.supersetNode.extraContainers) . | nindent 8 }}
{{- toYaml .Values.supersetNode.extraContainers | nindent 8 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
@@ -183,9 +180,6 @@ spec:
{{- with .Values.tolerations }}
tolerations: {{- toYaml . | nindent 8 }}
{{- end }}
{{- if .Values.supersetNode.terminationGracePeriodSeconds }}
terminationGracePeriodSeconds: {{ .Values.supersetNode.terminationGracePeriodSeconds }}
{{- end }}
{{- if .Values.imagePullSecrets }}
imagePullSecrets: {{- toYaml .Values.imagePullSecrets | nindent 8 }}
{{- end }}
+1 -4
View File
@@ -62,9 +62,6 @@ spec:
{{- if .Values.init.initContainers }}
initContainers: {{- tpl (toYaml .Values.init.initContainers) . | nindent 6 }}
{{- end }}
{{- with .Values.hostAliases }}
hostAliases: {{- toYaml . | nindent 6 }}
{{- end }}
containers:
- name: {{ template "superset.name" . }}-init-db
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
@@ -104,7 +101,7 @@ spec:
command: {{ tpl (toJson .Values.init.command) . }}
resources: {{- toYaml .Values.init.resources | nindent 10 }}
{{- if .Values.init.extraContainers }}
{{- tpl (toYaml .Values.init.extraContainers) . | nindent 6 }}
{{- toYaml .Values.init.extraContainers | nindent 6 }}
{{- end }}
{{- with .Values.nodeSelector }}
nodeSelector: {{- toYaml . | nindent 8 }}
+28 -106
View File
@@ -194,6 +194,11 @@ image:
imagePullSecrets: []
initImage:
repository: apache/superset
tag: dockerize
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 8088
@@ -269,7 +274,7 @@ supersetNode:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; exec /usr/bin/run-server.sh"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; /usr/bin/run-server.sh"
connections:
# -- Change in case of bringing your own redis and then also set redis.enabled:false
redis_host: "{{ .Release.Name }}-redis-headless"
@@ -298,29 +303,15 @@ supersetNode:
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/bash
- /bin/sh
- -c
- |
# opening a /dev/tcp fd performs a TCP connect without sending any
# payload (avoids postgres "incomplete startup packet" log noise);
# no external `dockerize`, `nc`, or busybox needed. SECONDS-based
# deadline mirrors the prior `dockerize -timeout 120s` behaviour.
SECONDS=0
until (exec 3<>/dev/tcp/"$DB_HOST"/"$DB_PORT") 2>/dev/null; do
if [ "$SECONDS" -ge 120 ]; then
echo "timeout waiting for postgres at $DB_HOST:$DB_PORT after 120s" >&2
exit 1
fi
echo "waiting for postgres at $DB_HOST:$DB_PORT (elapsed ${SECONDS}s)"
sleep 2
done
echo "postgres at $DB_HOST:$DB_PORT is up"
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -timeout 120s
resources:
limits:
memory: "256Mi"
@@ -369,12 +360,6 @@ supersetNode:
failureThreshold: 3
periodSeconds: 15
successThreshold: 1
# -- Container lifecycle hooks, e.g. a preStop sleep so the Service/Ingress
# stops routing to the pod before gunicorn receives SIGTERM
lifecycle: {}
# -- Pod termination grace period (seconds). Set greater than GUNICORN_TIMEOUT so
# in-flight requests can drain before SIGKILL
terminationGracePeriodSeconds: ~
# -- Resource settings for the supersetNode pods - these settings overwrite might existing values from the global resources object defined above.
resources: {}
# limits:
@@ -415,38 +400,22 @@ supersetWorker:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; exec celery --app=superset.tasks.celery_app:app worker"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app worker"
# -- If true, forces deployment to reload on each upgrade
forceReload: false
# -- Init container
# @default -- a container waiting for postgres and redis
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/bash
- /bin/sh
- -c
- |
# See supersetNode.initContainers for the rationale.
SECONDS=0
wait_for() {
local host=$1 port=$2 name=$3
until (exec 3<>/dev/tcp/"$host"/"$port") 2>/dev/null; do
if [ "$SECONDS" -ge 120 ]; then
echo "timeout waiting for $name at $host:$port after 120s" >&2
exit 1
fi
echo "waiting for $name at $host:$port (elapsed ${SECONDS}s)"
sleep 2
done
echo "$name at $host:$port is up"
}
wait_for "$DB_HOST" "$DB_PORT" postgres
wait_for "$REDIS_HOST" "$REDIS_PORT" redis
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
resources:
limits:
memory: "256Mi"
@@ -495,10 +464,6 @@ supersetWorker:
failureThreshold: 3
periodSeconds: 60
successThreshold: 1
# -- Container lifecycle hooks for the worker pod
lifecycle: {}
# -- Pod termination grace period (seconds) for the worker pod so in-flight tasks can drain before SIGKILL
terminationGracePeriodSeconds: ~
# -- No startup/readiness probes by default since we don't really care about its startup time (it doesn't serve traffic)
startupProbe: {}
# -- No startup/readiness probes by default since we don't really care about its startup time (it doesn't serve traffic)
@@ -523,38 +488,22 @@ supersetCeleryBeat:
command:
- "/bin/sh"
- "-c"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; exec celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid --schedule /tmp/celerybeat-schedule"
- ". {{ .Values.configMountPath }}/superset_bootstrap.sh; celery --app=superset.tasks.celery_app:app beat --pidfile /tmp/celerybeat.pid --schedule /tmp/celerybeat-schedule"
# -- If true, forces deployment to reload on each upgrade
forceReload: false
# -- List of init containers
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/bash
- /bin/sh
- -c
- |
# See supersetNode.initContainers for the rationale.
SECONDS=0
wait_for() {
local host=$1 port=$2 name=$3
until (exec 3<>/dev/tcp/"$host"/"$port") 2>/dev/null; do
if [ "$SECONDS" -ge 120 ]; then
echo "timeout waiting for $name at $host:$port after 120s" >&2
exit 1
fi
echo "waiting for $name at $host:$port (elapsed ${SECONDS}s)"
sleep 2
done
echo "$name at $host:$port is up"
}
wait_for "$DB_HOST" "$DB_PORT" postgres
wait_for "$REDIS_HOST" "$REDIS_PORT" redis
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
resources:
limits:
memory: "256Mi"
@@ -645,31 +594,15 @@ supersetCeleryFlower:
# @default -- a container waiting for postgres and redis
initContainers:
- name: wait-for-postgres-redis
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/bash
- /bin/sh
- -c
- |
# See supersetNode.initContainers for the rationale.
SECONDS=0
wait_for() {
local host=$1 port=$2 name=$3
until (exec 3<>/dev/tcp/"$host"/"$port") 2>/dev/null; do
if [ "$SECONDS" -ge 120 ]; then
echo "timeout waiting for $name at $host:$port after 120s" >&2
exit 1
fi
echo "waiting for $name at $host:$port (elapsed ${SECONDS}s)"
sleep 2
done
echo "$name at $host:$port is up"
}
wait_for "$DB_HOST" "$DB_PORT" postgres
wait_for "$REDIS_HOST" "$REDIS_PORT" redis
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -wait "tcp://$REDIS_HOST:$REDIS_PORT" -timeout 120s
resources:
limits:
memory: "256Mi"
@@ -831,26 +764,15 @@ init:
# @default -- a container waiting for postgres
initContainers:
- name: wait-for-postgres
image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
imagePullPolicy: "{{ .Values.image.pullPolicy }}"
image: "{{ .Values.initImage.repository }}:{{ .Values.initImage.tag }}"
imagePullPolicy: "{{ .Values.initImage.pullPolicy }}"
envFrom:
- secretRef:
name: "{{ tpl .Values.envFromSecret . }}"
command:
- /bin/bash
- /bin/sh
- -c
- |
# See supersetNode.initContainers for the rationale.
SECONDS=0
until (exec 3<>/dev/tcp/"$DB_HOST"/"$DB_PORT") 2>/dev/null; do
if [ "$SECONDS" -ge 120 ]; then
echo "timeout waiting for postgres at $DB_HOST:$DB_PORT after 120s" >&2
exit 1
fi
echo "waiting for postgres at $DB_HOST:$DB_PORT (elapsed ${SECONDS}s)"
sleep 2
done
echo "postgres at $DB_HOST:$DB_PORT is up"
- dockerize -wait "tcp://$DB_HOST:$DB_PORT" -timeout 120s
resources:
limits:
memory: "256Mi"
+49 -57
View File
@@ -38,20 +38,14 @@ dependencies = [
# no bounds for apache-superset-core until we have a stable version
"apache-superset-core",
"backoff>=1.8.0",
# cachetools is used directly by ``superset.db_engine_specs.aws_iam`` (TTLCache).
# It used to be installed transitively via ``google-auth`` (<2.53), but
# ``google-auth`` 2.53+ dropped it, so Superset must declare it
# explicitly to keep fresh ``pip install apache-superset`` working
# without the ``base.txt`` lock file (#40962).
"cachetools>=6.2.1, <7",
"celery>=5.3.6, <6.0.0",
"click>=8.4.0",
"click-option-group",
"colorama",
"flask-cors>=6.0.5, <7.0",
"croniter>=6.2.2",
"flask-cors>=6.0.0, <7.0",
"croniter>=0.3.28",
"cron-descriptor",
"cryptography>=48.0.0, <49.0.0",
"cryptography>=42.0.4, <47.0.0",
"deprecation>=2.1.0, <2.2.0",
"flask>=2.2.5, <4.0.0",
"flask-appbuilder>=5.2.1, <6.0.0",
@@ -59,27 +53,25 @@ dependencies = [
"flask-compress>=1.13, <2.0",
"flask-talisman>=1.0.0, <2.0",
"flask-login>=0.6.0, < 1.0",
"flask-migrate>=4.1.0, <5.0",
"flask-migrate>=3.1.0, <5.0",
"flask-session>=0.4.0, <1.0",
"flask-wtf>=1.3.0, <2.0",
"flask-wtf>=1.1.0, <2.0",
"geopy",
"greenlet<=3.5.1, >=3.5.1",
"greenlet>=3.0.3, <=3.5.0",
"gunicorn>=25.3.0, <26; sys_platform != 'win32'",
"hashids>=1.3.1, <2",
# holidays>=0.45 required for security fix
"holidays>=0.45, <1",
"humanize",
"isodate",
"jsonpath-ng>=1.8.0, <2",
"jsonpath-ng>=1.6.1, <2",
"Mako>=1.2.2",
"markdown>=3.10.2",
# marshmallow 4 compatibility: see superset/marshmallow_compatibility.py for a
# Flask-AppBuilder workaround. Tracking issue:
# https://github.com/apache/superset/issues/33162
"marshmallow>=3.0, <5",
# marshmallow>=4 has issues: https://github.com/apache/superset/issues/33162
"marshmallow>=3.0, <4",
"marshmallow-union>=0.1",
"msgpack>=1.2.0, <1.3",
"nh3>=0.3.5, <0.4",
"msgpack>=1.0.0, <1.2",
"nh3>=0.2.11, <0.4",
"numpy>1.23.5, <2.3",
"packaging",
# --------------------------
@@ -88,7 +80,7 @@ dependencies = [
"bottleneck", # recommended performance dependency for pandas, see https://pandas.pydata.org/docs/getting_started/install.html#performance-dependencies-recommended
# --------------------------
"parsedatetime",
"paramiko>=3.4.0, <4.0", # 4.0 removed DSSKey, still referenced by sshtunnel
"paramiko>=3.4.0",
"pgsanity",
"Pillow>=11.0.0, <13",
"polyline>=2.0.0, <3.0",
@@ -97,17 +89,18 @@ dependencies = [
"python-dateutil",
"python-dotenv", # optional dependencies for Flask but required for Superset, see https://flask.palletsprojects.com/en/stable/installation/#optional-dependencies
"pygeohash",
"pyarrow>=24.0.0, <25", # before upgrading pyarrow, check that all db dependencies support this, see e.g. https://github.com/apache/superset/pull/34693
"pyyaml>=6.0.3, <7.0.0",
"pyarrow>=16.1.0, <21", # before upgrading pyarrow, check that all db dependencies support this, see e.g. https://github.com/apache/superset/pull/34693
"pyyaml>=6.0.0, <7.0.0",
"PyJWT>=2.4.0, <3.0",
"redis>=5.0.0, <6.0",
"rison>=2.0.0, <3.0",
"selenium>=4.44.0, <5.0",
"selenium>=4.14.0, <5.0",
"shillelagh[gsheetsapi]>=1.4.4, <2.0",
"sshtunnel>=0.4.0, <0.5",
"simplejson>=4.1.1",
"simplejson>=3.15.0",
"slack_sdk>=3.19.0, <4",
"sqlalchemy>=1.4, <2",
"sqlalchemy-continuum>=1.6.0, <2.0.0",
"sqlalchemy-utils>=0.38.0, <0.43", # expanding lowerbound to work with pydoris
"sqlglot>=30.8.0, <31",
# newer pandas needs 0.9+
@@ -115,9 +108,9 @@ dependencies = [
"typing-extensions>=4, <5",
"waitress; sys_platform == 'win32'",
"watchdog>=6.0.0",
"wtforms>=3.2.2, <4",
"wtforms>=2.3.3, <4",
"wtforms-json",
"xlsxwriter>=3.2.9, <3.3",
"xlsxwriter>=3.0.7, <3.3",
]
[project.optional-dependencies]
@@ -126,12 +119,12 @@ athena = ["pyathena[pandas]>=2, <4"]
aurora-data-api = ["preset-sqlalchemy-aurora-data-api>=0.2.8,<0.3"]
bigquery = [
"pandas-gbq>=0.19.1",
"sqlalchemy-bigquery>=1.17.0",
"sqlalchemy-bigquery>=1.15.0",
"google-cloud-bigquery>=3.10.0",
]
clickhouse = ["clickhouse-connect>=1.1.1, <2.0"]
clickhouse = ["clickhouse-connect>=0.13.0, <2.0"]
cockroachdb = ["cockroachdb>=0.3.5, <0.4"]
crate = ["sqlalchemy-cratedb>=0.41.0, <1"]
crate = ["sqlalchemy-cratedb>=0.40.1, <1"]
d1 = [
"superset-engine-d1>=0.1.0",
"sqlalchemy-d1>=0.1.0",
@@ -143,80 +136,80 @@ databricks = [
"databricks-sqlalchemy==1.0.5",
]
db2 = ["ibm-db-sa>0.3.8, <=0.4.4"]
denodo = ["denodo-sqlalchemy>=2.0.5,<2.1.0"]
denodo = ["denodo-sqlalchemy>=1.0.6,<2.1.0"]
dremio = ["sqlalchemy-dremio>=1.2.1, <4"]
drill = ["sqlalchemy-drill>=1.1.10, <2"]
druid = ["pydruid>=0.6.5,<0.7"]
duckdb = ["duckdb>=1.5.2,<2", "duckdb-engine>=0.17.0"]
dynamodb = ["pydynamodb>=0.4.2"]
solr = ["sqlalchemy-solr >= 0.2.4.3"]
solr = ["sqlalchemy-solr >= 0.2.0"]
elasticsearch = ["elasticsearch-dbapi>=0.2.13, <0.3.0"]
exasol = ["sqlalchemy-exasol>=2.4.0, <8.0"]
excel = ["xlrd>=2.0.2, <2.1"]
exasol = ["sqlalchemy-exasol >= 2.4.0, < 8.0"]
excel = ["xlrd>=1.2.0, <1.3"]
fastmcp = [
"fastmcp>=3.2.4,<4.0",
# tiktoken backs the response-size-guard token estimator. Without
# it, the middleware falls back to a coarser character-based
# heuristic that under-counts JSON-heavy MCP responses.
"tiktoken>=0.13.0,<1.0",
"tiktoken>=0.7.0,<1.0",
]
firebird = ["sqlalchemy-firebird>=0.7.0, <2.2"]
firebolt = ["firebolt-sqlalchemy>=1.0.0, <2"]
gevent = ["gevent>=26.4.0"]
gevent = ["gevent>=23.9.1"]
gsheets = ["shillelagh[gsheetsapi]>=1.4.4, <2"]
hana = ["hdbcli==2.28.21", "sqlalchemy_hana==0.4.0"]
hana = ["hdbcli==2.28.20", "sqlalchemy_hana==0.4.0"]
hive = [
"pyhive[hive]>=0.6.5;python_version<'3.11'",
"pyhive[hive_pure_sasl]>=0.7.0",
"tableschema",
"thrift>=0.23.0, <1.0.0",
"thrift>=0.14.1, <1.0.0",
"thrift_sasl>=0.4.3, < 1.0.0",
]
impala = ["impyla>0.16.2, <0.23"]
kusto = ["sqlalchemy-kusto>=3.1.2, <4"]
kusto = ["sqlalchemy-kusto>=3.0.0, <4"]
kylin = ["kylinpy>=2.8.1, <2.9"]
mssql = ["pymssql>=2.3.13, <3"]
mssql = ["pymssql>=2.2.8, <3"]
# motherduck is an alias for duckdb - MotherDuck works via the duckdb driver
motherduck = ["apache-superset[duckdb]"]
mysql = ["mysqlclient>=2.1.0, <3"]
ocient = [
"sqlalchemy-ocient>=1.0.0",
"pyocient>=1.0.15, <4",
"pyocient>=1.0.15, <2",
"shapely",
"geojson",
]
oracle = ["oracledb>=2.0.0, <5"]
oracle = ["cx-Oracle>8.0.0, <8.4"]
parseable = ["sqlalchemy-parseable>=0.1.3,<0.2.0"]
pinot = ["pinotdb>=5.0.0, <10.0.0"]
playwright = ["playwright>=1.60.0, <2"]
playwright = ["playwright>=1.37.0, <2"]
postgres = ["psycopg2-binary==2.9.12"]
presto = ["pyhive[presto]>=0.6.5"]
trino = ["trino>=0.337.0"]
trino = ["trino>=0.328.0"]
prophet = ["prophet>=1.1.6, <2"]
redshift = ["sqlalchemy-redshift>=0.8.1, <0.9"]
risingwave = ["sqlalchemy-risingwave"]
shillelagh = ["shillelagh[all]>=1.4.4, <2"]
singlestore = ["sqlalchemy-singlestoredb>=1.1.1, <2"]
snowflake = ["snowflake-sqlalchemy>=1.2.4, <2"]
sqlite = ["syntaqlite>=0.1.0,<0.5.0"]
sqlite = ["syntaqlite>=0.1.0"]
spark = [
"pyhive[hive]>=0.6.5;python_version<'3.11'",
"pyhive[hive_pure_sasl]>=0.7",
"tableschema",
"thrift>=0.23.0, <1",
"thrift>=0.14.1, <1",
]
tdengine = [
"taospy>=2.7.21",
"taos-ws-py>=0.6.9"
"taos-ws-py>=0.3.8"
]
teradata = ["teradatasql>=16.20.0.23"]
thumbnails = [] # deprecated, will be removed in 7.0
vertica = ["sqlalchemy-vertica-python>= 0.6.3, < 0.7"]
vertica = ["sqlalchemy-vertica-python>= 0.5.9, < 0.7"]
netezza = ["nzalchemy>=11.0.2"]
starrocks = ["starrocks>=1.3.3, <2"]
starrocks = ["starrocks>=1.0.0"]
doris = ["pydoris>=1.0.0, <2.0.0"]
oceanbase = ["oceanbase_py>=0.0.1.2"]
ydb = ["ydb-sqlalchemy>=0.1.22", "ydb-sqlglot-plugin>=0.2.5"]
oceanbase = ["oceanbase_py>=0.0.1"]
ydb = ["ydb-sqlalchemy>=0.1.2"]
development = [
# no bounds for apache-superset-extensions-cli until a stable version
"apache-superset-extensions-cli",
@@ -224,26 +217,26 @@ development = [
"docker",
"flask-testing",
"freezegun",
"grpcio>=1.81.1",
"grpcio>=1.55.3",
"openapi-spec-validator",
"parameterized",
"pip",
"polib", # used by scripts/translations/ and their unit tests
"pre-commit",
"progress>=1.6.1,<2",
"progress>=1.5,<2",
"psutil",
"pyfakefs",
"pyinstrument>=5.1.2,<6",
"pyinstrument>=4.0.2,<6",
"pylint",
"pytest<8.0.0", # hairy issue with pytest >=8 where current_app proxies are not set in time
"pytest-asyncio",
"pytest-cov",
"pytest-mock",
"python-ldap>=3.4.7",
"python-ldap>=3.4.4",
"ruff",
"sqloxide",
"statsd",
"syntaqlite>=0.4.2,<0.5.0",
"syntaqlite>=0.1.0",
]
[project.urls]
@@ -375,6 +368,7 @@ select = [
ignore = [
"S101",
"PT004", # Fixtures that don't return values - underscore prefix conflicts with pytest usage
"PT006",
"T201",
"N999",
@@ -454,7 +448,6 @@ requirement_txt_file = "requirements/base.txt"
authorized_licenses = [
"academic free license (afl)",
"any-osi",
"apache-2.0",
"apache license 2.0",
"apache software",
"apache software, bsd",
@@ -464,7 +457,6 @@ authorized_licenses = [
"isc license (iscl)",
"isc license",
"mit",
"mit and psf-2.0",
"mit-cmu",
"mozilla public license 2.0 (mpl 2.0)",
"osi approved",
+1 -26
View File
@@ -18,30 +18,5 @@
testpaths =
tests
python_files = *_test.py test_*.py *_tests.py *viz/utils.py
# `-p no:warnings` temporarily disabled in favor of more finely tuned `filterwarnings`.
#addopts = -p no:warnings
addopts = -p no:warnings
asyncio_mode = auto
# `ignore` is effectively equivalent to `-p no:warnings`.
# Always print RemovedIn20Warning when SQLALCHEMY_WARN_20=1.
# Additionally, raise errors for refactored RemovedIn20Warning cases to prevent regression.
filterwarnings =
ignore
always::sqlalchemy.exc.RemovedIn20Warning
error:Passing a string to Connection.execute\(\) is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:"Query" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SavedQuery" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SqlaTable" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SqlMetric" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"TableColumn" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"TaggedObject" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:The ``as_declarative\(\)`` function is now available:sqlalchemy.exc.RemovedIn20Warning
# error:The autoload parameter is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The connection.execute\(\) method:sqlalchemy.exc.RemovedIn20Warning
# error:The current statement is being autocommitted using implicit autocommit:sqlalchemy.exc.RemovedIn20Warning
# error:The `database` package is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The ``declarative_base\(\)`` function is now available:sqlalchemy.exc.RemovedIn20Warning
# error:The Engine.execute\(\) method is considered legacy:sqlalchemy.exc.RemovedIn20Warning
error:The legacy calling style of select\(\) is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The "whens" argument to case:sqlalchemy.exc.RemovedIn20Warning
# error:"User" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
+7 -12
View File
@@ -26,11 +26,11 @@ filelock>=3.20.3,<4.0.0
brotli>=1.2.0,<2.0.0
numexpr>=2.9.0
# Security: CVE-2026-34073 (MEDIUM) - Improper Certificate Validation
cryptography>=48.0.0,<49.0.0
cryptography>=46.0.7,<47.0.0
# Security: Snyk - XSS vulnerability in Mako templates
mako>=1.3.11,<2.0.0
# Security: CVE-2024-52338 (CRITICAL) - Deserialization of untrusted data in IPC/Parquet readers
pyarrow>=24.0.0,<25.0.0
pyarrow>=20.0.0,<21.0.0
# Security: CVE-2026-27459 - pyopenssl certificate validation
pyopenssl>=26.0.0,<27.0.0
# Security: CVE-2026-25645 (MEDIUM) - Insecure Temporary File
@@ -44,10 +44,11 @@ async_timeout>=4.0.0,<5.0.0
# a bit of attention to bump.
apispec>=6.0.0,<6.7.0
# 1.4.1 introduced a memory regression that exhausts memory in the test suite
# (https://github.com/marshmallow-code/marshmallow-sqlalchemy/issues/665). 1.4.2
# claimed a fix but did not address the root cause; only 1.5.0 actually fixes it.
marshmallow-sqlalchemy>=1.5.0
# 1.4.1 appears to use much more memory, where the python test suite runs out of memory
# causing CI to fail. 1.4.0 is the last version that works.
# https://marshmallow-sqlalchemy.readthedocs.io/en/latest/changelog.html#id3
# Opened this issue https://github.com/marshmallow-code/marshmallow-sqlalchemy/issues/665
marshmallow-sqlalchemy>=1.3.0,<1.4.1
# needed for python 3.12 support
openapi-schema-validator>=0.6.3
@@ -57,9 +58,3 @@ openapi-schema-validator>=0.6.3
# Known affected packages: Preset's 'clients' package
# See docs/docs/contributing/pkg-resources-migration.md for details
setuptools<81
# google-auth 2.53+ dropped its transitive dependency on cachetools, which is
# imported directly by superset.db_engine_specs.aws_iam. We declare cachetools
# explicitly in pyproject.toml and pin google-auth to the post-drop range so
# the install path is internally consistent (#40962).
google-auth>=2.53.0,<3.0.0
+38 -30
View File
@@ -45,12 +45,12 @@ cachelib==0.13.0
# flask-caching
# flask-session
cachetools==6.2.1
# via apache-superset (pyproject.toml)
# via google-auth
cattrs==25.1.1
# via requests-cache
celery==5.5.2
# via apache-superset (pyproject.toml)
certifi==2026.5.20
certifi==2025.6.15
# via
# requests
# selenium
@@ -84,13 +84,12 @@ colorama==0.4.6
# flask-appbuilder
cron-descriptor==1.4.5
# via apache-superset (pyproject.toml)
croniter==6.2.2
croniter==6.0.0
# via apache-superset (pyproject.toml)
cryptography==48.0.1
cryptography==46.0.7
# via
# -r requirements/base.in
# apache-superset (pyproject.toml)
# google-auth
# paramiko
# pyopenssl
defusedxml==0.7.1
@@ -132,7 +131,7 @@ flask-caching==2.3.1
# via apache-superset (pyproject.toml)
flask-compress==1.17
# via apache-superset (pyproject.toml)
flask-cors==6.0.5
flask-cors==6.0.2
# via apache-superset (pyproject.toml)
flask-jwt-extended==4.7.1
# via flask-appbuilder
@@ -142,7 +141,7 @@ flask-login==0.6.3
# via
# apache-superset (pyproject.toml)
# flask-appbuilder
flask-migrate==4.1.0
flask-migrate==3.1.0
# via apache-superset (pyproject.toml)
flask-session==0.8.0
# via apache-superset (pyproject.toml)
@@ -152,7 +151,7 @@ flask-sqlalchemy==2.5.1
# flask-migrate
flask-talisman==1.1.0
# via apache-superset (pyproject.toml)
flask-wtf==1.3.0
flask-wtf==1.2.2
# via
# apache-superset (pyproject.toml)
# flask-appbuilder
@@ -160,11 +159,9 @@ geographiclib==2.0
# via geopy
geopy==2.4.1
# via apache-superset (pyproject.toml)
google-auth==2.53.0
# via
# -r requirements/base.in
# shillelagh
greenlet==3.5.1
google-auth==2.43.0
# via shillelagh
greenlet==3.1.1
# via
# apache-superset (pyproject.toml)
# shillelagh
@@ -179,7 +176,7 @@ holidays==0.82
# via apache-superset (pyproject.toml)
humanize==4.12.3
# via apache-superset (pyproject.toml)
idna==3.15
idna==3.10
# via
# email-validator
# requests
@@ -197,7 +194,7 @@ jinja2==3.1.6
# via
# flask
# flask-babel
jsonpath-ng==1.8.0
jsonpath-ng==1.7.0
# via apache-superset (pyproject.toml)
jsonschema==4.23.0
# via
@@ -211,7 +208,7 @@ kombu==5.5.3
# via celery
limits==5.1.0
# via flask-limiter
mako==1.3.12
mako==1.3.11
# via
# -r requirements/base.in
# apache-superset (pyproject.toml)
@@ -226,13 +223,13 @@ markupsafe==3.0.2
# mako
# werkzeug
# wtforms
marshmallow==4.3.0
marshmallow==3.26.2
# via
# apache-superset (pyproject.toml)
# flask-appbuilder
# marshmallow-sqlalchemy
# marshmallow-union
marshmallow-sqlalchemy==1.5.0
marshmallow-sqlalchemy==1.4.0
# via
# -r requirements/base.in
# flask-appbuilder
@@ -240,11 +237,11 @@ marshmallow-union==0.1.15
# via apache-superset (pyproject.toml)
mdurl==0.1.2
# via markdown-it-py
msgpack==1.2.1
msgpack==1.0.8
# via apache-superset (pyproject.toml)
msgspec==0.19.0
# via flask-session
nh3==0.3.5
nh3==0.2.21
# via apache-superset (pyproject.toml)
numexpr==2.10.2
# via -r requirements/base.in
@@ -273,6 +270,7 @@ packaging==25.0
# deprecation
# gunicorn
# limits
# marshmallow
# shillelagh
pandas==2.1.4
# via apache-superset (pyproject.toml)
@@ -288,19 +286,23 @@ pillow==12.2.0
# via apache-superset (pyproject.toml)
platformdirs==4.3.8
# via requests-cache
ply==3.11
# via jsonpath-ng
polyline==2.0.2
# via apache-superset (pyproject.toml)
prison==0.2.1
# via flask-appbuilder
prompt-toolkit==3.0.51
# via click-repl
pyarrow==24.0.0
pyarrow==20.0.0
# via
# -r requirements/base.in
# apache-superset (pyproject.toml)
# apache-superset-core
pyasn1==0.6.3
# via pyasn1-modules
# via
# pyasn1-modules
# rsa
pyasn1-modules==0.4.2
# via google-auth
pycparser==2.22
@@ -323,7 +325,7 @@ pyjwt==2.12.0
# redis
pynacl==1.6.2
# via paramiko
pyopenssl==26.2.0
pyopenssl==26.0.0
# via
# -r requirements/base.in
# shillelagh
@@ -344,11 +346,12 @@ python-dotenv==1.2.2
# via apache-superset (pyproject.toml)
pytz==2025.2
# via
# croniter
# flask-babel
# pandas
pyxlsb==1.0.10
# via pandas
pyyaml==6.0.3
pyyaml==6.0.2
# via
# apache-superset (pyproject.toml)
# apispec
@@ -375,13 +378,15 @@ rpds-py==0.25.0
# via
# jsonschema
# referencing
selenium==4.44.0
rsa==4.9.1
# via google-auth
selenium==4.32.0
# via apache-superset (pyproject.toml)
setuptools==80.9.0
# via -r requirements/base.in
shillelagh==1.4.4
# via apache-superset (pyproject.toml)
simplejson==4.1.1
simplejson==3.20.1
# via apache-superset (pyproject.toml)
six==1.17.0
# via
@@ -404,7 +409,10 @@ sqlalchemy==1.4.54
# flask-sqlalchemy
# marshmallow-sqlalchemy
# shillelagh
# sqlalchemy-continuum
# sqlalchemy-utils
sqlalchemy-continuum==1.6.0
# via apache-superset (pyproject.toml)
sqlalchemy-utils==0.42.0
# via
# apache-superset (pyproject.toml)
@@ -418,7 +426,7 @@ sshtunnel==0.4.0
# via apache-superset (pyproject.toml)
tabulate==0.10.0
# via apache-superset (pyproject.toml)
trio==0.33.0
trio==0.30.0
# via
# selenium
# trio-websocket
@@ -446,7 +454,7 @@ tzdata==2025.2
# pandas
url-normalize==2.2.1
# via requests-cache
urllib3==2.7.0
urllib3==2.6.3
# via
# -r requirements/base.in
# requests
@@ -475,7 +483,7 @@ wrapt==1.17.2
# via deprecated
wsproto==1.2.0
# via trio-websocket
wtforms==3.2.2
wtforms==3.2.1
# via
# apache-superset (pyproject.toml)
# flask-appbuilder
@@ -485,7 +493,7 @@ wtforms-json==0.3.5
# via apache-superset (pyproject.toml)
xlrd==2.0.1
# via pandas
xlsxwriter==3.2.9
xlsxwriter==3.0.9
# via
# apache-superset (pyproject.toml)
# pandas
+56 -48
View File
@@ -52,7 +52,7 @@ attrs==25.3.0
# referencing
# requests-cache
# trio
authlib==1.6.12
authlib==1.6.9
# via fastmcp
babel==2.17.0
# via
@@ -100,7 +100,7 @@ cachelib==0.13.0
cachetools==6.2.1
# via
# -c requirements/base-constraint.txt
# apache-superset
# google-auth
# py-key-value-aio
caio==0.9.25
# via aiofile
@@ -112,7 +112,7 @@ celery==5.5.2
# via
# -c requirements/base-constraint.txt
# apache-superset
certifi==2026.5.20
certifi==2025.6.15
# via
# -c requirements/base-constraint.txt
# httpcore
@@ -174,16 +174,15 @@ cron-descriptor==1.4.5
# via
# -c requirements/base-constraint.txt
# apache-superset
croniter==6.2.2
croniter==6.0.0
# via
# -c requirements/base-constraint.txt
# apache-superset
cryptography==48.0.1
cryptography==46.0.7
# via
# -c requirements/base-constraint.txt
# apache-superset
# authlib
# google-auth
# paramiko
# pyjwt
# pyopenssl
@@ -277,7 +276,7 @@ flask-compress==1.17
# via
# -c requirements/base-constraint.txt
# apache-superset
flask-cors==6.0.5
flask-cors==6.0.2
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -294,7 +293,7 @@ flask-login==0.6.3
# -c requirements/base-constraint.txt
# apache-superset
# flask-appbuilder
flask-migrate==4.1.0
flask-migrate==3.1.0
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -313,12 +312,12 @@ flask-talisman==1.1.0
# apache-superset
flask-testing==0.8.1
# via apache-superset
flask-wtf==1.3.0
flask-wtf==1.2.2
# via
# -c requirements/base-constraint.txt
# apache-superset
# flask-appbuilder
fonttools==4.60.2
fonttools==4.55.0
# via matplotlib
freezegun==1.5.1
# via apache-superset
@@ -332,7 +331,7 @@ geopy==2.4.1
# via
# -c requirements/base-constraint.txt
# apache-superset
gevent==26.4.0
gevent==24.2.1
# via apache-superset
google-api-core==2.23.0
# via
@@ -341,13 +340,12 @@ google-api-core==2.23.0
# google-cloud-core
# pandas-gbq
# sqlalchemy-bigquery
google-auth==2.53.0
google-auth==2.43.0
# via
# -c requirements/base-constraint.txt
# google-api-core
# google-auth-oauthlib
# google-cloud-bigquery
# google-cloud-bigquery-storage
# google-cloud-core
# pandas-gbq
# pydata-google-auth
@@ -362,7 +360,7 @@ google-cloud-bigquery==3.27.0
# apache-superset
# pandas-gbq
# sqlalchemy-bigquery
google-cloud-bigquery-storage==2.26.0
google-cloud-bigquery-storage==2.19.1
# via pandas-gbq
google-cloud-core==2.4.1
# via google-cloud-bigquery
@@ -374,7 +372,7 @@ googleapis-common-protos==1.66.0
# via
# google-api-core
# grpcio-status
greenlet==3.5.1
greenlet==3.1.1
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -383,7 +381,7 @@ greenlet==3.5.1
# sqlalchemy
griffelib==2.0.2
# via fastmcp
grpcio==1.81.1
grpcio==1.71.0
# via
# apache-superset
# google-api-core
@@ -423,7 +421,7 @@ humanize==4.12.3
# apache-superset
identify==2.5.36
# via pre-commit
idna==3.15
idna==3.10
# via
# -c requirements/base-constraint.txt
# anyio
@@ -472,7 +470,7 @@ jmespath==1.1.0
# via
# boto3
# botocore
jsonpath-ng==1.8.0
jsonpath-ng==1.7.0
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -508,9 +506,7 @@ limits==5.1.0
# via
# -c requirements/base-constraint.txt
# flask-limiter
lz4==4.4.5
# via trino
mako==1.3.12
mako==1.3.11
# via
# -c requirements/base-constraint.txt
# alembic
@@ -530,14 +526,14 @@ markupsafe==3.0.2
# mako
# werkzeug
# wtforms
marshmallow==4.3.0
marshmallow==3.26.2
# via
# -c requirements/base-constraint.txt
# apache-superset
# flask-appbuilder
# marshmallow-sqlalchemy
# marshmallow-union
marshmallow-sqlalchemy==1.5.0
marshmallow-sqlalchemy==1.4.0
# via
# -c requirements/base-constraint.txt
# flask-appbuilder
@@ -559,7 +555,7 @@ more-itertools==10.8.0
# via
# jaraco-classes
# jaraco-functools
msgpack==1.2.1
msgpack==1.0.8
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -569,7 +565,7 @@ msgspec==0.19.0
# flask-session
mysqlclient==2.2.6
# via apache-superset
nh3==0.3.5
nh3==0.2.21
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -611,8 +607,6 @@ ordered-set==4.1.0
# via
# -c requirements/base-constraint.txt
# flask-limiter
orjson==3.11.9
# via trino
outcome==1.3.0.post0
# via
# -c requirements/base-constraint.txt
@@ -631,6 +625,7 @@ packaging==25.0
# google-cloud-bigquery
# gunicorn
# limits
# marshmallow
# matplotlib
# pytest
# shillelagh
@@ -678,6 +673,10 @@ platformdirs==4.3.8
# virtualenv
pluggy==1.5.0
# via pytest
ply==3.11
# via
# -c requirements/base-constraint.txt
# jsonpath-ng
polib==1.2.0
# via apache-superset
polyline==2.0.2
@@ -690,7 +689,7 @@ prison==0.2.1
# via
# -c requirements/base-constraint.txt
# flask-appbuilder
progress==1.6.1
progress==1.6
# via apache-superset
prompt-toolkit==3.0.51
# via
@@ -702,7 +701,7 @@ proto-plus==1.25.0
# via
# google-api-core
# google-cloud-bigquery-storage
protobuf==5.29.6
protobuf==4.25.8
# via
# google-api-core
# google-cloud-bigquery-storage
@@ -715,7 +714,7 @@ psycopg2-binary==2.9.12
# via apache-superset
py-key-value-aio==0.4.4
# via fastmcp
pyarrow==24.0.0
pyarrow==20.0.0
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -727,6 +726,7 @@ pyasn1==0.6.3
# -c requirements/base-constraint.txt
# pyasn1-modules
# python-ldap
# rsa
pyasn1-modules==0.4.2
# via
# -c requirements/base-constraint.txt
@@ -767,7 +767,7 @@ pygments==2.20.0
# rich
pyhive==0.7.0
# via apache-superset
pyinstrument==5.1.2
pyinstrument==4.4.0
# via apache-superset
pyjwt==2.12.0
# via
@@ -783,7 +783,7 @@ pynacl==1.6.2
# via
# -c requirements/base-constraint.txt
# paramiko
pyopenssl==26.2.0
pyopenssl==26.0.0
# via
# -c requirements/base-constraint.txt
# shillelagh
@@ -837,13 +837,14 @@ python-dotenv==1.2.2
# apache-superset
# fastmcp
# pydantic-settings
python-ldap==3.4.7
python-ldap==3.4.4
# via apache-superset
python-multipart==0.0.29
python-multipart==0.0.20
# via mcp
pytz==2025.2
# via
# -c requirements/base-constraint.txt
# croniter
# flask-babel
# pandas
# trino
@@ -851,7 +852,7 @@ pyxlsb==1.0.10
# via
# -c requirements/base-constraint.txt
# pandas
pyyaml==6.0.3
pyyaml==6.0.2
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -913,13 +914,17 @@ rpds-py==0.25.0
# -c requirements/base-constraint.txt
# jsonschema
# referencing
rsa==4.9.1
# via
# -c requirements/base-constraint.txt
# google-auth
ruff==0.9.7
# via apache-superset
s3transfer==0.16.0
# via boto3
secretstorage==3.5.0
# via keyring
selenium==4.44.0
selenium==4.32.0
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -937,7 +942,7 @@ shillelagh==1.4.4
# via
# -c requirements/base-constraint.txt
# apache-superset
simplejson==4.1.1
simplejson==3.20.1
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -973,9 +978,14 @@ sqlalchemy==1.4.54
# marshmallow-sqlalchemy
# shillelagh
# sqlalchemy-bigquery
# sqlalchemy-continuum
# sqlalchemy-utils
sqlalchemy-bigquery==1.17.0
sqlalchemy-bigquery==1.15.0
# via apache-superset
sqlalchemy-continuum==1.6.0
# via
# -c requirements/base-constraint.txt
# apache-superset
sqlalchemy-utils==0.42.0
# via
# -c requirements/base-constraint.txt
@@ -999,13 +1009,13 @@ starlette==0.49.1
# via mcp
statsd==4.0.1
# via apache-superset
syntaqlite==0.4.2
syntaqlite==0.1.0
# via apache-superset
tabulate==0.10.0
# via
# -c requirements/base-constraint.txt
# apache-superset
tiktoken==0.13.0
tiktoken==0.12.0
# via apache-superset
tomli-w==1.2.0
# via apache-superset-extensions-cli
@@ -1015,9 +1025,9 @@ tqdm==4.67.1
# via
# cmdstanpy
# prophet
trino==0.337.0
trino==0.330.0
# via apache-superset
trio==0.33.0
trio==0.30.0
# via
# -c requirements/base-constraint.txt
# selenium
@@ -1035,7 +1045,6 @@ typing-extensions==4.15.0
# apache-superset-core
# cattrs
# exceptiongroup
# grpcio
# limits
# mcp
# opentelemetry-api
@@ -1067,7 +1076,7 @@ url-normalize==2.2.1
# via
# -c requirements/base-constraint.txt
# requests-cache
urllib3==2.7.0
urllib3==2.6.3
# via
# -c requirements/base-constraint.txt
# botocore
@@ -1085,7 +1094,7 @@ vine==5.1.0
# amqp
# celery
# kombu
virtualenv==20.36.1
virtualenv==20.29.2
# via pre-commit
watchdog==6.0.0
# via
@@ -1120,7 +1129,7 @@ wsproto==1.2.0
# via
# -c requirements/base-constraint.txt
# trio-websocket
wtforms==3.2.2
wtforms==3.2.1
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -1135,7 +1144,7 @@ xlrd==2.0.1
# via
# -c requirements/base-constraint.txt
# pandas
xlsxwriter==3.2.9
xlsxwriter==3.0.9
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -1150,4 +1159,3 @@ zstandard==0.23.0
# via
# -c requirements/base-constraint.txt
# flask-compress
# trino
+2 -2
View File
@@ -30,7 +30,7 @@ from flask import current_app
from flask_appbuilder import Model
from flask_migrate import downgrade, upgrade
from progress.bar import ChargingBar
from sqlalchemy import create_engine, inspect, text
from sqlalchemy import create_engine, inspect
from sqlalchemy.ext.automap import automap_base
from superset import db
@@ -154,7 +154,7 @@ def main( # noqa: C901
print(f"Migration goes from {down_revision} to {revision}")
current_revision = db.engine.execute(
text("SELECT version_num FROM alembic_version")
"SELECT version_num FROM alembic_version"
).scalar()
print(f"Current version of the DB is {current_revision}")
+1 -2
View File
@@ -33,7 +33,6 @@ PATTERNS = {
r"^setup\.py",
r"^pyproject\.toml$",
r"^requirements/.+\.txt",
r"^pyproject\.toml",
r"^.pylintrc",
],
"frontend": [
@@ -157,7 +156,7 @@ def main(event_type: str, sha: str, repo: str) -> None:
def get_git_sha() -> str:
return os.getenv("GITHUB_SHA") or subprocess.check_output( # noqa: S603
["git", "rev-parse", "HEAD"] # noqa: S603, S607
["git", "rev-parse", "HEAD"] # noqa: S607
).strip().decode("utf-8")
+1 -1
View File
@@ -55,7 +55,7 @@ if [ ${#js_ts_files[@]} -gt 0 ]; then
echo "$output" >&2
exit 1
}
if [ -n "$output" ]; then echo "$output"; fi
[ -n "$output" ] && echo "$output"
else
echo "No JavaScript/TypeScript files to lint"
fi

Some files were not shown because too many files have changed in this diff Show More