Adds a structured per-field change log alongside the foundational
shadow tables. Each save flush emits zero or more ``version_changes``
rows describing what changed relative to the previous version, with
shape ``[{kind, path, from_value, to_value, sequence}]`` keyed to
``version_transaction.id`` (FR-016 .. FR-021).
**Schema** — ``version_changes`` table, FK to ``version_transaction``
with ``ON DELETE CASCADE`` so retention drops dependent records
without explicit cleanup. Composite unique index on
``(transaction_id, entity_kind, entity_id, sequence)`` so the
listener can write monotonically and downstream readers see a
deterministic order.
**Diff engine** (``superset/versioning/diff.py``) — pure-function
diffing of pre-/post-state pairs:
- ``diff_scalar_fields`` for ordinary columns; emits one record per
changed field with JSON-safe ``from_value`` / ``to_value``.
- ``diff_json_field`` for ``json_metadata`` and ``params``, walking
the parsed structure and emitting per-sub-key records. Honours
an ``exclude_keys`` set
(``_DASHBOARD_JSON_METADATA_AUDIT_KEYS``: ``chart_configuration``,
``global_chart_configuration``, ``map_label_colors``,
``show_chart_timestamps``, ``color_namespace``;
``_CHART_PARAMS_AUDIT_KEYS``) so frontend-stamped sub-keys that
mutate on every save don't dominate the change log (FR-022).
- ``diff_dashboard_layout`` walks ``position_json`` structurally
and emits ``[verb, kind, id]`` records (verbs ``add``, ``remove``,
``move``, ``edit``; kinds from a ``CHART``/``ROW``/``COLUMN``/etc.
→ english map) so a UI can render "Added chart 'Foo'" without
re-parsing JSON. ``HEADER_ID`` is suppressed because it duplicates
the ``dashboard_title`` scalar record.
- ``fold_dashboard_layout_with_chart_changes`` deduplicates layout
records against M2M / chart-membership records by UUID so an
add-and-attach doesn't appear twice.
- ``_values_equivalent`` treats ``None`` and ``""`` as equal; this
matches the save path's habit of normalising nullable strings to
the empty string.
**Listener** — ``superset/versioning/changes.py`` registers a
``before_flush`` listener that captures pre-state for each dirty
entity and an ``after_flush`` listener that runs the diff engine
against the post-state and writes ``version_changes`` rows under
the resolved ``transaction_id``. Tracks processed transaction ids
on ``session.info`` so re-firings within a single transaction
(autoflush triggered by mid-commit queries) don't double-insert and
trip the unique constraint. Reads child rows via raw SELECT against
``table_columns`` / ``sql_metrics`` rather than ``dataset.columns``
because the live collection is stale during the restore path's raw
DELETE+INSERT cycle.
**Endpoint surface** — ``VersionDAO.list_change_records_batch``
batches the lookup across multiple transactions with a single
``WHERE transaction_id IN (...)`` query so the version-list
endpoint avoids N+1 round-trips. ``list_versions`` / ``get_version``
return entries with a populated ``changes`` array (empty for
``operation_type=0`` baseline rows).
**Tests** — ``test_diff.py`` covers the diff engine shape (39
unit cases across scalar, JSON, layout, child-collection, and
fold paths). ``change_records_tests.py`` exercises the listener
end-to-end with realistic save flows. ``perf_validation_tests.py``
is the T044 harness for SC-002/3/4 (list endpoint p95 < 1s,
restore < 3s, save overhead < 50ms).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds SQLAlchemy-Continuum as a dependency and wires it as the
canonical capture mechanism for chart, dashboard, and dataset edits.
**Schema** — three Alembic migrations, leaving the chain at one
foundation revision plus one child-shadow revision:
- ``version_transaction`` (renamed from Continuum's default
``transaction``; SQL-reserved-word workaround) carries the per-save
``user_id`` / ``issued_at`` and is the join target for all shadow
rows. Auto-incrementing PK; user_id has no FK so import / Celery /
CLI saves can write rows without an active Flask user.
- Parent shadow tables for the three entity types:
``dashboards_version``, ``slices_version``, ``tables_version``.
- Child shadow tables for dataset children + dashboard M2M:
``table_columns_version``, ``sql_metrics_version``,
``dashboard_slices_version`` (composite PK on the M2M shadow,
matching the live ``dashboard_slices`` reshape from
sc-105349-composite-association-pks).
**Models** — ``Dashboard``, ``Slice``, ``SqlaTable`` (and dataset
children ``TableColumn`` / ``SqlMetric``) gain ``__versioned__``
class attributes. The exclude lists carry both M2M relationships
(``owners``, ``roles``, ``dashboards``) and the ``AuditMixin``
columns (``changed_on`` / ``created_on`` / ``changed_by_fk`` /
``created_by_fk`` plus ``last_saved_at`` / ``last_saved_by_fk``
on ``Slice``) so auto-bumped audit fields cannot trigger a
version row on their own (FR-025).
**Plugins** — ``superset/versioning/factory.py`` ships three
Continuum plugins:
- ``VersionTransactionFactory`` renames the transaction table and
appends the unconditional ``user_id`` column.
- ``VersioningFlaskPlugin`` sources the acting user from Superset's
``g.user`` rather than ``flask_login.current_user`` (Superset's
JWT auth populates ``g.user`` but leaves ``current_user``
anonymous on API routes).
- ``SkipUnmodifiedPlugin`` filters Continuum's UPDATE operations,
marking content-equivalent re-saves as ``processed=True`` so they
don't mint no-op shadow rows (FR-026; see follow-up commits for
the test). Lives in this commit because it shares the file with
the other plugins.
**Save-path glue** — a ``before_flush`` baseline listener
(``superset/versioning/baseline.py``) inserts an ``operation_type=0``
shadow row the first time a pre-existing entity is saved, including
the slice-baseline-under-dashboard pattern that gives the dashboard
M2M shadow a row to join against. ``UpdateDashboardCommand`` wraps
its body in ``no_autoflush`` so ``process_tab_diff`` /
``process_native_filter_diff`` don't fire intermediate flushes that
would mint extra version rows. ``DatasetDAO.update_columns`` is
rewritten as a natural-key upsert keyed on ``column_name`` so child
edits flow through ORM events Continuum sees.
**DAO** — ``superset/daos/version.py`` exposes the read API used by
the version endpoints in the next commits:
``current_version_number`` (0-based index, unstable under retention
pruning), ``current_live_transaction_id`` (stable across pruning),
``current_live_version_uuid`` (deterministic UUIDv5), plus
``list_versions`` / ``get_version`` / ``restore_version`` and a
batch ``list_change_records_batch`` for N+1 avoidance.
**Initialization** — ``superset/initialization/__init__.py`` wires
``init_versioning()`` after ``make_versioned()`` runs and the
versioned mappers are configured. Registers the baseline listener
plus the change-record listener (the latter's body lives in the
next commit but the registration site is here because it shares
the init function).
**Tests** — version-capture and version-list integration tests for
each entity type, plus a ``VersionDAO`` unit test suite. Retention
test uses a backdated ``issued_at`` so it can drive
``_prune_old_versions_impl`` synchronously.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanups from PR review:
1. ``dashboard_roles.dashboard_id`` was created nullable in revision
e11ccdd12658 but was missing from ``TABLES_WITH_NULLABLE_FKS``. A
production database with a stray NULL ``dashboard_id`` row would have
failed the PK-add with a cryptic constraint violation. Fix by running
the NULL-FK cleanup on every affected table — it is a no-op DELETE on
tables whose FK columns are already NOT NULL, and it eliminates the
risk of further drift in the hardcoded set. ``dashboard_roles`` is
added to the documentation set; the runtime now does not consult it.
2. The unit-test parent-table name for ``rls_filter_roles`` and
``rls_filter_tables`` was ``rls_filter`` (does not exist) instead of
the real parent ``row_level_security_filters``. Test passes either
way (the in-memory FK is self-consistent), but the parameter is now
accurate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace synthetic id INTEGER PRIMARY KEY with composite PRIMARY KEY (fk1, fk2)
on the eight pure-junction tables: dashboard_roles, dashboard_slices,
dashboard_user, report_schedule_user, rls_filter_roles, rls_filter_tables,
slice_user, sqlatable_user. The redundant UNIQUE(fk1, fk2) on dashboard_slices
and report_schedule_user is dropped (subsumed by the new PK).
Migration handles dialect quirks: copy_from for tables with pre-existing
UNIQUE (so SQLite's anonymous-constraint reflection doesn't matter), wrapped-
subquery dedupe for MySQL (ERROR 1093), sa.Identity(always=False) on downgrade
to backfill the restored id column without NOT NULL violations, and distinct
PK names per direction (pk_<table> on upgrade, <table>_pkey on downgrade) to
avoid round-trip index-name collisions on Postgres.
ORM Table() definitions updated to match. UPDATING.md entry added with
operator runbook (BI-tool impact, pre-flight inventory queries, dedupe-row-
loss notice, pg_dump workaround, FK-NOT-NULL downgrade asymmetry note).
Tests: 8 schema-shape assertions (post-upgrade), 8 duplicate-rejection unit
tests, 8 distinct-pair sanity tests, 1 round-trip + idempotency test
(in-memory SQLite via Alembic MigrationContext).
Continuum-restore verification against the new shape is out of scope for this
PR; it is the responsibility of the versioning epic (sc-103156).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
