2 Commits

Author	SHA1	Message	Date
Mike Bridge	7ec3091d55	feat(versioning): change records + diff engine ... Adds a structured per-field change log alongside the foundational shadow tables. Each save flush emits zero or more ``version_changes`` rows describing what changed relative to the previous version, with shape ``[{kind, path, from_value, to_value, sequence}]`` keyed to ``version_transaction.id`` (FR-016 .. FR-021). **Schema** — ``version_changes`` table, FK to ``version_transaction`` with ``ON DELETE CASCADE`` so retention drops dependent records without explicit cleanup. Composite unique index on ``(transaction_id, entity_kind, entity_id, sequence)`` so the listener can write monotonically and downstream readers see a deterministic order. **Diff engine** (``superset/versioning/diff.py``) — pure-function diffing of pre-/post-state pairs: - ``diff_scalar_fields`` for ordinary columns; emits one record per changed field with JSON-safe ``from_value`` / ``to_value``. - ``diff_json_field`` for ``json_metadata`` and ``params``, walking the parsed structure and emitting per-sub-key records. Honours an ``exclude_keys`` set (``_DASHBOARD_JSON_METADATA_AUDIT_KEYS``: ``chart_configuration``, ``global_chart_configuration``, ``map_label_colors``, ``show_chart_timestamps``, ``color_namespace``; ``_CHART_PARAMS_AUDIT_KEYS``) so frontend-stamped sub-keys that mutate on every save don't dominate the change log (FR-022). - ``diff_dashboard_layout`` walks ``position_json`` structurally and emits ``[verb, kind, id]`` records (verbs ``add``, ``remove``, ``move``, ``edit``; kinds from a ``CHART``/``ROW``/``COLUMN``/etc. → english map) so a UI can render "Added chart 'Foo'" without re-parsing JSON. ``HEADER_ID`` is suppressed because it duplicates the ``dashboard_title`` scalar record. - ``fold_dashboard_layout_with_chart_changes`` deduplicates layout records against M2M / chart-membership records by UUID so an add-and-attach doesn't appear twice. - ``_values_equivalent`` treats ``None`` and ``""`` as equal; this matches the save path's habit of normalising nullable strings to the empty string. **Listener** — ``superset/versioning/changes.py`` registers a ``before_flush`` listener that captures pre-state for each dirty entity and an ``after_flush`` listener that runs the diff engine against the post-state and writes ``version_changes`` rows under the resolved ``transaction_id``. Tracks processed transaction ids on ``session.info`` so re-firings within a single transaction (autoflush triggered by mid-commit queries) don't double-insert and trip the unique constraint. Reads child rows via raw SELECT against ``table_columns`` / ``sql_metrics`` rather than ``dataset.columns`` because the live collection is stale during the restore path's raw DELETE+INSERT cycle. **Endpoint surface** — ``VersionDAO.list_change_records_batch`` batches the lookup across multiple transactions with a single ``WHERE transaction_id IN (...)`` query so the version-list endpoint avoids N+1 round-trips. ``list_versions`` / ``get_version`` return entries with a populated ``changes`` array (empty for ``operation_type=0`` baseline rows). **Tests** — ``test_diff.py`` covers the diff engine shape (39 unit cases across scalar, JSON, layout, child-collection, and fold paths). ``change_records_tests.py`` exercises the listener end-to-end with realistic save flows. ``perf_validation_tests.py`` is the T044 harness for SC-002/3/4 (list endpoint p95 < 1s, restore < 3s, save overhead < 50ms). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 15:37:19 -06:00
Mike Bridge	b68fea17af	feat(versioning): foundation — Continuum capture + parent/child shadow tables + VersionDAO ... Adds SQLAlchemy-Continuum as a dependency and wires it as the canonical capture mechanism for chart, dashboard, and dataset edits. **Schema** — three Alembic migrations, leaving the chain at one foundation revision plus one child-shadow revision: - ``version_transaction`` (renamed from Continuum's default ``transaction``; SQL-reserved-word workaround) carries the per-save ``user_id`` / ``issued_at`` and is the join target for all shadow rows. Auto-incrementing PK; user_id has no FK so import / Celery / CLI saves can write rows without an active Flask user. - Parent shadow tables for the three entity types: ``dashboards_version``, ``slices_version``, ``tables_version``. - Child shadow tables for dataset children + dashboard M2M: ``table_columns_version``, ``sql_metrics_version``, ``dashboard_slices_version`` (composite PK on the M2M shadow, matching the live ``dashboard_slices`` reshape from sc-105349-composite-association-pks). **Models** — ``Dashboard``, ``Slice``, ``SqlaTable`` (and dataset children ``TableColumn`` / ``SqlMetric``) gain ``__versioned__`` class attributes. The exclude lists carry both M2M relationships (``owners``, ``roles``, ``dashboards``) and the ``AuditMixin`` columns (``changed_on`` / ``created_on`` / ``changed_by_fk`` / ``created_by_fk`` plus ``last_saved_at`` / ``last_saved_by_fk`` on ``Slice``) so auto-bumped audit fields cannot trigger a version row on their own (FR-025). **Plugins** — ``superset/versioning/factory.py`` ships three Continuum plugins: - ``VersionTransactionFactory`` renames the transaction table and appends the unconditional ``user_id`` column. - ``VersioningFlaskPlugin`` sources the acting user from Superset's ``g.user`` rather than ``flask_login.current_user`` (Superset's JWT auth populates ``g.user`` but leaves ``current_user`` anonymous on API routes). - ``SkipUnmodifiedPlugin`` filters Continuum's UPDATE operations, marking content-equivalent re-saves as ``processed=True`` so they don't mint no-op shadow rows (FR-026; see follow-up commits for the test). Lives in this commit because it shares the file with the other plugins. **Save-path glue** — a ``before_flush`` baseline listener (``superset/versioning/baseline.py``) inserts an ``operation_type=0`` shadow row the first time a pre-existing entity is saved, including the slice-baseline-under-dashboard pattern that gives the dashboard M2M shadow a row to join against. ``UpdateDashboardCommand`` wraps its body in ``no_autoflush`` so ``process_tab_diff`` / ``process_native_filter_diff`` don't fire intermediate flushes that would mint extra version rows. ``DatasetDAO.update_columns`` is rewritten as a natural-key upsert keyed on ``column_name`` so child edits flow through ORM events Continuum sees. **DAO** — ``superset/daos/version.py`` exposes the read API used by the version endpoints in the next commits: ``current_version_number`` (0-based index, unstable under retention pruning), ``current_live_transaction_id`` (stable across pruning), ``current_live_version_uuid`` (deterministic UUIDv5), plus ``list_versions`` / ``get_version`` / ``restore_version`` and a batch ``list_change_records_batch`` for N+1 avoidance. **Initialization** — ``superset/initialization/__init__.py`` wires ``init_versioning()`` after ``make_versioned()`` runs and the versioned mappers are configured. Registers the baseline listener plus the change-record listener (the latter's body lives in the next commit but the registration site is here because it shares the init function). **Tests** — version-capture and version-list integration tests for each entity type, plus a ``VersionDAO`` unit test suite. Retention test uses a backdated ``issued_at`` so it can drive ``_prune_old_versions_impl`` synchronously. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>	2026-05-28 15:37:19 -06:00