Found by running fresh-install + round-trip against a real SQLite DB:
6 of the 8 affected tables had FK columns that were originally
declared nullable. PostgreSQL and MySQL implicitly promote the
constituent columns of an ``ALTER TABLE ... ADD PRIMARY KEY`` to
``NOT NULL``; SQLite does not (it's a long-standing SQLite quirk —
only ``INTEGER PRIMARY KEY`` enforces NOT NULL on a composite-PK
column). Result: a fresh SQLite install would accept
``INSERT INTO dashboard_slices (NULL, 5)`` despite both columns
being part of the composite PK.
Our integration tests previously masked this: the test fixture seeds
columns with ``nullable=False``, so the post-upgrade NOT NULL
assertion passed regardless of whether the migration enforced it.
Fix: add explicit ``batch_op.alter_column(fk, nullable=False)`` for
both FK columns inside the per-table batch_alter_table block. On
PostgreSQL and MySQL this is a no-op (PK already implies NOT NULL);
on SQLite it adds the missing NOT NULL declaration so a fresh
install matches the data-model.md "After" contract.
Verified end-to-end:
- Postgres + MySQL: column shape unchanged (still NOT NULL)
- SQLite fresh install + round-trip: all 8 tables now have NOT NULL
on FK columns, ``INSERT (NULL, 5)`` correctly rejected with
IntegrityError on dashboard_slices, dashboard_user, sqlatable_user
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two MySQL-only failures in the downgrade path, found by running the
full migration history against a fresh MySQL 8 container:
1. ``MySQLdb.OperationalError: (1553, "Cannot drop index 'PRIMARY':
needed in a foreign key constraint")``. InnoDB uses the composite
PK index to back the FK on the leftmost column. The downgrade
tried to drop the composite PK before dropping the FKs, orphaning
the FK's backing index. PostgreSQL and SQLite create separate
indexes for FK columns and don't trip on this.
2. ``Field 'id' doesn't have a default value`` on subsequent INSERT.
``sa.Identity(always=False)`` only emits ``AUTO_INCREMENT`` on
MySQL when the column is created with ``primary_key=True`` — our
portable path adds the column first then creates the PK separately,
so MySQL leaves the column without auto-generation. Existing rows
would all collide on id=0; future inserts fail because no default.
Postgres' ``GENERATED BY DEFAULT AS IDENTITY`` and SQLite's
``INTEGER PRIMARY KEY`` rowid alias don't have this gap.
Fix: extract ``_downgrade_mysql_table()`` that emits the canonical
MySQL idiom — drop FKs, then a single ALTER combining
``DROP PRIMARY KEY, ADD COLUMN id INT NOT NULL AUTO_INCREMENT,
ADD PRIMARY KEY (id)`` (which backfills existing rows with sequential
ids and preserves AUTO_INCREMENT), restore the redundant UNIQUE on
the 2 tables that originally had it, and re-add the FKs with their
original names. Postgres and SQLite keep the existing portable
``batch_alter_table`` path.
Raw SQL is unavoidable for the combined-ALTER form; per the
constitution it's allowed for dialect-specific DDL with no SQLA
equivalent, with triple-quoted strings for legibility.
Verified end-to-end: upgrade → downgrade → upgrade against a fresh
MySQL 8 container with INSERT-without-id sanity check showing the
restored ``id`` column auto-increments correctly.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
CI test-mysql failed with:
MySQLdb.OperationalError: (1826, "Duplicate foreign key constraint
name 'fk_dashboard_slices_slice_id_slices'")
Root cause: MySQL scopes foreign-key constraint names per-database,
not per-table (PostgreSQL and SQLite scope per-table). The
``batch_alter_table(... recreate="always", copy_from=...)`` path
used for ``dashboard_slices`` and ``report_schedule_user`` builds
``_alembic_tmp_<table>`` carrying the original FK names from
``copy_from`` while the original table still holds those names — MySQL
rejects the temp-table creation with ERROR 1826.
Fix: on MySQL only, drop the original FK constraints by name before
the ``batch_alter_table`` runs. The ``copy_from`` re-creates them on
the rebuilt table with their original names, so the post-migration
shape is unchanged. On PostgreSQL and SQLite the original code path
still runs unchanged.
Local SQLite tests (44 passed, 1 skipped) still pass; CI will validate
on MySQL.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Address Beto's review comments on apache/superset#39859: replace
``sa.text(f"...")`` SQL construction in the three pre-flight helpers
(``_delete_null_fk_rows``, ``_dedupe_by_min_id``, ``_assert_no_duplicates``)
with SQLAlchemy core constructs (``sa.delete``, ``sa.select``,
``sa.func``, ``.subquery()``, ``.notin_()``).
A small ``_table_clause()`` helper builds a lightweight ``TableClause``
exposing the columns the queries reference; the three helpers consume
it. Removes all ``# noqa: S608`` comments — they are no longer needed
because there is no string-interpolated SQL.
Verified the compiled SQL is identical on Postgres, MySQL, and SQLite,
including the MySQL ERROR 1093 workaround (the inner aggregation is
wrapped in a derived table via ``.subquery()``, producing
``... NOT IN (SELECT keep_id FROM (SELECT min(id) ...) AS keep_min)``).
Also drops the redundant ``f`` prefix on the two non-interpolating
lines of the ``_check_no_external_fks_to_id`` error message.
44 migration tests still pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Four operator-experience improvements from the second review pass:
1. ``TABLES_WITH_NULLABLE_FKS`` is now explicitly documented as an
informational set that is not consulted at runtime; the comment
explains the previous ``dashboard_roles`` omission was the bug
that motivated the always-run cleanup.
2. ``_delete_null_fk_rows`` docstring updated to match the
"always run" semantics (was still claiming "called only on tables
in TABLES_WITH_NULLABLE_FKS").
3. ``_check_no_external_fks_to_id`` now documents its scope
limitation: ``Inspector.get_table_names()`` returns the default
schema only, so cross-schema FKs in non-standard multi-schema
PostgreSQL deployments would not be caught. The single-schema
case (Superset's documented deployment) is fully covered.
4. ``_dedupe_by_min_id`` now logs a sample of up to 10 discarded
``(fk1, fk2, id)`` tuples at WARN before deletion, so operators
can audit which rows the ``MIN(id)`` policy drops. The keep-
original policy is correct in practice but discards later
re-grants on ownership tables; the sample makes that visible.
5. ``UPDATING.md`` documents the upgrade/downgrade primary-key
name divergence (``pk_<table>`` vs ``<table>_pkey``) so
operators using schema-comparison tools don't mistake it for
migration drift.
No schema or runtime-behaviour changes. All 44 migration tests pass.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two cleanups from PR review:
1. ``dashboard_roles.dashboard_id`` was created nullable in revision
e11ccdd12658 but was missing from ``TABLES_WITH_NULLABLE_FKS``. A
production database with a stray NULL ``dashboard_id`` row would have
failed the PK-add with a cryptic constraint violation. Fix by running
the NULL-FK cleanup on every affected table — it is a no-op DELETE on
tables whose FK columns are already NOT NULL, and it eliminates the
risk of further drift in the hardcoded set. ``dashboard_roles`` is
added to the documentation set; the runtime now does not consult it.
2. The unit-test parent-table name for ``rls_filter_roles`` and
``rls_filter_tables`` was ``rls_filter`` (does not exist) instead of
the real parent ``row_level_security_filters``. Test passes either
way (the in-memory FK is self-consistent), but the parameter is now
accurate.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Replace synthetic id INTEGER PRIMARY KEY with composite PRIMARY KEY (fk1, fk2)
on the eight pure-junction tables: dashboard_roles, dashboard_slices,
dashboard_user, report_schedule_user, rls_filter_roles, rls_filter_tables,
slice_user, sqlatable_user. The redundant UNIQUE(fk1, fk2) on dashboard_slices
and report_schedule_user is dropped (subsumed by the new PK).
Migration handles dialect quirks: copy_from for tables with pre-existing
UNIQUE (so SQLite's anonymous-constraint reflection doesn't matter), wrapped-
subquery dedupe for MySQL (ERROR 1093), sa.Identity(always=False) on downgrade
to backfill the restored id column without NOT NULL violations, and distinct
PK names per direction (pk_<table> on upgrade, <table>_pkey on downgrade) to
avoid round-trip index-name collisions on Postgres.
ORM Table() definitions updated to match. UPDATING.md entry added with
operator runbook (BI-tool impact, pre-flight inventory queries, dedupe-row-
loss notice, pg_dump workaround, FK-NOT-NULL downgrade asymmetry note).
Tests: 8 schema-shape assertions (post-upgrade), 8 duplicate-rejection unit
tests, 8 distinct-pair sanity tests, 1 round-trip + idempotency test
(in-memory SQLite via Alembic MigrationContext).
Continuum-restore verification against the new shape is out of scope for this
PR; it is the responsibility of the versioning epic (sc-103156).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
