Notes the breaking dependency change for downstream consumers: custom
db_engine_specs or extensions that touch SQLAlchemy internals directly
should check the 1.4->2.0 migration guide, and the optional connector
extras still capped below their own SQLAlchemy-2.0-only releases
(either pending #42891 or blocked entirely on upstream) keep pulling
1.4-line dialect versions until their own caps move.
session.query(Model).get(id) at superset/commands/sql_lab/estimate.py
was migrated to db.session.get(Database, id) in bc56ecbf15, but the
integration-test mocks in tests/integration_tests/sql_lab/ still
patched the old session.query().get() chain shape, so
QueryEstimationCommand picked up an auto-generated MagicMock instead
of the fixture and failed test-mysql/postgres/sqlite. Same mock-shape
fix already applied to the unit-test sibling in the same commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
session.query(Model).get(id) is deprecated under SQLAlchemy 2.0 in
favor of session.get(Model, id). Migrate the 9 remaining call sites
(flagged by review) and update the corresponding test mocks that
asserted against the old session.query(...).get(...) call shape.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
`generate_dashboard()` re-fetches with eager-loaded relationships via
`subqueryload(Dashboard.slices).subqueryload(Slice.editors)` (and
similarly for `.tags`, `Dashboard.editors`, `Dashboard.tags`), using a
`Dashboard` class resolved through a deferred `from
superset.models.dashboard import Dashboard` import inside the function --
deferred specifically so `@patch("superset.models.dashboard.Dashboard")`
in tests can substitute it.
`test_generate_dashboard_refetches_via_dao` and
`test_generate_dashboard_restricted_user_redacts_chart_datasource_name`
both patch the whole `Dashboard` class to control its constructor
(`mock_dashboard_cls.return_value = dashboard`), which leaves
`Dashboard.slices`/`.editors`/`.tags` as unconfigured auto-mocks. Passing
those to `subqueryload()` now raises `sqlalchemy.exc.ArgumentError:
Wildcard token cannot be followed by another entity` -- `ArgumentError`
subclasses `SQLAlchemyError`, which `generate_dashboard`'s re-fetch
already catches and treats as "re-fetch failed, return minimal
response", so both tests silently got the fallback response instead of
the real one they were asserting against.
Confirmed via a side-by-side instrumented run against SQLAlchemy 1.4.54:
the exact same mock chain there raised no error at all -- 1.4 didn't
eagerly validate `subqueryload()`'s argument type, so an unconfigured
MagicMock silently worked. This is a genuine SQLAlchemy 2.0 strictness
change in loader-option construction, but it doesn't affect real
production code (where `Dashboard` is never mocked and `.slices` is
always a real `InstrumentedAttribute`) -- it only breaks these two
tests' mocks, which were never fully specified to begin with.
Capture the real `Dashboard` class in a module-level `_RealDashboard`
import (before any per-test `@patch` can shadow it) and copy its
`slices`/`editors`/`tags` relationship attributes onto the mocked class
in `_setup_generate_dashboard_mocks`, so `subqueryload()` sees genuine
mapped attributes while `Dashboard(...)` construction stays mocked. Ran
the full 64-test file afterward, including the sibling
`..._refetch_sqlalchemy_error_rollback` test (which forces a real
`SQLAlchemyError` via `find_by_id.side_effect` and still correctly hits
the fallback path) -- no regressions.
test_get_sql_results_oauth2's `query = mocker.MagicMock(select_as_cta=False,
database=database)` left `limit` and `select_as_cta_used` as unconfigured
auto-mocks. `execute_sql_statements` unconditionally calls `apply_limit`
before ever reaching the mocked OAuth2 error, and `apply_limit` does
`query.limit > sql_max_row` once `SQLLAB_CTAS_NO_LIMIT` is false -- an
unconfigured MagicMock has no real `__gt__`, so this raised `TypeError:
'>' not supported between instances of 'MagicMock' and 'int'` instead of
ever reaching the OAuth2 path.
Root-caused with a side-by-side instrumented run against master pinned to
SQLAlchemy 1.4.54 + Flask-SQLAlchemy 2.5.1 (byte-identical sql_lab.py and
test file): there, `flask.current_app` inside `apply_limit` resolved to
an unrelated, already-initialized `superset.app` singleton (config
SQLLAB_CTAS_NO_LIMIT=True) instead of this test's own fixture app
(SQLLAB_CTAS_NO_LIMIT=False), so `apply_limit` took its
`select_as_cta_used and sqllab_ctas_no_limit` early-return path before
ever touching `query.limit` -- masking the gap by accident. Under
Flask-SQLAlchemy 3.x, `current_app` correctly resolves to the test's own
pushed app context, so the early return no longer fires and the
underspecified mock's real gap is exposed. This is not a SQLAlchemy 2.0
semantic break in application code; it's an under-specified test double
that only ever worked because of unrelated cross-test app-identity
bleed in the old environment.
Set `limit=None` and `select_as_cta_used=False` on the mock, matching
the real `Query` model's actual column defaults (`Column(Integer)` is
nullable/None; `Column(Boolean, default=False)`), so `apply_limit`
behaves the same way it would for a genuine freshly-queried `Query` row.
With the mock now correctly reaching the OAuth2 path inside the right
app context, the generated redirect URL correctly reflects this test's
own `SERVER_NAME=example.com` fixture config rather than Flask's
"localhost" default that leaked in from the wrong app previously --
update the two hardcoded `http://localhost/...` expectations to
`http://example.com/...` to match.
SQLAlchemy 2.0's declarative mapper now orders a model's mapped columns
with the class's own Column attributes first, followed by mixin-provided
columns (AuditMixinNullable's created_on/changed_on/created_by_fk/
changed_by_fk and the uuid mixin column) in mixin-declaration order,
appended at the end rather than interleaved at the front. This shifted
where `dbs.uuid`, `dbs.created_on`, `dbs.changed_on` (and the matching
`ssh_tunnels_1.*` aliases from the joined SSHTunnel) land in the compiled
SELECT list. SQLAlchemy 2.0 also renders the concatenation expression on
the left of an IN clause with an explicit parenthesized group.
Verified the actual compiled query has the identical column set (just
reordered) and an equivalent WHERE clause (same expression, extra
parens), then updated the hardcoded expected string in
test_database_filter to match.
shillelagh.exceptions.ProgrammingError is wrapped by SQLAlchemy's own
StatementError machinery, which appends a "Background on this error at:
https://sqlalche.me/e/<major><minor>/<code>" hint derived from the
running SQLAlchemy version. Under 1.4 that was .../e/14/f405; under 2.0
it's .../e/20/f405. test_dml and test_allowed_dbs hardcoded the 1.4 URL
in their expected exception strings; update both to the 2.0 path.
SQLAlchemy 2.0 changed the default poolclass for file-based SQLite engines
from NullPool to QueuePool. Under 1.4, every checkout created a fresh
low-level sqlite3 connection, so a pooled connection was never handed to a
thread other than the one that opened it. QueuePool reuses connections
across checkouts, including checkouts from background threads such as the
GTF task framework's deferred-flush timer (superset/tasks/context.py).
Combined with the test suite's `?check_same_thread=true` SQLite URIs, a
connection opened on the main thread can now be handed to a background
timer thread, which pysqlite rejects with "SQLite objects created in a
thread can only be used in that same thread." This silently broke the
deferred DB write in TaskContext._deferred_flush, intermittently failing
tests/integration_tests/tasks/test_throttling.py::test_throttle_behavior
under full-suite load (reproduced deterministically across two full
sqlite runs, passed in isolation, confirmed via QueuePool + NullPool
introspection against the pinned 1.4.54 and 2.0.51 wheels).
Pin poolclass=NullPool for the sqlite test lane to restore the 1.4
behavior. Superset's non-test default config uses `check_same_thread=false`
instead, which is unaffected by pooled cross-thread connection reuse, so
this is scoped to the test config rather than superset/config.py.
Root cause of test_get_invalid_table_table_metadata's 422-instead-of-200
on sqlite: SQLAlchemy 2.0's sqlite dialect raises NoSuchTableError from
reflection (get_columns/get_pk_constraint/get_indexes/get_foreign_keys/
get_table_comment) for a table that doesn't exist. SQLAlchemy 1.4's
sqlite dialect silently returned empty results instead - the API has
always relied on that specifically for sqlite to answer with an
empty-but-200 payload, which the test codifies explicitly (mysql and
other backends already expect 422 for the same request, since their
dialects already raised on missing tables pre-2.0 too - this is a
sqlite-only regression).
get_table_metadata() now catches NoSuchTableError, and only for
sqlite falls back to the same empty-metadata shape the old dialect
produced; every other backend re-raises unchanged, preserving the
existing 422 behavior there. Also had to stop select_star() from
re-triggering the same NoSuchTableError via its own internal
database.get_columns() re-fetch (it re-fetches whenever `cols` is
empty and either show_cols or latest_partition is requested) by
passing latest_partition=False in the missing-table case - a missing
table has no partitions to look up anyway.
Verified locally (sqlite): test_get_invalid_table_table_metadata
passes, and the full tests/integration_tests/databases/ suite (157
passed, 10 skipped, the 1 unrelated pre-existing count-mismatch
failure was local DB-file reuse pollution from earlier ad-hoc runs,
confirmed gone on a fresh DB) shows no regression.
Root cause of TestDashboardDAO.test_copy_dashboard_copies_native_filters's
"Instance '<Dashboard>' is not persisted" InvalidRequestError:
copy_dashboard() builds the new Dashboard, calls db.session.add(dash),
and returns it without ever flushing. Session.delete() on an object
that was add()-ed but never flushed (state.key is still None) has
always been invalid in SQLAlchemy - this isn't a 1.4->2.0 behavior
change - but it was silently masked whenever the caller happened to
run any other query afterward, since Session's default autoflush
flushes all pending objects (including this one) before executing that
query.
The sibling test, test_copy_dashboard_duplicate_slices, passes only by
accident: it queries db.session.query(Subject) right after
copy_dashboard() returns (to look up the admin's Subject row for an
assertion), which autoflushes `dash` as a side effect and gives it a
real id before its own db.session.delete(dash) cleanup runs.
test_copy_dashboard_copies_native_filters does no such incidental
query - it asserts on dash.params_dict (pure Python, no DB access) and
goes straight to db.session.delete(dash), so `dash` is still pending
and delete() rejects it outright.
Flushing explicitly at the end of copy_dashboard() (the DAO doesn't
commit - that's the @transaction-decorated command layer's job, per
superset/commands/dashboard/copy.py) makes the contract reliable:
every caller gets back a Dashboard with a real, persisted id, not one
that only works if they happen to touch the DB again afterward.
Verified locally (sqlite): tests/integration_tests/dashboards/dao_tests.py
(5/5) passes, including both copy_dashboard tests.
SQLAlchemy 2.0 removed the subtransactions= parameter from
Session.begin(); TestDatasource.setUp() called it unconditionally,
failing every test in the class with "TypeError: scoped_session.begin()
got an unexpected keyword argument 'subtransactions'" before the test
body ever ran.
Same fix as apache/superset#42866 (open at the time of this commit,
not yet merged to master despite this branch already having merged
master's tip) - applying it directly here since it's blocking all
further local verification of tests/integration_tests/datasource_tests.py
and is independent of the rest of this branch's SQLAlchemy 2.0 work.
Once #42866 merges to master, a future master-merge into this branch
will no-op on this file.
Verified locally (sqlite): all 31 tests in datasource_tests.py pass.
master currently has two unreconciled migration heads (4f145192b583,
the pivot-table-percent-display/report-retry merge, and c4a1b8e2d739,
the Databend secure->sslmode migration) - unrelated to this branch's
SQLAlchemy 2.0 work, just two migrations that landed around the same
time without a merge revision joining them. A fresh `superset db
upgrade` fails outright with "Multiple head revisions are present"
without this, which blocked local testing of the SQLAlchemy 2.0 fixes
on this branch after merging master's tip. Standard trivial Alembic
merge migration (empty upgrade/downgrade), generated via
`superset db merge heads`.
Root cause of the where_latest_partition CompileError across hive/
presto/trino tests ("Could not render literal value '2023-05-01' with
datatype TIMESTAMP"): superset/models/sql_types/presto_sql_types.py's
TimeStamp/Date TypeDecorator subclasses only override process_bind_param,
which returns the *final* literal SQL text ("TIMESTAMP '2023-05-01'") -
Presto/Trino don't support parameter binding for these types, so
process_bind_param has always done double duty as the literal-rendering
hook too.
TypeDecorator.literal_processor()'s standard composition, when only
process_bind_param is overridden (no process_literal_param), pipes its
output through the *impl* type's own literal_processor - i.e. it takes
the string "TIMESTAMP '2023-05-01'" and still runs it through the real
TIMESTAMP type's literal processor, which expects an actual datetime
and calls .isoformat() on it. That blows up with a CompileError. This
composition happens for the process_bind_param fallback path
regardless of whether SQLAlchemy 1.4 or 2.0 - confirmed empirically
that overriding process_literal_param instead doesn't help either,
since TypeDecorator's own literal_processor() chains *both* paths
through the impl processor when the impl has one.
Fixed by overriding literal_processor() directly on both classes -
against TypeDecorator's own docstring advice ("should not implement
this method"), but correct here since process_bind_param already
returns final SQL text rather than a value for further impl
processing to convert.
Separately, hive_tests.py/presto_tests.py hardcoded the expected
compiled SQL for an empty-column select() with two spaces before the
newline ("SELECT \n"); SQLAlchemy 2.0 renders it with one ("SELECT
\n") - a cosmetic Core-compiler change, fixed the expected strings to
match.
Verified locally (sqlite): tests/integration_tests/db_engine_specs/
and tests/unit_tests/db_engine_specs/ where_latest_partition tests
(hive, presto, trino, both integration and unit) all pass.
Critical regression: SQLAlchemy 2.0 changed URL.__str__() to always
substitute "***" for the password rather than rendering it verbatim
(SQLAlchemy 1.4's str(URL) rendered the real value). Every
build_sqlalchemy_uri() implementation across the db_engine_specs
(base/Postgres+MySQL+etc, ClickHouse, Databricks x2, Snowflake,
Databend, Couchbase) built a URL with the user's real password and
returned str(url) - which is exactly the string
superset/databases/schemas.py's pre-load hook writes into
data["sqlalchemy_uri"] when a database is created or edited via the
parameterized connection form. Under SQLAlchemy 2.0 that stores the
literal password "***" instead of the real one, breaking every new
connection made that way. Switched all of these to
render_as_string(hide_password=False), which is the 2.0-native way to
get the real, unmasked URL string.
Two more instances of the same str(URL) regression in
superset/models/core.py, both with real functional impact:
- Database.set_sqlalchemy_uri() intentionally replaces the real
password with Superset's own PASSWORD_MASK sentinel
("X" * 10, not a secret) before storing self.sqlalchemy_uri, so a
later edit can compare conn.password != PASSWORD_MASK to detect
"the user didn't touch the password field, keep the existing one."
str(conn) under 2.0 was substituting its own "***" for that
sentinel, so the stored URI no longer round-tripped to
PASSWORD_MASK - it round-tripped to the meaningless literal "***",
breaking password-preservation on every database edit.
- The per-process SQLAlchemy engine cache (superset/models/core.py,
_ENGINE_CACHE) keys on str(sqlalchemy_url) specifically so that a
password rotation naturally invalidates the cached engine (the
module comment states this explicitly). Under 2.0, str(url) always
masks to the same "***" regardless of the real password, so
rotating a database's password would silently keep reusing the old,
now-wrong cached engine/connection pool for the life of the worker
process.
Also fixed a separate, unrelated 1.4->2.0 break in
superset/db_engine_specs/duckdb.py: two build_sqlalchemy_uri variants
called the raw URL(...) constructor, which SQLAlchemy 2.0 turned into
a strict NamedTuple requiring username/password/host/port to be
passed explicitly (they used to default to None). That raised
"URL.__new__() missing 4 required positional arguments" outright.
Switched both to URL.create(), which keeps those optional.
Test-side: model_tests.py/db_engine_specs test files that asserted
str(uri) == "<scheme>://user:realpassword@host/..." were relying on
the old unmasked str() behavior; switched them to
uri.render_as_string(hide_password=False) to keep verifying the real
underlying value rather than relaxing what's being checked.
Verified locally (sqlite): tests/integration_tests/db_engine_specs/
and tests/unit_tests/db_engine_specs/ - the password-masking and
duckdb URL() failures are gone (18 -> 12 remaining, unrelated:
mysqlclient not importable on this Mac, 5 bigquery test_fetch_data
failures, and a where_latest_partition literal-rendering cluster
across hive/presto/trino, tracked separately).
Same removed-cascade_backrefs pattern as the ReportSchedule test
fixtures (see the "add missing db.session.add() in report-schedule
test fixtures" commit), found in real production code this time: the
legacy Explore "save as" -> "new dashboard" flow
(superset/views/core.py, the new_dashboard_name branch of
SliceAddView.save_or_overwrite_slice) constructs a brand-new, transient
Dashboard, then does `dash.slices.append(slc)` where `slc` is already
persistent. Appending a persistent Slice into a transient Dashboard's
`slices` collection also populates the reverse `Slice.dashboards`
backref - under SQLAlchemy 1.4 that implicitly cascaded the new
Dashboard into the session; under 2.0 (cascade_backrefs removed) it no
longer does, so `db.session.commit()` silently persisted nothing and
the "new dashboard" the user asked for was never created.
Confirmed directly: constructing a transient Dashboard, appending a
persistent Slice to its `.slices`, and committing left `dash.id` as
None (and logged
"SAWarning: Object of type <Dashboard> not in session, add operation
along 'Slice.dashboards' won't proceed" from Superset's own versioning
listener's flush). Adding `db.session.add(dash)` before the append
fixes it - verified the same probe then leaves `dash.id` populated
after commit.
No existing test exercises this specific new_dashboard_name path, so
verified via a standalone repro script against this branch's real
sqlalchemy==2.0.51/flask-sqlalchemy==3.1.1 rather than a test
assertion; ran tests/integration_tests/core_tests.py in full (47
passed, 2 skipped) to confirm no regression to the adjacent saveas/
overwrite paths that do have coverage.
Root cause of the "mypass%25123" != "mypass%123" and "p%40ss%21word"
!= "p@ss!word" round-trip failures in test_database_password_encoding.py:
Database.sqlalchemy_uri_decrypted manually percent-encoded the password
with urllib.parse.quote() before handing it to
URL.render_as_string(hide_password=False). Under SQLAlchemy 1.4,
render_as_string() rendered URL.password as a literal value, so the
manual pre-encoding was necessary. Under SQLAlchemy 2.0,
render_as_string() always percent-encodes the password itself - the
pre-encoded value gets encoded a second time (a literal "%" becomes
"%25", which decodes back to "%25" instead of "%" on the next parse).
Passing the raw password straight through and letting
render_as_string() do the (now single) encoding fixes the round-trip.
Confirmed the double-encoding mechanism directly against this branch's
sqlalchemy==2.0.51: URL.set(password=<pre-encoded>).render_as_string()
produces "mypass%2525123"; URL.set(password=<raw>).render_as_string()
correctly produces "mypass%25123".
Separately, model_tests.py::test_impersonate_user_trino (and the
mysqlclient-only test_adjust_engine_params_mysql, exercised on real CI
runners but skipped here where mysqlclient isn't importable) asserted
on str(url) for URLs containing a password. SQLAlchemy 2.0 also changed
URL.__str__() to hide the password by default (a deliberate hardening
change - 1.4 rendered it in full); switched those assertions to
render_as_string(hide_password=False) to compare against the real,
unmasked URL the engine was actually constructed with, rather than
relaxing what's being verified.
Verified locally (sqlite): test_database_password_encoding.py (5/5)
and model_tests.py (21 passed, 5 skipped - the mysqlclient-gated ones)
both pass clean.
Root cause: Flask-SQLAlchemy 2.x's SignallingSession.__init__ explicitly
passed `bind=<default engine>` into the SQLAlchemy Session constructor.
Flask-SQLAlchemy 3.x dropped that - it resolves the engine per-call via
Session.get_bind() (consulting the app's registered engines) instead of
fixing one at session-construction time, to support its reworked
multi-engine/multi-bind model. db.session.bind is therefore always None
under FSA 3.x; it was never a documented API, just an implementation
detail of FSA 2.x's Session subclass.
Confirmed locally: `db.session.bind` prints None while
`db.session.get_bind()` correctly resolves the live Engine, against
this branch's real flask-sqlalchemy==3.1.1/sqlalchemy==2.0.51 install.
Fixes the "AttributeError: 'NoneType' object has no attribute 'dialect'"
failures in dashboards/soft_delete_tests.py's partial-index dialect
detection, and pre-emptively fixes the same pattern in
versioning/id_reuse_tests.py (skipped under postgres/current in the CI
run that surfaced this, so not in the failure log there, but broken by
the same mechanism whenever it does run).
Verified locally against both sqlite and a real postgres:17-alpine
container: dashboards/soft_delete_tests.py and
versioning/id_reuse_tests.py pass clean on both backends after this fix.
Root cause of the "type filter_type_enum does not exist" postgres
failures across security/row_level_security_tests.py: no migration has
ever created a native Postgres enum type named filter_type_enum. The
2020-09-15 e5ef6828ac4e migration that added this column only ever
created a plain VARCHAR(255) - confirmed against a fresh `superset db
upgrade` on postgres:17-alpine (`\d row_level_security_filters` shows
filter_type as character varying, and `pg_type` has no
filter_type_enum row). The ORM model's `Enum(..., name="filter_type_enum")`
declaration was already mismatched with the real schema; that mismatch
was harmless under SQLAlchemy 1.4.
It stopped being harmless under SQLAlchemy 2.0: the new postgresql
"insertmanyvalues" execution strategy renders every bound parameter
with an explicit cast to its column type's DDL name, even for a
single-row INSERT via ORM flush (e.g. `SELECT p0::VARCHAR, p1::TEXT,
p2::filter_type_enum, ... ORDER BY sen_counter RETURNING ...`). Casting
to a type name that was never actually created fails outright with
psycopg2.errors.UndefinedObject, and poisons the session for the rest
of that transaction (surfacing as PendingRollbackError on every
subsequent statement in the same test).
native_enum=False makes SQLAlchemy render this column as a plain
VARCHAR with a CHECK constraint instead of a named Postgres enum type,
which is what the physical schema has always actually been.
Verified locally against a real postgres:17-alpine container running
this branch's exact sqlalchemy==2.0.51/flask-sqlalchemy==3.1.1: before
this fix, `test_model_view_rls_add_success` and friends failed with
the exact UndefinedObject/PendingRollbackError chain seen in CI; after
it, all 47 tests in security/row_level_security_tests.py pass.
Root cause of the ReportSchedule "is not persisted" InvalidRequestError
and the cascaded StaleDataError/PendingRollbackError fan-out seen across
charts/api_tests.py and dashboards/api_tests.py in CI: SQLAlchemy 2.0
removes the legacy cascade_backrefs behavior entirely. Under 1.4,
constructing `ReportSchedule(chart=chart)` where `chart` was already a
persistent, session-tracked object implicitly added the new
ReportSchedule to the session too, via the Slice.report_schedules
backref collection. That auto-cascade is gone in 2.0 (this is a
documented, intentional SQLAlchemy 1.4->2.0 removal, not a bug) - an
explicit db.session.add() is now required.
Without it, `create_chart_with_report`/`create_dashboard_with_report`
committed a ReportSchedule that was silently never persisted. The
actual test bodies then found no report attached (delete succeeded
with 200 instead of the expected 422 block), and fixture teardown's
`db.session.delete(report_schedule)` blew up with "Instance ... is not
persisted" since the object was transient all along. The dashboard
variant's cascading StaleDataError on dashboard_slices was a downstream
symptom of the same root cause (the "blocked" dashboard delete actually
went through).
Other ReportSchedule(...) construction sites in the test suite already
call db.session.add() explicitly (deletion_retention/*, charts/
soft_delete_tests.py, databases/api_tests.py, reports/utils.py) or
never persist the object at all (reports/alert_tests.py,
reports/commands_tests.py) - these two fixtures were the only ones
missing it. Production code is unaffected: BaseDAO.create() already
calls db.session.add() explicitly, so the real create-report code path
never relied on the removed cascade.
Verified locally (real flask-sqlalchemy==3.1.1 + sqlalchemy==2.0.51,
sqlite backend): tests/integration_tests/charts/api_tests.py (96/96)
and tests/integration_tests/dashboards/api_tests.py (156/156) both
pass clean after this fix, including the two previously-failing
report-block tests and the dashboard_slices StaleDataError case.
Root cause of the embedded-dashboard/task-framework test failures:
Flask-SQLAlchemy 3.x scopes db.session by the identity of the active
Flask app-context object (id(app_ctx)) instead of by thread/greenlet
identity like 2.x. Superset's codebase and test fixtures widely (and
often only implicitly, without an explicit commit()) assume a single
shared session per thread across nested app.app_context() blocks -
true under 2.x, false under 3.x once any code pushes a second context
on the same thread (e.g. the AppContextTask Celery wrapper below, or
test fixtures that each do their own `with app.app_context():`).
Verified locally (real flask-sqlalchemy==3.1.1 + sqlalchemy==2.0.51,
sqlite backend) against the exact tests failing in CI: restoring the
2.x scopefunc fixes tests/integration_tests/embedded/test_view.py
(the original symptom from the first #42542 break), tasks/test_event_
handlers.py, tasks/test_timeout.py, and one of two tasks/test_
throttling.py failures.
Also guard the Celery AppContextTask wrapper's app_context() push
with has_app_context(), so eager-mode task execution (e.g. .apply()
called from an existing request/test context) reuses the caller's
session instead of unconditionally splitting into a second, blind
one - the most impactful single instance of the pattern, since every
task run goes through it.
Two failures remain locally, both pre-existing and unrelated to this
scoping issue (confirmed present in CI before this fix too):
dashboards/soft_delete_tests.py's db.session.bind is None under FSA
3.x/SQLA 2.0, and a sqlite-specific check_same_thread cross-thread
connection reuse in test_throttling.py's timer-thread path.
Flask-SQLAlchemy 3.x scopes db.session by Flask app-context object
identity (id(app_ctx._get_current_object())) instead of thread
identity like 2.x. Pushing a redundant nested app_context() on a
thread that already has one active (as these task-manager call
sites did unconditionally) silently splits work across two distinct
Session objects under 3.x, where 2.x transparently shared one.
Verified locally against real flask-sqlalchemy==3.1.1 vs 2.5.1
installs: nested app_context() on one thread yields the same
Session under 2.x, different Session objects under 3.x.
sqlalchemy.orm.load_only() requires real ORM-mapped attribute objects
in SQLAlchemy 2.0 (ArgumentError: expected ORM mapped attribute for
loader strategy argument) -- passing "id"/"uuid" as plain strings, as
these two dynamically-generated import-mixin migrations did, no longer
works. Both call sites already have the actual dynamically-built model
class in scope (models["slices"]), so this just references its real
.id/.uuid attributes instead of their string names.
Three 2020-era migration scripts constructed MetaData(bind=bind), a
kwarg removed outright in SQLAlchemy 2.0 (TypeError: MetaData.__init__()
got an unexpected keyword argument 'bind'). Every actual usage already
passes autoload_with=bind per-table, so the MetaData-level bind was
redundant even before 2.0; dropped it.
Also fixes row["key"] string-indexed access on a raw Core Row result
in the same migration family -- SQLAlchemy 2.0's Row only supports
positional/attribute access directly, string-key lookups need
row._mapping["key"]. Verified both failure modes directly against a
real sqlalchemy==2.0.51 install before making this change.
Investigation-only, cherry-picked from mikebridge's
test/verify-sqlalchemy-continuum branch (backend-relevant hunks only,
skipping unrelated frontend formatting drift). Bumps sqlalchemy to
2.0.51 and flask-sqlalchemy to 3.1.1 directly, rather than the FSA
3.0.5 intermediate step attempted in PR #42542.
Notable fix beyond the version bumps: pessimistic_connection_handling
(superset/utils/core.py) now calls connection.rollback() after its
pool-checkout health-check SELECT. Under SQLAlchemy 2.0's autobegin
behavior that SELECT implicitly opens a transaction it previously
never closed, which is a plausible mechanism for the MySQL
lock-wait-timeout symptom from the original #42542 break.
Also updates two migration scripts (2018-07-26 add_implicit_tags,
2022-04-01 new_dataset_models_take_2) to Mapped[] typed relationship
annotations -- these define their own standalone declarative models
and were missed by discussion #40273's earlier unit-test-driven
deprecation-warning sweep, since migration scripts aren't exercised
by that suite.
Removes the now-obsolete SQLALCHEMY_WARN_20 pytest.ini filterwarnings
error lines, since sqlalchemy.exc.RemovedIn20Warning doesn't fire (or
exist in the same form) once SQLAlchemy 2.0 is actually installed.
Pushing to get a real test-sqlite/test-mysql CI signal, since local
sqlite runs don't reliably reproduce the original break either way.
Claude Code