session.query(Model).get(id) at superset/commands/sql_lab/estimate.py
was migrated to db.session.get(Database, id) in bc56ecbf15, but the
integration-test mocks in tests/integration_tests/sql_lab/ still
patched the old session.query().get() chain shape, so
QueryEstimationCommand picked up an auto-generated MagicMock instead
of the fixture and failed test-mysql/postgres/sqlite. Same mock-shape
fix already applied to the unit-test sibling in the same commit.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SQLAlchemy 2.0 changed the default poolclass for file-based SQLite engines
from NullPool to QueuePool. Under 1.4, every checkout created a fresh
low-level sqlite3 connection, so a pooled connection was never handed to a
thread other than the one that opened it. QueuePool reuses connections
across checkouts, including checkouts from background threads such as the
GTF task framework's deferred-flush timer (superset/tasks/context.py).
Combined with the test suite's `?check_same_thread=true` SQLite URIs, a
connection opened on the main thread can now be handed to a background
timer thread, which pysqlite rejects with "SQLite objects created in a
thread can only be used in that same thread." This silently broke the
deferred DB write in TaskContext._deferred_flush, intermittently failing
tests/integration_tests/tasks/test_throttling.py::test_throttle_behavior
under full-suite load (reproduced deterministically across two full
sqlite runs, passed in isolation, confirmed via QueuePool + NullPool
introspection against the pinned 1.4.54 and 2.0.51 wheels).
Pin poolclass=NullPool for the sqlite test lane to restore the 1.4
behavior. Superset's non-test default config uses `check_same_thread=false`
instead, which is unaffected by pooled cross-thread connection reuse, so
this is scoped to the test config rather than superset/config.py.
SQLAlchemy 2.0 removed the subtransactions= parameter from
Session.begin(); TestDatasource.setUp() called it unconditionally,
failing every test in the class with "TypeError: scoped_session.begin()
got an unexpected keyword argument 'subtransactions'" before the test
body ever ran.
Same fix as apache/superset#42866 (open at the time of this commit,
not yet merged to master despite this branch already having merged
master's tip) - applying it directly here since it's blocking all
further local verification of tests/integration_tests/datasource_tests.py
and is independent of the rest of this branch's SQLAlchemy 2.0 work.
Once #42866 merges to master, a future master-merge into this branch
will no-op on this file.
Verified locally (sqlite): all 31 tests in datasource_tests.py pass.
Root cause of the where_latest_partition CompileError across hive/
presto/trino tests ("Could not render literal value '2023-05-01' with
datatype TIMESTAMP"): superset/models/sql_types/presto_sql_types.py's
TimeStamp/Date TypeDecorator subclasses only override process_bind_param,
which returns the *final* literal SQL text ("TIMESTAMP '2023-05-01'") -
Presto/Trino don't support parameter binding for these types, so
process_bind_param has always done double duty as the literal-rendering
hook too.
TypeDecorator.literal_processor()'s standard composition, when only
process_bind_param is overridden (no process_literal_param), pipes its
output through the *impl* type's own literal_processor - i.e. it takes
the string "TIMESTAMP '2023-05-01'" and still runs it through the real
TIMESTAMP type's literal processor, which expects an actual datetime
and calls .isoformat() on it. That blows up with a CompileError. This
composition happens for the process_bind_param fallback path
regardless of whether SQLAlchemy 1.4 or 2.0 - confirmed empirically
that overriding process_literal_param instead doesn't help either,
since TypeDecorator's own literal_processor() chains *both* paths
through the impl processor when the impl has one.
Fixed by overriding literal_processor() directly on both classes -
against TypeDecorator's own docstring advice ("should not implement
this method"), but correct here since process_bind_param already
returns final SQL text rather than a value for further impl
processing to convert.
Separately, hive_tests.py/presto_tests.py hardcoded the expected
compiled SQL for an empty-column select() with two spaces before the
newline ("SELECT \n"); SQLAlchemy 2.0 renders it with one ("SELECT
\n") - a cosmetic Core-compiler change, fixed the expected strings to
match.
Verified locally (sqlite): tests/integration_tests/db_engine_specs/
and tests/unit_tests/db_engine_specs/ where_latest_partition tests
(hive, presto, trino, both integration and unit) all pass.
Root cause of the "mypass%25123" != "mypass%123" and "p%40ss%21word"
!= "p@ss!word" round-trip failures in test_database_password_encoding.py:
Database.sqlalchemy_uri_decrypted manually percent-encoded the password
with urllib.parse.quote() before handing it to
URL.render_as_string(hide_password=False). Under SQLAlchemy 1.4,
render_as_string() rendered URL.password as a literal value, so the
manual pre-encoding was necessary. Under SQLAlchemy 2.0,
render_as_string() always percent-encodes the password itself - the
pre-encoded value gets encoded a second time (a literal "%" becomes
"%25", which decodes back to "%25" instead of "%" on the next parse).
Passing the raw password straight through and letting
render_as_string() do the (now single) encoding fixes the round-trip.
Confirmed the double-encoding mechanism directly against this branch's
sqlalchemy==2.0.51: URL.set(password=<pre-encoded>).render_as_string()
produces "mypass%2525123"; URL.set(password=<raw>).render_as_string()
correctly produces "mypass%25123".
Separately, model_tests.py::test_impersonate_user_trino (and the
mysqlclient-only test_adjust_engine_params_mysql, exercised on real CI
runners but skipped here where mysqlclient isn't importable) asserted
on str(url) for URLs containing a password. SQLAlchemy 2.0 also changed
URL.__str__() to hide the password by default (a deliberate hardening
change - 1.4 rendered it in full); switched those assertions to
render_as_string(hide_password=False) to compare against the real,
unmasked URL the engine was actually constructed with, rather than
relaxing what's being verified.
Verified locally (sqlite): test_database_password_encoding.py (5/5)
and model_tests.py (21 passed, 5 skipped - the mysqlclient-gated ones)
both pass clean.
Root cause: Flask-SQLAlchemy 2.x's SignallingSession.__init__ explicitly
passed `bind=<default engine>` into the SQLAlchemy Session constructor.
Flask-SQLAlchemy 3.x dropped that - it resolves the engine per-call via
Session.get_bind() (consulting the app's registered engines) instead of
fixing one at session-construction time, to support its reworked
multi-engine/multi-bind model. db.session.bind is therefore always None
under FSA 3.x; it was never a documented API, just an implementation
detail of FSA 2.x's Session subclass.
Confirmed locally: `db.session.bind` prints None while
`db.session.get_bind()` correctly resolves the live Engine, against
this branch's real flask-sqlalchemy==3.1.1/sqlalchemy==2.0.51 install.
Fixes the "AttributeError: 'NoneType' object has no attribute 'dialect'"
failures in dashboards/soft_delete_tests.py's partial-index dialect
detection, and pre-emptively fixes the same pattern in
versioning/id_reuse_tests.py (skipped under postgres/current in the CI
run that surfaced this, so not in the failure log there, but broken by
the same mechanism whenever it does run).
Verified locally against both sqlite and a real postgres:17-alpine
container: dashboards/soft_delete_tests.py and
versioning/id_reuse_tests.py pass clean on both backends after this fix.
Root cause of the ReportSchedule "is not persisted" InvalidRequestError
and the cascaded StaleDataError/PendingRollbackError fan-out seen across
charts/api_tests.py and dashboards/api_tests.py in CI: SQLAlchemy 2.0
removes the legacy cascade_backrefs behavior entirely. Under 1.4,
constructing `ReportSchedule(chart=chart)` where `chart` was already a
persistent, session-tracked object implicitly added the new
ReportSchedule to the session too, via the Slice.report_schedules
backref collection. That auto-cascade is gone in 2.0 (this is a
documented, intentional SQLAlchemy 1.4->2.0 removal, not a bug) - an
explicit db.session.add() is now required.
Without it, `create_chart_with_report`/`create_dashboard_with_report`
committed a ReportSchedule that was silently never persisted. The
actual test bodies then found no report attached (delete succeeded
with 200 instead of the expected 422 block), and fixture teardown's
`db.session.delete(report_schedule)` blew up with "Instance ... is not
persisted" since the object was transient all along. The dashboard
variant's cascading StaleDataError on dashboard_slices was a downstream
symptom of the same root cause (the "blocked" dashboard delete actually
went through).
Other ReportSchedule(...) construction sites in the test suite already
call db.session.add() explicitly (deletion_retention/*, charts/
soft_delete_tests.py, databases/api_tests.py, reports/utils.py) or
never persist the object at all (reports/alert_tests.py,
reports/commands_tests.py) - these two fixtures were the only ones
missing it. Production code is unaffected: BaseDAO.create() already
calls db.session.add() explicitly, so the real create-report code path
never relied on the removed cascade.
Verified locally (real flask-sqlalchemy==3.1.1 + sqlalchemy==2.0.51,
sqlite backend): tests/integration_tests/charts/api_tests.py (96/96)
and tests/integration_tests/dashboards/api_tests.py (156/156) both
pass clean after this fix, including the two previously-failing
report-block tests and the dashboard_slices StaleDataError case.
