Compare commits

...
Author SHA1 Message Date
Superset Dev f0f55fbed3 fix(testcontainers): also monkeypatch oceanbase_py's get_columns
Fixing has_table() alone wasn't enough: get_columns() -- the actual
column-introspection call OceanBaseEngineSpec.get_columns() needs, and
what this test's second half exercises -- has the identical
connection.execute(raw string) bug, confirmed on real CI as the same
ObjectNotExecutableError. Patched the same way, reusing has_table()
correctly since self.has_table already resolves to the earlier patch.
2026-08-28 19:46:21 -07:00
Superset Dev b4c57ae9a5 fix(testcontainers): catch the not-found error in the has_table monkeypatch
Confirmed on real CI: even with exec_driver_sql fixing the SQLAlchemy 2.0
incompatibility, has_table() still failed -- DESCRIBE on a nonexistent
table raises 1146 ("table doesn't exist") rather than returning an empty
result set, and the original method never catches that at all. It can
only ever return True; checking for a table that doesn't exist (exactly
what create_all()'s checkfirst does first) raised instead of returning
False. Catches the error and returns False, same as any other dialect's
has_table() would.
2026-08-28 19:33:55 -07:00
Superset Dev 4589a78e52 fix(testcontainers): monkeypatch oceanbase_py's has_table instead of skipping checkfirst
checkfirst=False only sidestepped create_all()'s own call to has_table();
Inspector.get_columns() (used by OceanBaseEngineSpec.get_columns(), which
the second test needs to actually exercise) calls the same broken
has_table() internally and hit the identical ObjectNotExecutableError
(confirmed on real CI). Every other raw-SQL method in the same dialect
module correctly uses connection.exec_driver_sql(...) -- this looks like
an isolated oversight in just has_table(), not a deliberate design
choice, so this monkeypatches it to do the same thing the rest of the
dialect already does, fixing the root cause for both call sites instead
of routing around one of them.
2026-08-28 19:16:33 -07:00
Superset Dev c027d8ca6e fix(testcontainers): skip create_all's checkfirst for oceanbase
oceanbase_py's has_table() -- called by create_all()'s default
checkfirst=True before creating each table -- passes a raw string
straight to Connection.execute(), which SQLAlchemy 2.0 rejects outright
(ObjectNotExecutableError, confirmed on real CI). Added an optional
checkfirst override to the shared _pagination.py helper and used it here;
safe to skip the existence check since each test gets a genuinely fresh
container.
2026-08-28 19:03:23 -07:00
Superset Dev e053b6d92d fix(testcontainers): fix oceanbase driver install, drop vertica
The oceanbase driver install step reported success but never actually
installed oceanbase_py: --no-deps applied to `-e .[oceanbase]` blocks pip
from installing anything the extras marker pulls in, including
oceanbase_py itself, not just its conflicting transitive dependency.
Installing oceanbase_py as its own standalone package instead means
--no-deps only skips *its* dependencies, which is what was actually
intended. Confirmed via a manual workflow_dispatch run (nightly_only
dialects don't run on pull_request, so this needed a manual trigger to
catch at all).

Vertica dropped from this PR: the same workflow_dispatch run found
`vertica/vertica-ce` doesn't exist on Docker Hub. The only actively
maintained official image (`opentext/vertica-k8s`) is built to run under
the Vertica Kubernetes operator's orchestration, not as a standalone
single-container database -- a bare `docker run` likely won't bootstrap
a working instance on its own. Needs real investigation before it's
worth another attempt, same as Solr/IoTDB/TDengine/Parseable/Dremio
earlier in this series.
2026-08-28 18:54:06 -07:00
Superset Dev 65532b6bdd fix(testcontainers): dispose the firebird engine before the container tears down
Both tests passed on CI, but the job still failed: firebird-driver
registers its own atexit handler that talks to the Firebird subsystem to
shut it down cleanly, and it fired after the container was already gone,
segfaulting (exit code 139) rather than raising a catchable error.
Disposing the engine while the server is still up lets the driver close
out normally, so the later atexit call has nothing left to talk to.
2026-08-28 18:42:23 -07:00
Superset Dev 05e8d24d9d feat(ci): expand testcontainers coverage to databend, risingwave, firebird, ydb, oceanbase, vertica
Stacked on feat/testcontainers-nightly-only-gating. All six extras already
existed in pyproject.toml. oceanbase and vertica run nightly_only: true
(heavy first-boot and a ~12GB RAM floor, respectively), so they don't run
per-PR; databend/risingwave/firebird/ydb run on every PR like the rest of
this suite.

- oceanbase_py pins sqlalchemy-utils<0.39, which conflicts outright with
  Superset's own sqlalchemy-utils==0.42.1 pin -- kept out of the baseline
  dev install (same reason as db2's ibm-db-sa) and installed on demand,
  --no-deps, only for its own CI leg (it never actually imports
  sqlalchemy_utils itself, so the version mismatch is inert at runtime).
- databend: connects to the local standalone image's builtin `root` user
  (no password) with sslmode=disable, since Superset's default
  encryption_parameters assume TLS the local image doesn't have.
- risingwave: RisingWave's storage engine checkpoints asynchronously --
  a SELECT immediately after INSERT can see zero rows without an explicit
  FLUSH (confirmed on a real instance). Uses the shared _pagination.py
  helper's after_insert hook (originally added for CrateDB) to do that.
- firebird: sqlalchemy-firebird's driver is a pure-Python ctypes wrapper
  (py3-none-any wheel, confirmed by downloading it directly) that
  dynamically loads the native libfbclient from the host rather than
  bundling it -- CI installs that system package on demand. Also confirms
  in the test docstring that FirebirdEngineSpec's `limit_method =
  LimitMethod.FETCH_MANY` (comment: "uses FIRST to limit") is stale
  against the modern driver, which compiles real ROWS-based pagination.
- ydb: needed three real fixes to make a generic DockerContainer usable
  at all. (1) YDB's gRPC client does endpoint discovery and reconnects to
  whatever the server reports, which by default is the container's own
  internal Docker hostname -- fixed by binding the same port on the host
  as inside the container and advertising "localhost" as the container's
  own hostname, so the discovered endpoint is actually reachable. (2) The
  gRPC port opens before storage pools are fully initialized, so an early
  CREATE TABLE fails; the fixture retries a real metadata.create_all()
  probe rather than trusting the open port. (3) YDB rejects DDL inside an
  explicit transaction ("Scheme operations cannot be executed inside
  transaction") -- confirmed this only affects a raw text("CREATE
  TABLE..."), not metadata.create_all()'s own DDL execution path, which
  already does the right thing.
2026-08-28 18:35:55 -07:00
Superset Dev fec70404c4 fix(ci): move nightly_only gating from job-level to step-level if
A job-level `if:` can't reference `matrix` at all -- only github/inputs/
needs/vars contexts are available there, confirmed by actionlint and by
this exact commit's own CI run failing outright with "This run likely
failed because of a workflow file issue" (zero jobs registered). Moves
the same condition onto the step that actually runs the tests instead,
where matrix access is already used successfully by the existing db2
install step.
2026-08-28 18:35:19 -07:00
Superset Dev 72c6f95053 feat(ci): add nightly-only opt-out for heavy testcontainers dialects
A future dialect whose image is too heavy for per-PR CI (a multi-service
cluster, a many-GB image, a slow licensed installer) can set
nightly_only: true on its matrix entry to run only on the cron or a
manual workflow_dispatch, never on pull_request. No existing dialect
uses it yet -- this just lays the groundwork for candidates like SAP
HANA, Teradata, or Apache Druid.
2026-08-28 16:19:18 -07:00
Superset Dev d46e7986a2 fix(testcontainers): retry a real create-table-and-insert probe for starrocks readiness
The FE's query port accepts connections, and can even run metadata
statements like CREATE DATABASE, before the BE (execution backend) has
registered with it -- an actual CREATE TABLE/INSERT then fails with
"Backend node not found. Check if any backend node is down." (confirmed
on real CI). Replace the bare CREATE DATABASE readiness check with a
throwaway create-table-and-insert probe that exercises the exact
operations the tests below need.
2026-08-28 10:47:31 -07:00
Superset Dev 889d2a6a22 feat(ci): expand testcontainers coverage to postgres, mysql, clickhouse, starrocks
Stacked on feat/testcontainers-more-dialects. All four extras already
existed in pyproject.toml, so this only wires up tests -- no new
optional-dependency groups needed.

- postgres/mysql: straight copies of the timescaledb/mariadb pattern
  respectively, pointed at vanilla images instead of a fork/extension.
- clickhouse: connects over the container's HTTP port (8123), matching
  clickhouse-connect (Superset's driver), not the native TCP port (9000)
  the container's own docstring example uses. ClickHouse has no real
  primary-key concept and clickhouse-connect's DDL compiler rejects
  CREATE TABLE without an explicit engine, so _pagination.py gained an
  optional extra_table_args hook to pass MergeTree(order_by=...). Also
  works around a pre-existing quirk in db_engine_specs/clickhouse.py:
  its module-level type-formatting setup dereferences current_app.config,
  so importing it outside a Flask app context raises RuntimeError --
  tests/unit_tests/db_engine_specs/test_clickhouse.py already works
  around this with per-test local imports, but that suite also benefits
  from an autouse app_context fixture this suite doesn't have, so this
  test pushes one explicitly around the one-time import.
- starrocks: no dedicated testcontainers module, so a generic
  DockerContainer against the official allin1-ubuntu image (FE+BE in one
  container). Not verified locally (multiple-GB image, skipped to keep
  local Docker load low per session guidance); the fixture retries its
  first connection since the query port can accept TCP before StarRocks'
  query engine is fully initialized.
2026-08-28 10:47:31 -07:00
Superset Dev 75044575a1 fix(testcontainers): avoid table-qualified columns in mongodb pagination test
The previous fix (positional row access) only masked the symptom.
Root cause: SQLAlchemy always qualifies a Table-bound column as
"pilot_pagination.id" once there's a FROM clause, but pymongosql's
projection builder takes that qualified text completely literally as a
MongoDB field path -- {"pilot_pagination.id": 1} reads a *nested* field
under a top-level "pilot_pagination" key, which doesn't exist on these
flat documents, silently projecting None for every row instead of
raising (confirmed via pymongosql/sql/query_handler.py's
_extract_field_and_alias, which uses the raw expression text verbatim
as the Mongo field key regardless of any AS alias).

A bare column()/table() pair (not bound to each other via a real Table
object) compiles to an unqualified "id" reference, which resolves
correctly, while still exercising the dialect's own LIMIT/OFFSET
compilation through a real Core select() -- this test's actual intent.
2026-08-28 10:47:11 -07:00
Superset Dev 4028ff1e05 fix(testcontainers): use positional row access in mongodb pagination test
text() has no static column metadata of its own -- the row's key comes
entirely from whatever pymongosql's DBAPI cursor reports for the raw
compiled SQL string, which turned out to be the qualified
"pilot_pagination.id" (matching the SELECT list's column reference), not
the bare "id" that attribute access expected. Reproduced by compiling the
exact statement offline and inspecting text()'s (empty) column metadata;
confirmed the AttributeError on real CI runs of this branch and #43640.
2026-08-28 10:38:11 -07:00
rusackasandClaude Opus 4.8 67a193f1d9 fix(testcontainers): stop third-party dialect entry points from leaking global compiler state
sqlalchemy-monetdb mutates SQLAlchemy's shared, process-global
compiler.OPERATORS mapping in place on import instead of subclassing it
(OPERATORS = compiler.OPERATORS; OPERATORS[operators.ne] = " <> "). Once
Superset's own get_available_engine_specs() enumerates the "monetdb"
sqlalchemy.dialects entry point to build the "available databases" list
(which happens on every app boot, not just when MonetDB is actually
used), that changes `!=` rendering to `<>` for every dialect for the
rest of the process -- breaking test_where_operators identically across
the test-mysql/test-postgres/test-sqlite CI jobs, since installing the
monetdb extra now pulls sqlalchemy-monetdb into their shared dev
requirements.

Snapshot/restore sqlalchemy.sql.compiler.OPERATORS around each
third-party dialect entry point load so a misbehaving connector can't
leak global compiler state just because it was enumerated here.

Also fix the new mongodb testcontainers pagination test: pymongosql's
SQL-to-Mongo AST parser reads LIMIT/OFFSET off the compiled SQL text
ahead of parameter substitution, so a bound `LIMIT ?`/`OFFSET ?`
placeholder is silently dropped. Compile with literal_binds=True instead,
matching how Superset actually issues chart/SQL Lab queries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 05:56:24 -07:00
rusackasandClaude Opus 4.8 cd2638e27c fix(testcontainers): address bot + reviewer feedback on db2 baseline, mariadb host, mongodb pagination test, weak type assertions, monetdb readiness, and CI path coverage
- drop `db2` from the `testcontainers[...]` extra in development.in/.txt:
  it transitively pulls `ibm-db-sa`/`ibm-db` into the baseline dev lockfile,
  which has no Linux arm64 wheel and breaks the multi-platform dev Docker
  image build; the db2 CI leg already installs it on demand separately.
- mariadb: preserve a remote Docker daemon's real host, only rewrite the
  literal "localhost" case to 127.0.0.1.
- testcontainers.yml: widen pull_request paths to superset/db_engine_specs/**,
  pyproject.toml, and the requirements manifests so a driver/lockfile-only
  change can't bypass this coverage.
- mongodb: use Core `select(...).limit().offset()` instead of a literal SQL
  string, so the pagination test actually exercises the dialect's own
  compilation.
- monetdb/mongodb/yugabytedb: assert generic_type/sqla_type on the mapped
  column spec, not just that a spec was returned.
- monetdb: wait for the exposed port (not just a log line that predates
  `monetdbd start -n`) before considering the container ready.
- timescaledb: pin the image to a specific tag instead of the mutable
  `latest-pg16`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 02:35:46 -07:00
Superset Dev 58b37d0fa6 fix(testcontainers): disable implicit autoincrement on pagination test's id column
A single-column integer primary key implicitly becomes AUTO_INCREMENT on
MySQL/MariaDB. That column type treats an explicit 0 as NULL by default,
so inserting id=0 got silently reassigned to 1, colliding with the
explicit id=1 row in the same batch insert -- reproduced locally against
a real mariadb:11 container and confirmed fixed with autoincrement=False.

Supersedes the previous (incorrect) guess that a leftover table from an
earlier run was the cause.
2026-08-27 23:13:20 -07:00
Superset Dev 782104964e fix(testcontainers): drop stale pilot_pagination table before recreating
create_all() defaults to checkfirst=True, silently skipping creation (and
leaving old rows behind) if the table already exists from a prior run,
which collided with the fresh id=0-9 insert on primary key for mariadb.
2026-08-27 23:01:27 -07:00
Superset Dev 24e1c544ad fix(testcontainers): force 127.0.0.1 for mariadb, not localhost
MySqlContainer.get_connection_url() has no host override and defaults to
get_container_host_ip(), which returns the literal string "localhost" on
native Linux Docker (e.g. GitHub Actions runners). MySQLdb (mysqlclient)
treats a "localhost" host specially and attempts a Unix socket connection
instead of TCP, which fails since there's no local MySQL socket -- the
container is only reachable over the network. Confirmed on a real GHA
run: "Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock'".

This was never caught locally because of the separate, unrelated
mysqlclient linking issue on this dev machine, which crashed before ever
reaching a real connection attempt.
2026-08-27 22:50:55 -07:00
Superset Dev 1333ed12c7 feat(ci): expand testcontainers coverage to mariadb, timescaledb, yugabytedb, monetdb, mongodb
Adds five more dialects to the testcontainers suite, stacked on top of the
7-dialect pilot in feat/testcontainers-nightly-pilot: mariadb, timescaledb,
yugabytedb, monetdb, mongodb.

mariadb, timescaledb, and yugabytedb are all wire-compatible with an
existing base dialect (MySQL and Postgres respectively), so they reuse
testcontainers' MySqlContainer/PostgresContainer classes pointed at a
different image rather than needing new container-class wiring.
yugabytedb specifically cannot reuse PostgresContainer's built-in
readiness check, though: that execs `psql`, which the yugabyte image
doesn't ship (only its own `ysqlsh`) -- uses a generic DockerContainer
instead, started via `yugabyted start` and waiting on its own final
startup log line.

monetdb has no native testcontainers module; uses a generic DockerContainer
with the documented MDB_* environment variables. Publishes an amd64-only
image (confirmed running under Rosetta/QEMU emulation on Apple Silicon,
unlike CrateDB's harder x86-64-v3 CPU requirement).

mongodb needed a different data-setup approach, like elasticsearch before
it: documents get inserted via the native pymongo driver, not SQL INSERT,
since MongoDB is schemaless and Superset talks to it through pymongosql
(a SQL-to-MongoDB translation layer requiring a `?mode=superset` query
param). Two real bugs surfaced writing this one: testcontainers'
MongoDbContainer.get_connection_url() has no database path or query
string at all, so naively appending "&mode=superset" glued directly onto
the port number instead of starting a query string; and the root user
MongoDbContainer creates lives in the `admin` database, so connecting
with a different default database in the URL requires authSource=admin
or authentication fails outright. Confirmed pymongosql supports OFFSET
(maps to MongoDB's native `skip`), unlike Elasticsearch's SQL layer.

mariadb could not be verified locally in this environment: mysqlclient
(MySQLdb) has a pre-existing, unrelated native-library linking issue
against this machine's Homebrew-installed libmysqlclient. CI installs it
via apt on Linux, where this does not occur -- same accepted pattern
already used for crate/mssql/db2 in the base branch.
2026-08-27 22:44:32 -07:00
rusackasandClaude Opus 4.8 dff961a2f9 fix(deps): regenerate development.txt lockfile for ibm-db-sa
The db2 testcontainers extra pulls in ibm-db/ibm-db-sa transitively;
the pinned lockfile was missing those entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 03ebf2add4 fix(testcontainers): address remaining reviewer feedback (db2 arm64, cockroachdb DBAPI docs, stale workflow refs)
- requirements/development.in + testcontainers.yml: scope the `db2` extra
  (`ibm-db-sa`/`ibm-db`) out of the baseline dev install. `ibm-db` ships no
  Linux arm64 wheel, so bundling it there broke the multi-platform dev
  Docker image build on push. The testcontainers CI job now installs it
  directly, only for its own db2 matrix leg.
- cockroachdb.py: list `psycopg2-binary` alongside `sqlalchemy-cockroachdb`
  in the `pypi_packages` metadata, since a plain `cockroachdb://` URL can't
  connect without a DBAPI and sqlalchemy-cockroachdb doesn't install one.
- test_cockroachdb.py/test_crate.py/test_trino.py: fix stale docstring
  references to the old `nightly-testcontainers.yml` filename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 ce94f79a09 fix(testcontainers): address reviewer feedback on cockroachdb DBAPI, CI isolation, and test rigor
- pyproject.toml: pin psycopg2-binary alongside sqlalchemy-cockroachdb --
  the latter declares no DBAPI dependency of its own, so the documented
  `apache-superset[cockroachdb]` install couldn't actually connect.
- testcontainers.yml: scope the concurrency group by ref so a PR run and
  the nightly cron (or two different PRs) no longer cancel each other.
- test_cockroachdb.py: assert the actual generic/SQLAlchemy type, matching
  the Trino test, instead of only checking a column spec was found.
- pytest.ini + new `testcontainers` marker + _driver.py: exclude
  tests/testcontainers/ from a plain `pytest` run by default (it needs
  Docker), while the dedicated CI job now sets
  SUPERSET_TESTCONTAINERS_STRICT so a broken/missing driver import fails
  that job instead of silently skipping to a green, zero-tests-run result.
- UPDATING.md: the migration note now says to uninstall the old
  `cockroachdb` package outright, since reinstalling the extra alone can
  leave both packages registering the same dialect entry point.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:39 -07:00
rusackasandClaude Opus 4.8 80928bdb6c refactor(testcontainers): extract shared pagination-test body
The "paginated query returns correct rows in order" test body was
duplicated verbatim across cockroachdb/crate/db2/mssql/oracle/trino's
testcontainers suites. Factor the shared table-setup/assert logic into
_pagination.py, with an optional post-insert hook for CrateDB's
eventual-consistency REFRESH TABLE step; each call site keeps its own
test function and dialect-specific docstring.

Addresses bito-code-review feedback on #43502.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
rusackasandClaude Opus 4.8 6ce414928e test(db_engine_specs): fix test_test_connection_failed regression from mssql extra
Adding the mssql extra to development.in for testcontainers coverage
installs pymssql in the dev/test env, so the existing
mssql+pymssql://url probe in test_test_connection_failed no longer
hits the "driver not found" path it was meant to exercise -- it now
attempts a real (failing) connection instead. Swap it for a
dialect+driver URI whose dialect will never resolve to an installed
extra.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
Superset Dev f2d0b592c0 fix(cockroachdb): replace abandoned cockroachdb package with sqlalchemy-cockroachdb
The `cockroachdb` PyPI package (last released 2021) is abandoned, and its
SQLAlchemy dialect references sqlalchemy.dialects.postgresql.psycopg2's
PGCompiler_psycopg2, which SQLAlchemy 2.0 removed. Constructing a
cockroachdb:// engine has raised ImportError since the SQLAlchemy 2.0 bump
(#42803) landed, with nothing catching it: the driver isn't part of the
default dev/CI install, and the existing test_crdb.py only exercises
convert_dttm(), never a real engine.

Switches the `cockroachdb` extra to sqlalchemy-cockroachdb, the actively
maintained replacement already linked from CockroachDbEngineSpec's own
docs_url, adds it to the default dev install so this keeps getting
exercised, and adds a regression test that constructs a real engine
(verified to fail against the old package, pass against the new one).
2026-08-27 19:52:17 -07:00
Superset Dev 5a7fa2d1c6 feat(ci): expand testcontainers coverage to mssql, oracle, db2, elasticsearch
Adds four more dialects to the testcontainers suite (cockroachdb, crate,
trino from the initial pilot): mssql, oracle, db2, elasticsearch. All four
have native testcontainers-python container classes.

Restructures the workflow from one job running the whole suite to a
matrix, one job per dialect, running in parallel. A single slow container
would otherwise inflate wall-clock time for every dialect, not just its
own -- matrixing bounds total suite time by the slowest dialect instead of
the sum of all of them. Also renames the workflow file/name from
"Nightly-Testcontainers" to "Testcontainers" now that it runs on
pull_request (scoped via `paths`) in addition to the nightly cron.

Elasticsearch needed a different data-setup approach than the SQL-native
dialects: indices/documents get created via its REST API, not SQL INSERT,
matching how Superset actually encounters Elasticsearch in practice.
Confirmed empirically that Elasticsearch's SQL layer has no OFFSET support
at all (a real protocol limitation, already correctly documented via
ElasticSearchEngineSpec.supports_offset = False) and adjusted that
dialect's pagination test accordingly -- LIMIT/ORDER BY only, no OFFSET.

mssql and db2 could not be verified locally (no arm64 images for either;
this environment is Apple Silicon), same situation as crate's amd64-only
image from the initial pilot. Both are written against verified library
source (dialect names, connection URL construction) and will get their
first real execution on CI.
2026-08-27 19:51:23 -07:00
Superset Dev c0e32663a4 fix(deps): bump docker floor to 7.2.0 for testcontainers compatibility
docker==7.0.0 (the previously-unpinned floor) raises
`docker.errors.DockerException: ... Not supported URL scheme http+docker`
against the requests/urllib3 versions pinned elsewhere in this file --
confirmed on a real GHA run of #43502's nightly-testcontainers workflow,
where every test errored before any container was even pulled. 7.2.0 is
confirmed working, both locally and against a live container.
2026-08-27 19:51:23 -07:00
Superset Dev fa36e2565a temp: run nightly-testcontainers on this PR + add actions-timeline
Adds a scoped pull_request trigger so this workflow runs on this PR
itself, to get real GHA runner timing before deciding whether/how to
adopt this pattern more broadly. Also adds the actions-timeline job
(same pattern as superset-python-presto-hive.yml) to visualize per-step
duration.

Revert the pull_request trigger before merge -- it's here to measure
cost, not to become a permanent merge-blocking check.
2026-08-27 19:51:23 -07:00
rusackasandClaude Opus 4.8 eb91af8244 test(ci): assert mapped generic/sqla type in Trino column test
Address bot review feedback on #43502: the integration test only
checked that a ColumnSpec existed, not that INTEGER columns actually
mapped to the numeric generic type and an Integer sqla type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:51:23 -07:00
Superset Dev 051105098c feat(ci): add nightly testcontainers-based db_engine_specs tests
Pilot for real-container testing of db_engine_specs against actual
databases (CockroachDB, CrateDB, Trino), via testcontainers-python.
Existing unit tests mock the driver/dialect layer entirely, which cannot
catch real SQL-compilation or type-mapping bugs -- e.g. #42899, where
Trino emitted OFFSET before LIMIT for paginated queries.

Runs nightly (.github/workflows/nightly-testcontainers.yml), not on every
merge: container pulls and startup are slower and more flake-prone than
the existing mocked unit tests, and this measures that cost/signal
tradeoff before considering wider adoption.

Building this surfaced two real bugs, fixed/documented separately:
- The cockroachdb dialect was completely broken under SQLAlchemy 2.0 due
  to a dead upstream package -- fixed in #43501.
- testcontainers-python's TrinoContainer.get_connection_url() returns the
  container-internal port instead of the Docker-mapped host port; worked
  around locally, filed upstream.
2026-08-27 19:51:23 -07:00
35 changed files with 3175 additions and 17 deletions
+188
View File
@@ -0,0 +1,188 @@
# db_engine_specs tests against real databases (testcontainers)
name: Testcontainers
# Spins up real Docker containers (see tests/testcontainers/ for the current
# dialect list) via testcontainers-python, which catches real dialect/driver
# regressions -- the kind mocked db_engine_specs unit tests structurally
# cannot, e.g. apache/superset#42899 (Trino emitting OFFSET before LIMIT).
# Runs on a nightly cron (catches drift from a driver's own releases, not
# just from Superset's changes) and on pull_request, scoped via `paths` to
# only PRs that actually touch this test suite or the workflow itself, so
# unrelated PRs across the repo are never affected.
#
# A matrix entry can set `nightly_only: true` to run only on the cron (or a
# manual workflow_dispatch), never on pull_request -- for a dialect whose
# image is too heavy (a multi-service cluster, a many-GB image, a slow
# licensed installer) to justify adding its wall-clock/resource cost to
# every PR that merely touches this suite. Omit the field entirely for a
# normal dialect; it isn't nightly-only by default.
permissions:
contents: read
on:
schedule:
- cron: "0 5 * * *"
workflow_dispatch: {}
pull_request:
paths:
- ".github/workflows/testcontainers.yml"
- "tests/testcontainers/**"
- "superset/db_engine_specs/**"
- "pyproject.toml"
- "requirements/development.in"
- "requirements/development.txt"
concurrency:
# Scoped by ref, not just workflow name -- otherwise every PR run and the
# nightly cron share one group, and starting the workflow on another PR
# (or the nightly firing mid-PR-run) cancels an unrelated in-progress run.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
testcontainers:
runs-on: ubuntu-26.04
strategy:
fail-fast: false
matrix:
include:
# One job per dialect rather than one job for the whole suite: a
# single slow container would otherwise inflate the wall-clock
# time for every dialect, not just its own. Running in parallel
# means the suite's total time is bounded by the slowest dialect,
# not the sum of all of them. Db2's first-boot init is documented
# upstream as notably slow (a real instance bring-up, not just a
# process start) and untested locally here (no arm64 image), so
# it gets a wider timeout margin than the rest until real CI data
# says otherwise.
- dialect: cockroachdb
timeout: 10
- dialect: crate
timeout: 10
- dialect: trino
timeout: 10
- dialect: mssql
timeout: 10
- dialect: elasticsearch
timeout: 10
- dialect: oracle
timeout: 15
- dialect: db2
timeout: 25
- dialect: mariadb
timeout: 10
- dialect: timescaledb
timeout: 10
- dialect: yugabytedb
timeout: 10
- dialect: monetdb
timeout: 10
- dialect: mongodb
timeout: 10
- dialect: postgres
timeout: 10
- dialect: mysql
timeout: 10
- dialect: clickhouse
timeout: 10
# StarRocks' allin1-ubuntu image brings up both FE and BE in one
# container, which is a heavier bring-up than a single-process
# database -- wider margin until real CI data says otherwise.
- dialect: starrocks
timeout: 15
- dialect: databend
timeout: 10
- dialect: risingwave
timeout: 10
- dialect: firebird
timeout: 10
- dialect: ydb
timeout: 10
# OceanBase bootstraps a distributed-style cluster even in
# single-node MODE=MINI -- too heavy for every PR's CI budget, so
# it runs on the nightly cron / manual dispatch only.
- dialect: oceanbase
timeout: 20
nightly_only: true
timeout-minutes: ${{ matrix.timeout }}
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
# This job's matrix installs exactly one dialect's testcontainers
# driver for exactly this job, so treat that driver as required: a
# broken/missing import should fail the job, not silently skip to a
# misleadingly green, zero-tests-run result. See _driver.py.
SUPERSET_TESTCONTAINERS_STRICT: true
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
python-version: current
- name: Install db2 driver (ibm-db-sa)
# ibm-db (the db2 DBAPI) ships no Linux arm64 wheel, so it's kept out
# of the baseline dev install (requirements/development.in) to avoid
# breaking the multi-platform dev Docker image build. Install it here
# instead, only for this leg of the matrix.
if: matrix.dialect == 'db2'
run: uv pip install --system -e .[db2]
- name: Install oceanbase driver (oceanbase_py)
# oceanbase_py pins sqlalchemy-utils>=0.38.3,<0.39, which conflicts
# outright with Superset's own sqlalchemy-utils==0.42.1 pin -- kept
# out of the baseline dev install for the same reason as db2 above.
# Installed as its own standalone package (not via `-e .[oceanbase]`)
# so --no-deps only skips *oceanbase_py's* dependencies -- applied
# to `-e .[oceanbase]` instead, --no-deps blocks pip from installing
# anything the extras marker pulls in, including oceanbase_py
# itself, which "succeeds" without actually installing it
# (confirmed on real CI: the install step reported success, but the
# module was still missing). This job only needs oceanbase_py's
# dialect module importable, not its sqlalchemy-utils dependency
# satisfied, since nothing here calls into it.
if: matrix.dialect == 'oceanbase'
run: uv pip install --system --no-deps "oceanbase_py>=0.0.1.2"
- name: Install Firebird client library (libfbclient2)
# sqlalchemy-firebird's driver (firebird-driver) is a pure-Python
# ctypes wrapper (its wheel is py3-none-any) that dynamically loads
# the native Firebird client library from the host at import time
# -- it doesn't bundle that library itself, so it has to come from
# the system package manager, only for this leg of the matrix.
if: matrix.dialect == 'firebird'
run: |
sudo apt-get update
sudo apt-get install -y libfbclient2
- name: Run testcontainers db_engine_specs tests (${{ matrix.dialect }})
# A job-level `if:` can't reference `matrix` (only github/inputs/
# needs/vars are available there), so the nightly_only skip has to
# live on the step instead. A dialect without `nightly_only` set
# evaluates the left side true (unset is null, and `null != true`
# is true) and always runs; one WITH it set only runs on the cron
# or a manual dispatch, never on pull_request.
if: >-
matrix.nightly_only != true ||
github.event_name == 'schedule' ||
github.event_name == 'workflow_dispatch'
run: |
pytest --durations-min=2 -v -m testcontainers \
./tests/testcontainers/db_engine_specs/test_${{ matrix.dialect }}.py \
--junit-xml=test-results/junit-testcontainers-${{ matrix.dialect }}.xml
- name: Upload JUnit test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: junit-results-testcontainers-${{ matrix.dialect }}
path: test-results/
retention-days: 7
actions-timeline:
needs: [testcontainers]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
+1 -1
View File
@@ -25,7 +25,7 @@ assists people when migrating to a new version.
## Next
- `SAMPLES_ROW_LIMIT` is now the default for `/datasource/samples` requests without a valid explicit `per_page`, rather than a hard per-request ceiling; explicit limits are honored up to the existing global row-limit ceiling, matching `/chart/data` SAMPLES requests.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed should `pip uninstall cockroachdb && pip install sqlalchemy-cockroachdb` (or simply reinstall the extra) to restore CockroachDB connectivity.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed must `pip uninstall cockroachdb` before reinstalling the extra -- both packages register the same `cockroachdb` SQLAlchemy dialect entry point, so leaving the old one in place can still load the abandoned implementation.
### MCP tool results preserve stored string values
+19 -5
View File
@@ -148,10 +148,9 @@ clickhouse = ["clickhouse-connect>=1.7.2, <2.0"]
# 2.0). sqlalchemy-cockroachdb is the actively maintained replacement,
# already linked from CockroachDbEngineSpec.metadata's docs_url, and
# registers the same `cockroachdb` SQLAlchemy dialect entry point.
# sqlalchemy-cockroachdb depends only on SQLAlchemy itself, not on a DBAPI
# driver, so psycopg2-binary is pinned alongside it (matching the `postgres`
# extra) to keep this extra self-contained -- CockroachDB speaks the
# PostgreSQL wire protocol, so psycopg2 is what actually opens connections.
# sqlalchemy-cockroachdb itself declares no DBAPI dependency (its own docs
# require picking one), so pull in the same psycopg2-binary pin as the
# `postgres` extra -- CockroachDB speaks the Postgres wire protocol.
cockroachdb = ["sqlalchemy-cockroachdb>=2.0.0, <3", "psycopg2-binary==2.9.12"]
crate = ["sqlalchemy-cratedb>=0.43.1, <1"]
# sqlalchemy-d1's only release (0.1.0, Nov 2025) pins sqlalchemy<2,>=1.4,
@@ -223,6 +222,11 @@ impala = ["impyla>=0.24.0, <0.25"]
# superset/db_engine_specs/kusto.py's known_incompatibilities metadata.
kusto = ["sqlalchemy-kusto>=3.1.2, <4"]
kylin = ["kylinpy>=2.8.4, <2.9"]
# MariaDB is a MySQL fork implementing the same wire protocol - connects via
# the plain mysql dialect, same driver as mysql.
mariadb = ["apache-superset[mysql]"]
monetdb = ["sqlalchemy-monetdb>=2.1.0, <3", "pymonetdb>=1.9.1, <2"]
mongodb = ["pymongosql>=0.7.3, <1"]
mssql = ["pymssql>=2.3.13, <3"]
# motherduck is an alias for duckdb - MotherDuck works via the duckdb driver
motherduck = ["apache-superset[duckdb]"]
@@ -271,6 +275,9 @@ tdengine = [
"taos-ws-py>=0.7.0"
]
teradata = ["teradatasql>=20.0.0.66"]
# TimescaleDB is a genuine Postgres extension, not a fork - connects via the
# plain postgresql dialect, same driver as postgres.
timescaledb = ["apache-superset[postgres]"]
thumbnails = [] # deprecated, will be removed in 7.0
vertica = ["sqlalchemy-vertica-python>= 0.6.3, < 0.7"]
netezza = ["nzalchemy>= 11.1.2, < 11.2"]
@@ -278,11 +285,18 @@ starrocks = ["starrocks>=1.3.4, <2"]
doris = ["pydoris>=1.2.0, <2.0.0"]
oceanbase = ["oceanbase_py>=0.0.1.2"]
ydb = ["ydb-sqlalchemy>=0.1.22", "ydb-sqlglot-plugin>=0.2.8"]
# YugabyteDB's YSQL layer is fully Postgres-wire compatible - connects via
# the plain postgresql dialect, same driver as postgres.
yugabytedb = ["apache-superset[postgres]"]
development = [
# no bounds for apache-superset-extensions-cli until a stable version
"apache-superset-extensions-cli",
"boto3",
"docker",
# 7.0.0 raises `docker.errors.DockerException: ... Not supported URL
# scheme http+docker` against the requests/urllib3 versions pinned
# elsewhere in this file -- breaks testcontainers (tests/testcontainers/)
# before any container even starts. 7.2.0 is confirmed working.
"docker>=7.2.0",
"flask-testing",
"freezegun",
"grpcio>=1.82.1",
+7 -1
View File
@@ -19,7 +19,13 @@ testpaths =
tests
python_files = *_test.py test_*.py *_tests.py *viz/utils.py
# `-p no:warnings` temporarily disabled in favor of more finely tuned `filterwarnings`.
#addopts = -p no:warnings
# `not testcontainers` excludes tests/testcontainers/ by default: those spin up
# real Docker containers, and `testpaths = tests` would otherwise pull them into
# every plain `pytest` run. The dedicated CI job (testcontainers.yml) overrides
# this with an explicit `-m testcontainers` to run them.
addopts = -m "not testcontainers"
markers =
testcontainers: exercises a real database via testcontainers-python (needs Docker); excluded by default, see .github/workflows/testcontainers.yml
asyncio_mode = auto
# `ignore` is effectively equivalent to `-p no:warnings`.
+28 -1
View File
@@ -16,5 +16,32 @@
# specific language governing permissions and limitations
# under the License.
#
-e .[development,bigquery,cockroachdb,druid,duckdb,fastmcp,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
-e .[development,bigquery,clickhouse,cockroachdb,crate,databend,druid,duckdb,elasticsearch,fastmcp,firebird,gevent,gsheets,monetdb,mongodb,mssql,mysql,oracle,postgres,presto,prophet,risingwave,starrocks,trino,thumbnails,ydb]
-e ./superset-extensions-cli[test]
# testcontainers-backed db_engine_specs tests (tests/testcontainers/) --
# see .github/workflows/testcontainers.yml
#
# `db2` (the `ibm-db-sa`/`ibm-db` driver) and `oceanbase` (the `oceanbase_py`
# driver) are both deliberately left out of the baseline dev install above:
# `ibm-db` ships no Linux arm64 wheel, breaking the multi-platform
# (amd64+arm64) dev Docker image build; `oceanbase_py` pins
# `sqlalchemy-utils>=0.38.3,<0.39`, which conflicts outright with Superset's
# own `sqlalchemy-utils==0.42.1` pin -- there's no version of both that can
# coexist in one resolved environment. Both testcontainers CI jobs install
# their driver on demand instead, only for their own matrix leg -- see
# .github/workflows/testcontainers.yml.
#
# mariadb/timescaledb/yugabytedb need no testcontainers extra of their own:
# they reuse the postgres/mysql container classes pointed at a different
# image, and psycopg2-binary/mysqlclient are already pulled in above via
# the postgres/mysql extras. Plain postgres/mysql obviously need nothing
# extra either. clickhouse and starrocks also need no testcontainers extra:
# ClickHouseContainer has no driver import of its own (clickhouse-connect,
# pulled in above via the clickhouse extra, is all the test needs), and
# StarRocks has no dedicated testcontainers module at all -- its test uses
# a generic DockerContainer plus the same mysqlclient the mysql extra
# already provides. databend/risingwave/firebird/ydb are the same story:
# none has a dedicated testcontainers module, so each test uses a generic
# DockerContainer plus whatever driver its own extra above already
# provides.
testcontainers[cockroachdb,cratedb,mongodb,mssql,mysql,oracle,postgres,trino]>=4.15.0,<5
+153 -6
View File
@@ -12,10 +12,17 @@
# apache-superset
aiofile==3.9.0
# via py-key-value-aio
aiohappyeyeballs==2.7.1
# via aiohttp
aiohttp==3.14.3
# via ydb
aiosignal==1.4.0
# via aiohttp
alembic==1.15.2
# via
# -c requirements/base-constraint.txt
# flask-migrate
# starrocks
amqp==5.3.1
# via
# -c requirements/base-constraint.txt
@@ -24,6 +31,8 @@ annotated-types==0.7.0
# via
# -c requirements/base-constraint.txt
# pydantic
antlr4-python3-runtime==4.13.2
# via pymongosql
anyio==4.11.0
# via
# httpx
@@ -42,9 +51,12 @@ apsw==3.50.1.0
# shillelagh
astroid==3.3.10
# via pylint
asyncmy2==0.2.21
# via starrocks
attrs==25.3.0
# via
# -c requirements/base-constraint.txt
# aiohttp
# cattrs
# cyclopts
# jsonschema
@@ -65,6 +77,7 @@ backports-tarfile==1.2.0
backports-zstd==1.6.0
# via
# -c requirements/base-constraint.txt
# clickhouse-connect
# flask-compress
bcrypt==4.3.0
# via
@@ -117,8 +130,11 @@ celery==5.6.3
certifi==2026.5.20
# via
# -c requirements/base-constraint.txt
# clickhouse-connect
# elasticsearch
# httpcore
# httpx
# opensearch-py
# requests
cffi==2.0.0
# via
@@ -160,6 +176,8 @@ click-repl==0.3.0
# via
# -c requirements/base-constraint.txt
# celery
clickhouse-connect==1.7.2
# via apache-superset
cmdstanpy==1.1.0
# via prophet
colorama==0.4.6
@@ -171,6 +189,8 @@ contourpy==1.0.7
# via matplotlib
coverage==7.6.8
# via pytest-cov
crate==2.2.1
# via sqlalchemy-cratedb
cron-descriptor==1.4.5
# via
# -c requirements/base-constraint.txt
@@ -186,14 +206,20 @@ cryptography==50.0.0
# authlib
# google-auth
# joserfc
# oracledb
# paramiko
# pyjwt
# pymysql
# pyopenssl
# secretstorage
cycler==0.12.1
# via matplotlib
cyclopts==4.2.4
# via fastmcp-slim
databend-driver==0.34.2
# via databend-sqlalchemy
databend-sqlalchemy==0.5.5
# via apache-superset
db-dtypes==1.3.1
# via pandas-gbq
defusedxml==0.7.1
@@ -216,8 +242,11 @@ dnspython==2.7.0
# via
# -c requirements/base-constraint.txt
# email-validator
docker==7.0.0
# via apache-superset
# pymongo
docker==7.2.0
# via
# apache-superset
# testcontainers
docstring-parser==0.17.0
# via cyclopts
docutils==0.22.2
@@ -228,6 +257,10 @@ duckdb==1.5.5
# duckdb-engine
duckdb-engine==0.17.0
# via apache-superset
elasticsearch==7.17.13
# via elasticsearch-dbapi
elasticsearch-dbapi==0.2.13
# via apache-superset
email-validator==2.2.0
# via
# -c requirements/base-constraint.txt
@@ -237,6 +270,8 @@ et-xmlfile==2.0.0
# via
# -c requirements/base-constraint.txt
# openpyxl
events==0.5
# via opensearch-py
exceptiongroup==1.3.0
# via fastmcp-slim
fastmcp==3.4.7
@@ -247,6 +282,10 @@ filelock==3.20.3
# via
# -c requirements/base-constraint.txt
# virtualenv
firebird-base==2.0.3
# via firebird-driver
firebird-driver==2.0.3
# via sqlalchemy-firebird
flask==2.3.3
# via
# -c requirements/base-constraint.txt
@@ -327,12 +366,18 @@ fonttools==4.60.2
# via matplotlib
freezegun==1.5.1
# via apache-superset
frozenlist==1.8.0
# via
# aiohttp
# aiosignal
future==1.0.0
# via pyhive
geographiclib==2.0
# via
# -c requirements/base-constraint.txt
# geopy
geojson==3.3.0
# via sqlalchemy-cratedb
geopy==2.4.1
# via
# -c requirements/base-constraint.txt
@@ -389,6 +434,7 @@ grpcio==1.83.0
# apache-superset
# google-api-core
# grpcio-status
# ydb
grpcio-status==1.60.1
# via google-api-core
gunicorn==26.2.0
@@ -414,6 +460,7 @@ httpx==0.28.1
# via
# fastmcp-slim
# mcp
# testcontainers
httpx-sse==0.4.1
# via mcp
humanize==4.12.3
@@ -430,6 +477,7 @@ idna==3.15
# httpx
# requests
# url-normalize
# yarl
importlib-metadata==8.7.0
# via
# keyring
@@ -468,6 +516,7 @@ jmespath==1.1.0
# via
# boto3
# botocore
# pymongosql
joserfc==1.7.2
# via fastmcp-slim
jsonpath-ng==1.8.0
@@ -500,6 +549,8 @@ kombu==5.6.2
# via
# -c requirements/base-constraint.txt
# celery
lark==1.3.1
# via starrocks
lazy-object-proxy==1.10.0
# via openapi-spec-validator
limits==5.1.0
@@ -507,7 +558,9 @@ limits==5.1.0
# -c requirements/base-constraint.txt
# flask-limiter
lz4==4.4.5
# via trino
# via
# clickhouse-connect
# trino
mako==1.4.1
# via
# -c requirements/base-constraint.txt
@@ -567,6 +620,10 @@ msgspec==0.19.0
# via
# -c requirements/base-constraint.txt
# flask-session
multidict==6.7.1
# via
# aiohttp
# yarl
mysqlclient==2.2.8
# via apache-superset
nh3==0.3.6
@@ -605,14 +662,22 @@ openpyxl==3.1.5
# via
# -c requirements/base-constraint.txt
# pandas
opensearch-py==2.8.0
# via elasticsearch-dbapi
opentelemetry-api==1.39.1
# via fastmcp-slim
oracledb==4.0.2
# via
# apache-superset
# testcontainers
ordered-set==4.1.0
# via
# -c requirements/base-constraint.txt
# flask-limiter
orjson==3.11.9
# via trino
# via
# crate
# trino
packaging==25.0
# via
# -c requirements/base-constraint.txt
@@ -620,8 +685,8 @@ packaging==25.0
# apispec
# db-dtypes
# deprecation
# docker
# duckdb-engine
# elasticsearch-dbapi
# fastmcp-slim
# google-cloud-bigquery
# kombu
@@ -631,6 +696,8 @@ packaging==25.0
# pytest
# shillelagh
# sqlalchemy-bigquery
# sqlalchemy-firebird
# ydb
pandas==2.3.3
# via
# -c requirements/base-constraint.txt
@@ -692,16 +759,22 @@ prompt-toolkit==3.0.51
# via
# -c requirements/base-constraint.txt
# click-repl
propcache==0.5.2
# via
# aiohttp
# yarl
prophet==1.4.0
# via apache-superset
proto-plus==1.25.0
# via google-api-core
protobuf==5.29.6
# via
# firebird-base
# google-api-core
# googleapis-common-protos
# grpcio-status
# proto-plus
# ydb
psutil==6.1.0
# via
# apache-superset
@@ -775,6 +848,24 @@ pyjwt==2.13.0
# mcp
pylint==3.3.7
# via apache-superset
pymonetdb==1.9.1
# via
# apache-superset
# sqlalchemy-monetdb
pymongo==4.17.0
# via
# pymongosql
# testcontainers
pymongosql==0.7.3
# via apache-superset
pymssql==2.3.13
# via
# apache-superset
# testcontainers
pymysql==1.2.0
# via
# starrocks
# testcontainers
pynacl==1.6.2
# via
# -c requirements/base-constraint.txt
@@ -819,11 +910,13 @@ python-dateutil==2.9.0.post0
# botocore
# celery
# croniter
# firebird-driver
# flask-appbuilder
# freezegun
# google-cloud-bigquery
# holidays
# matplotlib
# opensearch-py
# pandas
# pyhive
# shillelagh
@@ -834,6 +927,7 @@ python-dotenv==1.2.2
# apache-superset
# fastmcp-slim
# pydantic-settings
# testcontainers
python-ldap==3.4.7
# via apache-superset
python-multipart==0.0.29
@@ -877,6 +971,7 @@ requests==2.33.0
# google-api-core
# google-cloud-bigquery
# jsonschema-path
# opensearch-py
# pydruid
# pyhive
# requests-cache
@@ -955,7 +1050,9 @@ sqlalchemy==2.0.52
# alembic
# apache-superset
# apache-superset-core
# databend-sqlalchemy
# duckdb-engine
# elasticsearch-dbapi
# flask-appbuilder
# flask-sqlalchemy
# marshmallow-sqlalchemy
@@ -963,7 +1060,14 @@ sqlalchemy==2.0.52
# sqlalchemy-bigquery
# sqlalchemy-cockroachdb
# sqlalchemy-continuum
# sqlalchemy-cratedb
# sqlalchemy-firebird
# sqlalchemy-monetdb
# sqlalchemy-risingwave
# sqlalchemy-utils
# starrocks
# testcontainers
# ydb-sqlalchemy
sqlalchemy-bigquery==1.17.2
# via apache-superset
sqlalchemy-cockroachdb==2.0.4
@@ -972,6 +1076,16 @@ sqlalchemy-continuum==1.7.0
# via
# -c requirements/base-constraint.txt
# apache-superset
sqlalchemy-cratedb==0.43.1
# via
# apache-superset
# testcontainers
sqlalchemy-firebird==2.2.0
# via apache-superset
sqlalchemy-monetdb==2.1.0
# via apache-superset
sqlalchemy-risingwave==2.1.0
# via apache-superset
sqlalchemy-utils==0.42.1
# via
# -c requirements/base-constraint.txt
@@ -983,6 +1097,7 @@ sqlglot==30.17.0
# -c requirements/base-constraint.txt
# apache-superset
# apache-superset-core
# ydb-sqlglot-plugin
sqloxide==0.1.51
# via apache-superset
sse-starlette==3.0.2
@@ -995,6 +1110,8 @@ starlette==1.3.1
# via
# fastmcp-slim
# mcp
starrocks==1.3.4
# via apache-superset
statsd==4.0.1
# via apache-superset
syntaqlite==0.9.0
@@ -1003,6 +1120,8 @@ tabulate==0.10.0
# via
# -c requirements/base-constraint.txt
# apache-superset
testcontainers==4.15.0
# via -r requirements/development.in
tiktoken==0.14.0
# via apache-superset
tomli-w==1.2.0
@@ -1014,10 +1133,14 @@ tqdm==4.67.1
# cmdstanpy
# prophet
trino==0.339.0
# via apache-superset
# via
# apache-superset
# testcontainers
typing-extensions==4.16.0
# via
# -c requirements/base-constraint.txt
# aiohttp
# aiosignal
# alembic
# anyio
# apache-superset
@@ -1029,6 +1152,7 @@ typing-extensions==4.16.0
# limits
# mcp
# opentelemetry-api
# oracledb
# py-key-value-aio
# pydantic
# pydantic-core
@@ -1037,6 +1161,7 @@ typing-extensions==4.16.0
# shillelagh
# sqlalchemy
# starlette
# testcontainers
# typing-inspection
typing-inspection==0.4.2
# via
@@ -1064,13 +1189,22 @@ urllib3==2.7.0
# via
# -c requirements/base-constraint.txt
# botocore
# clickhouse-connect
# crate
# docker
# elasticsearch
# opensearch-py
# requests
# requests-cache
# testcontainers
uvicorn==0.37.0
# via
# fastmcp-slim
# mcp
verlib2==0.3.2
# via
# crate
# sqlalchemy-cratedb
vine==5.1.0
# via
# -c requirements/base-constraint.txt
@@ -1104,6 +1238,7 @@ wrapt==1.17.2
# via
# -c requirements/base-constraint.txt
# deprecated
# testcontainers
wtforms==3.2.2
# via
# -c requirements/base-constraint.txt
@@ -1124,6 +1259,18 @@ xlsxwriter==3.2.9
# -c requirements/base-constraint.txt
# apache-superset
# pandas
yarl==1.24.5
# via aiohttp
ydb==3.31.4
# via
# ydb-dbapi
# ydb-sqlalchemy
ydb-dbapi==0.1.23
# via ydb-sqlalchemy
ydb-sqlalchemy==0.1.22
# via apache-superset
ydb-sqlglot-plugin==0.2.8
# via apache-superset
zipp==3.23.0
# via importlib-metadata
zope-event==5.0
+15
View File
@@ -41,6 +41,7 @@ import sqlalchemy.dialects
from flask import current_app as app
from sqlalchemy.engine.default import DefaultDialect
from sqlalchemy.exc import NoSuchModuleError
from sqlalchemy.sql import compiler as sqla_compiler
from superset import feature_flag_manager
from superset.db_engine_specs.base import BaseEngineSpec
@@ -159,6 +160,16 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
continue
# installed 3rd-party dialects
#
# `ep.load()` runs arbitrary module-level code in the third-party package.
# Some dialects (e.g. sqlalchemy-monetdb) mutate SQLAlchemy's shared,
# process-global `compiler.OPERATORS` mapping in place on import instead
# of subclassing it, which would otherwise silently change SQL rendering
# (e.g. `!=` -> `<>`) for every dialect for the rest of the process, not
# just the misbehaving one. Snapshot/restore around each load so a
# buggy connector can't leak global compiler state into unrelated
# dialects just because it was enumerated here.
operators_snapshot = dict(sqla_compiler.OPERATORS)
for ep in entry_points(group="sqlalchemy.dialects"):
try:
dialect = ep.load()
@@ -193,6 +204,10 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
if isinstance(driver, bytes):
driver = driver.decode()
drivers[backend].add(driver)
finally:
if sqla_compiler.OPERATORS != operators_snapshot:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(operators_snapshot)
dbs_denylist = app.config["DBS_AVAILABLE_DENYLIST"]
if not feature_flag_manager.is_feature_enabled("ENABLE_SUPERSET_META_DB"):
+4 -1
View File
@@ -44,7 +44,10 @@ class CockroachDbEngineSpec(PostgresEngineSpec):
DatabaseCategory.TRADITIONAL_RDBMS,
DatabaseCategory.OPEN_SOURCE,
],
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2"],
# sqlalchemy-cockroachdb declares no DBAPI dependency of its own (see
# the comment on the `cockroachdb` extra in pyproject.toml), so a
# plain `cockroachdb://` URL also needs psycopg2 installed to connect.
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2-binary"],
"connection_string": "cockroachdb://root@{hostname}:{port}/{database}?sslmode=disable",
"default_port": 26257,
"docs_url": "https://github.com/cockroachdb/sqlalchemy-cockroachdb",
@@ -2487,8 +2487,15 @@ class TestDatabaseApi(SupersetTestCase):
}
assert response == expected_response
# Uses a `dialect+driver://` URI (rather than the bare `broken://`
# above) to also cover engine-name extraction stripping the driver
# suffix. The dialect itself ("broken") must stay one that no
# installed extra ever registers a real SQLAlchemy plugin for --
# this PR's own testcontainers extras (mssql, oracle, db2, ...)
# install real drivers for those dialects, which would make this
# URI actually attempt a connection instead of failing to load.
data = {
"sqlalchemy_uri": "mssql+pymssql://url",
"sqlalchemy_uri": "broken+driver://url",
"database_name": "examples",
"impersonate_user": False,
"server_cert": None,
@@ -2500,7 +2507,7 @@ class TestDatabaseApi(SupersetTestCase):
expected_response = {
"errors": [
{
"message": "Could not load database driver for: mssql",
"message": "Could not load database driver for: broken",
"error_type": "GENERIC_COMMAND_ERROR",
"level": "warning",
"extra": {
+16
View File
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared import guard for the per-dialect testcontainers modules
(tests/testcontainers/db_engine_specs/test_*.py), each of which needs its
own optional `testcontainers[...]` driver submodule to even import.
"""
import importlib
import os
def require_driver(module_name: str) -> None:
"""
Import `module_name`, a dialect's `testcontainers` driver submodule.
Most environments treat that driver as optional: a bare local `pytest`
run, or another CI job that never installed the `testcontainers` extras,
should skip the module rather than fail collection outright.
The dedicated per-dialect CI job (.github/workflows/testcontainers.yml)
sets SUPERSET_TESTCONTAINERS_STRICT, because there the driver is not
optional -- that job's matrix installs exactly this one driver for
exactly this one module. A broken or missing import there means the job
is misconfigured, and should fail loudly instead of silently reporting
a misleadingly green, zero-tests-run result.
"""
if os.environ.get("SUPERSET_TESTCONTAINERS_STRICT"):
importlib.import_module(module_name)
else:
import pytest
pytest.importorskip(module_name)
@@ -0,0 +1,68 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared body for the "paginated query returns correct rows in order" test
that db_engine_specs.{cockroachdb,crate,db2,mssql,oracle,trino}'s
testcontainers suites each run against their own real instance: a plain
SQLAlchemy Core LIMIT/OFFSET query, compiled and executed for real. Mocked
tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can.
Each call site keeps its own test function (and dialect-specific docstring)
so failures still report against the right module; this only factors out
the identical table setup/assert body, via an optional post-insert hook for
dialects (CrateDB) that need one, and an optional extra-table-args hook for
dialects (ClickHouse) whose CREATE TABLE requires a schema item a plain
Column/primary key can't express.
"""
from collections.abc import Callable
from typing import Any
from sqlalchemy import Column, insert, Integer, MetaData, select, Table as SATable
from sqlalchemy.engine import Connection, Engine
def assert_paginated_query_returns_correct_rows_in_order(
engine: Engine,
after_insert: Callable[[Connection], None] | None = None,
extra_table_args: tuple[Any, ...] = (),
) -> None:
metadata = MetaData()
t = SATable(
"pilot_pagination",
metadata,
# autoincrement=False: a single-column integer primary key otherwise
# implicitly becomes AUTO_INCREMENT on MySQL/MariaDB. That column
# type treats an explicit 0 as NULL by default (NO_AUTO_VALUE_ON_ZERO
# is off), so the id=0 row below would silently get auto-assigned 1,
# colliding with the explicit id=1 row in the same batch insert.
Column("id", Integer, primary_key=True, autoincrement=False),
*extra_table_args,
)
metadata.create_all(engine)
with engine.begin() as conn:
conn.execute(insert(t), [{"id": i} for i in range(10)])
if after_insert is not None:
after_insert(conn)
with engine.connect() as conn:
stmt = select(t.c.id).order_by(t.c.id).limit(3).offset(4)
rows = conn.execute(stmt).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
@@ -0,0 +1,130 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.clickhouse against a real ClickHouse instance, spun
up on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Superset's recommended ClickHouse connector is `clickhouse-connect`
(`ClickHouseConnectEngineSpec`, engine "clickhousedb"), which talks HTTP,
not `ClickHouseContainer`'s own documented `clickhouse_driver` (a different,
native-TCP-protocol package Superset doesn't use at all). The container
exposes both the native TCP port (9000) and the HTTP port (8123); this test
connects over the HTTP port to match Superset's actual driver.
Unlike every other dialect in this suite, ClickHouse tables have no real
primary key/constraint concept -- CREATE TABLE requires an explicit engine
(e.g. MergeTree), or clickhouse-connect's DDL compiler raises a CompileError
rather than defaulting to one.
`superset.db_engine_specs.clickhouse` runs module-level setup code (default
type-formatting overrides) that dereferences `current_app.config` whenever
clickhouse-connect is installed, so importing it outside a Flask app context
raises RuntimeError the first time it's imported in a process.
`tests/unit_tests/db_engine_specs/test_clickhouse.py` gets an app context
for free from that suite's autouse fixture; this suite has no such fixture,
so this test pushes one explicitly around just that one-time import, reusing
the real app instance `tests/conftest.py` already builds for the rest of the
test run rather than constructing a second one.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.clickhouse")
require_driver("clickhouse_connect")
from clickhouse_connect.cc_sqlalchemy.ddl.tableengine import MergeTree # noqa: E402
from testcontainers.community.clickhouse import ClickHouseContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
HTTP_PORT = 8123
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with ClickHouseContainer("clickhouse/clickhouse-server:latest") as container:
host = container.get_container_host_ip()
port = container.get_exposed_port(HTTP_PORT)
yield create_engine(
f"clickhousedb://{container.username}:{container.password}"
f"@{host}:{port}/{container.dbname}"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(
engine, extra_table_args=(MergeTree(order_by="id"),)
)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
ClickHouseConnectEngineSpec.get_columns wraps a real SQLAlchemy
Inspector; this exercises that against actual server-reported column
metadata rather than a mocked Inspector.
"""
from tests.integration_tests.test_app import app
with app.app_context():
from superset.db_engine_specs.clickhouse import ClickHouseConnectEngineSpec
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
MergeTree(order_by="id"),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = ClickHouseConnectEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = ClickHouseConnectEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,97 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.cockroachdb against a real CockroachDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.cockroachdb import CockroachDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cockroachdb")
from testcontainers.community.cockroachdb import CockroachDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
# sqlalchemy-cockroachdb registers its dialect under the plain
# "cockroachdb" name; the container's own default ("cockroachdb+psycopg2")
# matches the abandoned `cockroachdb` package instead (see #43501).
with CockroachDBContainer(dialect="cockroachdb") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CockroachDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CockroachDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CockroachDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,103 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.crate against a real CrateDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
crate/crate only publishes an amd64 image (no arm64 build), and requires a
host CPU supporting the x86-64-v3 instruction set -- QEMU emulation on
Apple Silicon cannot satisfy that, so this file cannot run locally on an
Apple Silicon machine even with `docker pull --platform linux/amd64`. It
runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Connection, Engine
from superset.db_engine_specs.crate import CrateEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cratedb")
from testcontainers.community.cratedb import CrateDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with CrateDBContainer() as container:
yield create_engine(container.get_connection_url())
def _refresh_pilot_pagination(conn: Connection) -> None:
# CrateDB is eventually consistent: a row is not guaranteed visible to
# subsequent selects immediately after insert.
conn.exec_driver_sql("REFRESH TABLE pilot_pagination")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(
engine, after_insert=_refresh_pilot_pagination
)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CrateEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CrateEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CrateEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,114 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.databend against a real Databend instance, spun up
on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Databend has no dedicated testcontainers module, so this uses a generic
DockerContainer against the official `datafuselabs/databend` standalone
image. Superset's DatabendEngineSpec defaults to `sslmode=require`
(`encryption_parameters`), but the local standalone image has no TLS
listener, so this connects with `sslmode=disable` explicitly.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.databend import DatabendEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("databend_sqlalchemy")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
HTTP_PORT = 8000
DBNAME = "default"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("datafuselabs/databend")
container.with_exposed_ports(HTTP_PORT)
# The image's own startup banner documents this exact line as proof its
# HTTP query endpoint is bound and ready.
container.waiting_for(LogMessageWaitStrategy(f"listened at 0.0.0.0:{HTTP_PORT}"))
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(HTTP_PORT)
# "root" with no password is the image's builtin user -- confirmed
# directly against a running container, not from the image's own
# doc text, which only shows ${USER}/${PASSWORD} placeholders.
yield create_engine(f"databend://root:@{host}:{port}/{DBNAME}?sslmode=disable")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
DatabendEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = DatabendEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = DatabendEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.db2 against a real IBM Db2 instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
icr.io/db2_community/db2 only publishes amd64/ppc64le/s390x images (no
arm64 build), so this cannot run locally on an Apple Silicon machine. It
runs natively on GitHub Actions' x86_64 runners. Db2 is also a notably slow
starter (a full instance bring-up, not just a process start) -- expect this
module alone to take several minutes.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.db2 import Db2EngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.db2")
from testcontainers.community.db2 import Db2Container # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with Db2Container() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
Db2EngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = Db2EngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = Db2EngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,107 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.elasticsearch against a real Elasticsearch instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
Unlike the SQL-native dialects in this directory, Elasticsearch has no
CREATE TABLE / INSERT: indices and documents get created via its REST API
(elasticsearch-dbapi's SQLAlchemy dialect is read-focused, translating SQL
to the _sql endpoint), matching how Superset actually encounters
Elasticsearch in practice -- data arrives via ingestion tooling, not
through Superset itself.
"""
from collections.abc import Iterator
import pytest
import requests
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import Engine
from superset.db_engine_specs.elasticsearch import ElasticSearchEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.elasticsearch")
from testcontainers.community.elasticsearch import ElasticSearchContainer # noqa: E402
INDEX = "pilot_pagination"
def _index_document(
base_url: str, index: str, doc_id: int, body: dict[str, int]
) -> None:
response = requests.put(f"{base_url}/{index}/_doc/{doc_id}", json=body, timeout=10)
response.raise_for_status()
def _refresh(base_url: str, index: str) -> None:
response = requests.post(f"{base_url}/{index}/_refresh", timeout=10)
response.raise_for_status()
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with ElasticSearchContainer("elasticsearch:8.11.0") as container:
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
base_url = f"http://{host}:{port}"
for i in range(10):
_index_document(base_url, INDEX, i, {"id": i})
_refresh(base_url, INDEX)
yield create_engine(f"elasticsearch+http://{host}:{port}/")
def test_ordered_limited_query_returns_correct_rows(engine: Engine) -> None:
"""
A plain LIMIT query, compiled and executed against a real instance.
Mocked tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can. No OFFSET here: Elasticsearch's SQL layer genuinely
doesn't support it (a protocol limitation, not a bug -- confirmed
against a real instance, which raises a parsing_exception on OFFSET).
ElasticSearchEngineSpec.supports_offset = False documents this already.
"""
with engine.connect() as conn:
rows = conn.execute(
text(f"SELECT id FROM {INDEX} ORDER BY id LIMIT 3") # noqa: S608
).fetchall()
assert [row.id for row in rows] == [0, 1, 2]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
ElasticSearchEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported field mappings
rather than a mocked Inspector.
"""
inspector = inspect(engine)
columns = ElasticSearchEngineSpec.get_columns(inspector, Table(INDEX))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = ElasticSearchEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
@@ -0,0 +1,146 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.firebird against a real Firebird instance, spun up
on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Firebird references a database *file* rather than a server-managed named
database -- the connection URI is `firebird://user:pass@host:port/<path>`,
where <path> is the path to a .fdb file on the server. This uses the
well-known `jacobalberty/firebird` image, which creates that file (per
FIREBIRD_DATABASE) under /firebird/data on first boot.
FirebirdEngineSpec sets `limit_method = LimitMethod.FETCH_MANY` with a
comment claiming Firebird "uses FIRST to limit" -- stale relative to the
modern sqlalchemy-firebird driver, which compiles real ROWS-based
pagination (confirmed via an offline dialect compile: `SELECT ... ROWS
4 + 1 TO 4 + 3`, correctly ordered, not a Trino-style bug). That staleness
affects what Superset's own query layer emits, not what this suite's
direct dialect-compilation check exercises.
Could not be verified against a real running instance in this
environment: `firebird-driver` is a pure-Python ctypes wrapper (its wheel
is `py3-none-any`, confirmed by downloading it directly) that dynamically
loads the native Firebird client library (`libfbclient`) from the host at
import time -- it doesn't bundle that library itself. This machine has no
Homebrew formula or straightforward install path for it. The container
itself was confirmed to start and pass its own healthcheck; CI installs
the `libfbclient2` system package separately (see
.github/workflows/testcontainers.yml) for the actual client-library
dependency this driver needs.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.firebird import FirebirdEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("firebird.driver")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import HealthcheckWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
PORT = 3050
PASSWORD = "masterkey" # noqa: S105 -- fixed test-fixture password, not a secret
DB_FILE = "test.fdb"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("jacobalberty/firebird")
container.with_exposed_ports(PORT)
container.with_env("ISC_PASSWORD", PASSWORD)
container.with_env("FIREBIRD_DATABASE", DB_FILE)
# The image logs nothing beyond a single startup banner line and never
# prints a distinct "ready" message -- it ships its own Docker
# HEALTHCHECK instead, confirmed via `docker ps` reporting (healthy).
container.waiting_for(HealthcheckWaitStrategy())
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(PORT)
eng = create_engine(
f"firebird://sysdba:{PASSWORD}@{host}:{port}//firebird/data/{DB_FILE}"
)
yield eng
# firebird-driver registers its own atexit handler that talks to
# the Firebird subsystem to shut it down cleanly. Without disposing
# here first, that handler fires at interpreter exit against a
# server the container has *already* torn down -- confirmed on
# real CI as a segfault (exit code 139) after both tests had
# already passed. Disposing while the server is still up lets the
# driver close out normally, so the later atexit call has nothing
# left to talk to.
eng.dispose()
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
FirebirdEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = FirebirdEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = FirebirdEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,117 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mariadb against a real MariaDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MariaDB is a MySQL fork implementing the same wire protocol: connects via
the plain "mysql" dialect with mysqlclient, same as vanilla MySQL, just
pointed at the mariadb image instead of mysql:latest.
Could not be verified locally in this environment: mysqlclient (MySQLdb)
has a pre-existing, unrelated native-library linking issue against this
machine's Homebrew-installed libmysqlclient. CI installs it via apt on
Linux, where this does not occur.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mariadb import MariaDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mysql")
from testcontainers.community.mysql import MySqlContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MySqlContainer("mariadb:11") as container:
# get_connection_url() has no host override and defaults to
# get_container_host_ip(), which is the literal string "localhost"
# on native Linux Docker (e.g. GitHub Actions runners). MySQLdb
# (mysqlclient) treats a "localhost" host specially and attempts a
# Unix socket connection instead of TCP, which fails since there's
# no local MySQL socket -- the container is reached over the
# network. Only rewrite that specific local case to 127.0.0.1; a
# remote Docker daemon reports its own real host/IP here, which
# must be preserved so the suite can still reach it.
host = container.get_container_host_ip()
if host == "localhost":
host = "127.0.0.1"
port = container.get_exposed_port(container.port)
yield create_engine(
f"mysql://{container.username}:{container.password}"
f"@{host}:{port}/{container.dbname}"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MariaDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MariaDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MariaDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,127 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.monetdb against a real MonetDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
monetdb/monetdb publishes an amd64-only image, so this needs Rosetta/QEMU
emulation on Apple Silicon -- unlike CrateDB's x86-64-v3 CPU requirement,
this one actually runs fine under emulation (verified locally). No native
testcontainers module exists for MonetDB, so this uses a generic
DockerContainer with the documented MDB_* environment variables and waits
for the daemon's own startup log line.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.monetdb import MonetDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("sqlalchemy_monetdb")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import ( # noqa: E402
CompositeWaitStrategy,
LogMessageWaitStrategy,
PortWaitStrategy,
)
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
PORT = 50000
PASSWORD = "monetdb" # noqa: S105 -- fixed test-fixture password, not a secret
DBNAME = "test"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("monetdb/monetdb:latest")
container.with_exposed_ports(PORT)
container.with_env("MDB_DB_ADMIN_PASS", PASSWORD)
container.with_env("MDB_CREATE_DBS", DBNAME)
# The "Starting MonetDB daemon" log line is emitted before the image
# actually runs `monetdbd start -n`, so it alone isn't proof the server
# is accepting connections yet. Follow it with a port-connect check,
# which only succeeds once monetdbd is really listening.
container.waiting_for(
CompositeWaitStrategy(
LogMessageWaitStrategy(re.compile("Starting MonetDB daemon")),
PortWaitStrategy(PORT),
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(PORT)
yield create_engine(f"monetdb://monetdb:{PASSWORD}@{host}:{port}/{DBNAME}")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MonetDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MonetDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MonetDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,141 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mongodb against a real MongoDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MongoDB is schemaless, and Superset talks to it via `pymongosql`, a
SQL-to-MongoDB translation layer (dialect requires a `?mode=superset` query
param -- not part of testcontainers' own MongoDbContainer.get_connection_url()).
Documents get inserted via the native pymongo driver, not SQL INSERT,
matching how Superset actually encounters MongoDB in practice and avoiding
any assumption about pymongosql's own INSERT/DDL support.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
column,
create_engine,
inspect,
Integer,
select,
table,
text,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mongodb import MongoDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mongodb")
require_driver("pymongosql")
from testcontainers.community.mongodb import MongoDbContainer # noqa: E402
COLLECTION = "pilot_pagination"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MongoDbContainer("mongo:7.0.7") as container:
client = container.get_connection_client()
client[container.dbname][COLLECTION].insert_many([{"id": i} for i in range(10)])
# MongoDbContainer.get_connection_url() has no database path segment
# or query string at all (it only builds user:pass@host:port), so
# naively appending "&mode=superset" glues it straight onto the port
# number instead of starting a query string. Build the full URL
# ourselves instead of relying on string concatenation.
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
# authSource=admin is required: MongoDbContainer creates its root
# user via MONGO_INITDB_ROOT_USERNAME, which lives in the `admin`
# database, not in `dbname` -- without it, auth fails against
# whatever database is in the URL path.
yield create_engine(
f"mongodb://{container.username}:{container.password}@{host}:{port}"
f"/{container.dbname}?mode=superset&authSource=admin"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed
against a real instance. Mocked tests cannot catch a dialect compiling
this incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can. Unlike Elasticsearch's SQL
layer (which has no OFFSET support at all), pymongosql maps OFFSET to
MongoDB's native `skip`, so this dialect supports it.
Compiled with `literal_binds=True`, matching how Superset actually
issues chart/SQL Lab queries (see `models/helpers.py`'s
`get_query_str_extended`): pymongosql's SQL-to-Mongo AST parser reads
LIMIT/OFFSET straight off the compiled SQL text ahead of parameter
substitution, so a bound `LIMIT ?`/`OFFSET ?` placeholder is rejected
("invalid literal for int() with base 10: '?'") and the clause is
silently dropped -- unlike its WHERE-clause parameter handling, which
does substitute correctly. Literal binds sidestep that and exercise
the dialect's actual LIMIT/OFFSET compilation, per this test's intent.
Uses a bare `column("id")`/`table(...)` pair rather than a full
`Table`-bound column: SQLAlchemy always qualifies a Table-bound column
reference as `pilot_pagination.id` once there's a FROM clause, and
pymongosql's projection builder takes that qualified text completely
literally as a MongoDB field path -- `{"pilot_pagination.id": 1}` reads
a *nested* field under a top-level `pilot_pagination` key, which
doesn't exist on these flat documents, silently projecting None instead
of raising. An unbound column compiles unqualified ("id"), which
resolves correctly, while still exercising the dialect's own
LIMIT/OFFSET compilation via a real Core `select()`.
"""
id_col = column("id")
stmt = (
select(id_col)
.select_from(table(COLLECTION))
.order_by(id_col)
.limit(3)
.offset(4)
)
compiled = stmt.compile(engine, compile_kwargs={"literal_binds": True})
with engine.connect() as conn:
rows = conn.execute(text(str(compiled))).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MongoDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector, which
pymongosql implements by sampling real documents to infer column types
-- this exercises that against an actual running instance rather than
a mocked Inspector.
"""
inspector = inspect(engine)
columns = MongoDBEngineSpec.get_columns(inspector, Table(COLLECTION))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = MongoDBEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,93 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mssql against a real SQL Server instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
mcr.microsoft.com/mssql/server only publishes an amd64 image (SQL Server on
Linux has no ARM build), so this cannot run locally on an Apple Silicon
machine. It runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mssql import MssqlEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mssql")
from testcontainers.community.mssql import SqlServerContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with SqlServerContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MssqlEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MssqlEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MssqlEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,117 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mysql against a real MySQL instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Plain MySQL itself was never covered by this suite: MariaDB and StarRocks
both reuse `MySQLEngineSpec`'s plain "mysql" dialect via mysqlclient, but
neither stands in for vanilla MySQL server's own dialect quirks.
Could not be verified locally in this environment: mysqlclient (MySQLdb)
has a pre-existing, unrelated native-library linking issue against this
machine's Homebrew-installed libmysqlclient. CI installs it via apt on
Linux, where this does not occur.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mysql import MySQLEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mysql")
from testcontainers.community.mysql import MySqlContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MySqlContainer("mysql:8.0") as container:
# get_connection_url() has no host override and defaults to
# get_container_host_ip(), which is the literal string "localhost"
# on native Linux Docker (e.g. GitHub Actions runners). MySQLdb
# (mysqlclient) treats a "localhost" host specially and attempts a
# Unix socket connection instead of TCP, which fails since there's
# no local MySQL socket -- the container is reached over the
# network. Only rewrite that specific local case to 127.0.0.1; a
# remote Docker daemon reports its own real host/IP here, which
# must be preserved so the suite can still reach it.
host = container.get_container_host_ip()
if host == "localhost":
host = "127.0.0.1"
port = container.get_exposed_port(container.port)
yield create_engine(
f"mysql://{container.username}:{container.password}"
f"@{host}:{port}/{container.dbname}"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MySQLEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MySQLEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MySQLEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,221 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.oceanbase against a real OceanBase instance, spun up
on demand via testcontainers. Run via .github/workflows/testcontainers.yml,
on the nightly cron / manual dispatch only (see `nightly_only: true` on
this dialect's matrix entry) -- OceanBase bootstraps a distributed-style
cluster even in single-node MODE=MINI, a substantially heavier first-boot
than a single-process database, not a good fit for every PR's CI budget.
OceanBaseEngineSpec extends MySQLEngineSpec and its dialect
(oceanbase_py.sqlalchemy.dialect.OceanBaseDialect) extends
MySQLDialect_mysqldb directly with no custom DDL or LIMIT/OFFSET compiler,
so this follows the same mysqlclient-based pattern as MariaDB/MySQL/
StarRocks in this suite -- including the same "localhost" -> "127.0.0.1"
fix MySQLdb needs on native Linux Docker.
Could not be verified locally in this environment: mysqlclient (MySQLdb)
has a pre-existing, unrelated native-library linking issue against this
machine's Homebrew-installed libmysqlclient, and this dialect wasn't
pulled/run locally at all given its heavier resource footprint -- CI-only
verification, matching the nightly_only gating.
oceanbase_py.sqlalchemy.dialect.OceanBaseDialect has real bugs, all
confirmed on real CI, in both has_table() (called by create_all()'s
default checkfirst=True) and get_columns() (called by
OceanBaseEngineSpec.get_columns(), which this suite's second test needs
to actually exercise):
1. Both pass a raw string straight to Connection.execute() (e.g.
`connection.execute(f"DESCRIBE {full_name}")`), which SQLAlchemy 2.0
rejects outright (ObjectNotExecutableError). Every *other* raw-SQL
method in the same dialect module correctly uses
`connection.exec_driver_sql(...)` instead.
2. has_table() never catches the error DESCRIBE raises for a table that
doesn't exist (1146) -- so even with (1) fixed, it can only ever
return True, raising instead of returning False for exactly the case
checkfirst exists to handle.
This test monkeypatches both methods to do what the rest of the dialect's
raw-SQL methods already do, plus has_table()'s missing not-found
handling, rather than working around any of this from the test side.
"""
from collections.abc import Iterator
from typing import Any
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Connection, Engine, URL
from sqlalchemy.exc import NoSuchTableError, ProgrammingError
from superset.db_engine_specs.oceanbase import OceanBaseEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("oceanbase_py")
from oceanbase_py.sqlalchemy import datatype # noqa: E402
from oceanbase_py.sqlalchemy.dialect import OceanBaseDialect # noqa: E402
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
def _has_table(
self: OceanBaseDialect,
connection: Connection,
table_name: str,
schema: str | None = None,
**kw: object,
) -> bool:
if schema is None:
schema = self.default_schema_name
quote = self.identifier_preparer.quote_identifier
full_name = quote(table_name)
if schema:
full_name = f"{quote(schema)}.{full_name}"
try:
res = connection.exec_driver_sql(f"DESCRIBE {full_name}")
except ProgrammingError:
# The original never catches this at all -- DESCRIBE on a
# nonexistent table raises 1146 ("table doesn't exist") rather
# than returning an empty result set, so the unpatched method can
# only ever return True, and raises instead of returning False for
# exactly the case create_all()'s checkfirst exists to handle
# (confirmed on real CI).
return False
return res.first() is not None
def _get_columns(
self: OceanBaseDialect,
connection: Connection,
table_name: str,
schema: str | None = None,
**kw: object,
) -> list[dict[str, Any]]:
# Same connection.execute(raw string) anti-pattern as has_table(),
# confirmed on real CI as the same ObjectNotExecutableError -- this is
# the actual column-introspection call OceanBaseEngineSpec.get_columns
# (and this test) needs, so it gets the same exec_driver_sql fix.
if not self.has_table(connection, table_name, schema):
raise NoSuchTableError(f"schema={schema}, table={table_name}")
schema = schema or self._get_default_schema_name(connection)
quote = self.identifier_preparer.quote_identifier
full_name = quote(table_name)
if schema:
full_name = f"{quote(schema)}.{full_name}"
res = connection.exec_driver_sql(f"SHOW COLUMNS FROM {full_name}")
return [
{
"name": record.Field,
"type": datatype.parse_sql_type(record.Type),
"nullable": record.Null == "YES",
"default": record.Default,
}
for record in res
]
OceanBaseDialect.has_table = _has_table
OceanBaseDialect.get_columns = _get_columns
PORT = 2881
PASSWORD = "pilot" # noqa: S105 -- fixed test-fixture password, not a secret
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("oceanbase/oceanbase-ce")
container.with_exposed_ports(PORT)
container.with_env("MODE", "MINI")
container.with_env("OB_TENANT_PASSWORD", PASSWORD)
container.waiting_for(LogMessageWaitStrategy("boot success!"))
with container:
host = container.get_container_host_ip()
if host == "localhost":
host = "127.0.0.1"
port = container.get_exposed_port(PORT)
# OceanBase usernames for a MySQL-mode tenant use "user@tenant"
# (e.g. "root@test"), a literal "@" that URL.create() percent-encodes
# correctly -- an f-string would produce a second "@" that breaks
# the URL's own host/user boundary parsing.
yield create_engine(
URL.create(
"oceanbase",
username="root@test",
password=PASSWORD,
host=host,
port=int(port),
database="test",
)
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
OceanBaseEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = OceanBaseEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = OceanBaseEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.oracle against a real Oracle instance (gvenzl/oracle-free),
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
gvenzl/oracle-free ships with its datafiles pre-baked into the image, so
once the (large-ish, ~1GB) image is pulled, container startup is fast --
under 15s measured locally. Almost all the wall-clock cost here is the
image pull itself, same as any other dialect's container.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.oracle import OracleEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.oracle")
from testcontainers.community.oracle import OracleDbContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with OracleDbContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
OracleEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = OracleEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = OracleEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,99 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.postgres against a real PostgreSQL instance, spun up
on demand via testcontainers. Run via .github/workflows/testcontainers.yml
-- these exercise real SQL execution and dialect introspection, which
mocked unit tests structurally cannot.
Plain Postgres itself was never covered by this suite: CockroachDB,
TimescaleDB and YugabyteDB all speak the Postgres wire protocol and already
exercise `PostgresContainer`/the "postgresql" dialect, but none of them
stand in for vanilla PostgreSQL's own dialect quirks.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.postgres import PostgresEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.postgres")
from testcontainers.community.postgres import PostgresContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with PostgresContainer("postgres:17-alpine") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
PostgresEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = PostgresEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = PostgresEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,127 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.risingwave against a real RisingWave instance, spun
up on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
RisingWave speaks the Postgres wire protocol, but doesn't run the real
Postgres server binary or its POSTGRES_PASSWORD-style bootstrap env vars,
so this can't reuse `PostgresContainer` the way TimescaleDB/YugabyteDB do
-- it needs a generic DockerContainer against the official
`risingwavelabs/risingwave` single-binary playground image instead.
`RisingWaveDbEngineSpec` extends `PostgresEngineSpec`, and
`sqlalchemy-risingwave`'s dialect is a genuine subclass of SQLAlchemy's own
Postgres dialect (via psycopg2), so DDL/pagination compile with standard
Postgres semantics -- no ClickHouse-style mandatory table option needed.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
text,
)
from sqlalchemy.engine import Connection, Engine
from superset.db_engine_specs.risingwave import RisingWaveDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("sqlalchemy_risingwave")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
PORT = 4566
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("risingwavelabs/risingwave")
container.with_exposed_ports(PORT)
container.with_command("playground")
# The actual startup banner reads "RisingWave standalone mode is
# ready." -- confirmed against a real container's logs.
container.waiting_for(
LogMessageWaitStrategy(re.compile("RisingWave standalone mode is ready"))
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(PORT)
yield create_engine(f"risingwave://root@{host}:{port}/dev")
def _flush(conn: Connection) -> None:
# RisingWave's storage engine checkpoints asynchronously: without an
# explicit FLUSH, a SELECT immediately after INSERT can see zero rows
# -- confirmed against a real instance (a bare INSERT commits fine, but
# the data isn't visible to a subsequent query until flushed).
conn.execute(text("FLUSH"))
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine, after_insert=_flush)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
RisingWaveDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = RisingWaveDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = RisingWaveDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,156 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.starrocks against a real StarRocks instance, spun up
on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
StarRocks has no dedicated testcontainers module, so this uses a generic
DockerContainer against the official `starrocks/allin1-ubuntu` image, which
brings up both the FE (query frontend, MySQL wire protocol on port 9030)
and BE (execution backend) in a single container -- a heavier bring-up than
a single-process database. `root` has no password by default and no
database exists yet, so the fixture creates one itself before yielding an
engine pointed at it. The FE's query port accepts connections, and can even
run metadata statements like CREATE DATABASE, before the BE has registered
with it -- an actual CREATE TABLE/INSERT then fails with "Backend node not
found" -- so the fixture retries a real create-table-and-insert probe
against a throwaway table rather than trusting the open port or a bare
CREATE DATABASE as a readiness signal.
Not verified locally in this environment: the `allin1-ubuntu` image is
multiple GB and was skipped here to keep local Docker resource usage low,
per session guidance to lean on CI (which has no such constraint) for
dialects with unusually heavy images.
"""
import time
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
text,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.starrocks import StarRocksEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("starrocks")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import PortWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
QUERY_PORT = 9030
DBNAME = "pilot"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("starrocks/allin1-ubuntu")
container.with_exposed_ports(QUERY_PORT)
container.waiting_for(PortWaitStrategy(QUERY_PORT))
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(QUERY_PORT)
bootstrap_engine = create_engine(
f"starrocks://root:@{host}:{port}/default_catalog.information_schema"
)
# The FE's query port accepts connections, and can even run metadata
# statements like CREATE DATABASE, before any BE (execution backend)
# has registered with it -- an actual table create/insert then fails
# with "Backend node not found". Probe with the real operations the
# tests below need, in a throwaway table, so readiness is confirmed
# for what actually matters rather than just the FE's own port.
last_error: Exception | None = None
for _ in range(60):
try:
with bootstrap_engine.begin() as conn:
conn.execute(text(f"CREATE DATABASE IF NOT EXISTS {DBNAME}"))
probe_engine = create_engine(
f"starrocks://root:@{host}:{port}/default_catalog.{DBNAME}"
)
with probe_engine.begin() as conn:
conn.execute(
text("CREATE TABLE IF NOT EXISTS pilot_ready (id INT)")
)
conn.execute(text("INSERT INTO pilot_ready VALUES (1)"))
conn.execute(text("DROP TABLE pilot_ready"))
break
except Exception as ex: # noqa: BLE001 -- retry on any not-ready-yet error
last_error = ex
time.sleep(2)
else:
raise RuntimeError(
"StarRocks FE/BE never became ready to create and use a table"
) from last_error
yield create_engine(f"starrocks://root:@{host}:{port}/default_catalog.{DBNAME}")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
StarRocksEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = StarRocksEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = StarRocksEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,98 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.timescaledb against a real TimescaleDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
TimescaleDB is a genuine Postgres extension, not a fork: connects via the
plain "postgresql" dialect with psycopg2, same as vanilla Postgres, just
pointed at the timescale/timescaledb image instead of postgres:latest.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.timescaledb import TimescaleDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.postgres")
from testcontainers.community.postgres import PostgresContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with PostgresContainer("timescale/timescaledb:2.29.2-pg16") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TimescaleDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TimescaleDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TimescaleDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,104 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.trino against a real Trino instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Only Presto is covered by existing docker-compose-based integration CI;
Trino, despite sharing lineage with Presto, is not.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.trino import TrinoEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.trino")
from testcontainers.community.trino import TrinoContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with TrinoContainer() as container:
# TrinoContainer.get_connection_url() (testcontainers 4.15.0) returns
# the container-internal port (e.g. 8080) instead of the Docker-
# mapped host port, so the URL it builds cannot actually connect.
# Build it manually with get_exposed_port() instead. Filed upstream:
# https://github.com/testcontainers/testcontainers-python/issues
url = (
f"trino://{container.user}@{container.get_container_host_ip()}"
f":{container.get_exposed_port(container.port)}/memory/default"
)
yield create_engine(url)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. This is the exact bug class in apache/superset#42899,
where Trino emitted OFFSET before LIMIT for paginated queries -- a
dialect-compiler bug invisible to mocked tests, only catchable by
actually executing the compiled SQL.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TrinoEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TrinoEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TrinoEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,148 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.ydb against a real YDB instance, spun up on demand
via testcontainers. Run via .github/workflows/testcontainers.yml.
YDB has no dedicated testcontainers module, so this uses a generic
DockerContainer against the official `ydbplatform/local-ydb` image, which
needs no auth for local/anonymous access -- YDBEngineSpec's own
`sqlalchemy_uri_placeholder` ("ydb://{host}:{port}/{database_name}") has
no username/password at all.
"""
import time
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.ydb import YDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("ydb_sqlalchemy")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import PortWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
GRPC_PORT = 2136
DATABASE = "/local"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("ydbplatform/local-ydb")
container.with_exposed_ports(GRPC_PORT)
# YDB's gRPC client does endpoint discovery: it asks the server for its
# "real" endpoints and reconnects to whatever comes back, rather than
# just using the address it was originally given. By default that's
# the container's own internal Docker hostname (e.g. "6abbb4bb0ab7"),
# which isn't reachable from the host. Binding the same port number on
# the host as inside the container, plus advertising "localhost" as
# the container's own hostname, makes the discovered endpoint
# ("localhost:2136") actually resolve to something reachable.
container.with_bind_ports(GRPC_PORT, GRPC_PORT)
container.with_kwargs(hostname="localhost")
container.with_env("YDB_USE_IN_MEMORY_PDISKS", "true")
container.waiting_for(PortWaitStrategy(GRPC_PORT))
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(GRPC_PORT)
eng = create_engine(f"yql://{host}:{port}{DATABASE}")
# The gRPC port opens, and even a bare SELECT succeeds, before YDB's
# storage pools are fully initialized -- an actual CREATE TABLE can
# still fail with "database doesn't have storage pools at all to
# create tablet channels" (confirmed on a real instance). Probe with
# metadata.create_all()/drop_all() specifically, the same call the
# real tests below make: a raw `text("CREATE TABLE ...")` hits a
# separate, unrelated error ("Scheme operations cannot be executed
# inside transaction") that create_all()'s own DDL execution path
# doesn't, even with AUTOCOMMIT set on a manually-opened connection.
probe_metadata = MetaData()
SATable("pilot_ready", probe_metadata, Column("id", Integer, primary_key=True))
last_error: Exception | None = None
for _ in range(30):
try:
probe_metadata.create_all(eng)
probe_metadata.drop_all(eng)
break
except Exception as ex: # noqa: BLE001 -- retry on any not-ready-yet error
last_error = ex
time.sleep(2)
else:
raise RuntimeError(
"YDB never became ready to create and use a table"
) from last_error
yield eng
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
YDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = YDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = YDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,119 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.yugabytedb against a real YugabyteDB instance, spun
up on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
YugabyteDB's YSQL layer is fully Postgres-wire compatible (postgresql+
psycopg2, port 5433), but the image doesn't ship a plain `psql` binary
(only its own `ysqlsh`), so testcontainers' PostgresContainer can't be
reused directly -- its built-in readiness check execs `psql`, which would
fail here. This uses a generic DockerContainer, starting the node via
`yugabyted start --background=false` and waiting for yugabyted's own final
startup message instead.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.yugabytedb import YugabyteDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
YSQL_PORT = 5433
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("yugabytedb/yugabyte:latest")
container.with_exposed_ports(YSQL_PORT)
container.with_command("bin/yugabyted start --background=false")
container.waiting_for(
LogMessageWaitStrategy(
re.compile("Data placement constraint successfully verified")
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(YSQL_PORT)
# Default single-node credentials/database, per yugabyted's own
# documented quickstart defaults -- no env vars needed to set them.
yield create_engine(
f"postgresql+psycopg2://yugabyte:yugabyte@{host}:{port}/yugabyte"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
YugabyteDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = YugabyteDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = YugabyteDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -137,6 +137,56 @@ def test_get_available_engine_specs_keeps_valid_third_party_dialect(
assert available[SqliteEngineSpec] == {"valid_driver"}
def test_get_available_engine_specs_restores_compiler_operators(
mocker: MockerFixture,
) -> None:
"""
A third-party ``sqlalchemy.dialects`` entry point that mutates SQLAlchemy's
shared, process-global ``compiler.OPERATORS`` mapping on import (as
``sqlalchemy-monetdb`` does, in place, rather than subclassing) must not be
allowed to leak that change into every other dialect for the rest of the
process.
Regression test: enumerating a real "monetdb" entry point here (to build the
"available databases" list) silently changed ``!=`` rendering to ``<>`` for
postgres/mysql/sqlite/etc. too, for the remainder of the process.
"""
from sqlalchemy.sql import compiler as sqla_compiler, operators
mocker.patch(
"superset.db_engine_specs.load_engine_specs",
return_value=iter([]),
)
pristine = dict(sqla_compiler.OPERATORS)
assert pristine[operators.ne] != " <> "
class MisbehavingDialect(DefaultDialect):
name = "misbehaving"
driver = "misbehaving_driver"
def load_and_mutate_globally() -> type[MisbehavingDialect]:
# Mirrors sqlalchemy-monetdb's `base.py`: grabs a reference to the
# shared dict (not a copy) and mutates it in place.
sqla_compiler.OPERATORS[operators.ne] = " <> "
return MisbehavingDialect
entry_point = mocker.MagicMock()
entry_point.name = "misbehaving"
entry_point.load.side_effect = load_and_mutate_globally
mocker.patch(
"superset.db_engine_specs.entry_points",
return_value=[entry_point],
)
try:
get_available_engine_specs()
assert sqla_compiler.OPERATORS[operators.ne] == pristine[operators.ne]
finally:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(pristine)
@pytest.mark.parametrize(
"app",
[{"DBS_AVAILABLE_DENYLIST": {"databricks": {"pyhive", "pyodbc"}}}],