Compare commits

...
Author SHA1 Message Date
Superset Dev 75044575a1 fix(testcontainers): avoid table-qualified columns in mongodb pagination test
The previous fix (positional row access) only masked the symptom.
Root cause: SQLAlchemy always qualifies a Table-bound column as
"pilot_pagination.id" once there's a FROM clause, but pymongosql's
projection builder takes that qualified text completely literally as a
MongoDB field path -- {"pilot_pagination.id": 1} reads a *nested* field
under a top-level "pilot_pagination" key, which doesn't exist on these
flat documents, silently projecting None for every row instead of
raising (confirmed via pymongosql/sql/query_handler.py's
_extract_field_and_alias, which uses the raw expression text verbatim
as the Mongo field key regardless of any AS alias).

A bare column()/table() pair (not bound to each other via a real Table
object) compiles to an unqualified "id" reference, which resolves
correctly, while still exercising the dialect's own LIMIT/OFFSET
compilation through a real Core select() -- this test's actual intent.
2026-08-28 10:47:11 -07:00
Superset Dev 4028ff1e05 fix(testcontainers): use positional row access in mongodb pagination test
text() has no static column metadata of its own -- the row's key comes
entirely from whatever pymongosql's DBAPI cursor reports for the raw
compiled SQL string, which turned out to be the qualified
"pilot_pagination.id" (matching the SELECT list's column reference), not
the bare "id" that attribute access expected. Reproduced by compiling the
exact statement offline and inspecting text()'s (empty) column metadata;
confirmed the AttributeError on real CI runs of this branch and #43640.
2026-08-28 10:38:11 -07:00
rusackasandClaude Opus 4.8 67a193f1d9 fix(testcontainers): stop third-party dialect entry points from leaking global compiler state
sqlalchemy-monetdb mutates SQLAlchemy's shared, process-global
compiler.OPERATORS mapping in place on import instead of subclassing it
(OPERATORS = compiler.OPERATORS; OPERATORS[operators.ne] = " <> "). Once
Superset's own get_available_engine_specs() enumerates the "monetdb"
sqlalchemy.dialects entry point to build the "available databases" list
(which happens on every app boot, not just when MonetDB is actually
used), that changes `!=` rendering to `<>` for every dialect for the
rest of the process -- breaking test_where_operators identically across
the test-mysql/test-postgres/test-sqlite CI jobs, since installing the
monetdb extra now pulls sqlalchemy-monetdb into their shared dev
requirements.

Snapshot/restore sqlalchemy.sql.compiler.OPERATORS around each
third-party dialect entry point load so a misbehaving connector can't
leak global compiler state just because it was enumerated here.

Also fix the new mongodb testcontainers pagination test: pymongosql's
SQL-to-Mongo AST parser reads LIMIT/OFFSET off the compiled SQL text
ahead of parameter substitution, so a bound `LIMIT ?`/`OFFSET ?`
placeholder is silently dropped. Compile with literal_binds=True instead,
matching how Superset actually issues chart/SQL Lab queries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 05:56:24 -07:00
rusackasandClaude Opus 4.8 cd2638e27c fix(testcontainers): address bot + reviewer feedback on db2 baseline, mariadb host, mongodb pagination test, weak type assertions, monetdb readiness, and CI path coverage
- drop `db2` from the `testcontainers[...]` extra in development.in/.txt:
  it transitively pulls `ibm-db-sa`/`ibm-db` into the baseline dev lockfile,
  which has no Linux arm64 wheel and breaks the multi-platform dev Docker
  image build; the db2 CI leg already installs it on demand separately.
- mariadb: preserve a remote Docker daemon's real host, only rewrite the
  literal "localhost" case to 127.0.0.1.
- testcontainers.yml: widen pull_request paths to superset/db_engine_specs/**,
  pyproject.toml, and the requirements manifests so a driver/lockfile-only
  change can't bypass this coverage.
- mongodb: use Core `select(...).limit().offset()` instead of a literal SQL
  string, so the pagination test actually exercises the dialect's own
  compilation.
- monetdb/mongodb/yugabytedb: assert generic_type/sqla_type on the mapped
  column spec, not just that a spec was returned.
- monetdb: wait for the exposed port (not just a log line that predates
  `monetdbd start -n`) before considering the container ready.
- timescaledb: pin the image to a specific tag instead of the mutable
  `latest-pg16`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 02:35:46 -07:00
Superset Dev 58b37d0fa6 fix(testcontainers): disable implicit autoincrement on pagination test's id column
A single-column integer primary key implicitly becomes AUTO_INCREMENT on
MySQL/MariaDB. That column type treats an explicit 0 as NULL by default,
so inserting id=0 got silently reassigned to 1, colliding with the
explicit id=1 row in the same batch insert -- reproduced locally against
a real mariadb:11 container and confirmed fixed with autoincrement=False.

Supersedes the previous (incorrect) guess that a leftover table from an
earlier run was the cause.
2026-08-27 23:13:20 -07:00
Superset Dev 782104964e fix(testcontainers): drop stale pilot_pagination table before recreating
create_all() defaults to checkfirst=True, silently skipping creation (and
leaving old rows behind) if the table already exists from a prior run,
which collided with the fresh id=0-9 insert on primary key for mariadb.
2026-08-27 23:01:27 -07:00
Superset Dev 24e1c544ad fix(testcontainers): force 127.0.0.1 for mariadb, not localhost
MySqlContainer.get_connection_url() has no host override and defaults to
get_container_host_ip(), which returns the literal string "localhost" on
native Linux Docker (e.g. GitHub Actions runners). MySQLdb (mysqlclient)
treats a "localhost" host specially and attempts a Unix socket connection
instead of TCP, which fails since there's no local MySQL socket -- the
container is only reachable over the network. Confirmed on a real GHA
run: "Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock'".

This was never caught locally because of the separate, unrelated
mysqlclient linking issue on this dev machine, which crashed before ever
reaching a real connection attempt.
2026-08-27 22:50:55 -07:00
Superset Dev 1333ed12c7 feat(ci): expand testcontainers coverage to mariadb, timescaledb, yugabytedb, monetdb, mongodb
Adds five more dialects to the testcontainers suite, stacked on top of the
7-dialect pilot in feat/testcontainers-nightly-pilot: mariadb, timescaledb,
yugabytedb, monetdb, mongodb.

mariadb, timescaledb, and yugabytedb are all wire-compatible with an
existing base dialect (MySQL and Postgres respectively), so they reuse
testcontainers' MySqlContainer/PostgresContainer classes pointed at a
different image rather than needing new container-class wiring.
yugabytedb specifically cannot reuse PostgresContainer's built-in
readiness check, though: that execs `psql`, which the yugabyte image
doesn't ship (only its own `ysqlsh`) -- uses a generic DockerContainer
instead, started via `yugabyted start` and waiting on its own final
startup log line.

monetdb has no native testcontainers module; uses a generic DockerContainer
with the documented MDB_* environment variables. Publishes an amd64-only
image (confirmed running under Rosetta/QEMU emulation on Apple Silicon,
unlike CrateDB's harder x86-64-v3 CPU requirement).

mongodb needed a different data-setup approach, like elasticsearch before
it: documents get inserted via the native pymongo driver, not SQL INSERT,
since MongoDB is schemaless and Superset talks to it through pymongosql
(a SQL-to-MongoDB translation layer requiring a `?mode=superset` query
param). Two real bugs surfaced writing this one: testcontainers'
MongoDbContainer.get_connection_url() has no database path or query
string at all, so naively appending "&mode=superset" glued directly onto
the port number instead of starting a query string; and the root user
MongoDbContainer creates lives in the `admin` database, so connecting
with a different default database in the URL requires authSource=admin
or authentication fails outright. Confirmed pymongosql supports OFFSET
(maps to MongoDB's native `skip`), unlike Elasticsearch's SQL layer.

mariadb could not be verified locally in this environment: mysqlclient
(MySQLdb) has a pre-existing, unrelated native-library linking issue
against this machine's Homebrew-installed libmysqlclient. CI installs it
via apt on Linux, where this does not occur -- same accepted pattern
already used for crate/mssql/db2 in the base branch.
2026-08-27 22:44:32 -07:00
rusackasandClaude Opus 4.8 dff961a2f9 fix(deps): regenerate development.txt lockfile for ibm-db-sa
The db2 testcontainers extra pulls in ibm-db/ibm-db-sa transitively;
the pinned lockfile was missing those entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 03ebf2add4 fix(testcontainers): address remaining reviewer feedback (db2 arm64, cockroachdb DBAPI docs, stale workflow refs)
- requirements/development.in + testcontainers.yml: scope the `db2` extra
  (`ibm-db-sa`/`ibm-db`) out of the baseline dev install. `ibm-db` ships no
  Linux arm64 wheel, so bundling it there broke the multi-platform dev
  Docker image build on push. The testcontainers CI job now installs it
  directly, only for its own db2 matrix leg.
- cockroachdb.py: list `psycopg2-binary` alongside `sqlalchemy-cockroachdb`
  in the `pypi_packages` metadata, since a plain `cockroachdb://` URL can't
  connect without a DBAPI and sqlalchemy-cockroachdb doesn't install one.
- test_cockroachdb.py/test_crate.py/test_trino.py: fix stale docstring
  references to the old `nightly-testcontainers.yml` filename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 ce94f79a09 fix(testcontainers): address reviewer feedback on cockroachdb DBAPI, CI isolation, and test rigor
- pyproject.toml: pin psycopg2-binary alongside sqlalchemy-cockroachdb --
  the latter declares no DBAPI dependency of its own, so the documented
  `apache-superset[cockroachdb]` install couldn't actually connect.
- testcontainers.yml: scope the concurrency group by ref so a PR run and
  the nightly cron (or two different PRs) no longer cancel each other.
- test_cockroachdb.py: assert the actual generic/SQLAlchemy type, matching
  the Trino test, instead of only checking a column spec was found.
- pytest.ini + new `testcontainers` marker + _driver.py: exclude
  tests/testcontainers/ from a plain `pytest` run by default (it needs
  Docker), while the dedicated CI job now sets
  SUPERSET_TESTCONTAINERS_STRICT so a broken/missing driver import fails
  that job instead of silently skipping to a green, zero-tests-run result.
- UPDATING.md: the migration note now says to uninstall the old
  `cockroachdb` package outright, since reinstalling the extra alone can
  leave both packages registering the same dialect entry point.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:39 -07:00
rusackasandClaude Opus 4.8 80928bdb6c refactor(testcontainers): extract shared pagination-test body
The "paginated query returns correct rows in order" test body was
duplicated verbatim across cockroachdb/crate/db2/mssql/oracle/trino's
testcontainers suites. Factor the shared table-setup/assert logic into
_pagination.py, with an optional post-insert hook for CrateDB's
eventual-consistency REFRESH TABLE step; each call site keeps its own
test function and dialect-specific docstring.

Addresses bito-code-review feedback on #43502.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
rusackasandClaude Opus 4.8 6ce414928e test(db_engine_specs): fix test_test_connection_failed regression from mssql extra
Adding the mssql extra to development.in for testcontainers coverage
installs pymssql in the dev/test env, so the existing
mssql+pymssql://url probe in test_test_connection_failed no longer
hits the "driver not found" path it was meant to exercise -- it now
attempts a real (failing) connection instead. Swap it for a
dialect+driver URI whose dialect will never resolve to an installed
extra.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
Superset Dev f2d0b592c0 fix(cockroachdb): replace abandoned cockroachdb package with sqlalchemy-cockroachdb
The `cockroachdb` PyPI package (last released 2021) is abandoned, and its
SQLAlchemy dialect references sqlalchemy.dialects.postgresql.psycopg2's
PGCompiler_psycopg2, which SQLAlchemy 2.0 removed. Constructing a
cockroachdb:// engine has raised ImportError since the SQLAlchemy 2.0 bump
(#42803) landed, with nothing catching it: the driver isn't part of the
default dev/CI install, and the existing test_crdb.py only exercises
convert_dttm(), never a real engine.

Switches the `cockroachdb` extra to sqlalchemy-cockroachdb, the actively
maintained replacement already linked from CockroachDbEngineSpec's own
docs_url, adds it to the default dev install so this keeps getting
exercised, and adds a regression test that constructs a real engine
(verified to fail against the old package, pass against the new one).
2026-08-27 19:52:17 -07:00
Superset Dev 5a7fa2d1c6 feat(ci): expand testcontainers coverage to mssql, oracle, db2, elasticsearch
Adds four more dialects to the testcontainers suite (cockroachdb, crate,
trino from the initial pilot): mssql, oracle, db2, elasticsearch. All four
have native testcontainers-python container classes.

Restructures the workflow from one job running the whole suite to a
matrix, one job per dialect, running in parallel. A single slow container
would otherwise inflate wall-clock time for every dialect, not just its
own -- matrixing bounds total suite time by the slowest dialect instead of
the sum of all of them. Also renames the workflow file/name from
"Nightly-Testcontainers" to "Testcontainers" now that it runs on
pull_request (scoped via `paths`) in addition to the nightly cron.

Elasticsearch needed a different data-setup approach than the SQL-native
dialects: indices/documents get created via its REST API, not SQL INSERT,
matching how Superset actually encounters Elasticsearch in practice.
Confirmed empirically that Elasticsearch's SQL layer has no OFFSET support
at all (a real protocol limitation, already correctly documented via
ElasticSearchEngineSpec.supports_offset = False) and adjusted that
dialect's pagination test accordingly -- LIMIT/ORDER BY only, no OFFSET.

mssql and db2 could not be verified locally (no arm64 images for either;
this environment is Apple Silicon), same situation as crate's amd64-only
image from the initial pilot. Both are written against verified library
source (dialect names, connection URL construction) and will get their
first real execution on CI.
2026-08-27 19:51:23 -07:00
Superset Dev c0e32663a4 fix(deps): bump docker floor to 7.2.0 for testcontainers compatibility
docker==7.0.0 (the previously-unpinned floor) raises
`docker.errors.DockerException: ... Not supported URL scheme http+docker`
against the requests/urllib3 versions pinned elsewhere in this file --
confirmed on a real GHA run of #43502's nightly-testcontainers workflow,
where every test errored before any container was even pulled. 7.2.0 is
confirmed working, both locally and against a live container.
2026-08-27 19:51:23 -07:00
Superset Dev fa36e2565a temp: run nightly-testcontainers on this PR + add actions-timeline
Adds a scoped pull_request trigger so this workflow runs on this PR
itself, to get real GHA runner timing before deciding whether/how to
adopt this pattern more broadly. Also adds the actions-timeline job
(same pattern as superset-python-presto-hive.yml) to visualize per-step
duration.

Revert the pull_request trigger before merge -- it's here to measure
cost, not to become a permanent merge-blocking check.
2026-08-27 19:51:23 -07:00
rusackasandClaude Opus 4.8 eb91af8244 test(ci): assert mapped generic/sqla type in Trino column test
Address bot review feedback on #43502: the integration test only
checked that a ColumnSpec existed, not that INTEGER columns actually
mapped to the numeric generic type and an Integer sqla type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:51:23 -07:00
Superset Dev 051105098c feat(ci): add nightly testcontainers-based db_engine_specs tests
Pilot for real-container testing of db_engine_specs against actual
databases (CockroachDB, CrateDB, Trino), via testcontainers-python.
Existing unit tests mock the driver/dialect layer entirely, which cannot
catch real SQL-compilation or type-mapping bugs -- e.g. #42899, where
Trino emitted OFFSET before LIMIT for paginated queries.

Runs nightly (.github/workflows/nightly-testcontainers.yml), not on every
merge: container pulls and startup are slower and more flake-prone than
the existing mocked unit tests, and this measures that cost/signal
tradeoff before considering wider adoption.

Building this surfaced two real bugs, fixed/documented separately:
- The cockroachdb dialect was completely broken under SQLAlchemy 2.0 due
  to a dead upstream package -- fixed in #43501.
- testcontainers-python's TrinoContainer.get_connection_url() returns the
  container-internal port instead of the Docker-mapped host port; worked
  around locally, filed upstream.
2026-08-27 19:51:23 -07:00
26 changed files with 1757 additions and 16 deletions
+121
View File
@@ -0,0 +1,121 @@
# db_engine_specs tests against real databases (testcontainers)
name: Testcontainers
# Spins up real Docker containers (see tests/testcontainers/ for the current
# dialect list) via testcontainers-python, which catches real dialect/driver
# regressions -- the kind mocked db_engine_specs unit tests structurally
# cannot, e.g. apache/superset#42899 (Trino emitting OFFSET before LIMIT).
# Runs on a nightly cron (catches drift from a driver's own releases, not
# just from Superset's changes) and on pull_request, scoped via `paths` to
# only PRs that actually touch this test suite or the workflow itself, so
# unrelated PRs across the repo are never affected.
permissions:
contents: read
on:
schedule:
- cron: "0 5 * * *"
workflow_dispatch: {}
pull_request:
paths:
- ".github/workflows/testcontainers.yml"
- "tests/testcontainers/**"
- "superset/db_engine_specs/**"
- "pyproject.toml"
- "requirements/development.in"
- "requirements/development.txt"
concurrency:
# Scoped by ref, not just workflow name -- otherwise every PR run and the
# nightly cron share one group, and starting the workflow on another PR
# (or the nightly firing mid-PR-run) cancels an unrelated in-progress run.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
testcontainers:
runs-on: ubuntu-26.04
strategy:
fail-fast: false
matrix:
include:
# One job per dialect rather than one job for the whole suite: a
# single slow container would otherwise inflate the wall-clock
# time for every dialect, not just its own. Running in parallel
# means the suite's total time is bounded by the slowest dialect,
# not the sum of all of them. Db2's first-boot init is documented
# upstream as notably slow (a real instance bring-up, not just a
# process start) and untested locally here (no arm64 image), so
# it gets a wider timeout margin than the rest until real CI data
# says otherwise.
- dialect: cockroachdb
timeout: 10
- dialect: crate
timeout: 10
- dialect: trino
timeout: 10
- dialect: mssql
timeout: 10
- dialect: elasticsearch
timeout: 10
- dialect: oracle
timeout: 15
- dialect: db2
timeout: 25
- dialect: mariadb
timeout: 10
- dialect: timescaledb
timeout: 10
- dialect: yugabytedb
timeout: 10
- dialect: monetdb
timeout: 10
- dialect: mongodb
timeout: 10
timeout-minutes: ${{ matrix.timeout }}
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
# This job's matrix installs exactly one dialect's testcontainers
# driver for exactly this job, so treat that driver as required: a
# broken/missing import should fail the job, not silently skip to a
# misleadingly green, zero-tests-run result. See _driver.py.
SUPERSET_TESTCONTAINERS_STRICT: true
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
python-version: current
- name: Install db2 driver (ibm-db-sa)
# ibm-db (the db2 DBAPI) ships no Linux arm64 wheel, so it's kept out
# of the baseline dev install (requirements/development.in) to avoid
# breaking the multi-platform dev Docker image build. Install it here
# instead, only for this leg of the matrix.
if: matrix.dialect == 'db2'
run: uv pip install --system -e .[db2]
- name: Run testcontainers db_engine_specs tests (${{ matrix.dialect }})
run: |
pytest --durations-min=2 -v -m testcontainers \
./tests/testcontainers/db_engine_specs/test_${{ matrix.dialect }}.py \
--junit-xml=test-results/junit-testcontainers-${{ matrix.dialect }}.xml
- name: Upload JUnit test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: junit-results-testcontainers-${{ matrix.dialect }}
path: test-results/
retention-days: 7
actions-timeline:
needs: [testcontainers]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
+1 -1
View File
@@ -25,7 +25,7 @@ assists people when migrating to a new version.
## Next
- `SAMPLES_ROW_LIMIT` is now the default for `/datasource/samples` requests without a valid explicit `per_page`, rather than a hard per-request ceiling; explicit limits are honored up to the existing global row-limit ceiling, matching `/chart/data` SAMPLES requests.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed should `pip uninstall cockroachdb && pip install sqlalchemy-cockroachdb` (or simply reinstall the extra) to restore CockroachDB connectivity.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed must `pip uninstall cockroachdb` before reinstalling the extra -- both packages register the same `cockroachdb` SQLAlchemy dialect entry point, so leaving the old one in place can still load the abandoned implementation.
### MCP tool results preserve stored string values
+19 -5
View File
@@ -148,10 +148,9 @@ clickhouse = ["clickhouse-connect>=1.7.2, <2.0"]
# 2.0). sqlalchemy-cockroachdb is the actively maintained replacement,
# already linked from CockroachDbEngineSpec.metadata's docs_url, and
# registers the same `cockroachdb` SQLAlchemy dialect entry point.
# sqlalchemy-cockroachdb depends only on SQLAlchemy itself, not on a DBAPI
# driver, so psycopg2-binary is pinned alongside it (matching the `postgres`
# extra) to keep this extra self-contained -- CockroachDB speaks the
# PostgreSQL wire protocol, so psycopg2 is what actually opens connections.
# sqlalchemy-cockroachdb itself declares no DBAPI dependency (its own docs
# require picking one), so pull in the same psycopg2-binary pin as the
# `postgres` extra -- CockroachDB speaks the Postgres wire protocol.
cockroachdb = ["sqlalchemy-cockroachdb>=2.0.0, <3", "psycopg2-binary==2.9.12"]
crate = ["sqlalchemy-cratedb>=0.43.1, <1"]
# sqlalchemy-d1's only release (0.1.0, Nov 2025) pins sqlalchemy<2,>=1.4,
@@ -223,6 +222,11 @@ impala = ["impyla>=0.24.0, <0.25"]
# superset/db_engine_specs/kusto.py's known_incompatibilities metadata.
kusto = ["sqlalchemy-kusto>=3.1.2, <4"]
kylin = ["kylinpy>=2.8.4, <2.9"]
# MariaDB is a MySQL fork implementing the same wire protocol - connects via
# the plain mysql dialect, same driver as mysql.
mariadb = ["apache-superset[mysql]"]
monetdb = ["sqlalchemy-monetdb>=2.1.0, <3", "pymonetdb>=1.9.1, <2"]
mongodb = ["pymongosql>=0.7.3, <1"]
mssql = ["pymssql>=2.3.13, <3"]
# motherduck is an alias for duckdb - MotherDuck works via the duckdb driver
motherduck = ["apache-superset[duckdb]"]
@@ -271,6 +275,9 @@ tdengine = [
"taos-ws-py>=0.7.0"
]
teradata = ["teradatasql>=20.0.0.66"]
# TimescaleDB is a genuine Postgres extension, not a fork - connects via the
# plain postgresql dialect, same driver as postgres.
timescaledb = ["apache-superset[postgres]"]
thumbnails = [] # deprecated, will be removed in 7.0
vertica = ["sqlalchemy-vertica-python>= 0.6.3, < 0.7"]
netezza = ["nzalchemy>= 11.1.2, < 11.2"]
@@ -278,11 +285,18 @@ starrocks = ["starrocks>=1.3.4, <2"]
doris = ["pydoris>=1.2.0, <2.0.0"]
oceanbase = ["oceanbase_py>=0.0.1.2"]
ydb = ["ydb-sqlalchemy>=0.1.22", "ydb-sqlglot-plugin>=0.2.8"]
# YugabyteDB's YSQL layer is fully Postgres-wire compatible - connects via
# the plain postgresql dialect, same driver as postgres.
yugabytedb = ["apache-superset[postgres]"]
development = [
# no bounds for apache-superset-extensions-cli until a stable version
"apache-superset-extensions-cli",
"boto3",
"docker",
# 7.0.0 raises `docker.errors.DockerException: ... Not supported URL
# scheme http+docker` against the requests/urllib3 versions pinned
# elsewhere in this file -- breaks testcontainers (tests/testcontainers/)
# before any container even starts. 7.2.0 is confirmed working.
"docker>=7.2.0",
"flask-testing",
"freezegun",
"grpcio>=1.82.1",
+7 -1
View File
@@ -19,7 +19,13 @@ testpaths =
tests
python_files = *_test.py test_*.py *_tests.py *viz/utils.py
# `-p no:warnings` temporarily disabled in favor of more finely tuned `filterwarnings`.
#addopts = -p no:warnings
# `not testcontainers` excludes tests/testcontainers/ by default: those spin up
# real Docker containers, and `testpaths = tests` would otherwise pull them into
# every plain `pytest` run. The dedicated CI job (testcontainers.yml) overrides
# this with an explicit `-m testcontainers` to run them.
addopts = -m "not testcontainers"
markers =
testcontainers: exercises a real database via testcontainers-python (needs Docker); excluded by default, see .github/workflows/testcontainers.yml
asyncio_mode = auto
# `ignore` is effectively equivalent to `-p no:warnings`.
+15 -1
View File
@@ -16,5 +16,19 @@
# specific language governing permissions and limitations
# under the License.
#
-e .[development,bigquery,cockroachdb,druid,duckdb,fastmcp,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
-e .[development,bigquery,cockroachdb,crate,druid,duckdb,elasticsearch,fastmcp,gevent,gsheets,monetdb,mongodb,mssql,mysql,oracle,postgres,presto,prophet,trino,thumbnails]
-e ./superset-extensions-cli[test]
# testcontainers-backed db_engine_specs tests (tests/testcontainers/) --
# see .github/workflows/testcontainers.yml
#
# `db2` (the `ibm-db-sa`/`ibm-db` driver) is deliberately left out of the
# baseline dev install above: `ibm-db` ships no Linux arm64 wheel, so
# including it here breaks the multi-platform (amd64+arm64) dev Docker
# image build. The testcontainers CI job installs it on demand, only for
# the db2 matrix leg -- see .github/workflows/testcontainers.yml.
#
# mariadb/timescaledb/yugabytedb need no testcontainers extra of their own:
# they reuse the postgres/mysql container classes pointed at a different
# image, and psycopg2-binary/mysqlclient are already pulled in above via
# the postgres/mysql extras.
testcontainers[cockroachdb,cratedb,mongodb,mssql,mysql,oracle,postgres,trino]>=4.15.0,<5
+78 -5
View File
@@ -24,6 +24,8 @@ annotated-types==0.7.0
# via
# -c requirements/base-constraint.txt
# pydantic
antlr4-python3-runtime==4.13.2
# via pymongosql
anyio==4.11.0
# via
# httpx
@@ -117,8 +119,10 @@ celery==5.6.3
certifi==2026.5.20
# via
# -c requirements/base-constraint.txt
# elasticsearch
# httpcore
# httpx
# opensearch-py
# requests
cffi==2.0.0
# via
@@ -171,6 +175,8 @@ contourpy==1.0.7
# via matplotlib
coverage==7.6.8
# via pytest-cov
crate==2.2.1
# via sqlalchemy-cratedb
cron-descriptor==1.4.5
# via
# -c requirements/base-constraint.txt
@@ -186,8 +192,10 @@ cryptography==50.0.0
# authlib
# google-auth
# joserfc
# oracledb
# paramiko
# pyjwt
# pymysql
# pyopenssl
# secretstorage
cycler==0.12.1
@@ -216,8 +224,11 @@ dnspython==2.7.0
# via
# -c requirements/base-constraint.txt
# email-validator
docker==7.0.0
# via apache-superset
# pymongo
docker==7.2.0
# via
# apache-superset
# testcontainers
docstring-parser==0.17.0
# via cyclopts
docutils==0.22.2
@@ -228,6 +239,10 @@ duckdb==1.5.5
# duckdb-engine
duckdb-engine==0.17.0
# via apache-superset
elasticsearch==7.17.13
# via elasticsearch-dbapi
elasticsearch-dbapi==0.2.13
# via apache-superset
email-validator==2.2.0
# via
# -c requirements/base-constraint.txt
@@ -237,6 +252,8 @@ et-xmlfile==2.0.0
# via
# -c requirements/base-constraint.txt
# openpyxl
events==0.5
# via opensearch-py
exceptiongroup==1.3.0
# via fastmcp-slim
fastmcp==3.4.7
@@ -333,6 +350,8 @@ geographiclib==2.0
# via
# -c requirements/base-constraint.txt
# geopy
geojson==3.3.0
# via sqlalchemy-cratedb
geopy==2.4.1
# via
# -c requirements/base-constraint.txt
@@ -414,6 +433,7 @@ httpx==0.28.1
# via
# fastmcp-slim
# mcp
# testcontainers
httpx-sse==0.4.1
# via mcp
humanize==4.12.3
@@ -468,6 +488,7 @@ jmespath==1.1.0
# via
# boto3
# botocore
# pymongosql
joserfc==1.7.2
# via fastmcp-slim
jsonpath-ng==1.8.0
@@ -605,14 +626,22 @@ openpyxl==3.1.5
# via
# -c requirements/base-constraint.txt
# pandas
opensearch-py==2.8.0
# via elasticsearch-dbapi
opentelemetry-api==1.39.1
# via fastmcp-slim
oracledb==4.0.2
# via
# apache-superset
# testcontainers
ordered-set==4.1.0
# via
# -c requirements/base-constraint.txt
# flask-limiter
orjson==3.11.9
# via trino
# via
# crate
# trino
packaging==25.0
# via
# -c requirements/base-constraint.txt
@@ -620,8 +649,8 @@ packaging==25.0
# apispec
# db-dtypes
# deprecation
# docker
# duckdb-engine
# elasticsearch-dbapi
# fastmcp-slim
# google-cloud-bigquery
# kombu
@@ -775,6 +804,22 @@ pyjwt==2.13.0
# mcp
pylint==3.3.7
# via apache-superset
pymonetdb==1.9.1
# via
# apache-superset
# sqlalchemy-monetdb
pymongo==4.17.0
# via
# pymongosql
# testcontainers
pymongosql==0.7.3
# via apache-superset
pymssql==2.3.13
# via
# apache-superset
# testcontainers
pymysql==1.2.0
# via testcontainers
pynacl==1.6.2
# via
# -c requirements/base-constraint.txt
@@ -824,6 +869,7 @@ python-dateutil==2.9.0.post0
# google-cloud-bigquery
# holidays
# matplotlib
# opensearch-py
# pandas
# pyhive
# shillelagh
@@ -834,6 +880,7 @@ python-dotenv==1.2.2
# apache-superset
# fastmcp-slim
# pydantic-settings
# testcontainers
python-ldap==3.4.7
# via apache-superset
python-multipart==0.0.29
@@ -877,6 +924,7 @@ requests==2.33.0
# google-api-core
# google-cloud-bigquery
# jsonschema-path
# opensearch-py
# pydruid
# pyhive
# requests-cache
@@ -956,6 +1004,7 @@ sqlalchemy==2.0.52
# apache-superset
# apache-superset-core
# duckdb-engine
# elasticsearch-dbapi
# flask-appbuilder
# flask-sqlalchemy
# marshmallow-sqlalchemy
@@ -963,7 +1012,10 @@ sqlalchemy==2.0.52
# sqlalchemy-bigquery
# sqlalchemy-cockroachdb
# sqlalchemy-continuum
# sqlalchemy-cratedb
# sqlalchemy-monetdb
# sqlalchemy-utils
# testcontainers
sqlalchemy-bigquery==1.17.2
# via apache-superset
sqlalchemy-cockroachdb==2.0.4
@@ -972,6 +1024,12 @@ sqlalchemy-continuum==1.7.0
# via
# -c requirements/base-constraint.txt
# apache-superset
sqlalchemy-cratedb==0.43.1
# via
# apache-superset
# testcontainers
sqlalchemy-monetdb==2.1.0
# via apache-superset
sqlalchemy-utils==0.42.1
# via
# -c requirements/base-constraint.txt
@@ -1003,6 +1061,8 @@ tabulate==0.10.0
# via
# -c requirements/base-constraint.txt
# apache-superset
testcontainers==4.15.0
# via -r requirements/development.in
tiktoken==0.14.0
# via apache-superset
tomli-w==1.2.0
@@ -1014,7 +1074,9 @@ tqdm==4.67.1
# cmdstanpy
# prophet
trino==0.339.0
# via apache-superset
# via
# apache-superset
# testcontainers
typing-extensions==4.16.0
# via
# -c requirements/base-constraint.txt
@@ -1029,6 +1091,7 @@ typing-extensions==4.16.0
# limits
# mcp
# opentelemetry-api
# oracledb
# py-key-value-aio
# pydantic
# pydantic-core
@@ -1037,6 +1100,7 @@ typing-extensions==4.16.0
# shillelagh
# sqlalchemy
# starlette
# testcontainers
# typing-inspection
typing-inspection==0.4.2
# via
@@ -1064,13 +1128,21 @@ urllib3==2.7.0
# via
# -c requirements/base-constraint.txt
# botocore
# crate
# docker
# elasticsearch
# opensearch-py
# requests
# requests-cache
# testcontainers
uvicorn==0.37.0
# via
# fastmcp-slim
# mcp
verlib2==0.3.2
# via
# crate
# sqlalchemy-cratedb
vine==5.1.0
# via
# -c requirements/base-constraint.txt
@@ -1104,6 +1176,7 @@ wrapt==1.17.2
# via
# -c requirements/base-constraint.txt
# deprecated
# testcontainers
wtforms==3.2.2
# via
# -c requirements/base-constraint.txt
+15
View File
@@ -41,6 +41,7 @@ import sqlalchemy.dialects
from flask import current_app as app
from sqlalchemy.engine.default import DefaultDialect
from sqlalchemy.exc import NoSuchModuleError
from sqlalchemy.sql import compiler as sqla_compiler
from superset import feature_flag_manager
from superset.db_engine_specs.base import BaseEngineSpec
@@ -159,6 +160,16 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
continue
# installed 3rd-party dialects
#
# `ep.load()` runs arbitrary module-level code in the third-party package.
# Some dialects (e.g. sqlalchemy-monetdb) mutate SQLAlchemy's shared,
# process-global `compiler.OPERATORS` mapping in place on import instead
# of subclassing it, which would otherwise silently change SQL rendering
# (e.g. `!=` -> `<>`) for every dialect for the rest of the process, not
# just the misbehaving one. Snapshot/restore around each load so a
# buggy connector can't leak global compiler state into unrelated
# dialects just because it was enumerated here.
operators_snapshot = dict(sqla_compiler.OPERATORS)
for ep in entry_points(group="sqlalchemy.dialects"):
try:
dialect = ep.load()
@@ -193,6 +204,10 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
if isinstance(driver, bytes):
driver = driver.decode()
drivers[backend].add(driver)
finally:
if sqla_compiler.OPERATORS != operators_snapshot:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(operators_snapshot)
dbs_denylist = app.config["DBS_AVAILABLE_DENYLIST"]
if not feature_flag_manager.is_feature_enabled("ENABLE_SUPERSET_META_DB"):
+4 -1
View File
@@ -44,7 +44,10 @@ class CockroachDbEngineSpec(PostgresEngineSpec):
DatabaseCategory.TRADITIONAL_RDBMS,
DatabaseCategory.OPEN_SOURCE,
],
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2"],
# sqlalchemy-cockroachdb declares no DBAPI dependency of its own (see
# the comment on the `cockroachdb` extra in pyproject.toml), so a
# plain `cockroachdb://` URL also needs psycopg2 installed to connect.
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2-binary"],
"connection_string": "cockroachdb://root@{hostname}:{port}/{database}?sslmode=disable",
"default_port": 26257,
"docs_url": "https://github.com/cockroachdb/sqlalchemy-cockroachdb",
@@ -2487,8 +2487,15 @@ class TestDatabaseApi(SupersetTestCase):
}
assert response == expected_response
# Uses a `dialect+driver://` URI (rather than the bare `broken://`
# above) to also cover engine-name extraction stripping the driver
# suffix. The dialect itself ("broken") must stay one that no
# installed extra ever registers a real SQLAlchemy plugin for --
# this PR's own testcontainers extras (mssql, oracle, db2, ...)
# install real drivers for those dialects, which would make this
# URI actually attempt a connection instead of failing to load.
data = {
"sqlalchemy_uri": "mssql+pymssql://url",
"sqlalchemy_uri": "broken+driver://url",
"database_name": "examples",
"impersonate_user": False,
"server_cert": None,
@@ -2500,7 +2507,7 @@ class TestDatabaseApi(SupersetTestCase):
expected_response = {
"errors": [
{
"message": "Could not load database driver for: mssql",
"message": "Could not load database driver for: broken",
"error_type": "GENERIC_COMMAND_ERROR",
"level": "warning",
"extra": {
+16
View File
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared import guard for the per-dialect testcontainers modules
(tests/testcontainers/db_engine_specs/test_*.py), each of which needs its
own optional `testcontainers[...]` driver submodule to even import.
"""
import importlib
import os
def require_driver(module_name: str) -> None:
"""
Import `module_name`, a dialect's `testcontainers` driver submodule.
Most environments treat that driver as optional: a bare local `pytest`
run, or another CI job that never installed the `testcontainers` extras,
should skip the module rather than fail collection outright.
The dedicated per-dialect CI job (.github/workflows/testcontainers.yml)
sets SUPERSET_TESTCONTAINERS_STRICT, because there the driver is not
optional -- that job's matrix installs exactly this one driver for
exactly this one module. A broken or missing import there means the job
is misconfigured, and should fail loudly instead of silently reporting
a misleadingly green, zero-tests-run result.
"""
if os.environ.get("SUPERSET_TESTCONTAINERS_STRICT"):
importlib.import_module(module_name)
else:
import pytest
pytest.importorskip(module_name)
@@ -0,0 +1,63 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared body for the "paginated query returns correct rows in order" test
that db_engine_specs.{cockroachdb,crate,db2,mssql,oracle,trino}'s
testcontainers suites each run against their own real instance: a plain
SQLAlchemy Core LIMIT/OFFSET query, compiled and executed for real. Mocked
tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can.
Each call site keeps its own test function (and dialect-specific docstring)
so failures still report against the right module; this only factors out
the identical table setup/assert body, via an optional post-insert hook for
dialects (CrateDB) that need one.
"""
from collections.abc import Callable
from sqlalchemy import Column, insert, Integer, MetaData, select, Table as SATable
from sqlalchemy.engine import Connection, Engine
def assert_paginated_query_returns_correct_rows_in_order(
engine: Engine,
after_insert: Callable[[Connection], None] | None = None,
) -> None:
metadata = MetaData()
t = SATable(
"pilot_pagination",
metadata,
# autoincrement=False: a single-column integer primary key otherwise
# implicitly becomes AUTO_INCREMENT on MySQL/MariaDB. That column
# type treats an explicit 0 as NULL by default (NO_AUTO_VALUE_ON_ZERO
# is off), so the id=0 row below would silently get auto-assigned 1,
# colliding with the explicit id=1 row in the same batch insert.
Column("id", Integer, primary_key=True, autoincrement=False),
)
metadata.create_all(engine)
with engine.begin() as conn:
conn.execute(insert(t), [{"id": i} for i in range(10)])
if after_insert is not None:
after_insert(conn)
with engine.connect() as conn:
stmt = select(t.c.id).order_by(t.c.id).limit(3).offset(4)
rows = conn.execute(stmt).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
@@ -0,0 +1,97 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.cockroachdb against a real CockroachDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.cockroachdb import CockroachDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cockroachdb")
from testcontainers.community.cockroachdb import CockroachDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
# sqlalchemy-cockroachdb registers its dialect under the plain
# "cockroachdb" name; the container's own default ("cockroachdb+psycopg2")
# matches the abandoned `cockroachdb` package instead (see #43501).
with CockroachDBContainer(dialect="cockroachdb") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CockroachDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CockroachDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CockroachDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,103 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.crate against a real CrateDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
crate/crate only publishes an amd64 image (no arm64 build), and requires a
host CPU supporting the x86-64-v3 instruction set -- QEMU emulation on
Apple Silicon cannot satisfy that, so this file cannot run locally on an
Apple Silicon machine even with `docker pull --platform linux/amd64`. It
runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Connection, Engine
from superset.db_engine_specs.crate import CrateEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cratedb")
from testcontainers.community.cratedb import CrateDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with CrateDBContainer() as container:
yield create_engine(container.get_connection_url())
def _refresh_pilot_pagination(conn: Connection) -> None:
# CrateDB is eventually consistent: a row is not guaranteed visible to
# subsequent selects immediately after insert.
conn.exec_driver_sql("REFRESH TABLE pilot_pagination")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(
engine, after_insert=_refresh_pilot_pagination
)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CrateEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CrateEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CrateEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.db2 against a real IBM Db2 instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
icr.io/db2_community/db2 only publishes amd64/ppc64le/s390x images (no
arm64 build), so this cannot run locally on an Apple Silicon machine. It
runs natively on GitHub Actions' x86_64 runners. Db2 is also a notably slow
starter (a full instance bring-up, not just a process start) -- expect this
module alone to take several minutes.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.db2 import Db2EngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.db2")
from testcontainers.community.db2 import Db2Container # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with Db2Container() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
Db2EngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = Db2EngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = Db2EngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,107 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.elasticsearch against a real Elasticsearch instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
Unlike the SQL-native dialects in this directory, Elasticsearch has no
CREATE TABLE / INSERT: indices and documents get created via its REST API
(elasticsearch-dbapi's SQLAlchemy dialect is read-focused, translating SQL
to the _sql endpoint), matching how Superset actually encounters
Elasticsearch in practice -- data arrives via ingestion tooling, not
through Superset itself.
"""
from collections.abc import Iterator
import pytest
import requests
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import Engine
from superset.db_engine_specs.elasticsearch import ElasticSearchEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.elasticsearch")
from testcontainers.community.elasticsearch import ElasticSearchContainer # noqa: E402
INDEX = "pilot_pagination"
def _index_document(
base_url: str, index: str, doc_id: int, body: dict[str, int]
) -> None:
response = requests.put(f"{base_url}/{index}/_doc/{doc_id}", json=body, timeout=10)
response.raise_for_status()
def _refresh(base_url: str, index: str) -> None:
response = requests.post(f"{base_url}/{index}/_refresh", timeout=10)
response.raise_for_status()
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with ElasticSearchContainer("elasticsearch:8.11.0") as container:
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
base_url = f"http://{host}:{port}"
for i in range(10):
_index_document(base_url, INDEX, i, {"id": i})
_refresh(base_url, INDEX)
yield create_engine(f"elasticsearch+http://{host}:{port}/")
def test_ordered_limited_query_returns_correct_rows(engine: Engine) -> None:
"""
A plain LIMIT query, compiled and executed against a real instance.
Mocked tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can. No OFFSET here: Elasticsearch's SQL layer genuinely
doesn't support it (a protocol limitation, not a bug -- confirmed
against a real instance, which raises a parsing_exception on OFFSET).
ElasticSearchEngineSpec.supports_offset = False documents this already.
"""
with engine.connect() as conn:
rows = conn.execute(
text(f"SELECT id FROM {INDEX} ORDER BY id LIMIT 3") # noqa: S608
).fetchall()
assert [row.id for row in rows] == [0, 1, 2]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
ElasticSearchEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported field mappings
rather than a mocked Inspector.
"""
inspector = inspect(engine)
columns = ElasticSearchEngineSpec.get_columns(inspector, Table(INDEX))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = ElasticSearchEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
@@ -0,0 +1,117 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mariadb against a real MariaDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MariaDB is a MySQL fork implementing the same wire protocol: connects via
the plain "mysql" dialect with mysqlclient, same as vanilla MySQL, just
pointed at the mariadb image instead of mysql:latest.
Could not be verified locally in this environment: mysqlclient (MySQLdb)
has a pre-existing, unrelated native-library linking issue against this
machine's Homebrew-installed libmysqlclient. CI installs it via apt on
Linux, where this does not occur.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mariadb import MariaDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mysql")
from testcontainers.community.mysql import MySqlContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MySqlContainer("mariadb:11") as container:
# get_connection_url() has no host override and defaults to
# get_container_host_ip(), which is the literal string "localhost"
# on native Linux Docker (e.g. GitHub Actions runners). MySQLdb
# (mysqlclient) treats a "localhost" host specially and attempts a
# Unix socket connection instead of TCP, which fails since there's
# no local MySQL socket -- the container is reached over the
# network. Only rewrite that specific local case to 127.0.0.1; a
# remote Docker daemon reports its own real host/IP here, which
# must be preserved so the suite can still reach it.
host = container.get_container_host_ip()
if host == "localhost":
host = "127.0.0.1"
port = container.get_exposed_port(container.port)
yield create_engine(
f"mysql://{container.username}:{container.password}"
f"@{host}:{port}/{container.dbname}"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MariaDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MariaDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MariaDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,127 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.monetdb against a real MonetDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
monetdb/monetdb publishes an amd64-only image, so this needs Rosetta/QEMU
emulation on Apple Silicon -- unlike CrateDB's x86-64-v3 CPU requirement,
this one actually runs fine under emulation (verified locally). No native
testcontainers module exists for MonetDB, so this uses a generic
DockerContainer with the documented MDB_* environment variables and waits
for the daemon's own startup log line.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.monetdb import MonetDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("sqlalchemy_monetdb")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import ( # noqa: E402
CompositeWaitStrategy,
LogMessageWaitStrategy,
PortWaitStrategy,
)
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
PORT = 50000
PASSWORD = "monetdb" # noqa: S105 -- fixed test-fixture password, not a secret
DBNAME = "test"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("monetdb/monetdb:latest")
container.with_exposed_ports(PORT)
container.with_env("MDB_DB_ADMIN_PASS", PASSWORD)
container.with_env("MDB_CREATE_DBS", DBNAME)
# The "Starting MonetDB daemon" log line is emitted before the image
# actually runs `monetdbd start -n`, so it alone isn't proof the server
# is accepting connections yet. Follow it with a port-connect check,
# which only succeeds once monetdbd is really listening.
container.waiting_for(
CompositeWaitStrategy(
LogMessageWaitStrategy(re.compile("Starting MonetDB daemon")),
PortWaitStrategy(PORT),
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(PORT)
yield create_engine(f"monetdb://monetdb:{PASSWORD}@{host}:{port}/{DBNAME}")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MonetDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MonetDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MonetDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,141 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mongodb against a real MongoDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MongoDB is schemaless, and Superset talks to it via `pymongosql`, a
SQL-to-MongoDB translation layer (dialect requires a `?mode=superset` query
param -- not part of testcontainers' own MongoDbContainer.get_connection_url()).
Documents get inserted via the native pymongo driver, not SQL INSERT,
matching how Superset actually encounters MongoDB in practice and avoiding
any assumption about pymongosql's own INSERT/DDL support.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
column,
create_engine,
inspect,
Integer,
select,
table,
text,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mongodb import MongoDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mongodb")
require_driver("pymongosql")
from testcontainers.community.mongodb import MongoDbContainer # noqa: E402
COLLECTION = "pilot_pagination"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MongoDbContainer("mongo:7.0.7") as container:
client = container.get_connection_client()
client[container.dbname][COLLECTION].insert_many([{"id": i} for i in range(10)])
# MongoDbContainer.get_connection_url() has no database path segment
# or query string at all (it only builds user:pass@host:port), so
# naively appending "&mode=superset" glues it straight onto the port
# number instead of starting a query string. Build the full URL
# ourselves instead of relying on string concatenation.
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
# authSource=admin is required: MongoDbContainer creates its root
# user via MONGO_INITDB_ROOT_USERNAME, which lives in the `admin`
# database, not in `dbname` -- without it, auth fails against
# whatever database is in the URL path.
yield create_engine(
f"mongodb://{container.username}:{container.password}@{host}:{port}"
f"/{container.dbname}?mode=superset&authSource=admin"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed
against a real instance. Mocked tests cannot catch a dialect compiling
this incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can. Unlike Elasticsearch's SQL
layer (which has no OFFSET support at all), pymongosql maps OFFSET to
MongoDB's native `skip`, so this dialect supports it.
Compiled with `literal_binds=True`, matching how Superset actually
issues chart/SQL Lab queries (see `models/helpers.py`'s
`get_query_str_extended`): pymongosql's SQL-to-Mongo AST parser reads
LIMIT/OFFSET straight off the compiled SQL text ahead of parameter
substitution, so a bound `LIMIT ?`/`OFFSET ?` placeholder is rejected
("invalid literal for int() with base 10: '?'") and the clause is
silently dropped -- unlike its WHERE-clause parameter handling, which
does substitute correctly. Literal binds sidestep that and exercise
the dialect's actual LIMIT/OFFSET compilation, per this test's intent.
Uses a bare `column("id")`/`table(...)` pair rather than a full
`Table`-bound column: SQLAlchemy always qualifies a Table-bound column
reference as `pilot_pagination.id` once there's a FROM clause, and
pymongosql's projection builder takes that qualified text completely
literally as a MongoDB field path -- `{"pilot_pagination.id": 1}` reads
a *nested* field under a top-level `pilot_pagination` key, which
doesn't exist on these flat documents, silently projecting None instead
of raising. An unbound column compiles unqualified ("id"), which
resolves correctly, while still exercising the dialect's own
LIMIT/OFFSET compilation via a real Core `select()`.
"""
id_col = column("id")
stmt = (
select(id_col)
.select_from(table(COLLECTION))
.order_by(id_col)
.limit(3)
.offset(4)
)
compiled = stmt.compile(engine, compile_kwargs={"literal_binds": True})
with engine.connect() as conn:
rows = conn.execute(text(str(compiled))).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MongoDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector, which
pymongosql implements by sampling real documents to infer column types
-- this exercises that against an actual running instance rather than
a mocked Inspector.
"""
inspector = inspect(engine)
columns = MongoDBEngineSpec.get_columns(inspector, Table(COLLECTION))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = MongoDBEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,93 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mssql against a real SQL Server instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
mcr.microsoft.com/mssql/server only publishes an amd64 image (SQL Server on
Linux has no ARM build), so this cannot run locally on an Apple Silicon
machine. It runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mssql import MssqlEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mssql")
from testcontainers.community.mssql import SqlServerContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with SqlServerContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MssqlEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MssqlEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MssqlEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.oracle against a real Oracle instance (gvenzl/oracle-free),
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
gvenzl/oracle-free ships with its datafiles pre-baked into the image, so
once the (large-ish, ~1GB) image is pulled, container startup is fast --
under 15s measured locally. Almost all the wall-clock cost here is the
image pull itself, same as any other dialect's container.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.oracle import OracleEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.oracle")
from testcontainers.community.oracle import OracleDbContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with OracleDbContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
OracleEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = OracleEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = OracleEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,98 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.timescaledb against a real TimescaleDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
TimescaleDB is a genuine Postgres extension, not a fork: connects via the
plain "postgresql" dialect with psycopg2, same as vanilla Postgres, just
pointed at the timescale/timescaledb image instead of postgres:latest.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.timescaledb import TimescaleDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.postgres")
from testcontainers.community.postgres import PostgresContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with PostgresContainer("timescale/timescaledb:2.29.2-pg16") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TimescaleDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TimescaleDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TimescaleDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,104 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.trino against a real Trino instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Only Presto is covered by existing docker-compose-based integration CI;
Trino, despite sharing lineage with Presto, is not.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.trino import TrinoEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.trino")
from testcontainers.community.trino import TrinoContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with TrinoContainer() as container:
# TrinoContainer.get_connection_url() (testcontainers 4.15.0) returns
# the container-internal port (e.g. 8080) instead of the Docker-
# mapped host port, so the URL it builds cannot actually connect.
# Build it manually with get_exposed_port() instead. Filed upstream:
# https://github.com/testcontainers/testcontainers-python/issues
url = (
f"trino://{container.user}@{container.get_container_host_ip()}"
f":{container.get_exposed_port(container.port)}/memory/default"
)
yield create_engine(url)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. This is the exact bug class in apache/superset#42899,
where Trino emitted OFFSET before LIMIT for paginated queries -- a
dialect-compiler bug invisible to mocked tests, only catchable by
actually executing the compiled SQL.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TrinoEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TrinoEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TrinoEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,119 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.yugabytedb against a real YugabyteDB instance, spun
up on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
YugabyteDB's YSQL layer is fully Postgres-wire compatible (postgresql+
psycopg2, port 5433), but the image doesn't ship a plain `psql` binary
(only its own `ysqlsh`), so testcontainers' PostgresContainer can't be
reused directly -- its built-in readiness check execs `psql`, which would
fail here. This uses a generic DockerContainer, starting the node via
`yugabyted start --background=false` and waiting for yugabyted's own final
startup message instead.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.yugabytedb import YugabyteDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
YSQL_PORT = 5433
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("yugabytedb/yugabyte:latest")
container.with_exposed_ports(YSQL_PORT)
container.with_command("bin/yugabyted start --background=false")
container.waiting_for(
LogMessageWaitStrategy(
re.compile("Data placement constraint successfully verified")
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(YSQL_PORT)
# Default single-node credentials/database, per yugabyted's own
# documented quickstart defaults -- no env vars needed to set them.
yield create_engine(
f"postgresql+psycopg2://yugabyte:yugabyte@{host}:{port}/yugabyte"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
YugabyteDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = YugabyteDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = YugabyteDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -137,6 +137,56 @@ def test_get_available_engine_specs_keeps_valid_third_party_dialect(
assert available[SqliteEngineSpec] == {"valid_driver"}
def test_get_available_engine_specs_restores_compiler_operators(
mocker: MockerFixture,
) -> None:
"""
A third-party ``sqlalchemy.dialects`` entry point that mutates SQLAlchemy's
shared, process-global ``compiler.OPERATORS`` mapping on import (as
``sqlalchemy-monetdb`` does, in place, rather than subclassing) must not be
allowed to leak that change into every other dialect for the rest of the
process.
Regression test: enumerating a real "monetdb" entry point here (to build the
"available databases" list) silently changed ``!=`` rendering to ``<>`` for
postgres/mysql/sqlite/etc. too, for the remainder of the process.
"""
from sqlalchemy.sql import compiler as sqla_compiler, operators
mocker.patch(
"superset.db_engine_specs.load_engine_specs",
return_value=iter([]),
)
pristine = dict(sqla_compiler.OPERATORS)
assert pristine[operators.ne] != " <> "
class MisbehavingDialect(DefaultDialect):
name = "misbehaving"
driver = "misbehaving_driver"
def load_and_mutate_globally() -> type[MisbehavingDialect]:
# Mirrors sqlalchemy-monetdb's `base.py`: grabs a reference to the
# shared dict (not a copy) and mutates it in place.
sqla_compiler.OPERATORS[operators.ne] = " <> "
return MisbehavingDialect
entry_point = mocker.MagicMock()
entry_point.name = "misbehaving"
entry_point.load.side_effect = load_and_mutate_globally
mocker.patch(
"superset.db_engine_specs.entry_points",
return_value=[entry_point],
)
try:
get_available_engine_specs()
assert sqla_compiler.OPERATORS[operators.ne] == pristine[operators.ne]
finally:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(pristine)
@pytest.mark.parametrize(
"app",
[{"DBS_AVAILABLE_DENYLIST": {"databricks": {"pyhive", "pyodbc"}}}],