Compare commits

..
Author SHA1 Message Date
Superset Dev 75044575a1 fix(testcontainers): avoid table-qualified columns in mongodb pagination test
The previous fix (positional row access) only masked the symptom.
Root cause: SQLAlchemy always qualifies a Table-bound column as
"pilot_pagination.id" once there's a FROM clause, but pymongosql's
projection builder takes that qualified text completely literally as a
MongoDB field path -- {"pilot_pagination.id": 1} reads a *nested* field
under a top-level "pilot_pagination" key, which doesn't exist on these
flat documents, silently projecting None for every row instead of
raising (confirmed via pymongosql/sql/query_handler.py's
_extract_field_and_alias, which uses the raw expression text verbatim
as the Mongo field key regardless of any AS alias).

A bare column()/table() pair (not bound to each other via a real Table
object) compiles to an unqualified "id" reference, which resolves
correctly, while still exercising the dialect's own LIMIT/OFFSET
compilation through a real Core select() -- this test's actual intent.
2026-08-28 10:47:11 -07:00
Superset Dev 4028ff1e05 fix(testcontainers): use positional row access in mongodb pagination test
text() has no static column metadata of its own -- the row's key comes
entirely from whatever pymongosql's DBAPI cursor reports for the raw
compiled SQL string, which turned out to be the qualified
"pilot_pagination.id" (matching the SELECT list's column reference), not
the bare "id" that attribute access expected. Reproduced by compiling the
exact statement offline and inspecting text()'s (empty) column metadata;
confirmed the AttributeError on real CI runs of this branch and #43640.
2026-08-28 10:38:11 -07:00
rusackasandClaude Opus 4.8 67a193f1d9 fix(testcontainers): stop third-party dialect entry points from leaking global compiler state
sqlalchemy-monetdb mutates SQLAlchemy's shared, process-global
compiler.OPERATORS mapping in place on import instead of subclassing it
(OPERATORS = compiler.OPERATORS; OPERATORS[operators.ne] = " <> "). Once
Superset's own get_available_engine_specs() enumerates the "monetdb"
sqlalchemy.dialects entry point to build the "available databases" list
(which happens on every app boot, not just when MonetDB is actually
used), that changes `!=` rendering to `<>` for every dialect for the
rest of the process -- breaking test_where_operators identically across
the test-mysql/test-postgres/test-sqlite CI jobs, since installing the
monetdb extra now pulls sqlalchemy-monetdb into their shared dev
requirements.

Snapshot/restore sqlalchemy.sql.compiler.OPERATORS around each
third-party dialect entry point load so a misbehaving connector can't
leak global compiler state just because it was enumerated here.

Also fix the new mongodb testcontainers pagination test: pymongosql's
SQL-to-Mongo AST parser reads LIMIT/OFFSET off the compiled SQL text
ahead of parameter substitution, so a bound `LIMIT ?`/`OFFSET ?`
placeholder is silently dropped. Compile with literal_binds=True instead,
matching how Superset actually issues chart/SQL Lab queries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 05:56:24 -07:00
rusackasandClaude Opus 4.8 cd2638e27c fix(testcontainers): address bot + reviewer feedback on db2 baseline, mariadb host, mongodb pagination test, weak type assertions, monetdb readiness, and CI path coverage
- drop `db2` from the `testcontainers[...]` extra in development.in/.txt:
  it transitively pulls `ibm-db-sa`/`ibm-db` into the baseline dev lockfile,
  which has no Linux arm64 wheel and breaks the multi-platform dev Docker
  image build; the db2 CI leg already installs it on demand separately.
- mariadb: preserve a remote Docker daemon's real host, only rewrite the
  literal "localhost" case to 127.0.0.1.
- testcontainers.yml: widen pull_request paths to superset/db_engine_specs/**,
  pyproject.toml, and the requirements manifests so a driver/lockfile-only
  change can't bypass this coverage.
- mongodb: use Core `select(...).limit().offset()` instead of a literal SQL
  string, so the pagination test actually exercises the dialect's own
  compilation.
- monetdb/mongodb/yugabytedb: assert generic_type/sqla_type on the mapped
  column spec, not just that a spec was returned.
- monetdb: wait for the exposed port (not just a log line that predates
  `monetdbd start -n`) before considering the container ready.
- timescaledb: pin the image to a specific tag instead of the mutable
  `latest-pg16`.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-28 02:35:46 -07:00
Superset Dev 58b37d0fa6 fix(testcontainers): disable implicit autoincrement on pagination test's id column
A single-column integer primary key implicitly becomes AUTO_INCREMENT on
MySQL/MariaDB. That column type treats an explicit 0 as NULL by default,
so inserting id=0 got silently reassigned to 1, colliding with the
explicit id=1 row in the same batch insert -- reproduced locally against
a real mariadb:11 container and confirmed fixed with autoincrement=False.

Supersedes the previous (incorrect) guess that a leftover table from an
earlier run was the cause.
2026-08-27 23:13:20 -07:00
Superset Dev 782104964e fix(testcontainers): drop stale pilot_pagination table before recreating
create_all() defaults to checkfirst=True, silently skipping creation (and
leaving old rows behind) if the table already exists from a prior run,
which collided with the fresh id=0-9 insert on primary key for mariadb.
2026-08-27 23:01:27 -07:00
Superset Dev 24e1c544ad fix(testcontainers): force 127.0.0.1 for mariadb, not localhost
MySqlContainer.get_connection_url() has no host override and defaults to
get_container_host_ip(), which returns the literal string "localhost" on
native Linux Docker (e.g. GitHub Actions runners). MySQLdb (mysqlclient)
treats a "localhost" host specially and attempts a Unix socket connection
instead of TCP, which fails since there's no local MySQL socket -- the
container is only reachable over the network. Confirmed on a real GHA
run: "Can't connect to local MySQL server through socket
'/var/run/mysqld/mysqld.sock'".

This was never caught locally because of the separate, unrelated
mysqlclient linking issue on this dev machine, which crashed before ever
reaching a real connection attempt.
2026-08-27 22:50:55 -07:00
Superset Dev 1333ed12c7 feat(ci): expand testcontainers coverage to mariadb, timescaledb, yugabytedb, monetdb, mongodb
Adds five more dialects to the testcontainers suite, stacked on top of the
7-dialect pilot in feat/testcontainers-nightly-pilot: mariadb, timescaledb,
yugabytedb, monetdb, mongodb.

mariadb, timescaledb, and yugabytedb are all wire-compatible with an
existing base dialect (MySQL and Postgres respectively), so they reuse
testcontainers' MySqlContainer/PostgresContainer classes pointed at a
different image rather than needing new container-class wiring.
yugabytedb specifically cannot reuse PostgresContainer's built-in
readiness check, though: that execs `psql`, which the yugabyte image
doesn't ship (only its own `ysqlsh`) -- uses a generic DockerContainer
instead, started via `yugabyted start` and waiting on its own final
startup log line.

monetdb has no native testcontainers module; uses a generic DockerContainer
with the documented MDB_* environment variables. Publishes an amd64-only
image (confirmed running under Rosetta/QEMU emulation on Apple Silicon,
unlike CrateDB's harder x86-64-v3 CPU requirement).

mongodb needed a different data-setup approach, like elasticsearch before
it: documents get inserted via the native pymongo driver, not SQL INSERT,
since MongoDB is schemaless and Superset talks to it through pymongosql
(a SQL-to-MongoDB translation layer requiring a `?mode=superset` query
param). Two real bugs surfaced writing this one: testcontainers'
MongoDbContainer.get_connection_url() has no database path or query
string at all, so naively appending "&mode=superset" glued directly onto
the port number instead of starting a query string; and the root user
MongoDbContainer creates lives in the `admin` database, so connecting
with a different default database in the URL requires authSource=admin
or authentication fails outright. Confirmed pymongosql supports OFFSET
(maps to MongoDB's native `skip`), unlike Elasticsearch's SQL layer.

mariadb could not be verified locally in this environment: mysqlclient
(MySQLdb) has a pre-existing, unrelated native-library linking issue
against this machine's Homebrew-installed libmysqlclient. CI installs it
via apt on Linux, where this does not occur -- same accepted pattern
already used for crate/mssql/db2 in the base branch.
2026-08-27 22:44:32 -07:00
rusackasandClaude Opus 4.8 dff961a2f9 fix(deps): regenerate development.txt lockfile for ibm-db-sa
The db2 testcontainers extra pulls in ibm-db/ibm-db-sa transitively;
the pinned lockfile was missing those entries.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 03ebf2add4 fix(testcontainers): address remaining reviewer feedback (db2 arm64, cockroachdb DBAPI docs, stale workflow refs)
- requirements/development.in + testcontainers.yml: scope the `db2` extra
  (`ibm-db-sa`/`ibm-db`) out of the baseline dev install. `ibm-db` ships no
  Linux arm64 wheel, so bundling it there broke the multi-platform dev
  Docker image build on push. The testcontainers CI job now installs it
  directly, only for its own db2 matrix leg.
- cockroachdb.py: list `psycopg2-binary` alongside `sqlalchemy-cockroachdb`
  in the `pypi_packages` metadata, since a plain `cockroachdb://` URL can't
  connect without a DBAPI and sqlalchemy-cockroachdb doesn't install one.
- test_cockroachdb.py/test_crate.py/test_trino.py: fix stale docstring
  references to the old `nightly-testcontainers.yml` filename.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:53:00 -07:00
rusackasandClaude Opus 4.8 ce94f79a09 fix(testcontainers): address reviewer feedback on cockroachdb DBAPI, CI isolation, and test rigor
- pyproject.toml: pin psycopg2-binary alongside sqlalchemy-cockroachdb --
  the latter declares no DBAPI dependency of its own, so the documented
  `apache-superset[cockroachdb]` install couldn't actually connect.
- testcontainers.yml: scope the concurrency group by ref so a PR run and
  the nightly cron (or two different PRs) no longer cancel each other.
- test_cockroachdb.py: assert the actual generic/SQLAlchemy type, matching
  the Trino test, instead of only checking a column spec was found.
- pytest.ini + new `testcontainers` marker + _driver.py: exclude
  tests/testcontainers/ from a plain `pytest` run by default (it needs
  Docker), while the dedicated CI job now sets
  SUPERSET_TESTCONTAINERS_STRICT so a broken/missing driver import fails
  that job instead of silently skipping to a green, zero-tests-run result.
- UPDATING.md: the migration note now says to uninstall the old
  `cockroachdb` package outright, since reinstalling the extra alone can
  leave both packages registering the same dialect entry point.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:39 -07:00
rusackasandClaude Opus 4.8 80928bdb6c refactor(testcontainers): extract shared pagination-test body
The "paginated query returns correct rows in order" test body was
duplicated verbatim across cockroachdb/crate/db2/mssql/oracle/trino's
testcontainers suites. Factor the shared table-setup/assert logic into
_pagination.py, with an optional post-insert hook for CrateDB's
eventual-consistency REFRESH TABLE step; each call site keeps its own
test function and dialect-specific docstring.

Addresses bito-code-review feedback on #43502.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
rusackasandClaude Opus 4.8 6ce414928e test(db_engine_specs): fix test_test_connection_failed regression from mssql extra
Adding the mssql extra to development.in for testcontainers coverage
installs pymssql in the dev/test env, so the existing
mssql+pymssql://url probe in test_test_connection_failed no longer
hits the "driver not found" path it was meant to exercise -- it now
attempts a real (failing) connection instead. Swap it for a
dialect+driver URI whose dialect will never resolve to an installed
extra.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:52:17 -07:00
Superset Dev f2d0b592c0 fix(cockroachdb): replace abandoned cockroachdb package with sqlalchemy-cockroachdb
The `cockroachdb` PyPI package (last released 2021) is abandoned, and its
SQLAlchemy dialect references sqlalchemy.dialects.postgresql.psycopg2's
PGCompiler_psycopg2, which SQLAlchemy 2.0 removed. Constructing a
cockroachdb:// engine has raised ImportError since the SQLAlchemy 2.0 bump
(#42803) landed, with nothing catching it: the driver isn't part of the
default dev/CI install, and the existing test_crdb.py only exercises
convert_dttm(), never a real engine.

Switches the `cockroachdb` extra to sqlalchemy-cockroachdb, the actively
maintained replacement already linked from CockroachDbEngineSpec's own
docs_url, adds it to the default dev install so this keeps getting
exercised, and adds a regression test that constructs a real engine
(verified to fail against the old package, pass against the new one).
2026-08-27 19:52:17 -07:00
Superset Dev 5a7fa2d1c6 feat(ci): expand testcontainers coverage to mssql, oracle, db2, elasticsearch
Adds four more dialects to the testcontainers suite (cockroachdb, crate,
trino from the initial pilot): mssql, oracle, db2, elasticsearch. All four
have native testcontainers-python container classes.

Restructures the workflow from one job running the whole suite to a
matrix, one job per dialect, running in parallel. A single slow container
would otherwise inflate wall-clock time for every dialect, not just its
own -- matrixing bounds total suite time by the slowest dialect instead of
the sum of all of them. Also renames the workflow file/name from
"Nightly-Testcontainers" to "Testcontainers" now that it runs on
pull_request (scoped via `paths`) in addition to the nightly cron.

Elasticsearch needed a different data-setup approach than the SQL-native
dialects: indices/documents get created via its REST API, not SQL INSERT,
matching how Superset actually encounters Elasticsearch in practice.
Confirmed empirically that Elasticsearch's SQL layer has no OFFSET support
at all (a real protocol limitation, already correctly documented via
ElasticSearchEngineSpec.supports_offset = False) and adjusted that
dialect's pagination test accordingly -- LIMIT/ORDER BY only, no OFFSET.

mssql and db2 could not be verified locally (no arm64 images for either;
this environment is Apple Silicon), same situation as crate's amd64-only
image from the initial pilot. Both are written against verified library
source (dialect names, connection URL construction) and will get their
first real execution on CI.
2026-08-27 19:51:23 -07:00
Superset Dev c0e32663a4 fix(deps): bump docker floor to 7.2.0 for testcontainers compatibility
docker==7.0.0 (the previously-unpinned floor) raises
`docker.errors.DockerException: ... Not supported URL scheme http+docker`
against the requests/urllib3 versions pinned elsewhere in this file --
confirmed on a real GHA run of #43502's nightly-testcontainers workflow,
where every test errored before any container was even pulled. 7.2.0 is
confirmed working, both locally and against a live container.
2026-08-27 19:51:23 -07:00
Superset Dev fa36e2565a temp: run nightly-testcontainers on this PR + add actions-timeline
Adds a scoped pull_request trigger so this workflow runs on this PR
itself, to get real GHA runner timing before deciding whether/how to
adopt this pattern more broadly. Also adds the actions-timeline job
(same pattern as superset-python-presto-hive.yml) to visualize per-step
duration.

Revert the pull_request trigger before merge -- it's here to measure
cost, not to become a permanent merge-blocking check.
2026-08-27 19:51:23 -07:00
rusackasandClaude Opus 4.8 eb91af8244 test(ci): assert mapped generic/sqla type in Trino column test
Address bot review feedback on #43502: the integration test only
checked that a ColumnSpec existed, not that INTEGER columns actually
mapped to the numeric generic type and an Integer sqla type.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-27 19:51:23 -07:00
Superset Dev 051105098c feat(ci): add nightly testcontainers-based db_engine_specs tests
Pilot for real-container testing of db_engine_specs against actual
databases (CockroachDB, CrateDB, Trino), via testcontainers-python.
Existing unit tests mock the driver/dialect layer entirely, which cannot
catch real SQL-compilation or type-mapping bugs -- e.g. #42899, where
Trino emitted OFFSET before LIMIT for paginated queries.

Runs nightly (.github/workflows/nightly-testcontainers.yml), not on every
merge: container pulls and startup are slower and more flake-prone than
the existing mocked unit tests, and this measures that cost/signal
tradeoff before considering wider adoption.

Building this surfaced two real bugs, fixed/documented separately:
- The cockroachdb dialect was completely broken under SQLAlchemy 2.0 due
  to a dead upstream package -- fixed in #43501.
- testcontainers-python's TrinoContainer.get_connection_url() returns the
  container-internal port instead of the Docker-mapped host port; worked
  around locally, filed upstream.
2026-08-27 19:51:23 -07:00
96 changed files with 2048 additions and 2247 deletions
+2 -2
View File
@@ -67,7 +67,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
uses: github/codeql-action/init@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -78,6 +78,6 @@ jobs:
# queries: security-extended,security-and-quality
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4.37.8
uses: github/codeql-action/analyze@ff2f1c621b7f889edc0d3c761ac2e6a3f8cdb0dd # v4.37.7
with:
category: "/language:${{matrix.language}}"
+121
View File
@@ -0,0 +1,121 @@
# db_engine_specs tests against real databases (testcontainers)
name: Testcontainers
# Spins up real Docker containers (see tests/testcontainers/ for the current
# dialect list) via testcontainers-python, which catches real dialect/driver
# regressions -- the kind mocked db_engine_specs unit tests structurally
# cannot, e.g. apache/superset#42899 (Trino emitting OFFSET before LIMIT).
# Runs on a nightly cron (catches drift from a driver's own releases, not
# just from Superset's changes) and on pull_request, scoped via `paths` to
# only PRs that actually touch this test suite or the workflow itself, so
# unrelated PRs across the repo are never affected.
permissions:
contents: read
on:
schedule:
- cron: "0 5 * * *"
workflow_dispatch: {}
pull_request:
paths:
- ".github/workflows/testcontainers.yml"
- "tests/testcontainers/**"
- "superset/db_engine_specs/**"
- "pyproject.toml"
- "requirements/development.in"
- "requirements/development.txt"
concurrency:
# Scoped by ref, not just workflow name -- otherwise every PR run and the
# nightly cron share one group, and starting the workflow on another PR
# (or the nightly firing mid-PR-run) cancels an unrelated in-progress run.
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
jobs:
testcontainers:
runs-on: ubuntu-26.04
strategy:
fail-fast: false
matrix:
include:
# One job per dialect rather than one job for the whole suite: a
# single slow container would otherwise inflate the wall-clock
# time for every dialect, not just its own. Running in parallel
# means the suite's total time is bounded by the slowest dialect,
# not the sum of all of them. Db2's first-boot init is documented
# upstream as notably slow (a real instance bring-up, not just a
# process start) and untested locally here (no arm64 image), so
# it gets a wider timeout margin than the rest until real CI data
# says otherwise.
- dialect: cockroachdb
timeout: 10
- dialect: crate
timeout: 10
- dialect: trino
timeout: 10
- dialect: mssql
timeout: 10
- dialect: elasticsearch
timeout: 10
- dialect: oracle
timeout: 15
- dialect: db2
timeout: 25
- dialect: mariadb
timeout: 10
- dialect: timescaledb
timeout: 10
- dialect: yugabytedb
timeout: 10
- dialect: monetdb
timeout: 10
- dialect: mongodb
timeout: 10
timeout-minutes: ${{ matrix.timeout }}
env:
PYTHONPATH: ${{ github.workspace }}
SUPERSET_TESTENV: true
SUPERSET_SECRET_KEY: not-a-secret
# This job's matrix installs exactly one dialect's testcontainers
# driver for exactly this job, so treat that driver as required: a
# broken/missing import should fail the job, not silently skip to a
# misleadingly green, zero-tests-run result. See _driver.py.
SUPERSET_TESTCONTAINERS_STRICT: true
steps:
- name: Checkout
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false
- name: Setup Python
uses: ./.github/actions/setup-backend/
with:
python-version: current
- name: Install db2 driver (ibm-db-sa)
# ibm-db (the db2 DBAPI) ships no Linux arm64 wheel, so it's kept out
# of the baseline dev install (requirements/development.in) to avoid
# breaking the multi-platform dev Docker image build. Install it here
# instead, only for this leg of the matrix.
if: matrix.dialect == 'db2'
run: uv pip install --system -e .[db2]
- name: Run testcontainers db_engine_specs tests (${{ matrix.dialect }})
run: |
pytest --durations-min=2 -v -m testcontainers \
./tests/testcontainers/db_engine_specs/test_${{ matrix.dialect }}.py \
--junit-xml=test-results/junit-testcontainers-${{ matrix.dialect }}.xml
- name: Upload JUnit test results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7
with:
name: junit-results-testcontainers-${{ matrix.dialect }}
path: test-results/
retention-days: 7
actions-timeline:
needs: [testcontainers]
if: always()
runs-on: ubuntu-26.04
permissions:
actions: read
steps:
- uses: Kesin11/actions-timeline@57fc93f20c6da7fbc14063c6d24a2a5627c799ad # v3.2.0
+1 -1
View File
@@ -25,7 +25,7 @@ assists people when migrating to a new version.
## Next
- `SAMPLES_ROW_LIMIT` is now the default for `/datasource/samples` requests without a valid explicit `per_page`, rather than a hard per-request ceiling; explicit limits are honored up to the existing global row-limit ceiling, matching `/chart/data` SAMPLES requests.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed should `pip uninstall cockroachdb && pip install sqlalchemy-cockroachdb` (or simply reinstall the extra) to restore CockroachDB connectivity.
- The `cockroachdb` extra (`pip install apache-superset[cockroachdb]`) now installs `sqlalchemy-cockroachdb` instead of the abandoned `cockroachdb` package, whose SQLAlchemy dialect could not be imported under SQLAlchemy 2.0. Existing environments with the old package installed must `pip uninstall cockroachdb` before reinstalling the extra -- both packages register the same `cockroachdb` SQLAlchemy dialect entry point, so leaving the old one in place can still load the abandoned implementation.
### MCP tool results preserve stored string values
+20 -6
View File
@@ -148,10 +148,9 @@ clickhouse = ["clickhouse-connect>=1.7.2, <2.0"]
# 2.0). sqlalchemy-cockroachdb is the actively maintained replacement,
# already linked from CockroachDbEngineSpec.metadata's docs_url, and
# registers the same `cockroachdb` SQLAlchemy dialect entry point.
# sqlalchemy-cockroachdb depends only on SQLAlchemy itself, not on a DBAPI
# driver, so psycopg2-binary is pinned alongside it (matching the `postgres`
# extra) to keep this extra self-contained -- CockroachDB speaks the
# PostgreSQL wire protocol, so psycopg2 is what actually opens connections.
# sqlalchemy-cockroachdb itself declares no DBAPI dependency (its own docs
# require picking one), so pull in the same psycopg2-binary pin as the
# `postgres` extra -- CockroachDB speaks the Postgres wire protocol.
cockroachdb = ["sqlalchemy-cockroachdb>=2.0.0, <3", "psycopg2-binary==2.9.12"]
crate = ["sqlalchemy-cratedb>=0.43.1, <1"]
# sqlalchemy-d1's only release (0.1.0, Nov 2025) pins sqlalchemy<2,>=1.4,
@@ -196,7 +195,7 @@ excel = ["xlrd>=2.0.2, <2.1"]
excel-export = ["boto3"]
fastmcp = [
"fastmcp>=3.4.7,<4.0",
"mcp>=1.29.1,<3.0",
"mcp>=1.29.1,<2.0",
# tiktoken backs the response-size-guard token estimator. Without
# it, the middleware falls back to a coarser character-based
# heuristic that under-counts JSON-heavy MCP responses.
@@ -223,6 +222,11 @@ impala = ["impyla>=0.24.0, <0.25"]
# superset/db_engine_specs/kusto.py's known_incompatibilities metadata.
kusto = ["sqlalchemy-kusto>=3.1.2, <4"]
kylin = ["kylinpy>=2.8.4, <2.9"]
# MariaDB is a MySQL fork implementing the same wire protocol - connects via
# the plain mysql dialect, same driver as mysql.
mariadb = ["apache-superset[mysql]"]
monetdb = ["sqlalchemy-monetdb>=2.1.0, <3", "pymonetdb>=1.9.1, <2"]
mongodb = ["pymongosql>=0.7.3, <1"]
mssql = ["pymssql>=2.3.13, <3"]
# motherduck is an alias for duckdb - MotherDuck works via the duckdb driver
motherduck = ["apache-superset[duckdb]"]
@@ -271,6 +275,9 @@ tdengine = [
"taos-ws-py>=0.7.0"
]
teradata = ["teradatasql>=20.0.0.66"]
# TimescaleDB is a genuine Postgres extension, not a fork - connects via the
# plain postgresql dialect, same driver as postgres.
timescaledb = ["apache-superset[postgres]"]
thumbnails = [] # deprecated, will be removed in 7.0
vertica = ["sqlalchemy-vertica-python>= 0.6.3, < 0.7"]
netezza = ["nzalchemy>= 11.1.2, < 11.2"]
@@ -278,11 +285,18 @@ starrocks = ["starrocks>=1.3.4, <2"]
doris = ["pydoris>=1.2.0, <2.0.0"]
oceanbase = ["oceanbase_py>=0.0.1.2"]
ydb = ["ydb-sqlalchemy>=0.1.22", "ydb-sqlglot-plugin>=0.2.8"]
# YugabyteDB's YSQL layer is fully Postgres-wire compatible - connects via
# the plain postgresql dialect, same driver as postgres.
yugabytedb = ["apache-superset[postgres]"]
development = [
# no bounds for apache-superset-extensions-cli until a stable version
"apache-superset-extensions-cli",
"boto3",
"docker",
# 7.0.0 raises `docker.errors.DockerException: ... Not supported URL
# scheme http+docker` against the requests/urllib3 versions pinned
# elsewhere in this file -- breaks testcontainers (tests/testcontainers/)
# before any container even starts. 7.2.0 is confirmed working.
"docker>=7.2.0",
"flask-testing",
"freezegun",
"grpcio>=1.82.1",
+7 -1
View File
@@ -19,7 +19,13 @@ testpaths =
tests
python_files = *_test.py test_*.py *_tests.py *viz/utils.py
# `-p no:warnings` temporarily disabled in favor of more finely tuned `filterwarnings`.
#addopts = -p no:warnings
# `not testcontainers` excludes tests/testcontainers/ by default: those spin up
# real Docker containers, and `testpaths = tests` would otherwise pull them into
# every plain `pytest` run. The dedicated CI job (testcontainers.yml) overrides
# this with an explicit `-m testcontainers` to run them.
addopts = -m "not testcontainers"
markers =
testcontainers: exercises a real database via testcontainers-python (needs Docker); excluded by default, see .github/workflows/testcontainers.yml
asyncio_mode = auto
# `ignore` is effectively equivalent to `-p no:warnings`.
+15 -1
View File
@@ -16,5 +16,19 @@
# specific language governing permissions and limitations
# under the License.
#
-e .[development,bigquery,cockroachdb,druid,duckdb,fastmcp,gevent,gsheets,mysql,postgres,presto,prophet,trino,thumbnails]
-e .[development,bigquery,cockroachdb,crate,druid,duckdb,elasticsearch,fastmcp,gevent,gsheets,monetdb,mongodb,mssql,mysql,oracle,postgres,presto,prophet,trino,thumbnails]
-e ./superset-extensions-cli[test]
# testcontainers-backed db_engine_specs tests (tests/testcontainers/) --
# see .github/workflows/testcontainers.yml
#
# `db2` (the `ibm-db-sa`/`ibm-db` driver) is deliberately left out of the
# baseline dev install above: `ibm-db` ships no Linux arm64 wheel, so
# including it here breaks the multi-platform (amd64+arm64) dev Docker
# image build. The testcontainers CI job installs it on demand, only for
# the db2 matrix leg -- see .github/workflows/testcontainers.yml.
#
# mariadb/timescaledb/yugabytedb need no testcontainers extra of their own:
# they reuse the postgres/mysql container classes pointed at a different
# image, and psycopg2-binary/mysqlclient are already pulled in above via
# the postgres/mysql extras.
testcontainers[cockroachdb,cratedb,mongodb,mssql,mysql,oracle,postgres,trino]>=4.15.0,<5
+78 -5
View File
@@ -24,6 +24,8 @@ annotated-types==0.7.0
# via
# -c requirements/base-constraint.txt
# pydantic
antlr4-python3-runtime==4.13.2
# via pymongosql
anyio==4.11.0
# via
# httpx
@@ -117,8 +119,10 @@ celery==5.6.3
certifi==2026.5.20
# via
# -c requirements/base-constraint.txt
# elasticsearch
# httpcore
# httpx
# opensearch-py
# requests
cffi==2.0.0
# via
@@ -171,6 +175,8 @@ contourpy==1.0.7
# via matplotlib
coverage==7.6.8
# via pytest-cov
crate==2.2.1
# via sqlalchemy-cratedb
cron-descriptor==1.4.5
# via
# -c requirements/base-constraint.txt
@@ -186,8 +192,10 @@ cryptography==50.0.0
# authlib
# google-auth
# joserfc
# oracledb
# paramiko
# pyjwt
# pymysql
# pyopenssl
# secretstorage
cycler==0.12.1
@@ -216,8 +224,11 @@ dnspython==2.7.0
# via
# -c requirements/base-constraint.txt
# email-validator
docker==7.0.0
# via apache-superset
# pymongo
docker==7.2.0
# via
# apache-superset
# testcontainers
docstring-parser==0.17.0
# via cyclopts
docutils==0.22.2
@@ -228,6 +239,10 @@ duckdb==1.5.5
# duckdb-engine
duckdb-engine==0.17.0
# via apache-superset
elasticsearch==7.17.13
# via elasticsearch-dbapi
elasticsearch-dbapi==0.2.13
# via apache-superset
email-validator==2.2.0
# via
# -c requirements/base-constraint.txt
@@ -237,6 +252,8 @@ et-xmlfile==2.0.0
# via
# -c requirements/base-constraint.txt
# openpyxl
events==0.5
# via opensearch-py
exceptiongroup==1.3.0
# via fastmcp-slim
fastmcp==3.4.7
@@ -333,6 +350,8 @@ geographiclib==2.0
# via
# -c requirements/base-constraint.txt
# geopy
geojson==3.3.0
# via sqlalchemy-cratedb
geopy==2.4.1
# via
# -c requirements/base-constraint.txt
@@ -414,6 +433,7 @@ httpx==0.28.1
# via
# fastmcp-slim
# mcp
# testcontainers
httpx-sse==0.4.1
# via mcp
humanize==4.12.3
@@ -468,6 +488,7 @@ jmespath==1.1.0
# via
# boto3
# botocore
# pymongosql
joserfc==1.7.2
# via fastmcp-slim
jsonpath-ng==1.8.0
@@ -605,14 +626,22 @@ openpyxl==3.1.5
# via
# -c requirements/base-constraint.txt
# pandas
opensearch-py==2.8.0
# via elasticsearch-dbapi
opentelemetry-api==1.39.1
# via fastmcp-slim
oracledb==4.0.2
# via
# apache-superset
# testcontainers
ordered-set==4.1.0
# via
# -c requirements/base-constraint.txt
# flask-limiter
orjson==3.11.9
# via trino
# via
# crate
# trino
packaging==25.0
# via
# -c requirements/base-constraint.txt
@@ -620,8 +649,8 @@ packaging==25.0
# apispec
# db-dtypes
# deprecation
# docker
# duckdb-engine
# elasticsearch-dbapi
# fastmcp-slim
# google-cloud-bigquery
# kombu
@@ -775,6 +804,22 @@ pyjwt==2.13.0
# mcp
pylint==3.3.7
# via apache-superset
pymonetdb==1.9.1
# via
# apache-superset
# sqlalchemy-monetdb
pymongo==4.17.0
# via
# pymongosql
# testcontainers
pymongosql==0.7.3
# via apache-superset
pymssql==2.3.13
# via
# apache-superset
# testcontainers
pymysql==1.2.0
# via testcontainers
pynacl==1.6.2
# via
# -c requirements/base-constraint.txt
@@ -824,6 +869,7 @@ python-dateutil==2.9.0.post0
# google-cloud-bigquery
# holidays
# matplotlib
# opensearch-py
# pandas
# pyhive
# shillelagh
@@ -834,6 +880,7 @@ python-dotenv==1.2.2
# apache-superset
# fastmcp-slim
# pydantic-settings
# testcontainers
python-ldap==3.4.7
# via apache-superset
python-multipart==0.0.29
@@ -877,6 +924,7 @@ requests==2.33.0
# google-api-core
# google-cloud-bigquery
# jsonschema-path
# opensearch-py
# pydruid
# pyhive
# requests-cache
@@ -956,6 +1004,7 @@ sqlalchemy==2.0.52
# apache-superset
# apache-superset-core
# duckdb-engine
# elasticsearch-dbapi
# flask-appbuilder
# flask-sqlalchemy
# marshmallow-sqlalchemy
@@ -963,7 +1012,10 @@ sqlalchemy==2.0.52
# sqlalchemy-bigquery
# sqlalchemy-cockroachdb
# sqlalchemy-continuum
# sqlalchemy-cratedb
# sqlalchemy-monetdb
# sqlalchemy-utils
# testcontainers
sqlalchemy-bigquery==1.17.2
# via apache-superset
sqlalchemy-cockroachdb==2.0.4
@@ -972,6 +1024,12 @@ sqlalchemy-continuum==1.7.0
# via
# -c requirements/base-constraint.txt
# apache-superset
sqlalchemy-cratedb==0.43.1
# via
# apache-superset
# testcontainers
sqlalchemy-monetdb==2.1.0
# via apache-superset
sqlalchemy-utils==0.42.1
# via
# -c requirements/base-constraint.txt
@@ -1003,6 +1061,8 @@ tabulate==0.10.0
# via
# -c requirements/base-constraint.txt
# apache-superset
testcontainers==4.15.0
# via -r requirements/development.in
tiktoken==0.14.0
# via apache-superset
tomli-w==1.2.0
@@ -1014,7 +1074,9 @@ tqdm==4.67.1
# cmdstanpy
# prophet
trino==0.339.0
# via apache-superset
# via
# apache-superset
# testcontainers
typing-extensions==4.16.0
# via
# -c requirements/base-constraint.txt
@@ -1029,6 +1091,7 @@ typing-extensions==4.16.0
# limits
# mcp
# opentelemetry-api
# oracledb
# py-key-value-aio
# pydantic
# pydantic-core
@@ -1037,6 +1100,7 @@ typing-extensions==4.16.0
# shillelagh
# sqlalchemy
# starlette
# testcontainers
# typing-inspection
typing-inspection==0.4.2
# via
@@ -1064,13 +1128,21 @@ urllib3==2.7.0
# via
# -c requirements/base-constraint.txt
# botocore
# crate
# docker
# elasticsearch
# opensearch-py
# requests
# requests-cache
# testcontainers
uvicorn==0.37.0
# via
# fastmcp-slim
# mcp
verlib2==0.3.2
# via
# crate
# sqlalchemy-cratedb
vine==5.1.0
# via
# -c requirements/base-constraint.txt
@@ -1104,6 +1176,7 @@ wrapt==1.17.2
# via
# -c requirements/base-constraint.txt
# deprecated
# testcontainers
wtforms==3.2.2
# via
# -c requirements/base-constraint.txt
+9 -9
View File
@@ -108,7 +108,7 @@
"json-stringify-pretty-compact": "^4.0.0",
"lodash": "^4.18.1",
"lodash-es": "^4.18.1",
"mapbox-gl": "^3.29.0",
"mapbox-gl": "^3.28.1",
"markdown-to-jsx": "^9.10.2",
"match-sorter": "^8.3.0",
"memoize-one": "^6.0.0",
@@ -234,7 +234,7 @@
"eslint-plugin-lodash": "^8.0.0",
"eslint-plugin-no-only-tests": "^3.4.0",
"eslint-plugin-react-prefer-function-component": "^5.0.0",
"eslint-plugin-react-you-might-not-need-an-effect": "^1.0.2",
"eslint-plugin-react-you-might-not-need-an-effect": "^1.0.1",
"eslint-plugin-storybook": "10.5.10",
"eslint-plugin-testing-library": "^7.16.2",
"eslint-plugin-theme-colors": "file:eslint-rules/eslint-plugin-theme-colors",
@@ -20208,9 +20208,9 @@
"license": "MIT"
},
"node_modules/eslint-plugin-react-you-might-not-need-an-effect": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/eslint-plugin-react-you-might-not-need-an-effect/-/eslint-plugin-react-you-might-not-need-an-effect-1.0.2.tgz",
"integrity": "sha512-HgYol2zhH3KbnW9Q4FY/FcIINbsYVL6rwQESChqBC2s9SLWtw1wg05+J7SCrr3BfQpNY2ReYhf8xjpd2JhKJOQ==",
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/eslint-plugin-react-you-might-not-need-an-effect/-/eslint-plugin-react-you-might-not-need-an-effect-1.0.1.tgz",
"integrity": "sha512-oOhQTYhor88Xp8RVytq25tvBfiAjU0r9SCDC51Qop+3Wg5BR1xGMAkM+/dV4MZbcMhdaU1L9bkv6LC95JmiTig==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -28488,9 +28488,9 @@
}
},
"node_modules/mapbox-gl": {
"version": "3.29.0",
"resolved": "https://registry.npmjs.org/mapbox-gl/-/mapbox-gl-3.29.0.tgz",
"integrity": "sha512-Fnh1WLsZMfihwRZY5scp456iQuZo9G97tTpb26bf/Ejsi/L7O+4dE9+I03VoOor2ul3DEOp6F2P3273omyVsNw==",
"version": "3.28.1",
"resolved": "https://registry.npmjs.org/mapbox-gl/-/mapbox-gl-3.28.1.tgz",
"integrity": "sha512-f8bCHFzZ51bKig7rnD7e08aoFLOV3MNFZduspZ4lgOgiaNVp9sw4NSWcgo3IWTyekYKKXjiICD2BP2o4DiYfxw==",
"license": "SEE LICENSE IN LICENSE.txt",
"workspaces": [
"src/style-spec",
@@ -43490,7 +43490,7 @@
"license": "Apache-2.0",
"dependencies": {
"@math.gl/web-mercator": "^4.1.0",
"mapbox-gl": "^3.29.0",
"mapbox-gl": "^3.28.1",
"maplibre-gl": "^5.24.0",
"react-map-gl": "^8.1.2",
"supercluster": "^9.0.0"
+2 -2
View File
@@ -185,7 +185,7 @@
"json-stringify-pretty-compact": "^4.0.0",
"lodash": "^4.18.1",
"lodash-es": "^4.18.1",
"mapbox-gl": "^3.29.0",
"mapbox-gl": "^3.28.1",
"markdown-to-jsx": "^9.10.2",
"match-sorter": "^8.3.0",
"memoize-one": "^6.0.0",
@@ -311,7 +311,7 @@
"eslint-plugin-lodash": "^8.0.0",
"eslint-plugin-no-only-tests": "^3.4.0",
"eslint-plugin-react-prefer-function-component": "^5.0.0",
"eslint-plugin-react-you-might-not-need-an-effect": "^1.0.2",
"eslint-plugin-react-you-might-not-need-an-effect": "^1.0.1",
"eslint-plugin-storybook": "10.5.10",
"eslint-plugin-testing-library": "^7.16.2",
"eslint-plugin-theme-colors": "file:eslint-rules/eslint-plugin-theme-colors",
@@ -65,7 +65,6 @@ export type AntdExposedProps = Pick<
| 'onOpenChange'
| 'optionRender'
| 'placeholder'
| 'prefix'
| 'showArrow'
| 'showSearch'
| 'tokenSeparators'
@@ -119,20 +119,6 @@ export function retrieveErrorMessage(
return statusError || parseStringResponse(str);
}
function getFirstValidationError(message: JsonObject): string | undefined {
const [firstError] = Object.values(message);
if (typeof firstError === 'string') {
return firstError;
}
if (Array.isArray(firstError)) {
return firstError.find((item): item is string => typeof item === 'string');
}
return undefined;
}
export function parseErrorJson(responseJson: JsonObject): ClientErrorObject {
let error = { ...responseJson };
// Backwards compatibility for old error renderers with the new error object
@@ -140,12 +126,13 @@ export function parseErrorJson(responseJson: JsonObject): ClientErrorObject {
error.error = error.description = error.errors[0].message;
error.link = error.errors[0]?.extra?.link;
}
// Marshmallow field validation returns arrays for string messages, but
// serializes lazy translation messages as strings instead.
// Marshmallow field validation returns the error message in the format
// of { message: { field1: [msg1, msg2], field2: [msg], } }
if (!error.error && error.message) {
if (typeof error.message === 'object') {
error.error =
getFirstValidationError(error.message) || t('Invalid input');
Object.values(error.message as Record<string, string[]>)[0]?.[0] ||
t('Invalid input');
}
if (typeof error.message === 'string') {
if (checkForHtml(error.message)) {
@@ -244,24 +244,6 @@ test('parseErrorJson with message', () => {
});
});
test('parseErrorJson preserves string-valued validation messages', () => {
const calculatedColumnError =
'Custom SQL fields cannot be parsed as a single SQL statement.';
expect(
parseErrorJson({
message: {
'columns.0.expression': calculatedColumnError,
},
}),
).toEqual({
message: {
'columns.0.expression': calculatedColumnError,
},
error: calculatedColumnError,
});
});
test('parseErrorJson with HTML message', () => {
expect(
parseErrorJson({
@@ -47,9 +47,6 @@ const getCrossFilterDataMask =
) =>
(value: string) => {
const selected = Object.values(selectedValues);
if (!labelMap[value] && !selected.includes(value)) {
return undefined;
}
let values: string[];
if (selected.includes(value)) {
values = selected.filter(v => v !== value);
@@ -180,46 +180,3 @@ test('cross-filter does nothing when emitCrossFilters is false', () => {
expect(setDataMask).not.toHaveBeenCalled();
});
test('cross-filter does nothing when name is missing from labelMap', () => {
const setDataMask = jest.fn();
const props = buildProps({
groupby: ['topics'],
labelMap: {
cancellations: ['cancellations'],
},
selectedValues: {},
setDataMask,
});
const handlers = allEventHandlers(props);
// e.g. Pie "Other" category is not present in labelMap
handlers.click({ name: 'Other' });
expect(setDataMask).not.toHaveBeenCalled();
});
test('cross-filter still deselects a previously selected value that is missing from labelMap', () => {
const setDataMask = jest.fn();
const props = buildProps({
groupby: ['topics'],
labelMap: {
cancellations: ['cancellations'],
},
// "Other" was selected before it dropped out of labelMap (e.g. a stale
// cross-filter from an earlier render or dashboard state).
selectedValues: { 0: 'Other' },
setDataMask,
});
const handlers = allEventHandlers(props);
handlers.click({ name: 'Other' });
expect(setDataMask).toHaveBeenCalledWith(
expect.objectContaining({
extraFormData: {
filters: [],
},
}),
);
});
@@ -30,7 +30,7 @@
},
"dependencies": {
"@math.gl/web-mercator": "^4.1.0",
"mapbox-gl": "^3.29.0",
"mapbox-gl": "^3.28.1",
"maplibre-gl": "^5.24.0",
"react-map-gl": "^8.1.2",
"supercluster": "^9.0.0"
@@ -355,31 +355,25 @@ describe('SqlEditor', () => {
}),
);
// findByRole('button', { name }) walks every stylesheet rule via nwsapi to
// compute the accessible name, which can crash on an unrelated antd Tabs
// "more" button style; findByLabelText matches the same aria-label without
// that traversal.
test('enables the save dataset button when the latest query succeeded', async () => {
const { findByLabelText } = setupWithLatestQuery({
state: QueryState.Success,
});
expect(await findByLabelText('Save dataset')).toBeEnabled();
const { findByRole } = setupWithLatestQuery({ state: QueryState.Success });
expect(await findByRole('button', { name: 'Save dataset' })).toBeEnabled();
});
test('disables the save dataset button when the latest query failed', async () => {
const { findByLabelText } = setupWithLatestQuery({
const { findByRole } = setupWithLatestQuery({
state: QueryState.Failed,
results: undefined,
});
expect(await findByLabelText('Save dataset')).toBeDisabled();
expect(await findByRole('button', { name: 'Save dataset' })).toBeDisabled();
});
test('disables the save dataset button when the results are not loaded', async () => {
const { findByLabelText } = setupWithLatestQuery({
const { findByRole } = setupWithLatestQuery({
state: QueryState.Success,
results: undefined,
});
expect(await findByLabelText('Save dataset')).toBeDisabled();
expect(await findByRole('button', { name: 'Save dataset' })).toBeDisabled();
});
test('renders an Extension if provided', async () => {
@@ -53,22 +53,6 @@ test('RowCountLabel renders limit with danger and tooltip', async () => {
expect(tooltip).toHaveTextContent('The row limit');
});
test('RowCountLabel uses a caller-provided limitReachedMessage instead of the default', async () => {
render(
<RowCountLabel
rowcount={100}
limit={100}
limitReachedMessage="Custom limit message"
/>,
);
const expectedText = '100 rows';
expect(screen.getByText(expectedText)).toBeInTheDocument();
userEvent.hover(screen.getByText(expectedText));
const tooltip = await screen.findByRole('tooltip');
expect(tooltip).toHaveTextContent('Custom limit message');
expect(tooltip).not.toHaveTextContent('The row limit set for the chart');
});
test('RowCountLabel renders loading', () => {
render(<RowCountLabel loading />);
const expectedText = 'Loading...';
@@ -26,9 +26,6 @@ type RowCountLabelProps = {
limit?: number;
loading?: boolean;
label?: JSX.Element;
// Overrides the default "chart" wording for panes (e.g. samples) where the
// limit reached isn't the chart's own row_limit.
limitReachedMessage?: React.ReactNode;
};
const limitReachedMsg = t(
@@ -36,13 +33,7 @@ const limitReachedMsg = t(
);
export default function RowCountLabel(props: RowCountLabelProps) {
const {
rowcount = 0,
limit = null,
loading,
label,
limitReachedMessage,
} = props;
const { rowcount = 0, limit = null, loading, label } = props;
const limitReached = limit && rowcount >= limit;
const type =
limitReached || (rowcount === 0 && !loading) ? 'error' : 'default';
@@ -59,10 +50,7 @@ export default function RowCountLabel(props: RowCountLabelProps) {
</Label>
);
return limitReached ? (
<Tooltip
id="tt-rowcount-tooltip"
title={<span>{limitReachedMessage ?? limitReachedMsg}</span>}
>
<Tooltip id="tt-rowcount-tooltip" title={<span>{limitReachedMsg}</span>}>
{label || labelText}
</Tooltip>
) : (
@@ -117,7 +117,7 @@ const StyledDiv = styled.div`
${
isMobileConsumptionEnabled()
? `@media (max-width: ${theme.screenSMMax}px) {
.slice-header .header-title {
[data-test='slice-header'] .header-title {
-webkit-line-clamp: unset;
display: block;
white-space: normal;
@@ -79,15 +79,6 @@ type PropertiesModalProps = {
addSuccessToast: (message: string) => void;
addDangerToast: (message: string) => void;
onlyApply?: boolean;
renderExtraFields?: (context: {
assetId: number;
assetType: 'dashboard';
accessorCount: number;
}) => {
content: React.ReactNode;
saveDisabled?: boolean;
saveTooltip?: string;
};
};
type DashboardInfo = {
@@ -116,7 +107,6 @@ const PropertiesModal = ({
onlyApply = false,
onSubmit = () => {},
show = false,
renderExtraFields,
}: PropertiesModalProps) => {
const dispatch = useDispatch();
const [form] = Form.useForm();
@@ -133,17 +123,6 @@ const PropertiesModal = ({
});
const [editors, setEditors] = useState<Subject[]>([]);
const [viewers, setViewers] = useState<Subject[]>([]);
const extraFields = useMemo(
() =>
renderExtraFields?.({
assetId: dashboardId,
assetType: 'dashboard',
accessorCount: editors.length + viewers.length,
}),
[renderExtraFields, dashboardId, editors.length, viewers.length],
);
const saveLabel = onlyApply ? t('Apply') : t('Save');
const [tags, setTags] = useState<TagType[]>([]);
const [customCss, setCustomCss] = useState('');
@@ -719,21 +698,15 @@ const PropertiesModal = ({
}}
title={t('Dashboard properties')}
isEditMode
saveDisabled={
dashboardInfo?.isManagedExternally ||
hasErrors ||
extraFields?.saveDisabled
}
saveDisabled={dashboardInfo?.isManagedExternally || hasErrors}
saveLoading={isApplying}
contentLoading={isLoading}
errorTooltip={
extraFields?.saveDisabled && extraFields?.saveTooltip
? extraFields.saveTooltip
: dashboardInfo?.isManagedExternally
? t(
"This dashboard is managed externally, and can't be edited in Superset",
)
: errorTooltip
dashboardInfo?.isManagedExternally
? t(
"This dashboard is managed externally, and can't be edited in Superset",
)
: errorTooltip
}
saveText={saveLabel}
wrapProps={{ 'data-test': 'properties-edit-modal' }}
@@ -796,7 +769,6 @@ const PropertiesModal = ({
onChangeViewers={handleOnChangeViewers}
onChangeTags={handleChangeTags}
onClearTags={handleClearTags}
renderExtraFields={extraFields}
/>
),
},
@@ -38,11 +38,6 @@ interface AccessSectionProps {
onChangeViewers: (viewers: SubjectPickerValue[]) => void;
onChangeTags: (tags: { label: string; value: number }[]) => void;
onClearTags: () => void;
renderExtraFields?: {
content: React.ReactNode;
saveDisabled?: boolean;
saveTooltip?: string;
};
}
const AccessSection = ({
@@ -54,7 +49,6 @@ const AccessSection = ({
onChangeViewers,
onChangeTags,
onClearTags,
renderExtraFields,
}: AccessSectionProps) => {
const enableViewers = isFeatureEnabled(FeatureFlag.EnableViewers);
@@ -140,7 +134,6 @@ const AccessSection = ({
/>
</ModalFormField>
)}
{renderExtraFields?.content}
</>
);
};
@@ -208,18 +208,6 @@ test('Should render', () => {
expect(screen.getByTestId('slice-header')).toBeInTheDocument();
});
test('Should expose a class hook, not just data-test, for fullscreen styling', () => {
const props = createProps();
render(<SliceHeader {...props} />, {
useRedux: true,
useRouter: true,
initialState,
});
// The production build strips data-test attributes, so CSS that targets the
// header must hang off a class instead.
expect(screen.getByTestId('slice-header')).toHaveClass('slice-header');
});
test('Should render - default props', () => {
const props = createProps();
@@ -270,11 +270,7 @@ const SliceHeader = forwardRef<HTMLDivElement, SliceHeaderProps>(
);
return (
<ChartHeaderStyles
className="slice-header"
data-test="slice-header"
ref={ref}
>
<ChartHeaderStyles data-test="slice-header" ref={ref}>
<div className="header-title" ref={headerRef}>
<Tooltip title={headerTooltip}>
{/* this div ensures the hover event triggers correctly and prevents flickering */}
@@ -19,13 +19,20 @@
import { css, SupersetTheme } from '@apache-superset/core/theme';
export const fullscreenStyles = (theme: SupersetTheme) => css`
.dashboard-component-chart-holder:fullscreen {
[data-test='dashboard-component-chart-holder']:fullscreen {
background-color: ${theme.colorBgBase};
width: 100vw;
height: 100vh;
box-sizing: border-box;
display: flex;
flex-direction: column;
padding: ${theme.sizeUnit * 4}px;
overflow: visible;
position: relative;
pointer-events: auto;
z-index: ${theme.zIndexPopupBase};
opacity: 1;
visibility: visible;
/* Ensure children take up available space */
.dashboard-chart,
@@ -51,8 +58,13 @@ export const fullscreenStyles = (theme: SupersetTheme) => css`
}
}
/* Keep the header above the chart it shares the fullscreen layer with */
.dashboard-component-chart-holder:fullscreen .slice-header {
/* Interaction and Header fixes */
[data-test='dashboard-component-chart-holder']:fullscreen * {
pointer-events: auto;
}
[data-test='dashboard-component-chart-holder']:fullscreen
[data-test='slice-header'] {
z-index: ${theme.zIndexPopupBase};
position: relative;
}
@@ -337,17 +337,13 @@ const ChartHolder = ({
)}
>
<AntdThemeProvider
getPopupContainer={(triggerNode?: HTMLElement) => {
// Only the fullscreen element's subtree is painted, so popups
// have to be portaled into it rather than to document.body.
// Resolve it directly instead of matching a selector: the
// production build strips data-test attributes.
const fullscreenElement =
document.fullscreenElement as HTMLElement | null;
return triggerNode && fullscreenElement?.contains(triggerNode)
? fullscreenElement
: document.body;
}}
getPopupContainer={(triggerNode: HTMLElement) =>
document.fullscreenElement
? (triggerNode?.closest?.(
'[data-test="dashboard-component-chart-holder"]',
) as HTMLElement) || document.body
: document.body
}
>
{!editMode && (
<AnchorLink
@@ -97,29 +97,3 @@ test('does not render DeckglLayerVisibilityTooltip for standard filter type', ()
screen.queryByTestId('deckgl-layer-visibility-tooltip-icon'),
).not.toBeInTheDocument();
});
test('does not mark a defaultToFirstItem-only filter as required', () => {
render(
<FilterControl
filter={{
...nativeFilter,
controlValues: { defaultToFirstItem: true },
}}
onFilterSelectionChange={jest.fn()}
/>,
);
expect(screen.queryByText('*')).not.toBeInTheDocument();
});
test('marks an enableEmptyFilter filter as required', () => {
render(
<FilterControl
filter={{
...nativeFilter,
controlValues: { enableEmptyFilter: true },
}}
onFilterSelectionChange={jest.fn()}
/>,
);
expect(screen.getByText('*')).toBeInTheDocument();
});
@@ -65,7 +65,9 @@ const FilterControl = ({
isFilterInScope(filter) &&
checkIsMissingRequiredValue(filter, filter.dataMask?.filterState);
const validateStatus = isMissingRequiredValue ? 'error' : undefined;
const isRequired = !!filter.controlValues?.enableEmptyFilter;
const isRequired =
!!filter.controlValues?.enableEmptyFilter ||
!!filter.controlValues?.defaultToFirstItem;
const inverseSelection = !!filter.controlValues?.inverseSelection;
const {
@@ -211,16 +211,6 @@ test('checkIsMissingRequiredValue returns false for non-required filter with und
expect(checkIsMissingRequiredValue(filter, filterState)).toBe(false);
});
test('checkIsMissingRequiredValue returns false when only defaultToFirstItem is set', () => {
const filter = createFilter('test-filter', {
enableEmptyFilter: false,
controlValues: { defaultToFirstItem: true },
});
expect(checkIsMissingRequiredValue(filter, { value: null })).toBe(false);
expect(checkIsMissingRequiredValue(filter, { value: undefined })).toBe(false);
});
test('checkIsMissingRequiredValue returns falsy for filter without controlValues', () => {
const filter = { id: 'test-filter' } as Filter;
const filterState: FilterState = { value: undefined };
@@ -309,48 +299,6 @@ test('checkIsApplyDisabled returns true when required filter is missing value in
);
});
test('checkIsApplyDisabled enables Apply after clearing a cascading defaultToFirstItem child', () => {
// Regression: a child filter that is dependent on a parent and configured with
// "Select first filter value by default" but NOT "Filter value is required"
// must stay clearable — clearing it may not disable Apply.
const parent = createFilter('parent', {
enableEmptyFilter: true,
controlValues: { defaultToFirstItem: true },
});
const child = createFilter('child', {
enableEmptyFilter: false,
controlValues: { defaultToFirstItem: true },
});
const dataMaskSelected: DataMaskStateWithId = {
parent: {
id: 'parent',
filterState: { value: ['USA'] },
extraFormData: createExtraFormDataWithFilter('country', ['USA']),
},
child: {
id: 'child',
filterState: { value: null },
extraFormData: {},
},
};
const dataMaskApplied: DataMaskStateWithId = {
parent: {
id: 'parent',
filterState: { value: ['USA'] },
extraFormData: createExtraFormDataWithFilter('country', ['USA']),
},
child: {
id: 'child',
filterState: { value: ['CA'] },
extraFormData: createExtraFormDataWithFilter('state', ['CA']),
},
};
expect(
checkIsApplyDisabled(dataMaskSelected, dataMaskApplied, [parent, child]),
).toBe(false);
});
test('checkIsApplyDisabled enables Apply when Selected has a filter value not yet in Applied', () => {
// Regression: when a required filter's default isn't applied (Applied missing
// the entry) and the user types a value, Selected gains an entry Applied
@@ -48,10 +48,9 @@ export const checkIsMissingRequiredValue = (
filter: FilterElement,
filterState?: FilterState,
) => {
// Only `enableEmptyFilter` ("Filter value is required") makes a value
// mandatory. `defaultToFirstItem` merely seeds an initial selection, so a
// filter cleared by the user must stay clearable, with Apply enabled.
const isRequired = !!filter.controlValues?.enableEmptyFilter;
const isRequired =
!!filter.controlValues?.enableEmptyFilter ||
!!filter.controlValues?.defaultToFirstItem;
if (!isRequired) return false;
@@ -16,13 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import {
render,
screen,
waitFor,
userEvent,
fireEvent,
} from 'spec/helpers/testing-library';
import { render, screen, waitFor } from 'spec/helpers/testing-library';
import Control, { ControlProps } from 'src/explore/components/Control';
const defaultProps: ControlProps = {
@@ -83,72 +77,3 @@ test('call setControlValue if isVisible is false', async () => {
expect(defaultProps.actions.setControlValue).toHaveBeenCalled(),
);
});
test('shows the description icon while the control is hovered', async () => {
render(
setup({
label: 'My checkbox',
description: 'Help text',
}),
);
expect(
screen.queryByRole('button', { name: 'Show info tooltip' }),
).not.toBeInTheDocument();
await userEvent.hover(screen.getByTestId('checkbox'));
expect(
screen.getByRole('button', { name: 'Show info tooltip' }),
).toBeInTheDocument();
await userEvent.unhover(screen.getByTestId('checkbox'));
expect(
screen.queryByRole('button', { name: 'Show info tooltip' }),
).not.toBeInTheDocument();
});
test('shows the description icon while the control has keyboard focus', () => {
render(
setup({
label: 'My checkbox',
description: 'Help text',
}),
);
expect(
screen.queryByRole('button', { name: 'Show info tooltip' }),
).not.toBeInTheDocument();
fireEvent.focus(screen.getByRole('checkbox'));
const infoIcon = screen.getByRole('button', { name: 'Show info tooltip' });
expect(infoIcon).toBeInTheDocument();
fireEvent.blur(screen.getByRole('checkbox'), { relatedTarget: infoIcon });
expect(
screen.getByRole('button', { name: 'Show info tooltip' }),
).toBeInTheDocument();
fireEvent.blur(infoIcon, { relatedTarget: document.body });
expect(
screen.queryByRole('button', { name: 'Show info tooltip' }),
).not.toBeInTheDocument();
});
test('keeps the description icon visible when the pointer leaves a focused control', () => {
render(
setup({
label: 'My checkbox',
description: 'Help text',
}),
);
fireEvent.focus(screen.getByRole('checkbox'));
expect(
screen.getByRole('button', { name: 'Show info tooltip' }),
).toBeInTheDocument();
fireEvent.mouseLeave(screen.getByTestId('checkbox'));
expect(
screen.getByRole('button', { name: 'Show info tooltip' }),
).toBeInTheDocument();
});
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import { ReactNode, useCallback, useState, useEffect, FocusEvent } from 'react';
import { ReactNode, useCallback, useState, useEffect } from 'react';
import { isEqual } from 'lodash-es';
import {
ControlType,
@@ -70,18 +70,7 @@ export default function Control(props: ControlProps) {
} = props;
const [hovered, setHovered] = useState(false);
const [focused, setFocused] = useState(false);
const wasVisible = usePrevious(isVisible);
const handleBlur = (event: FocusEvent<HTMLDivElement>) => {
if (
!(event.relatedTarget instanceof Node) ||
!event.currentTarget.contains(event.relatedTarget)
) {
setFocused(false);
}
};
const onChange = useCallback(
(value: any, errors: any[]) => setControlValue(name, value, errors),
[name, setControlValue],
@@ -130,15 +119,9 @@ export default function Control(props: ControlProps) {
style={hidden ? { display: 'none' } : undefined}
onMouseEnter={() => setHovered(true)}
onMouseLeave={() => setHovered(false)}
onFocus={() => setFocused(true)}
onBlur={handleBlur}
>
<ErrorBoundary>
<ControlComponent
onChange={onChange}
hovered={hovered || focused}
{...props}
/>
<ControlComponent onChange={onChange} hovered={hovered} {...props} />
</ErrorBoundary>
</StyledControl>
);
@@ -1,112 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import {
render,
screen,
userEvent,
fireEvent,
} from 'spec/helpers/testing-library';
import ControlHeader from './ControlHeader';
const description = 'This control filters the whole chart.';
test('does not render the description icon until the control is hovered', () => {
const { rerender } = render(
<ControlHeader
name="time_range"
label="Date Range"
description={description}
/>,
);
expect(
screen.queryByRole('button', { name: 'Show info tooltip' }),
).not.toBeInTheDocument();
rerender(
<ControlHeader
name="time_range"
label="Date Range"
description={description}
hovered
/>,
);
expect(
screen.getByRole('button', { name: 'Show info tooltip' }),
).toBeInTheDocument();
});
test('notifies onDescriptionHoverChange when the info icon is hovered', async () => {
const onDescriptionHoverChange = jest.fn();
render(
<ControlHeader
name="time_range"
label="Date Range"
description={description}
hovered
onDescriptionHoverChange={onDescriptionHoverChange}
/>,
);
const infoIcon = screen.getByRole('button', { name: 'Show info tooltip' });
await userEvent.hover(infoIcon);
expect(onDescriptionHoverChange).toHaveBeenCalledWith(true);
await userEvent.unhover(infoIcon);
expect(onDescriptionHoverChange).toHaveBeenCalledWith(false);
});
test('notifies onDescriptionHoverChange when the info icon is focused', () => {
const onDescriptionHoverChange = jest.fn();
render(
<ControlHeader
name="time_range"
label="Date Range"
description={description}
hovered
onDescriptionHoverChange={onDescriptionHoverChange}
/>,
);
const infoIcon = screen.getByRole('button', { name: 'Show info tooltip' });
fireEvent.focus(infoIcon);
expect(onDescriptionHoverChange).toHaveBeenCalledWith(true);
fireEvent.blur(infoIcon);
expect(onDescriptionHoverChange).toHaveBeenCalledWith(false);
});
test('activates tooltipOnClick from the keyboard', () => {
const tooltipOnClick = jest.fn();
render(
<ControlHeader
name="time_range"
label="Date Range"
description={description}
hovered
tooltipOnClick={tooltipOnClick}
/>,
);
fireEvent.keyDown(screen.getByRole('button', { name: 'Show info tooltip' }), {
key: 'Enter',
});
expect(tooltipOnClick).toHaveBeenCalledTimes(1);
});
@@ -38,7 +38,6 @@ export type ControlHeaderProps = {
tooltipOnClick?: () => void;
warning?: string;
danger?: string;
onDescriptionHoverChange?: (hovered: boolean) => void;
// Allow extra props from control spread patterns (e.g. {...this.props})
[key: string]: unknown;
};
@@ -72,7 +71,6 @@ const ControlHeader: FC<ControlHeaderProps> = ({
tooltipOnClick = () => {},
warning,
danger,
onDescriptionHoverChange,
}) => {
const theme = useTheme();
@@ -91,44 +89,24 @@ const ControlHeader: FC<ControlHeaderProps> = ({
position: absolute;
top: 50%;
right: 0;
z-index: 1;
padding-left: ${theme.sizeUnit}px;
transform: translate(100%, -50%);
white-space: nowrap;
pointer-events: auto;
`}
>
{description && (
<>
<span>
<Tooltip
id="description-tooltip"
title={description}
placement="top"
mouseLeaveDelay={0}
trigger={['hover', 'focus']}
>
{/* Same role="button" pattern as the label text: a real <button>
is not valid inside FormLabel's <label>. */}
<span
// eslint-disable-next-line jsx-a11y/prefer-tag-over-role
role="button"
tabIndex={0}
data-test={`${name}-description-icon`}
aria-label={t('Show info tooltip')}
onMouseEnter={() => onDescriptionHoverChange?.(true)}
onMouseLeave={() => onDescriptionHoverChange?.(false)}
onFocus={() => onDescriptionHoverChange?.(true)}
onBlur={() => onDescriptionHoverChange?.(false)}
<Icons.InfoCircleOutlined
css={iconStyles}
onClick={tooltipOnClick}
onKeyDown={handleKeyboardActivation(tooltipOnClick)}
css={css`
cursor: pointer;
`}
>
<Icons.InfoCircleOutlined css={iconStyles} />
</span>
/>
</Tooltip>{' '}
</>
</span>
)}
{renderTrigger && (
<span>
@@ -68,8 +68,6 @@ export const TableControls = ({
canDownload,
rowLimit,
rowLimitOptions,
effectiveRowLimit,
limitReachedMessage,
onRowLimitChange,
onDownloadCSV,
onDownloadXLSX,
@@ -113,19 +111,14 @@ export const TableControls = ({
value={rowLimit}
onChange={onRowLimitChange}
options={rowLimitOptions ?? []}
// Labelled as the applied limit to avoid a second row count next to RowCountLabel.
prefix={t('Limit')}
css={css`
min-width: 160px;
min-width: 110px;
`}
/>
)}
<RowCountLabel
rowcount={rowcount}
limit={effectiveRowLimit ?? rowLimit}
limitReachedMessage={limitReachedMessage}
loading={isLoading}
/>
{(!onRowLimitChange || rowcount < (rowLimit ?? Infinity)) && (
<RowCountLabel rowcount={rowcount} loading={isLoading} />
)}
{canDownload && onDownloadCSV && onDownloadXLSX && (
<DownloadDropdown
onDownloadCSV={onDownloadCSV}
@@ -136,12 +136,6 @@ export const SamplesPane = ({
const columns = useGridColumns(colnames, coltypes, data);
const keywordFilter = useKeywordFilter(filterText);
// Samples aren't capped by a chart's row_limit, just this pane's own
// page-size selector, so RowCountLabel's default "chart" wording is wrong here.
const limitReachedMessage = t(
'The sample row limit was reached. This %s may contain more rows.',
datasetLabelLower(),
);
const handleInputChange = useCallback(
(input: string) => setFilterText(input),
@@ -167,7 +161,6 @@ export const SamplesPane = ({
canDownload={canDownload}
rowLimit={rowLimit}
rowLimitOptions={ROW_LIMIT_OPTIONS}
limitReachedMessage={limitReachedMessage}
onRowLimitChange={handleRowLimitChange}
/>
<ErrorAlertWrapper>
@@ -204,7 +197,6 @@ export const SamplesPane = ({
canDownload={canDownload}
rowLimit={rowLimit}
rowLimitOptions={ROW_LIMIT_OPTIONS}
limitReachedMessage={limitReachedMessage}
onRowLimitChange={handleRowLimitChange}
/>
<GridContainer>
@@ -56,8 +56,6 @@ export const SingleQueryResultPane = ({
columnDisplayNames,
rowLimit,
rowLimitOptions,
effectiveRowLimit,
limitReachedMessage,
onRowLimitChange,
onDownloadCSV,
onDownloadXLSX,
@@ -88,8 +86,6 @@ export const SingleQueryResultPane = ({
canDownload={canDownload}
rowLimit={rowLimit}
rowLimitOptions={rowLimitOptions}
effectiveRowLimit={effectiveRowLimit}
limitReachedMessage={limitReachedMessage}
onRowLimitChange={onRowLimitChange}
onDownloadCSV={onDownloadCSV}
onDownloadXLSX={onDownloadXLSX}
@@ -84,17 +84,6 @@ export const useResultsPane = ({
// Never exceed the chart's own row_limit
const effectiveRowLimit = Math.min(rowLimit, chartRowLimit);
// When this pane's own row-limit selector is stricter than the chart's
// row_limit, it - not the chart - is what caps the result, so
// RowCountLabel's default "chart" wording would be misleading (the chart's
// configured row_limit was never actually reached).
const limitReachedMessage =
rowLimit < chartRowLimit
? t(
'The row limit selected for this pane was reached. There may be more matching rows.',
)
: undefined;
const cappedFormData = useMemo(
() => ({ ...queryFormData, row_limit: effectiveRowLimit }),
[queryFormData, effectiveRowLimit],
@@ -247,8 +236,6 @@ export const useResultsPane = ({
columnDisplayNames={columnDisplayNames}
rowLimit={rowLimit}
rowLimitOptions={ROW_LIMIT_OPTIONS}
effectiveRowLimit={effectiveRowLimit}
limitReachedMessage={limitReachedMessage}
onRowLimitChange={handleRowLimitChange}
/>
</StyledDiv>
@@ -1,94 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import {
act,
render,
screen,
sleep,
userEvent,
} from 'spec/helpers/testing-library';
import { GenericDataType } from '@apache-superset/core/common';
import {
TableControls,
ROW_LIMIT_OPTIONS,
} from '../components/DataTableControls';
import { TableControlsProps } from '../types';
const setup = (overrides: Partial<TableControlsProps> = {}) =>
render(
<TableControls
data={[]}
columnNames={['name']}
columnTypes={[GenericDataType.String]}
rowcount={0}
onInputChange={jest.fn()}
isLoading={false}
canDownload
rowLimit={100}
rowLimitOptions={ROW_LIMIT_OPTIONS}
onRowLimitChange={jest.fn()}
{...overrides}
/>,
{ useRedux: true },
);
test('shows the row count when the result fills the selected row limit', () => {
setup({ rowcount: 100, rowLimit: 100 });
expect(screen.getByTestId('row-count-label')).toHaveTextContent('100 rows');
});
test('warns that the row limit was reached when the result fills it', async () => {
setup({ rowcount: 100, rowLimit: 100 });
userEvent.hover(screen.getByTestId('row-count-label'));
expect(await screen.findByRole('tooltip')).toHaveTextContent(
'The row limit set for the chart was reached',
);
});
test('does not warn when the result is smaller than the selected row limit', async () => {
setup({ rowcount: 42, rowLimit: 100 });
expect(screen.getByTestId('row-count-label')).toHaveTextContent('42 rows');
userEvent.hover(screen.getByTestId('row-count-label'));
// Wait past antd's 0.1s mouseEnterDelay so a regression that made the
// tooltip appear would be caught here instead of racing the delay.
await act(() => sleep(150));
expect(screen.queryByRole('tooltip')).not.toBeInTheDocument();
});
test("warns when the chart's own row limit truncates below the selected row limit", async () => {
setup({ rowcount: 250, rowLimit: 1000, effectiveRowLimit: 250 });
userEvent.hover(screen.getByTestId('row-count-label'));
expect(await screen.findByRole('tooltip')).toHaveTextContent(
'The row limit set for the chart was reached',
);
});
test('labels the row limit selector so it is not read as a second row count', () => {
setup({ rowcount: 100, rowLimit: 100 });
expect(screen.getByText('Limit')).toBeInTheDocument();
});
@@ -16,12 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import {
screen,
render,
waitFor,
userEvent,
} from 'spec/helpers/testing-library';
import { screen, render, waitFor } from 'spec/helpers/testing-library';
import { setupAGGridModules } from '@superset-ui/core/components/ThemedAgGridReact';
import { getChartDataRequest } from 'src/components/Chart/chartAction';
import { ResultsPaneOnDashboard } from '../components';
@@ -162,42 +157,6 @@ describe('useResultsPane query data reuse', () => {
expect(screen.queryByText('Sci-Fi')).not.toBeInTheDocument();
expect(screen.getByText('2 rows')).toBeVisible();
expect(mockedGetChartDataRequest).not.toHaveBeenCalled();
userEvent.hover(screen.getByText('2 rows'));
expect(await screen.findByRole('tooltip')).toHaveTextContent(
'The row limit set for the chart was reached',
);
});
test("warns about this pane's own row limit, not the chart's, when the pane's selector is what caps the result", async () => {
// chart row_limit (2000) is well above this pane's default 1000-row
// selector, so the selector - not the chart - is what truncates here.
const props = createResultsPaneOnDashboardProps({
sliceId: 208,
rowLimit: 2000,
queriesResponse: [
{
colnames: ['genre'],
coltypes: [1],
data: Array.from({ length: 1500 }, (_, i) => ({
genre: `genre-${i}`,
})),
rowcount: 1500,
},
],
});
render(<ResultsPaneOnDashboard {...props} />, { useRedux: true });
const rowCountLabel = await screen.findByTestId('row-count-label');
expect(rowCountLabel).toHaveTextContent('1k rows');
userEvent.hover(rowCountLabel);
const tooltip = await screen.findByRole('tooltip');
expect(tooltip).toHaveTextContent(
'The row limit selected for this pane was reached',
);
expect(tooltip).not.toHaveTextContent('for the chart');
});
test('renders an empty (0 rows) result from reused data without an API call', async () => {
@@ -84,12 +84,6 @@ export interface TableControlsProps extends DrillControlsProps {
canDownload: boolean;
rowLimit?: number;
rowLimitOptions?: { value: number; label: string }[];
// Effective result limit, capped by the chart's row limit.
// Defaults to `rowLimit` and controls the "row limit reached" warning.
effectiveRowLimit?: number;
// Overrides RowCountLabel's default "chart" wording for panes (e.g.
// samples) where the limit reached isn't the chart's own row_limit.
limitReachedMessage?: React.ReactNode;
onRowLimitChange?: (limit: number) => void;
}
@@ -110,9 +104,5 @@ export interface SingleQueryResultPaneProp
columnDisplayNames?: Record<string, string>;
rowLimit?: number;
rowLimitOptions?: { value: number; label: string }[];
effectiveRowLimit?: number;
// Overrides RowCountLabel's default "chart" wording when the pane's own
// row-limit selector, not the chart's row_limit, is what capped the result.
limitReachedMessage?: React.ReactNode;
onRowLimitChange?: (limit: number) => void;
}
@@ -16,14 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import {
type ReactNode,
ChangeEvent,
useMemo,
useState,
useCallback,
useEffect,
} from 'react';
import { ChangeEvent, useMemo, useState, useCallback, useEffect } from 'react';
import {
Input,
@@ -63,12 +56,6 @@ export type PropertiesModalProps = {
permissionsError?: string;
addSuccessToast: (msg: string) => void;
addDangerToast: (msg: string) => void;
/** Optional render prop for injecting extra fields (e.g. folder selector). */
renderExtraFields?: (context: {
assetId: number;
assetType: 'chart';
accessorCount: number;
}) => { content: ReactNode; saveDisabled?: boolean; saveTooltip?: string };
};
function PropertiesModal({
@@ -78,7 +65,6 @@ function PropertiesModal({
show,
addSuccessToast,
addDangerToast,
renderExtraFields,
}: PropertiesModalProps) {
const [submitting, setSubmitting] = useState(false);
// values of form inputs
@@ -101,25 +87,6 @@ function PropertiesModal({
>(null);
const [tags, setTags] = useState<TagType[]>([]);
const chartId = slice.slice_id;
const extraFields = useMemo(
() =>
chartId
? renderExtraFields?.({
assetId: chartId,
assetType: 'chart',
accessorCount:
(selectedEditors?.length ?? 0) + (selectedViewers?.length ?? 0),
})
: undefined,
[
chartId,
renderExtraFields,
selectedEditors?.length,
selectedViewers?.length,
],
);
// Validation setup
const modalSections = useMemo(
() => [
@@ -314,20 +281,14 @@ function PropertiesModal({
title={t('Chart properties')}
isEditMode
saveDisabled={
submitting ||
!name ||
slice.is_managed_externally ||
hasErrors ||
extraFields?.saveDisabled
submitting || !name || slice.is_managed_externally || hasErrors
}
errorTooltip={
extraFields?.saveDisabled && extraFields?.saveTooltip
? extraFields.saveTooltip
: slice.is_managed_externally
? t(
"This chart is managed externally, and can't be edited in Superset",
)
: errorTooltip
slice.is_managed_externally
? t(
"This chart is managed externally, and can't be edited in Superset",
)
: errorTooltip
}
wrapProps={{ 'data-test': 'properties-edit-modal' }}
>
@@ -434,7 +395,6 @@ function PropertiesModal({
/>
</ModalFormField>
)}
{extraFields?.content}
</>
),
},
@@ -147,7 +147,6 @@ export default function DateFilterLabel(props: DateFilterControlProps) {
onOpenPopover = noOp,
onClosePopover = noOp,
isOverflowingFilterBar = false,
hovered: isControlHovered = false,
} = props;
const defaultTimeFilter = useDefaultTimeFilter();
@@ -162,16 +161,9 @@ export default function DateFilterLabel(props: DateFilterControlProps) {
const [validTimeRange, setValidTimeRange] = useState<boolean>(false);
const [evalResponse, setEvalResponse] = useState<string>(value);
const [tooltipTitle, setTooltipTitle] = useState<ReactNode | null>(t(value));
const [isDescriptionHovered, setIsDescriptionHovered] = useState(false);
const theme = useTheme();
const [labelRef, labelIsTruncated] = useCSSTextTruncation<HTMLSpanElement>();
useEffect(() => {
if (!isControlHovered) {
setIsDescriptionHovered(false);
}
}, [isControlHovered]);
useEffect(() => {
if (value === NO_TIME_RANGE) {
setActualTimeRange(NO_TIME_RANGE);
@@ -376,12 +368,7 @@ export default function DateFilterLabel(props: DateFilterControlProps) {
}
overlayClassName="time-range-popover"
>
<Tooltip
placement="top"
title={isDescriptionHovered ? null : tooltipTitle}
mouseLeaveDelay={0}
overlayStyle={{ pointerEvents: 'none' }}
>
<Tooltip placement="top" title={tooltipTitle}>
{/* Wrap in a span so the Popover gets a stable DOM ref target;
DateLabel forwards its ref to an inner span used for measuring
text truncation, which would otherwise become the popover's
@@ -403,10 +390,7 @@ export default function DateFilterLabel(props: DateFilterControlProps) {
return (
<>
<ControlHeader
{...props}
onDescriptionHoverChange={setIsDescriptionHovered}
/>
<ControlHeader {...props} />
{popoverContent}
</>
);
@@ -18,35 +18,16 @@
*/
import thunk from 'redux-thunk';
import { Provider } from 'react-redux';
import configureMockStore from 'redux-mock-store';
import configureStore from 'redux-mock-store';
import {
render,
screen,
userEvent,
waitFor,
fireEvent,
} from 'spec/helpers/testing-library';
import { render, screen, userEvent } from 'spec/helpers/testing-library';
import { NO_TIME_RANGE, fetchTimeRange } from '@superset-ui/core';
import { NO_TIME_RANGE } from '@superset-ui/core';
import DateFilterLabel from '..';
import { DateFilterControlProps } from '../types';
import { DateFilterTestKey } from '../utils';
jest.mock('@superset-ui/core', () => ({
...jest.requireActual('@superset-ui/core'),
fetchTimeRange: jest.fn(),
}));
const mockedFetchTimeRange = fetchTimeRange as jest.MockedFunction<
typeof fetchTimeRange
>;
const FIELD_TOOLTIP = '2024-01-01 ≤ col < 2024-01-08';
const DESCRIPTION_TOOLTIP =
'This control filters the whole chart based on the selected time range.';
const mockStore = configureMockStore([thunk]);
const mockStore = configureStore([thunk]);
const defaultProps = {
onChange: jest.fn(),
@@ -54,11 +35,6 @@ const defaultProps = {
onOpenPopover: jest.fn(),
};
beforeEach(() => {
mockedFetchTimeRange.mockReset();
mockedFetchTimeRange.mockResolvedValue({ value: FIELD_TOOLTIP });
});
function setup(
props: Omit<DateFilterControlProps, 'name'> = defaultProps,
store: any = mockStore({}),
@@ -160,61 +136,3 @@ test('DateFilter should properly handle isOverflowingFilterBar prop changes', ()
expect(popoverAfterRerender?.parentElement).toBe(trigger.parentElement);
expect(popoverAfterRerender?.parentElement).not.toBe(document.body);
});
test('hovering the description icon does not show the date range tooltip', async () => {
const tooltipOnClick = jest.fn();
render(
setup({
...defaultProps,
value: 'Last week',
label: 'Date Range',
description: DESCRIPTION_TOOLTIP,
hovered: true,
tooltipOnClick,
}),
);
await waitFor(() => {
expect(screen.getByText('Last week')).toBeInTheDocument();
});
await userEvent.hover(screen.getByText('Last week'));
expect(await screen.findByRole('tooltip')).toHaveTextContent(FIELD_TOOLTIP);
const descriptionIcon = screen.getByRole('button', {
name: 'Show info tooltip',
});
fireEvent.focus(descriptionIcon);
await waitFor(() => {
expect(screen.getByRole('tooltip')).toHaveTextContent(DESCRIPTION_TOOLTIP);
expect(screen.getByRole('tooltip')).not.toHaveTextContent(FIELD_TOOLTIP);
expect(screen.getAllByRole('tooltip')).toHaveLength(1);
});
fireEvent.blur(descriptionIcon);
await waitFor(() => {
expect(screen.getByRole('tooltip')).toHaveTextContent(FIELD_TOOLTIP);
expect(screen.getAllByRole('tooltip')).toHaveLength(1);
});
await userEvent.unhover(screen.getByText('Last week'));
await waitFor(() => {
expect(screen.queryByRole('tooltip')).not.toBeInTheDocument();
});
await userEvent.hover(descriptionIcon);
const tooltip = await screen.findByRole('tooltip');
expect(tooltip).toHaveTextContent(DESCRIPTION_TOOLTIP);
expect(tooltip).not.toHaveTextContent(FIELD_TOOLTIP);
expect(screen.getAllByRole('tooltip')).toHaveLength(1);
await userEvent.unhover(descriptionIcon);
await waitFor(() => {
expect(screen.queryByRole('tooltip')).not.toBeInTheDocument();
});
fireEvent.keyDown(descriptionIcon, { key: 'Enter' });
expect(tooltipOnClick).toHaveBeenCalled();
});
@@ -16,8 +16,6 @@
* specific language governing permissions and limitations
* under the License.
*/
import { ReactNode } from 'react';
export type SelectOptionType = {
value: string;
label: string;
@@ -115,8 +113,4 @@ export interface DateFilterControlProps {
onOpenPopover?: () => void;
onClosePopover?: () => void;
isOverflowingFilterBar?: boolean;
hovered?: boolean;
description?: ReactNode;
label?: ReactNode;
tooltipOnClick?: () => void;
}
@@ -951,167 +951,3 @@ test('filters the subject select by column verbose_name as well as column_name',
expect(within(dropdown).getByText('total_count')).toBeInTheDocument();
expect(within(dropdown).queryByText('Full Name')).not.toBeInTheDocument();
});
const COLUMN_VALUES_ENDPOINT =
'glob:*/api/v1/datasource/*/column/value/values/*';
let columnValues: { result: unknown[]; limit: number } = {
result: [],
limit: 10000,
};
fetchMock.get(COLUMN_VALUES_ENDPOINT, () => columnValues);
const setupWithFilterValues = (result: unknown[], limit = 10000) => {
columnValues = { result, limit };
const onChange = jest.fn();
const validHandler = jest.fn();
const spy = jest.spyOn(redux, 'useSelector');
spy.mockReturnValue({});
const props = {
adhocFilter: new AdhocFilter({
expressionType: ExpressionTypes.Simple,
subject: 'value',
operatorId: Operators.In,
operator: OPERATOR_ENUM_TO_OPERATOR_TYPE[Operators.In].operation,
comparator: [],
clause: Clauses.Where,
}),
onChange,
options,
datasource: {
...TestDataset,
columns: [{ column_name: 'value', type: 'VARCHAR', id: 3 }],
filter_select: true,
},
partitionColumn: 'test',
validHandler,
};
render(
<AdhocFilterEditPopoverSimpleTabContent {...(props as unknown as Props)} />,
);
return props;
};
const openComparator = async () => {
const comparator = screen.getByRole('combobox', {
name: 'Comparator option',
});
userEvent.click(comparator);
return comparator;
};
test('loads comparator values from the server', async () => {
setupWithFilterValues(['alpha', 'beta']);
await openComparator();
expect(await screen.findByTitle('alpha')).toBeInTheDocument();
});
test('sends the typed text to the server rather than filtering the loaded page', async () => {
// The loaded page is bounded, so matching client-side cannot reach a value
// beyond the row limit. The search has to reach the database.
setupWithFilterValues(['alpha']);
const comparator = await openComparator();
userEvent.type(comparator, 'gamma');
await waitFor(
() => {
const searched = fetchMock.callHistory
.calls(COLUMN_VALUES_ENDPOINT)
.map(call => String(call.url));
expect(searched.some(url => url.includes('q=gamma'))).toBe(true);
},
{ timeout: 3000 },
);
});
test('lets a value the server did not return still be selected', async () => {
// Even with server-side search a match can fall outside the page; typing the
// exact value has to remain a way through.
setupWithFilterValues([]);
const comparator = await openComparator();
userEvent.type(comparator, 'not-in-the-page');
expect(await screen.findByTitle('not-in-the-page')).toBeInTheDocument();
});
test('does not query for values when the dataset disables them', async () => {
fetchMock.clearHistory();
setup({
adhocFilter: new AdhocFilter({
expressionType: ExpressionTypes.Simple,
subject: 'value',
operatorId: Operators.In,
operator: OPERATOR_ENUM_TO_OPERATOR_TYPE[Operators.In].operation,
comparator: [],
clause: Clauses.Where,
}),
});
await openComparator();
expect(fetchMock.callHistory.calls(COLUMN_VALUES_ENDPOINT)).toHaveLength(0);
});
test('stores the picked value, not the option object', async () => {
// AsyncSelect is labelInValue: taking its argument at face value puts
// {label, value} into the comparator, and the engine then fails to render it
// as a literal.
const props = setupWithFilterValues(['Michael']);
await openComparator();
userEvent.click(await screen.findByTitle('Michael'));
await waitFor(() => expect(props.onChange).toHaveBeenCalled());
const [filter] = props.onChange.mock.calls.at(-1);
expect(filter.comparator).toEqual(['Michael']);
});
test('can remove a value that was saved earlier', async () => {
// Reopening the popover restores the comparator from the saved filter, and
// the value is not in the freshly loaded page. Removing it has to still work.
columnValues = { result: [], limit: 10000 };
const onChange = jest.fn();
const validHandler = jest.fn();
jest.spyOn(redux, 'useSelector').mockReturnValue({});
render(
<AdhocFilterEditPopoverSimpleTabContent
{...({
adhocFilter: new AdhocFilter({
expressionType: ExpressionTypes.Simple,
subject: 'value',
operatorId: Operators.In,
operator: OPERATOR_ENUM_TO_OPERATOR_TYPE[Operators.In].operation,
comparator: ['Michael'],
clause: Clauses.Where,
}),
onChange,
options,
datasource: {
...TestDataset,
columns: [{ column_name: 'value', type: 'VARCHAR', id: 3 }],
filter_select: true,
},
partitionColumn: 'test',
validHandler,
} as unknown as Props)}
/>,
);
// Remove it the way a user does: the tag's own close control.
userEvent.click(await screen.findByLabelText('close'));
await waitFor(() => expect(onChange).toHaveBeenCalled());
const [filter] = onChange.mock.calls.at(-1);
expect(filter.comparator).toEqual([]);
});
test('says the list is partial when the server capped it', async () => {
setupWithFilterValues(['alpha', 'beta'], 2);
await openComparator();
expect(
await screen.findByText(/Only the first 2 values are listed/),
).toBeInTheDocument();
});
test('does not say the list is partial when it is complete', async () => {
setupWithFilterValues(['alpha', 'beta'], 10000);
await openComparator();
expect(await screen.findByTitle('alpha')).toBeInTheDocument();
expect(screen.queryByText(/Only the first/)).not.toBeInTheDocument();
});
@@ -16,25 +16,13 @@
* specific language governing permissions and limitations
* under the License.
*/
import {
FC,
ChangeEvent,
useCallback,
useEffect,
useMemo,
useState,
useRef,
} from 'react';
import { FC, ChangeEvent, useEffect, useState, useRef } from 'react';
import {
AsyncSelect,
Input,
InputRef,
Select,
Tooltip,
type AsyncSelectRef,
type LabeledValue,
type SelectOptionsTypePage,
type SelectValue,
} from '@superset-ui/core/components';
import { t } from '@apache-superset/core/translation';
@@ -69,7 +57,7 @@ import { useDatePickerInAdhocFilter } from '../utils';
import { useDefaultTimeFilter } from '../../DateFilterControl/utils';
import { Clauses, ExpressionTypes } from '../types';
const SelectWithLabel = styled(AsyncSelect)<{ labelText: string }>`
const SelectWithLabel = styled(Select)<{ labelText: string }>`
.ant-select-content::after {
content: ${({ labelText }) => labelText || '\\A0'};
display: inline-block;
@@ -79,30 +67,6 @@ const SelectWithLabel = styled(AsyncSelect)<{ labelText: string }>`
}
`;
// The server answers with one bounded page, not an offset window: paging would
// need a stable ORDER BY, and ordering a high-cardinality column is the full
// scan this search exists to avoid. A page size no response can reach keeps
// AsyncSelect from asking for a second page.
const COMPARATOR_PAGE_SIZE = 1_000_000;
const toLabeledValue = (value: unknown): LabeledValue => ({
value: value as LabeledValue['value'],
label: optionLabel(value as null | number | boolean | string),
});
// The reverse of toLabeledValue: what AsyncSelect emits is labelled, and the
// comparator has to be the raw value or the engine cannot render it as a
// literal.
const unwrapComparator = (value: unknown): unknown => {
if (Array.isArray(value)) {
return value.map(unwrapComparator);
}
if (value !== null && typeof value === 'object' && 'value' in value) {
return (value as LabeledValue).value;
}
return value;
};
export interface SimpleExpressionType {
expressionType: keyof typeof ExpressionTypes;
column: ColumnMeta;
@@ -383,9 +347,11 @@ const AdhocFilterEditPopoverSimpleTabContent: FC<Props> = props => {
} = useSimpleTabFilterProps(props);
const [comparator, setComparator] = useState(props.adhocFilter.comparator);
const comparatorInputRef = useRef<InputRef | null>(null);
const comparatorSelectRef = useRef<AsyncSelectRef>(null);
const [loadedOptionCount, setLoadedOptionCount] = useState(0);
const [optionsTruncated, setOptionsTruncated] = useState(false);
const [suggestions, setSuggestions] = useState<
Record<'label' | 'value', any>[]
>([]);
const [loadingComparatorSuggestions, setLoadingComparatorSuggestions] =
useState<boolean>(false);
const [hasFocusedComparator, setHasFocusedComparator] =
useState<boolean>(false);
@@ -421,8 +387,18 @@ const AdhocFilterEditPopoverSimpleTabContent: FC<Props> = props => {
/>
);
const createSuggestionsPlaceholder = () =>
loadedOptionCount ? t('%s option(s)', loadedOptionCount) : '';
const getOptionsRemaining = () => {
// if select is multi/value is array, we show the options not selected
const valuesFromSuggestionsLength = Array.isArray(comparator)
? comparator.filter(v => suggestions.includes(v)).length
: 0;
return suggestions ? suggestions.length - valuesFromSuggestionsLength : 0;
};
const createSuggestionsPlaceholder = () => {
const optionsRemaining = getOptionsRemaining();
const placeholder = t('%s option(s)', optionsRemaining);
return optionsRemaining ? placeholder : '';
};
const handleSubjectChange = (subject: string) => {
setComparator(undefined);
@@ -479,63 +455,21 @@ const AdhocFilterEditPopoverSimpleTabContent: FC<Props> = props => {
operatorId !== undefined &&
DISABLE_INPUT_OPERATORS.includes(operatorId as Operators);
const canSuggestComparatorValues = Boolean(
subjectString &&
props.datasource?.filter_select &&
props.adhocFilter.clause !== Clauses.Having,
);
const hasComparatorOptions =
(operatorId && MULTI_OPERATORS.has(operatorId as Operators)) ||
canSuggestComparatorValues;
// AsyncSelect is labelInValue, so the value it is given has to be labelled
// too. Handed a bare value it still renders, but `handleOnDeselect` then
// compares `element.value` against entries that have no `.value`, matches
// nothing, and the tag cannot be removed.
//
// Memoised because AsyncSelect resets its internal selection whenever the
// identity of `value` changes. A fresh array every render would wipe out
// each pick as soon as it was made.
const comparatorSelectValue = useMemo(
() =>
Array.isArray(comparator)
? comparator.map(toLabeledValue)
: isDefined(comparator) && comparator !== ''
? toLabeledValue(comparator)
: undefined,
[comparator],
);
const handleComparatorChange = useCallback(
(value: unknown) => {
onComparatorChange(unwrapComparator(value) as string);
},
// eslint-disable-next-line react-hooks/exhaustive-deps
[props.adhocFilter, props.onChange],
);
suggestions.length > 0;
const comparatorSelectProps = {
allowClear: true,
allowNewOptions: true,
ariaLabel: t('Comparator option'),
pageSize: COMPARATOR_PAGE_SIZE,
// A capped list reads as the whole set unless it says otherwise, so an
// absent value looks like a value that does not exist. Only shown when the
// list is actually cut short.
helperText: optionsTruncated
? t(
'Only the first %s values are listed. Type to search all of them, ' +
'or enter a value that is not listed.',
loadedOptionCount,
)
: undefined,
mode:
operatorId && MULTI_OPERATORS.has(operatorId as Operators)
? ('multiple' as const)
: ('single' as const),
value: comparatorSelectValue as SelectValue,
onChange: handleComparatorChange,
loading: loadingComparatorSuggestions,
value: comparator as SelectValue,
onChange: onComparatorChange,
notFoundContent: t('Type a value here'),
placeholder: createSuggestionsPlaceholder(),
};
@@ -561,89 +495,76 @@ const AdhocFilterEditPopoverSimpleTabContent: FC<Props> = props => {
onChange: onDatePickerChange,
});
// Element-level array operators (Contains any / Contains all) search inside
// the array, so suggest individual elements; whole-array operators (=, In, …)
// keep the default distinct-array suggestions.
const arrayElements =
props.adhocFilter.operatorId === Operators.ContainsAny ||
props.adhocFilter.operatorId === Operators.ContainsAll;
// AsyncSelect throws away every loaded option when the identity of its
// `options` callback changes, so this depends on plain values rather than on
// `props.datasource`, whose identity the parent does not guarantee.
const datasourceType = props.datasource?.type;
const datasourceId = props.datasource?.id;
const loadComparatorOptions = useCallback(
async (search: string): Promise<SelectOptionsTypePage> => {
const col = subjectString;
if (!col || !canSuggestComparatorValues) {
return { data: [], totalCount: 0 };
}
const params = new URLSearchParams();
if (arrayElements) {
params.set('array_elements', 'true');
}
if (search) {
params.set('q', search);
}
const query = params.toString();
try {
const { json } = await SupersetClient.get({
endpoint:
`/api/v1/datasource/${datasourceType}/${datasourceId}` +
`/column/${encodeURIComponent(col)}/values/${query ? `?${query}` : ''}`,
});
const data = json.result.map((suggestion: unknown) => {
// Complex column values arrive as JS arrays or objects: whole arrays
// for MULTI_VALUE columns (e.g. [5, 6, 7]) and Map/Tuple objects for
// nested-container columns (e.g. {"a": ["x","y"]}). A raw
// array/object is neither a valid single-select value (antd collapses
// an array to its first element) nor renderable as a React child (an
// object throws). Render it as its literal string, which is also
// exactly what the backend's parse_array_literal expects for the
// whole-array operators.
if (suggestion !== null && typeof suggestion === 'object') {
const literal = JSON.stringify(suggestion);
return { value: literal, label: literal };
}
return {
value: suggestion as null | number | boolean | string,
label: optionLabel(suggestion as null | number | boolean | string),
};
});
setLoadedOptionCount(data.length);
setOptionsTruncated(isDefined(json.limit) && data.length >= json.limit);
// The count has to exceed what was returned. AsyncSelect treats
// `loaded >= totalCount` as "that is every value", sets allValuesLoaded
// and from then on serves searches by filtering the loaded page
// client-side -- which is the behaviour this whole change exists to
// replace. Pagination is held off by COMPARATOR_PAGE_SIZE instead.
return { data, totalCount: data.length + 1 };
} catch {
setLoadedOptionCount(0);
setOptionsTruncated(false);
return { data: [], totalCount: 0 };
}
},
[
subjectString,
canSuggestComparatorValues,
datasourceType,
datasourceId,
arrayElements,
],
);
// Options are cached per search term inside AsyncSelect; a different column
// or a switch to element-level suggestions invalidates all of them.
useEffect(() => {
comparatorSelectRef.current?.clearCache();
}, [subjectString, arrayElements]);
const refreshComparatorSuggestions = () => {
const { datasource } = props;
const col = props.adhocFilter.subject;
const having = props.adhocFilter.clause === Clauses.Having;
if (col && datasource && datasource.filter_select && !having) {
const controller = new AbortController();
const { signal } = controller;
if (loadingComparatorSuggestions) {
controller.abort();
}
// Element-level array operators (Contains any / Contains all) search
// inside the array, so suggest individual elements; whole-array
// operators (=, In, …) keep the default distinct-array suggestions.
const { operatorId } = props.adhocFilter;
const arrayElements =
operatorId === Operators.ContainsAny ||
operatorId === Operators.ContainsAll;
setLoadingComparatorSuggestions(true);
SupersetClient.get({
signal,
endpoint: `/api/v1/datasource/${datasource.type}/${datasource.id}/column/${col}/values/${
arrayElements ? '?array_elements=true' : ''
}`,
})
.then(({ json }) => {
setSuggestions(
json.result.map((suggestion: unknown) => {
// Complex column values arrive as JS arrays or objects: whole
// arrays for MULTI_VALUE columns (e.g. [5, 6, 7]) and Map/Tuple
// objects for nested-container columns (e.g. {"a": ["x","y"]}).
// A raw array/object is neither a valid single-select value
// (antd collapses an array to its first element) nor renderable
// as a React child (an object throws). Render it as its literal
// string, which is also exactly what the backend's
// parse_array_literal expects for the whole-array operators.
if (suggestion !== null && typeof suggestion === 'object') {
const literal = JSON.stringify(suggestion);
return { value: literal, label: literal };
}
return {
value: suggestion as null | number | boolean | string,
label: optionLabel(
suggestion as null | number | boolean | string,
),
};
}),
);
setLoadingComparatorSuggestions(false);
})
.catch(() => {
setSuggestions([]);
setLoadingComparatorSuggestions(false);
});
}
};
if (!datePicker) {
refreshComparatorSuggestions();
}
// loadingComparatorSuggestions intentionally omitted - set inside effect, would cause infinite loop
// eslint-disable-next-line react-hooks/exhaustive-deps
}, [
props.adhocFilter.subject,
props.adhocFilter.clause,
props.adhocFilter.operatorId,
props.datasource,
datePicker,
]);
useEffect(() => {
if (isFeatureEnabled(FeatureFlag.EnableAdvancedDataTypes)) {
@@ -749,12 +670,11 @@ const AdhocFilterEditPopoverSimpleTabContent: FC<Props> = props => {
}
>
<SelectWithLabel
ref={comparatorSelectRef}
css={css`
margin-top: ${theme.marginXS}px;
`}
labelText={labelText}
options={loadComparatorOptions}
options={suggestions}
{...comparatorSelectProps}
/>
</Tooltip>
@@ -16,12 +16,10 @@
* specific language governing permissions and limitations
* under the License.
*/
import { useCallback, useState } from 'react';
import {
AppSection,
Behavior,
ChartProps,
type DataMask,
type FilterState,
} from '@superset-ui/core';
import { supersetTheme } from '@apache-superset/core/theme';
@@ -879,41 +877,10 @@ describe('SelectFilterPlugin', () => {
expect(await screen.findByTitle('brand-new')).toBeInTheDocument();
});
test('says the list is capped when it hits the row limit', async () => {
// 3 rows of data against a limit of 3: the user is looking at a page, not
// at every value the column has.
getWrapper({ rowLimit: 3 });
userEvent.click(screen.getAllByRole('combobox')[0]);
expect(
await screen.findByText(/Only the first 3 values are listed/),
).toBeInTheDocument();
});
test('offers the ways out that the filter actually supports', async () => {
getWrapper({ rowLimit: 3, creatable: true, searchAllOptions: true });
userEvent.click(screen.getAllByRole('combobox')[0]);
expect(
await screen.findByText(/Type to search all of them/),
).toBeInTheDocument();
expect(
screen.getByText(/You can enter a value that is not listed/),
).toBeInTheDocument();
});
test('says nothing when the whole column fits under the limit', async () => {
getWrapper();
userEvent.click(screen.getAllByRole('combobox')[0]);
expect(await screen.findByRole('combobox')).toBeInTheDocument();
expect(screen.queryByText(/Only the first/)).not.toBeInTheDocument();
});
test('shows create option when searchAllOptions is true', async () => {
// Server-side search returns a bounded page, so a value that exists in the
// data can still be missing from the dropdown. Suppressing the create
// option there leaves the user with no way to apply it at all.
test('does not show create option when searchAllOptions is true', () => {
getWrapper({ creatable: true, searchAllOptions: true });
userEvent.type(screen.getByRole('combobox'), 'brand-new');
expect(await screen.findByTitle('brand-new')).toBeInTheDocument();
expect(screen.queryByTitle('brand-new')).not.toBeInTheDocument();
});
});
@@ -1426,153 +1393,6 @@ test('preserves dependent filter value restored from URL when it exists in data'
});
});
test('keeps a dependent filter empty after the user clears it', async () => {
// Regression: a dependent filter with "Select first filter value by default"
// used to re-apply the first option as soon as the cleared value round-tripped
// through the filter bar, making it impossible to clear.
jest.useRealTimers();
const setDataMaskMock = jest.fn();
const testProps = {
...selectMultipleProps,
formData: {
...selectMultipleProps.formData,
multiSelect: false,
enableEmptyFilter: false,
defaultToFirstItem: true,
// Non-empty extraFormData is what marks this filter as dependent
extraFormData: {
filters: [{ col: 'region', op: 'IN', val: ['North America'] }],
},
},
};
// The filter bar feeds every dispatched dataMask back into the plugin as the
// controlled `filterState` prop; the harness reproduces that round-trip.
const ControlledSelectFilter = () => {
const [filterState, setFilterState] = useState<FilterState>({
value: ['boy'],
});
const handleDataMask = useCallback((dataMask: DataMask) => {
setDataMaskMock(dataMask);
setFilterState(prev => ({ ...prev, ...dataMask.filterState }));
}, []);
return (
// @ts-expect-error
<SelectFilterPlugin
// @ts-expect-error
{...transformProps({ ...testProps, filterState })}
setDataMask={handleDataMask}
showOverflow={false}
/>
);
};
render(<ControlledSelectFilter />, {
useRedux: true,
initialState: {
nativeFilters: {
filters: {
'test-filter': {
name: 'Test Filter',
},
},
},
dataMask: {
'test-filter': {
extraFormData: {},
filterState: { value: ['boy'] },
},
},
},
});
userEvent.click(
screen.getByRole('img', {
name: /close-circle/i,
hidden: true,
}),
);
await waitFor(() =>
expect(setDataMaskMock).toHaveBeenLastCalledWith(
expect.objectContaining({
extraFormData: {},
filterState: expect.objectContaining({ value: null }),
}),
),
);
// Let the re-validation effects settle: the value must not come back
await act(async () => {
await Promise.resolve();
});
expect(setDataMaskMock).toHaveBeenLastCalledWith(
expect.objectContaining({
filterState: expect.objectContaining({ value: null }),
}),
);
expect(screen.queryByTitle('boy')).not.toBeInTheDocument();
});
test('keeps a dependent filter empty when it mounts with a cleared value', async () => {
// Regression: after a reload the cleared state comes back as `value: null` on
// a fresh component, so the in-memory "user cleared this" ref is gone. The
// first item must still not be re-applied.
const setDataMaskMock = jest.fn();
const testProps = {
...selectMultipleProps,
formData: {
...selectMultipleProps.formData,
multiSelect: false,
enableEmptyFilter: false,
defaultToFirstItem: true,
extraFormData: {
filters: [{ col: 'region', op: 'IN', val: ['North America'] }],
},
},
filterState: { value: null },
};
render(
// @ts-expect-error
<SelectFilterPlugin
// @ts-expect-error
{...transformProps(testProps)}
setDataMask={setDataMaskMock}
showOverflow={false}
/>,
{
useRedux: true,
initialState: {
nativeFilters: {
filters: {
'test-filter': {
name: 'Test Filter',
},
},
},
dataMask: {
'test-filter': {
extraFormData: {},
filterState: { value: null },
},
},
},
},
);
// Let the re-validation effect run before asserting it did nothing
await act(async () => {
await Promise.resolve();
});
expect(setDataMaskMock).toHaveBeenCalled();
expect(setDataMaskMock).not.toHaveBeenCalledWith(
expect.objectContaining({
filterState: expect.objectContaining({ value: ['boy'] }),
}),
);
});
test('resets dependent filter to first item when value does not exist in data', async () => {
const setDataMaskMock = jest.fn();
const testProps = {
@@ -157,6 +157,7 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
const [col] = groupby;
const [initialColtypeMap] = useState(coltypeMap);
const [search, setSearch] = useState('');
const prevDataRef = useRef(data);
const userClearedRef = useRef(false);
const [dataMask, dispatchDataMask] = useImmerReducer(reducer, {
extraFormData: {},
@@ -271,10 +272,7 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
type: 'ownState',
ownState: {
coltypeMap: initialColtypeMap,
// The dropdown offers `stripSurroundingQuotes(search)` as the
// creatable option, so the server has to be asked for the same
// string or the two disagree about what was searched for.
search: stripSurroundingQuotes(search).trim(),
search,
},
});
}
@@ -284,10 +282,8 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
const handleBlur = useCallback(() => {
unsetFocusedFilter();
if (search) {
onSearch('');
}
}, [onSearch, search, unsetFocusedFilter]);
onSearch('');
}, [onSearch, unsetFocusedFilter]);
const handleChange = useCallback(
(value?: SelectValue | number | string) => {
@@ -309,25 +305,6 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
? t('No data')
: tn('%s option', '%s options', data.length, data.length);
// A capped list reads as the whole set, so a value sitting past the row
// limit looks like a value that does not exist. Each sentence is only added
// when it is actually true of this filter's configuration.
const rowLimit = Number(formData.rowLimit) || 0;
const helperText = useMemo(() => {
if (!rowLimit || data.length < rowLimit) {
return undefined;
}
return [
t('Only the first %s values are listed.', data.length),
searchAllOptions ? t('Type to search all of them.') : undefined,
creatable !== false
? t('You can enter a value that is not listed.')
: undefined,
]
.filter(Boolean)
.join(' ');
}, [creatable, data.length, rowLimit, searchAllOptions]);
const formItemExtra = useMemo(() => {
if (filterState.validateMessage) {
return (
@@ -360,6 +337,7 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
const unquotedSearch = stripSurroundingQuotes(search);
if (
unquotedSearch &&
!searchAllOptions &&
creatable !== false &&
!hasOption(unquotedSearch, uniqueOptions, true)
) {
@@ -369,7 +347,7 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
];
}
return uniqueOptions;
}, [search, uniqueOptions, creatable]);
}, [search, uniqueOptions, creatable, searchAllOptions]);
const sortComparator = useCallback(
(a: LabeledValue, b: LabeledValue) => {
@@ -452,6 +430,26 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
clearAllTrigger,
]);
useEffect(() => {
const prev = prevDataRef.current;
const curr = data;
const hasDataChanged =
prev?.length !== curr?.length ||
prev?.some((row, i) => {
const prevVal = row[col];
const currVal = curr[i][col];
return typeof prevVal === 'bigint' || typeof currVal === 'bigint'
? prevVal?.toString() !== currVal?.toString()
: prevVal !== currVal;
});
// If data actually changed (e.g., due to parent filter), reset flag
if (hasDataChanged) {
prevDataRef.current = data;
}
}, [data, col]);
useEffect(() => {
if (
filterState.value?.every((value?: any) =>
@@ -464,17 +462,13 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
? (groupby.map(col => data[0][col]) as string[])
: null;
// Skip default value update when clearAllTrigger is active.
// `null` is a persisted "user cleared this" state, as opposed to
// `undefined` for "never set", so it must not be re-defaulted either —
// `userClearedRef` alone would not survive a reload.
// Skip default value update when clearAllTrigger is active
if (
!clearAllTrigger &&
defaultToFirstItem &&
!userClearedRef.current &&
Object.keys(formData?.extraFormData || {}).length &&
filterState.value !== undefined &&
filterState.value !== null &&
firstItem !== null &&
filterState.value !== firstItem
) {
@@ -640,7 +634,7 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
name={formData.nativeFilterId}
allowClear
autoClearSearchValue
allowNewOptions={creatable !== false}
allowNewOptions={!searchAllOptions && creatable !== false}
allowNewOptionsOnPaste={multiSelect && searchAllOptions}
allowSelectAll={!searchAllOptions}
value={multiSelect ? filterState.value || [] : filterState.value}
@@ -649,7 +643,6 @@ export default function PluginFilterSelect(props: PluginFilterSelectProps) {
showSearch={showSearch}
mode={multiSelect ? 'multiple' : 'single'}
placeholder={placeholderText}
helperText={helperText}
onClear={() => onSearch('')}
onSearch={onSearch}
onBlur={handleBlur}
@@ -117,38 +117,6 @@ describe('Select buildQuery', () => {
]);
});
test('should not sort by the searched column', () => {
// Ordering by a high-cardinality column makes the engine sort every match
// before applying the row limit; the dropdown re-sorts the page anyway.
const queryContext = buildQuery(
{ ...formData, sortAscending: true },
{
ownState: {
search: 'abc',
coltypeMap: { my_col: GenericDataType.String },
},
},
);
const [query] = queryContext.queries;
expect(query.orderby).toEqual([]);
});
test('should keep the sort metric while searching', () => {
// A sort metric decides which rows come back, so dropping it would change
// the result set rather than just its order.
const queryContext = buildQuery(
{ ...formData, sortMetric: 'my_metric', sortAscending: false },
{
ownState: {
search: 'abc',
coltypeMap: { my_col: GenericDataType.String },
},
},
);
const [query] = queryContext.queries;
expect(query.orderby).toEqual([['my_metric', false]]);
});
test('should add text search parameter for numeric to query filter', () => {
const queryContext = buildQuery(formData, {
ownState: {
@@ -54,13 +54,6 @@ const buildQuery: BuildQuery<PluginFilterSelectQueryFormData> = (
}
const sortColumns = sortMetric ? [sortMetric] : columns;
// Sorting by the searched column makes the engine scan and sort every
// match before applying the row limit, which is the dominant cost of
// search-as-you-type on a high-cardinality column. The dropdown re-sorts
// the returned page client-side, so the server sort buys nothing here. A
// sort metric is different: it selects *which* rows come back, so it has
// to stay.
const skipOrderBy = !!search && !sortMetric;
const query: QueryObject[] = [
{
...baseQueryObject,
@@ -68,7 +61,7 @@ const buildQuery: BuildQuery<PluginFilterSelectQueryFormData> = (
metrics: sortMetric ? [sortMetric] : [],
filters: filters.concat(extraFilters),
orderby:
!skipOrderBy && (sortMetric || sortAscending !== undefined)
sortMetric || sortAscending !== undefined
? sortColumns.map(column => [column, !!sortAscending])
: [],
},
+1 -10
View File
@@ -585,17 +585,8 @@ class ChartDataRestApi(ChartRestApi):
query["timing"] = query_result.timing.as_public_dict()
if security_manager.is_guest_user():
# Guests may see the generated SQL only when the role attached to
# their guest token has been granted "can view query on Dashboard",
# mirroring the permission the frontend uses to expose the
# "View query" action. Stacktraces and driver errors stay redacted
# regardless, as those leak details of the deployment itself.
can_view_query = security_manager.can_access(
"can_view_query", "Dashboard"
)
for query in queries:
if not can_view_query:
query.pop("query", None)
query.pop("query", None)
query.pop("stacktrace", None)
if query.get("error"):
query["error"] = sanitize_error_message(query["error"])
+4 -11
View File
@@ -91,10 +91,8 @@ class ExportChartsCommand(ExportModelsCommand):
def enable_tag_export(cls) -> None:
cls._include_tags = True
def run(
self, seen: set[str] | None = None
) -> Iterator[tuple[str, Callable[[], str]]]:
yield from super().run(seen=seen)
def run(self) -> Iterator[tuple[str, Callable[[], str]]]:
yield from super().run()
# Tags are exported once for all requested charts (rather than per
# chart in `_export`) so a multi-chart export doesn't lose tags to
@@ -110,17 +108,12 @@ class ExportChartsCommand(ExportModelsCommand):
@staticmethod
def _export(
model: Slice, export_related: bool = True, seen: set[str] | None = None
model: Slice, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided
if seen is None:
seen = set()
yield (
ExportChartsCommand._file_name(model),
lambda: ExportChartsCommand._file_content(model),
)
if model.table and export_related:
# Pass the shared seen set to the dataset export command
yield from ExportDatasetsCommand([model.table.id]).run(seen=seen)
yield from ExportDatasetsCommand([model.table.id]).run()
+6 -20
View File
@@ -383,12 +383,8 @@ class ExportDashboardsCommand(ExportModelsCommand):
@staticmethod
# ruff: noqa: C901
def _export(
model: Dashboard, export_related: bool = True, seen: set[str] | None = None
model: Dashboard, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided
if seen is None:
seen = set()
yield (
ExportDashboardsCommand._file_name(model),
lambda: ExportDashboardsCommand._file_content(model),
@@ -399,11 +395,8 @@ class ExportDashboardsCommand(ExportModelsCommand):
dashboard_ids = model.id
command = ExportChartsCommand(chart_ids)
command.disable_tag_export()
try:
# Pass the shared seen set to the chart export command
yield from command.run(seen=seen)
finally:
command.enable_tag_export()
yield from command.run()
command.enable_tag_export()
if feature_flag_manager.is_feature_enabled("TAGGING_SYSTEM"):
yield from ExportTagsCommand(
dashboard_ids=dashboard_ids, chart_ids=chart_ids
@@ -413,8 +406,7 @@ class ExportDashboardsCommand(ExportModelsCommand):
if model.theme:
from superset.commands.theme.export import ExportThemesCommand
# Pass the shared seen set to the theme export command
yield from ExportThemesCommand([model.theme.id]).run(seen=seen)
yield from ExportThemesCommand([model.theme.id]).run()
payload = model.export_to_dict(
recursive=False,
@@ -443,10 +435,7 @@ class ExportDashboardsCommand(ExportModelsCommand):
if dataset_id is not None:
dataset = DatasetDAO.find_by_id(dataset_id)
if dataset:
# Pass the shared seen set to the dataset export command
yield from ExportDatasetsCommand([dataset_id]).run(
seen=seen
)
yield from ExportDatasetsCommand([dataset_id]).run()
# Export datasets referenced by display controls
for customization in (
@@ -457,7 +446,4 @@ class ExportDashboardsCommand(ExportModelsCommand):
if dataset_id is not None:
dataset = DatasetDAO.find_by_id(dataset_id)
if dataset:
# Pass the shared seen set to the dataset export command
yield from ExportDatasetsCommand([dataset_id]).run(
seen=seen
)
yield from ExportDatasetsCommand([dataset_id]).run()
+1 -5
View File
@@ -113,12 +113,8 @@ class ExportDatabasesCommand(ExportModelsCommand):
@staticmethod
def _export(
model: Database, export_related: bool = True, seen: set[str] | None = None
model: Database, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided
if seen is None:
seen = set()
yield (
ExportDatabasesCommand._file_name(model),
lambda: ExportDatabasesCommand._file_content(model),
@@ -219,8 +219,6 @@ class UploadCommand(BaseCommand):
database_id=self._model_id,
editors=editors,
schema=self._schema,
# Ensure catalog is set
catalog=self._model.get_default_catalog(),
)
db.session.add(sqla_table)
+24 -37
View File
@@ -89,12 +89,8 @@ class ExportDatasetsCommand(ExportModelsCommand):
@staticmethod
def _export(
model: SqlaTable, export_related: bool = True, seen: set[str] | None = None
model: SqlaTable, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided
if seen is None:
seen = set()
yield (
ExportDatasetsCommand._file_name(model),
lambda: ExportDatasetsCommand._file_content(model),
@@ -107,41 +103,32 @@ class ExportDatasetsCommand(ExportModelsCommand):
)
file_path = f"databases/{db_file_name}.yaml"
# Only yield the database file if not already seen. This is
# critical to fix the issue where databases were being
# duplicated and potentially overwritten when charts from
# different databases were exported.
if file_path not in seen:
payload = model.database.export_to_dict(
payload = model.database.export_to_dict(
recursive=False,
include_parent_ref=False,
include_defaults=True,
export_uuids=True,
)
# TODO (betodealmeida): move this logic to export_to_dict once this
# becomes the default export endpoint
if payload.get("extra"):
try:
payload["extra"] = json.loads(payload["extra"])
except json.JSONDecodeError:
logger.info("Unable to decode `extra` field: %s", payload["extra"])
if ssh_tunnel := model.database.ssh_tunnel:
ssh_tunnel_payload = ssh_tunnel.export_to_dict(
recursive=False,
include_parent_ref=False,
include_defaults=True,
export_uuids=True,
export_uuids=False,
)
# TODO (betodealmeida): move this logic to export_to_dict once this
# becomes the default export endpoint
if payload.get("extra"):
try:
payload["extra"] = json.loads(payload["extra"])
except json.JSONDecodeError:
logger.info(
"Unable to decode `extra` field: %s", payload["extra"]
)
payload["ssh_tunnel"] = mask_password_info(ssh_tunnel_payload)
if ssh_tunnel := model.database.ssh_tunnel:
ssh_tunnel_payload = ssh_tunnel.export_to_dict(
recursive=False,
include_parent_ref=False,
include_defaults=True,
export_uuids=False,
)
payload["ssh_tunnel"] = mask_password_info(ssh_tunnel_payload)
payload["version"] = EXPORT_VERSION
payload["version"] = EXPORT_VERSION
yield (
file_path,
lambda: yaml.safe_dump(
payload, sort_keys=False, allow_unicode=True
),
)
yield (
file_path,
lambda: yaml.safe_dump(payload, sort_keys=False, allow_unicode=True),
)
+2 -2
View File
@@ -384,7 +384,7 @@ class UpdateDatasetCommand(UpdateMixin, BaseCommand):
)
exceptions.append(
ValidationError(
[message],
message,
field_name=f"{label}.{idx}.expression",
)
)
@@ -412,7 +412,7 @@ class UpdateDatasetCommand(UpdateMixin, BaseCommand):
)
exceptions.append(
ValidationError(
[message],
message,
field_name="fetch_values_predicate",
)
)
+11 -29
View File
@@ -47,45 +47,27 @@ class ExportModelsCommand(BaseCommand):
@staticmethod
def _file_content(model: Model) -> str:
raise NotImplementedError("Subclasses MUST implement _file_content")
raise NotImplementedError("Subclasses MUST implement _export")
@staticmethod
def _export(
model: Model, export_related: bool = True, seen: set[str] | None = None
model: Model, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
raise NotImplementedError("Subclasses MUST implement _export")
def run(
self, seen: set[str] | None = None
) -> Iterator[tuple[str, Callable[[], str]]]:
def run(self) -> Iterator[tuple[str, Callable[[], str]]]:
self.validate()
# Use provided seen set or create new one
if seen is None:
seen = set()
should_add_metadata = True
else:
# If seen set is provided, we're being called from another command
should_add_metadata = False
# Only add metadata if this is the root command
if should_add_metadata:
metadata = {
"version": EXPORT_VERSION,
"type": self.dao.model_cls.__name__, # type: ignore
"timestamp": datetime.now(tz=timezone.utc).isoformat(),
}
if METADATA_FILE_NAME not in seen:
yield (
METADATA_FILE_NAME,
lambda: yaml.safe_dump(metadata, sort_keys=False),
)
seen.add(METADATA_FILE_NAME)
metadata = {
"version": EXPORT_VERSION,
"type": self.dao.model_cls.__name__, # type: ignore
"timestamp": datetime.now(tz=timezone.utc).isoformat(),
}
yield METADATA_FILE_NAME, lambda: yaml.safe_dump(metadata, sort_keys=False)
seen = {METADATA_FILE_NAME}
for model in self._models:
for file_name, file_content in self._export(
model, self.export_related, seen
):
for file_name, file_content in self._export(model, self.export_related):
if file_name not in seen:
yield file_name, file_content
seen.add(file_name)
+17 -25
View File
@@ -67,12 +67,8 @@ class ExportSavedQueriesCommand(ExportModelsCommand):
@staticmethod
def _export(
model: SavedQuery, export_related: bool = True, seen: set[str] | None = None
model: SavedQuery, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided
if seen is None:
seen = set()
yield (
ExportSavedQueriesCommand._file_name(model),
lambda: ExportSavedQueriesCommand._file_content(model),
@@ -83,25 +79,21 @@ class ExportSavedQueriesCommand(ExportModelsCommand):
database_slug = secure_filename(model.database.database_name)
file_name = f"databases/{database_slug}.yaml"
# Only yield if not already seen (similar to dataset export)
if file_name not in seen:
payload = model.database.export_to_dict(
recursive=False,
include_parent_ref=False,
include_defaults=True,
export_uuids=True,
)
# TODO (betodealmeida): move this logic to export_to_dict once this
# becomes the default export endpoint
if "extra" in payload:
try:
payload["extra"] = json.loads(payload["extra"])
except (TypeError, json.JSONDecodeError):
logger.info(
"Unable to decode `extra` field: %s", payload["extra"]
)
payload = model.database.export_to_dict(
recursive=False,
include_parent_ref=False,
include_defaults=True,
export_uuids=True,
)
# TODO (betodealmeida): move this logic to export_to_dict once this
# becomes the default export endpoint
if "extra" in payload:
try:
payload["extra"] = json.loads(payload["extra"])
except json.JSONDecodeError:
logger.info("Unable to decode `extra` field: %s", payload["extra"])
payload["version"] = EXPORT_VERSION
payload["version"] = EXPORT_VERSION
file_content = yaml.safe_dump(payload, sort_keys=False)
yield file_name, lambda: file_content
file_content = yaml.safe_dump(payload, sort_keys=False)
yield file_name, lambda: file_content
+1 -3
View File
@@ -46,9 +46,7 @@ class ExportTagsCommand(ExportModelsCommand):
self.dashboard_ids = dashboard_ids
self.chart_ids = chart_ids
def run(
self, seen: set[str] | None = None
) -> Iterator[tuple[str, Callable[[], str]]]:
def run(self) -> Iterator[tuple[str, Callable[[], str]]]:
if not feature_flag_manager.is_feature_enabled("TAGGING_SYSTEM"):
return
+1 -5
View File
@@ -67,12 +67,8 @@ class ExportThemesCommand(ExportModelsCommand):
@staticmethod
def _export(
model: Theme, export_related: bool = True, seen: set[str] | None = None
model: Theme, export_related: bool = True
) -> Iterator[tuple[str, Callable[[], str]]]:
# Initialize seen set if not provided (for consistency)
if seen is None:
seen = set()
yield (
ExportThemesCommand._file_name(model),
lambda: ExportThemesCommand._file_content(model),
-1
View File
@@ -964,7 +964,6 @@ class AnnotationDatasource(BaseDatasource):
limit: int = 10000,
denormalize_column: bool = False,
array_elements: bool = False,
search: str | None = None,
) -> list[Any]:
raise NotImplementedError()
+2 -23
View File
@@ -41,9 +41,6 @@ from superset.views.base_api import BaseSupersetApi, statsd_metrics
logger = logging.getLogger(__name__)
# Cache lifetime for search-filtered column values, in seconds.
SEARCH_CACHE_TIMEOUT = 60
class DatasourceRestApi(BaseSupersetApi):
allow_browser_login = True
@@ -90,14 +87,6 @@ class DatasourceRestApi(BaseSupersetApi):
type: string
name: column_name
description: The name of the column to get values for
- in: query
schema:
type: string
name: q
description: >-
Optional case-insensitive substring; only values containing it are
returned. Lets the client search the full column rather than the
truncated first page.
responses:
200:
description: A List of distinct values for the column
@@ -147,10 +136,6 @@ class DatasourceRestApi(BaseSupersetApi):
# Element-level operators (Contains any / Contains all) request the
# distinct array *elements* rather than distinct whole arrays.
array_elements = parse_boolean_string(request.args.get("array_elements"))
# Server-side search. Without it the client can only match against the
# bounded first page, so a value beyond ``FILTER_SELECT_ROW_LIMIT`` is
# unfindable on a high-cardinality column.
search = (request.args.get("q") or "").strip() or None
# Cache distinct column-value results so a dashboard with many filters
# backed by the same (often heavy) virtual dataset doesn't re-execute
@@ -184,7 +169,6 @@ class DatasourceRestApi(BaseSupersetApi):
"limit": row_limit,
"denorm": denormalize_column,
"elements": array_elements,
"q": search,
"rls": security_manager.get_rls_cache_key(datasource),
"changed_on": str(getattr(datasource, "changed_on", "")),
},
@@ -200,7 +184,7 @@ class DatasourceRestApi(BaseSupersetApi):
logger.debug(
"column-values cache HIT: uid=%s col=%s", datasource.uid, column_name
)
response = self.response(200, result=cached, limit=row_limit)
response = self.response(200, result=cached)
response.headers["X-Cache-Status"] = "HIT"
return response
@@ -210,7 +194,6 @@ class DatasourceRestApi(BaseSupersetApi):
limit=row_limit,
denormalize_column=denormalize_column,
array_elements=array_elements,
search=search,
)
except KeyError:
return self.response(
@@ -242,15 +225,11 @@ class DatasourceRestApi(BaseSupersetApi):
timeout = datasource.cache_timeout or app.config.get(
"CACHE_DEFAULT_TIMEOUT", 300
)
if search:
# Every distinct search term is its own key, so a few users typing
# would otherwise pin one entry per keystroke for the full timeout.
timeout = min(timeout, SEARCH_CACHE_TIMEOUT)
cache_manager.data_cache.set(cache_key, payload, timeout=timeout)
logger.debug(
"column-values cache MISS: uid=%s col=%s", datasource.uid, column_name
)
response = self.response(200, result=payload, limit=row_limit)
response = self.response(200, result=payload)
response.headers["X-Cache-Status"] = "MISS"
return response
+15
View File
@@ -41,6 +41,7 @@ import sqlalchemy.dialects
from flask import current_app as app
from sqlalchemy.engine.default import DefaultDialect
from sqlalchemy.exc import NoSuchModuleError
from sqlalchemy.sql import compiler as sqla_compiler
from superset import feature_flag_manager
from superset.db_engine_specs.base import BaseEngineSpec
@@ -159,6 +160,16 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
continue
# installed 3rd-party dialects
#
# `ep.load()` runs arbitrary module-level code in the third-party package.
# Some dialects (e.g. sqlalchemy-monetdb) mutate SQLAlchemy's shared,
# process-global `compiler.OPERATORS` mapping in place on import instead
# of subclassing it, which would otherwise silently change SQL rendering
# (e.g. `!=` -> `<>`) for every dialect for the rest of the process, not
# just the misbehaving one. Snapshot/restore around each load so a
# buggy connector can't leak global compiler state into unrelated
# dialects just because it was enumerated here.
operators_snapshot = dict(sqla_compiler.OPERATORS)
for ep in entry_points(group="sqlalchemy.dialects"):
try:
dialect = ep.load()
@@ -193,6 +204,10 @@ def get_available_engine_specs() -> dict[type[BaseEngineSpec], set[str]]: # noq
if isinstance(driver, bytes):
driver = driver.decode()
drivers[backend].add(driver)
finally:
if sqla_compiler.OPERATORS != operators_snapshot:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(operators_snapshot)
dbs_denylist = app.config["DBS_AVAILABLE_DENYLIST"]
if not feature_flag_manager.is_feature_enabled("ENABLE_SUPERSET_META_DB"):
+4 -1
View File
@@ -44,7 +44,10 @@ class CockroachDbEngineSpec(PostgresEngineSpec):
DatabaseCategory.TRADITIONAL_RDBMS,
DatabaseCategory.OPEN_SOURCE,
],
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2"],
# sqlalchemy-cockroachdb declares no DBAPI dependency of its own (see
# the comment on the `cockroachdb` extra in pyproject.toml), so a
# plain `cockroachdb://` URL also needs psycopg2 installed to connect.
"pypi_packages": ["sqlalchemy-cockroachdb", "psycopg2-binary"],
"connection_string": "cockroachdb://root@{hostname}:{port}/{database}?sslmode=disable",
"default_port": 26257,
"docs_url": "https://github.com/cockroachdb/sqlalchemy-cockroachdb",
-9
View File
@@ -30,7 +30,6 @@ from superset.commands.temporary_cache.exceptions import (
TemporaryCacheResourceNotFoundError,
)
from superset.constants import MODEL_API_RW_METHOD_PERMISSION_MAP
from superset.exceptions import SupersetTemplateException
from superset.explore.form_data.schemas import FormDataPostSchema, FormDataPutSchema
from superset.extensions import event_logger
from superset.views.base_api import BaseSupersetApi, requires_json, statsd_metrics
@@ -111,8 +110,6 @@ class ExploreFormDataRestApi(BaseSupersetApi):
return self.response(403, message=str(ex))
except TemporaryCacheResourceNotFoundError as ex:
return self.response(404, message=str(ex))
except SupersetTemplateException as ex:
return self.response(ex.status, message=str(ex))
@expose("/form_data/<string:key>", methods=("PUT",))
@protect()
@@ -186,8 +183,6 @@ class ExploreFormDataRestApi(BaseSupersetApi):
return self.response(403, message=str(ex))
except TemporaryCacheResourceNotFoundError as ex:
return self.response(404, message=str(ex))
except SupersetTemplateException as ex:
return self.response(ex.status, message=str(ex))
@expose("/form_data/<string:key>", methods=("GET",))
@protect()
@@ -239,8 +234,6 @@ class ExploreFormDataRestApi(BaseSupersetApi):
return self.response(403, message=str(ex))
except TemporaryCacheResourceNotFoundError as ex:
return self.response(404, message=str(ex))
except SupersetTemplateException as ex:
return self.response(ex.status, message=str(ex))
@expose("/form_data/<string:key>", methods=("DELETE",))
@protect()
@@ -293,5 +286,3 @@ class ExploreFormDataRestApi(BaseSupersetApi):
return self.response(403, message=str(ex))
except TemporaryCacheResourceNotFoundError as ex:
return self.response(404, message=str(ex))
except SupersetTemplateException as ex:
return self.response(ex.status, message=str(ex))
+1 -10
View File
@@ -16,8 +16,6 @@
# under the License.
from typing import Optional
from jinja2.exceptions import TemplateError
from superset import security_manager
from superset.commands.chart.exceptions import (
ChartAccessDeniedError,
@@ -35,7 +33,6 @@ from superset.commands.exceptions import (
from superset.daos.chart import ChartDAO
from superset.daos.dataset import DatasetDAO
from superset.daos.query import QueryDAO
from superset.exceptions import SupersetTemplateException
from superset.utils.core import DatasourceType
@@ -56,13 +53,7 @@ def check_query_access(query_id: int) -> Optional[bool]:
# Access checks below, no need to validate them twice as they can be expensive.
query = QueryDAO.find_by_id(query_id, skip_base_filter=True)
if query:
try:
security_manager.raise_for_access(query=query)
except TemplateError as ex:
# raise_for_access() Jinja-renders the query's SQL to resolve
# the tables it touches; a malformed template surfaces here as
# a raw jinja2 exception rather than a Superset one.
raise SupersetTemplateException(str(ex)) from ex
security_manager.raise_for_access(query=query)
return True
raise QueryNotFoundValidationError()
-39
View File
@@ -198,41 +198,6 @@ def get_effective_hours_offset(
R_SUFFIX = "__right_suffix"
# Escape character for LIKE patterns built from user-supplied search text.
# Deliberately not a backslash: dialects that escape backslashes when rendering
# string literals would emit a two-character ESCAPE clause, which is a syntax
# error on engines that honour standard-conforming strings.
LIKE_ESCAPE_CHAR = "!"
def escape_like_pattern(value: str) -> str:
"""
Neutralize LIKE wildcards in user-supplied search text.
Without this a user typing ``%`` or ``_`` would match every row, which is
both wrong and, on a large table, a scan the search was meant to avoid.
"""
return (
value.replace(LIKE_ESCAPE_CHAR, LIKE_ESCAPE_CHAR * 2)
.replace("%", f"{LIKE_ESCAPE_CHAR}%")
.replace("_", f"{LIKE_ESCAPE_CHAR}_")
)
def build_like_predicate(
expr: ColumnElement[Any],
search: str,
) -> ColumnElement[Any]:
"""
Build a case-insensitive containment predicate for ``expr``.
``lower(expr) LIKE lower('%term%')`` is used rather than ``ILIKE`` because
the latter is not portable across engines.
"""
pattern = f"%{escape_like_pattern(search)}%".lower()
return sa.func.lower(expr).like(pattern, escape=LIKE_ESCAPE_CHAR)
def _normalize_mssql_virtual_dataset_sql(
sql: str, parsed_script: SQLScript, engine: str
) -> str:
@@ -4045,7 +4010,6 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
limit: int = 10000,
denormalize_column: bool = False,
array_elements: bool = False,
search: str | None = None,
) -> list[Any]:
# denormalize column name before querying for values
# unless disabled in the dataset configuration
@@ -4083,9 +4047,6 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
.select_from(tbl)
.distinct()
)
if search:
qry = qry.where(build_like_predicate(value_expr, search))
if limit:
qry = qry.limit(limit)
+3 -11
View File
@@ -162,21 +162,13 @@ def memoized_func(key: str, cache: Cache = cache_manager.cache) -> Callable[...,
def wrapped_f(*args: Any, **kwargs: Any) -> Any:
should_cache = kwargs.pop("cache", True)
force = kwargs.pop("force", False)
# always popped, even when caching is skipped, so it is never forwarded
# to the decorated function as an unexpected keyword argument.
cache_timeout = kwargs.pop("cache_timeout", None)
cache_timeout = kwargs.pop(
"cache_timeout", app.config["CACHE_DEFAULT_TIMEOUT"]
)
if not should_cache:
return f(*args, **kwargs)
# callers may explicitly pass ``cache_timeout=None`` (eg, when a database
# has no custom metadata cache timeout configured), which should fall back
# to the default timeout rather than be forwarded to the cache backend.
# the config lookup happens here so the uncached path stays independent
# of the Flask app config.
if cache_timeout is None:
cache_timeout = app.config["CACHE_DEFAULT_TIMEOUT"]
# format the key using args/kwargs passed to the decorated function
signature = inspect.signature(f)
bound_args = signature.bind(*args, **kwargs)
@@ -34,7 +34,6 @@ import pytest
from flask import g, Response
from flask.ctx import AppContext
from superset import security_manager
from superset.charts.data.api import ChartDataRestApi
from superset.commands.chart.data.get_data_command import ChartDataCommand
from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
@@ -99,25 +98,6 @@ INCOMPATIBLE_ADHOC_COLUMN_FIXTURE: AdhocColumn = {
}
def _override_view_query_permission(granted: bool) -> Any:
"""
Answer ("can_view_query", "Dashboard") with ``granted`` and let every other
permission check fall through to the real security manager, so the rest of
the request keeps its normal access rules.
"""
real_can_access = security_manager.can_access
def can_access(permission_name: str, view_name: str) -> bool:
if (permission_name, view_name) == ("can_view_query", "Dashboard"):
return granted
return real_can_access(permission_name, view_name)
return mock.patch(
"superset.charts.data.api.security_manager.can_access",
side_effect=can_access,
)
def _query_timing() -> QueryTiming:
return QueryTiming(
query_planning_ns=0,
@@ -1592,40 +1572,19 @@ class TestGetChartDataApi(BaseTestChartDataApi):
@pytest.mark.usefixtures("load_birth_names_dashboard_with_slices")
def test_chart_data_as_guest_user(self, is_guest_user, has_guest_access):
"""
Chart data API: Test response does not include the SQL query for embedded
users whose role lacks "can view query on Dashboard".
Chart data API: Test response does not inlcude the SQL query for embedded
users.
"""
g.user.rls = []
is_guest_user.return_value = True
has_guest_access.return_value = True
with _override_view_query_permission(granted=False):
rv = self.client.post(CHART_DATA_URI, json=self.query_context_payload)
rv = self.client.post(CHART_DATA_URI, json=self.query_context_payload)
data = json.loads(rv.data.decode("utf-8"))
result = data["result"]
excluded_key = "query"
assert all([excluded_key not in query for query in result]) # noqa: C419
@mock.patch("superset.security.manager.SupersetSecurityManager.has_guest_access")
@mock.patch("superset.security.manager.SupersetSecurityManager.is_guest_user")
@pytest.mark.usefixtures("load_birth_names_dashboard_with_slices")
def test_chart_data_as_guest_user_allowed_to_view_query(
self, is_guest_user, has_guest_access
):
"""
Chart data API: Test response includes the SQL query for embedded users
whose role carries "can view query on Dashboard".
"""
g.user.rls = []
is_guest_user.return_value = True
has_guest_access.return_value = True
with _override_view_query_permission(granted=True):
rv = self.client.post(CHART_DATA_URI, json=self.query_context_payload)
data = json.loads(rv.data.decode("utf-8"))
result = data["result"]
assert all("query" in query for query in result)
def test_chart_data_table_chart_with_time_grain_filter(self):
"""
Chart data API: Test that a table chart that's not using a temporal column can
@@ -533,110 +533,6 @@ class TestExportDashboardsCommand(SupersetTestCase):
{"dashboard_title": "World Bank's Data"},
)
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@patch("superset.security.manager.g")
@patch("superset.views.base.g")
def test_export_dashboard_cross_database_charts(self, mock_g1, mock_g2):
"""
Test that dashboards with charts from multiple databases export correctly.
This reproduces issue #37113 where charts from different databases were missing.
"""
mock_g1.user = security_manager.find_user("admin")
mock_g2.user = security_manager.find_user("admin")
# Create a second database for testing
second_db = Database(database_name="test_db_2", sqlalchemy_uri="sqlite://")
db.session.add(second_db)
# Create a dataset in the second database
second_dataset = SqlaTable(
table_name="second_dataset",
database=second_db,
database_id=second_db.id,
columns=[],
)
db.session.add(second_dataset)
# Flush so `second_dataset.id` is populated before it's read below;
# otherwise the chart would be constructed with `datasource_id=None`
# and never actually link back to this dataset.
db.session.flush()
# Create a chart using the second database's dataset
chart_from_second_db = Slice(
slice_name="Chart from Second Database",
datasource_type="table",
datasource_id=second_dataset.id,
datasource_name=second_dataset.table_name,
viz_type="bar",
params=json.dumps({"viz_type": "bar"}),
)
db.session.add(chart_from_second_db)
# Get the example dashboard and add the new chart
example_dashboard = (
db.session.query(Dashboard).filter_by(slug="world_health").one()
)
# Store original charts count
original_charts_count = len(example_dashboard.slices)
# Add the new chart from different database to the dashboard
example_dashboard.slices.append(chart_from_second_db)
db.session.commit()
try:
# Export the dashboard
command = ExportDashboardsCommand([example_dashboard.id])
contents = dict(command.run())
# Verify all databases are exported
db_files = [key for key in contents.keys() if key.startswith("databases/")]
assert len(db_files) >= 2, (
f"Expected at least 2 database files, got {db_files}"
)
# Verify the second database is included
assert "databases/test_db_2.yaml" in contents.keys(), (
f"Second database not found in export. Keys: {list(contents.keys())}"
)
# Verify all charts are exported (original + new one)
chart_files = [key for key in contents.keys() if key.startswith("charts/")]
assert len(chart_files) == original_charts_count + 1, (
f"Expected {original_charts_count + 1} charts, got {len(chart_files)}"
)
# Verify the new chart from second database is included
chart_from_second_db_file = None
for key in chart_files:
if f"Chart_from_Second_Database_{chart_from_second_db.id}" in key:
chart_from_second_db_file = key
break
assert chart_from_second_db_file is not None, (
f"Chart from second database not found in export. "
f"Chart files: {chart_files}"
)
# Verify the dataset from second database is included
dataset_files = [
key for key in contents.keys() if key.startswith("datasets/")
]
second_dataset_file = (
f"datasets/test_db_2/second_dataset_{second_dataset.id}.yaml"
)
assert second_dataset_file in contents.keys(), (
f"Second dataset not found. Dataset files: {dataset_files}"
)
finally:
# Clean up, even if an assertion above failed, so a failing run
# doesn't leave extra Database/Slice/SqlaTable rows for later tests.
example_dashboard.slices.remove(chart_from_second_db)
db.session.delete(chart_from_second_db)
db.session.delete(second_dataset)
db.session.delete(second_db)
db.session.commit()
class TestImportDashboardsCommand(SupersetTestCase):
def test_import_v0_dashboard_cli_export(self):
@@ -2487,8 +2487,15 @@ class TestDatabaseApi(SupersetTestCase):
}
assert response == expected_response
# Uses a `dialect+driver://` URI (rather than the bare `broken://`
# above) to also cover engine-name extraction stripping the driver
# suffix. The dialect itself ("broken") must stay one that no
# installed extra ever registers a real SQLAlchemy plugin for --
# this PR's own testcontainers extras (mssql, oracle, db2, ...)
# install real drivers for those dialects, which would make this
# URI actually attempt a connection instead of failing to load.
data = {
"sqlalchemy_uri": "mssql+pymssql://url",
"sqlalchemy_uri": "broken+driver://url",
"database_name": "examples",
"impersonate_user": False,
"server_cert": None,
@@ -2500,7 +2507,7 @@ class TestDatabaseApi(SupersetTestCase):
expected_response = {
"errors": [
{
"message": "Could not load database driver for: mssql",
"message": "Could not load database driver for: broken",
"error_type": "GENERIC_COMMAND_ERROR",
"level": "warning",
"extra": {
@@ -146,36 +146,6 @@ def test_csv_upload_dataset():
assert user_is_editor(security_manager.find_user("admin"), dataset)
@only_postgresql
@pytest.mark.usefixtures("setup_csv_upload_with_context_schema")
def test_csv_upload_dataset_catalog():
admin_user = security_manager.find_user(username="admin")
upload_database = get_upload_db()
with override_user(admin_user):
UploadCommand(
upload_database.id,
CSV_UPLOAD_TABLE_W_SCHEMA,
create_csv_file(CSV_FILE_1),
"public",
CSVReader({}),
).run()
dataset = (
db.session.query(SqlaTable)
.filter_by(
database_id=upload_database.id,
table_name=CSV_UPLOAD_TABLE_W_SCHEMA,
)
.one()
)
catalog = upload_database.get_default_catalog()
assert dataset.catalog == catalog
assert dataset.schema_perm == (
f"[{upload_database.database_name}].[{catalog}].[public]"
)
@pytest.mark.usefixtures("setup_csv_upload_with_context")
def test_csv_upload_with_index():
admin_user = security_manager.find_user(username="admin")
@@ -155,7 +155,6 @@ class TestDatasourceApi(SupersetTestCase):
limit=10000,
denormalize_column=False,
array_elements=False,
search=None,
)
@pytest.mark.usefixtures("app_context", "virtual_dataset")
@@ -171,79 +170,6 @@ class TestDatasourceApi(SupersetTestCase):
)
assert values_for_column_mock.call_args.kwargs["array_elements"] is True
@pytest.mark.usefixtures("app_context", "virtual_dataset")
def test_get_column_values_search_filters_server_side(self):
"""``?q=`` narrows the values in the database rather than client-side,
which is what makes a value beyond the row limit reachable at all."""
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
rv = self.client.get(
f"api/v1/datasource/table/{table.id}/column/col2/values/?q=b"
)
assert rv.status_code == 200
assert json.loads(rv.data.decode("utf-8"))["result"] == ["b"]
@pytest.mark.usefixtures("app_context", "virtual_dataset")
def test_get_column_values_search_is_case_insensitive(self):
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
rv = self.client.get(
f"api/v1/datasource/table/{table.id}/column/col2/values/?q=B"
)
assert rv.status_code == 200
assert json.loads(rv.data.decode("utf-8"))["result"] == ["b"]
@pytest.mark.usefixtures("app_context", "virtual_dataset")
def test_get_column_values_search_escapes_wildcards(self):
"""A literal ``%`` must not be treated as "match everything"."""
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
rv = self.client.get(
f"api/v1/datasource/table/{table.id}/column/col2/values/?q=%25"
)
assert rv.status_code == 200
assert json.loads(rv.data.decode("utf-8"))["result"] == []
@pytest.mark.usefixtures("app_context", "virtual_dataset")
@patch("superset.models.helpers.ExploreMixin.values_for_column")
def test_get_column_values_blank_search_is_ignored(self, values_for_column_mock):
"""Whitespace is not a search term; it must not narrow the list."""
values_for_column_mock.return_value = []
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
self.client.get(
f"api/v1/datasource/table/{table.id}/column/col2/values/?q=%20%20"
)
assert values_for_column_mock.call_args.kwargs["search"] is None
@pytest.mark.usefixtures("app_context", "virtual_dataset")
def test_get_column_values_returns_applied_limit(self):
"""The client needs the limit to tell a short list from a truncated
one, so it can say the list is partial instead of implying it is whole."""
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
rv = self.client.get(f"api/v1/datasource/table/{table.id}/column/col2/values/")
assert rv.status_code == 200
assert json.loads(rv.data.decode("utf-8"))["limit"] == 10000
@pytest.mark.usefixtures("app_context", "virtual_dataset")
@patch("superset.models.helpers.ExploreMixin.values_for_column")
def test_get_column_values_cache_isolated_per_search(self, values_for_column_mock):
"""Search terms must partition the cache; sharing one entry would serve
the results of somebody else's search."""
cache_manager.data_cache.clear()
values_for_column_mock.return_value = ["x"]
self.login(ADMIN_USERNAME)
table = self.get_virtual_dataset()
url = f"api/v1/datasource/table/{table.id}/column/col2/values/"
self.client.get(url)
self.client.get(f"{url}?q=a")
self.client.get(f"{url}?q=b")
self.client.get(f"{url}?q=a")
assert values_for_column_mock.call_count == 3
@pytest.mark.usefixtures("app_context", "virtual_dataset")
@patch("superset.db_engine_specs.base.BaseEngineSpec.denormalize_name")
def test_get_column_values_not_denormalize_column(self, denormalize_name_mock):
@@ -265,7 +191,6 @@ class TestDatasourceApi(SupersetTestCase):
limit=10000,
denormalize_column=True,
array_elements=False,
search=None,
)
@pytest.mark.usefixtures("app_context", "virtual_dataset")
+16
View File
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,16 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
@@ -0,0 +1,47 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared import guard for the per-dialect testcontainers modules
(tests/testcontainers/db_engine_specs/test_*.py), each of which needs its
own optional `testcontainers[...]` driver submodule to even import.
"""
import importlib
import os
def require_driver(module_name: str) -> None:
"""
Import `module_name`, a dialect's `testcontainers` driver submodule.
Most environments treat that driver as optional: a bare local `pytest`
run, or another CI job that never installed the `testcontainers` extras,
should skip the module rather than fail collection outright.
The dedicated per-dialect CI job (.github/workflows/testcontainers.yml)
sets SUPERSET_TESTCONTAINERS_STRICT, because there the driver is not
optional -- that job's matrix installs exactly this one driver for
exactly this one module. A broken or missing import there means the job
is misconfigured, and should fail loudly instead of silently reporting
a misleadingly green, zero-tests-run result.
"""
if os.environ.get("SUPERSET_TESTCONTAINERS_STRICT"):
importlib.import_module(module_name)
else:
import pytest
pytest.importorskip(module_name)
@@ -0,0 +1,63 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Shared body for the "paginated query returns correct rows in order" test
that db_engine_specs.{cockroachdb,crate,db2,mssql,oracle,trino}'s
testcontainers suites each run against their own real instance: a plain
SQLAlchemy Core LIMIT/OFFSET query, compiled and executed for real. Mocked
tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can.
Each call site keeps its own test function (and dialect-specific docstring)
so failures still report against the right module; this only factors out
the identical table setup/assert body, via an optional post-insert hook for
dialects (CrateDB) that need one.
"""
from collections.abc import Callable
from sqlalchemy import Column, insert, Integer, MetaData, select, Table as SATable
from sqlalchemy.engine import Connection, Engine
def assert_paginated_query_returns_correct_rows_in_order(
engine: Engine,
after_insert: Callable[[Connection], None] | None = None,
) -> None:
metadata = MetaData()
t = SATable(
"pilot_pagination",
metadata,
# autoincrement=False: a single-column integer primary key otherwise
# implicitly becomes AUTO_INCREMENT on MySQL/MariaDB. That column
# type treats an explicit 0 as NULL by default (NO_AUTO_VALUE_ON_ZERO
# is off), so the id=0 row below would silently get auto-assigned 1,
# colliding with the explicit id=1 row in the same batch insert.
Column("id", Integer, primary_key=True, autoincrement=False),
)
metadata.create_all(engine)
with engine.begin() as conn:
conn.execute(insert(t), [{"id": i} for i in range(10)])
if after_insert is not None:
after_insert(conn)
with engine.connect() as conn:
stmt = select(t.c.id).order_by(t.c.id).limit(3).offset(4)
rows = conn.execute(stmt).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
@@ -0,0 +1,97 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.cockroachdb against a real CockroachDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.cockroachdb import CockroachDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cockroachdb")
from testcontainers.community.cockroachdb import CockroachDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
# sqlalchemy-cockroachdb registers its dialect under the plain
# "cockroachdb" name; the container's own default ("cockroachdb+psycopg2")
# matches the abandoned `cockroachdb` package instead (see #43501).
with CockroachDBContainer(dialect="cockroachdb") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CockroachDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CockroachDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CockroachDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,103 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.crate against a real CrateDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
crate/crate only publishes an amd64 image (no arm64 build), and requires a
host CPU supporting the x86-64-v3 instruction set -- QEMU emulation on
Apple Silicon cannot satisfy that, so this file cannot run locally on an
Apple Silicon machine even with `docker pull --platform linux/amd64`. It
runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Connection, Engine
from superset.db_engine_specs.crate import CrateEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.cratedb")
from testcontainers.community.cratedb import CrateDBContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with CrateDBContainer() as container:
yield create_engine(container.get_connection_url())
def _refresh_pilot_pagination(conn: Connection) -> None:
# CrateDB is eventually consistent: a row is not guaranteed visible to
# subsequent selects immediately after insert.
conn.exec_driver_sql("REFRESH TABLE pilot_pagination")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(
engine, after_insert=_refresh_pilot_pagination
)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
CrateEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = CrateEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = CrateEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.db2 against a real IBM Db2 instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
icr.io/db2_community/db2 only publishes amd64/ppc64le/s390x images (no
arm64 build), so this cannot run locally on an Apple Silicon machine. It
runs natively on GitHub Actions' x86_64 runners. Db2 is also a notably slow
starter (a full instance bring-up, not just a process start) -- expect this
module alone to take several minutes.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.db2 import Db2EngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.db2")
from testcontainers.community.db2 import Db2Container # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with Db2Container() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
Db2EngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = Db2EngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = Db2EngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,107 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.elasticsearch against a real Elasticsearch instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
Unlike the SQL-native dialects in this directory, Elasticsearch has no
CREATE TABLE / INSERT: indices and documents get created via its REST API
(elasticsearch-dbapi's SQLAlchemy dialect is read-focused, translating SQL
to the _sql endpoint), matching how Superset actually encounters
Elasticsearch in practice -- data arrives via ingestion tooling, not
through Superset itself.
"""
from collections.abc import Iterator
import pytest
import requests
from sqlalchemy import create_engine, inspect, text
from sqlalchemy.engine import Engine
from superset.db_engine_specs.elasticsearch import ElasticSearchEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.elasticsearch")
from testcontainers.community.elasticsearch import ElasticSearchContainer # noqa: E402
INDEX = "pilot_pagination"
def _index_document(
base_url: str, index: str, doc_id: int, body: dict[str, int]
) -> None:
response = requests.put(f"{base_url}/{index}/_doc/{doc_id}", json=body, timeout=10)
response.raise_for_status()
def _refresh(base_url: str, index: str) -> None:
response = requests.post(f"{base_url}/{index}/_refresh", timeout=10)
response.raise_for_status()
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with ElasticSearchContainer("elasticsearch:8.11.0") as container:
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
base_url = f"http://{host}:{port}"
for i in range(10):
_index_document(base_url, INDEX, i, {"id": i})
_refresh(base_url, INDEX)
yield create_engine(f"elasticsearch+http://{host}:{port}/")
def test_ordered_limited_query_returns_correct_rows(engine: Engine) -> None:
"""
A plain LIMIT query, compiled and executed against a real instance.
Mocked tests cannot catch a dialect compiling this incorrectly (see
apache/superset#42899, where Trino emitted OFFSET before LIMIT) -- only
real execution can. No OFFSET here: Elasticsearch's SQL layer genuinely
doesn't support it (a protocol limitation, not a bug -- confirmed
against a real instance, which raises a parsing_exception on OFFSET).
ElasticSearchEngineSpec.supports_offset = False documents this already.
"""
with engine.connect() as conn:
rows = conn.execute(
text(f"SELECT id FROM {INDEX} ORDER BY id LIMIT 3") # noqa: S608
).fetchall()
assert [row.id for row in rows] == [0, 1, 2]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
ElasticSearchEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported field mappings
rather than a mocked Inspector.
"""
inspector = inspect(engine)
columns = ElasticSearchEngineSpec.get_columns(inspector, Table(INDEX))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = ElasticSearchEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
@@ -0,0 +1,117 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mariadb against a real MariaDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MariaDB is a MySQL fork implementing the same wire protocol: connects via
the plain "mysql" dialect with mysqlclient, same as vanilla MySQL, just
pointed at the mariadb image instead of mysql:latest.
Could not be verified locally in this environment: mysqlclient (MySQLdb)
has a pre-existing, unrelated native-library linking issue against this
machine's Homebrew-installed libmysqlclient. CI installs it via apt on
Linux, where this does not occur.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mariadb import MariaDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mysql")
from testcontainers.community.mysql import MySqlContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MySqlContainer("mariadb:11") as container:
# get_connection_url() has no host override and defaults to
# get_container_host_ip(), which is the literal string "localhost"
# on native Linux Docker (e.g. GitHub Actions runners). MySQLdb
# (mysqlclient) treats a "localhost" host specially and attempts a
# Unix socket connection instead of TCP, which fails since there's
# no local MySQL socket -- the container is reached over the
# network. Only rewrite that specific local case to 127.0.0.1; a
# remote Docker daemon reports its own real host/IP here, which
# must be preserved so the suite can still reach it.
host = container.get_container_host_ip()
if host == "localhost":
host = "127.0.0.1"
port = container.get_exposed_port(container.port)
yield create_engine(
f"mysql://{container.username}:{container.password}"
f"@{host}:{port}/{container.dbname}"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MariaDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MariaDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MariaDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,127 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.monetdb against a real MonetDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
monetdb/monetdb publishes an amd64-only image, so this needs Rosetta/QEMU
emulation on Apple Silicon -- unlike CrateDB's x86-64-v3 CPU requirement,
this one actually runs fine under emulation (verified locally). No native
testcontainers module exists for MonetDB, so this uses a generic
DockerContainer with the documented MDB_* environment variables and waits
for the daemon's own startup log line.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.monetdb import MonetDbEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
require_driver("sqlalchemy_monetdb")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import ( # noqa: E402
CompositeWaitStrategy,
LogMessageWaitStrategy,
PortWaitStrategy,
)
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
PORT = 50000
PASSWORD = "monetdb" # noqa: S105 -- fixed test-fixture password, not a secret
DBNAME = "test"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("monetdb/monetdb:latest")
container.with_exposed_ports(PORT)
container.with_env("MDB_DB_ADMIN_PASS", PASSWORD)
container.with_env("MDB_CREATE_DBS", DBNAME)
# The "Starting MonetDB daemon" log line is emitted before the image
# actually runs `monetdbd start -n`, so it alone isn't proof the server
# is accepting connections yet. Follow it with a port-connect check,
# which only succeeds once monetdbd is really listening.
container.waiting_for(
CompositeWaitStrategy(
LogMessageWaitStrategy(re.compile("Starting MonetDB daemon")),
PortWaitStrategy(PORT),
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(PORT)
yield create_engine(f"monetdb://monetdb:{PASSWORD}@{host}:{port}/{DBNAME}")
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MonetDbEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MonetDbEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MonetDbEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,141 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mongodb against a real MongoDB instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
MongoDB is schemaless, and Superset talks to it via `pymongosql`, a
SQL-to-MongoDB translation layer (dialect requires a `?mode=superset` query
param -- not part of testcontainers' own MongoDbContainer.get_connection_url()).
Documents get inserted via the native pymongo driver, not SQL INSERT,
matching how Superset actually encounters MongoDB in practice and avoiding
any assumption about pymongosql's own INSERT/DDL support.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
column,
create_engine,
inspect,
Integer,
select,
table,
text,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mongodb import MongoDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mongodb")
require_driver("pymongosql")
from testcontainers.community.mongodb import MongoDbContainer # noqa: E402
COLLECTION = "pilot_pagination"
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with MongoDbContainer("mongo:7.0.7") as container:
client = container.get_connection_client()
client[container.dbname][COLLECTION].insert_many([{"id": i} for i in range(10)])
# MongoDbContainer.get_connection_url() has no database path segment
# or query string at all (it only builds user:pass@host:port), so
# naively appending "&mode=superset" glues it straight onto the port
# number instead of starting a query string. Build the full URL
# ourselves instead of relying on string concatenation.
host = container.get_container_host_ip()
port = container.get_exposed_port(container.port)
# authSource=admin is required: MongoDbContainer creates its root
# user via MONGO_INITDB_ROOT_USERNAME, which lives in the `admin`
# database, not in `dbname` -- without it, auth fails against
# whatever database is in the URL path.
yield create_engine(
f"mongodb://{container.username}:{container.password}@{host}:{port}"
f"/{container.dbname}?mode=superset&authSource=admin"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed
against a real instance. Mocked tests cannot catch a dialect compiling
this incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can. Unlike Elasticsearch's SQL
layer (which has no OFFSET support at all), pymongosql maps OFFSET to
MongoDB's native `skip`, so this dialect supports it.
Compiled with `literal_binds=True`, matching how Superset actually
issues chart/SQL Lab queries (see `models/helpers.py`'s
`get_query_str_extended`): pymongosql's SQL-to-Mongo AST parser reads
LIMIT/OFFSET straight off the compiled SQL text ahead of parameter
substitution, so a bound `LIMIT ?`/`OFFSET ?` placeholder is rejected
("invalid literal for int() with base 10: '?'") and the clause is
silently dropped -- unlike its WHERE-clause parameter handling, which
does substitute correctly. Literal binds sidestep that and exercise
the dialect's actual LIMIT/OFFSET compilation, per this test's intent.
Uses a bare `column("id")`/`table(...)` pair rather than a full
`Table`-bound column: SQLAlchemy always qualifies a Table-bound column
reference as `pilot_pagination.id` once there's a FROM clause, and
pymongosql's projection builder takes that qualified text completely
literally as a MongoDB field path -- `{"pilot_pagination.id": 1}` reads
a *nested* field under a top-level `pilot_pagination` key, which
doesn't exist on these flat documents, silently projecting None instead
of raising. An unbound column compiles unqualified ("id"), which
resolves correctly, while still exercising the dialect's own
LIMIT/OFFSET compilation via a real Core `select()`.
"""
id_col = column("id")
stmt = (
select(id_col)
.select_from(table(COLLECTION))
.order_by(id_col)
.limit(3)
.offset(4)
)
compiled = stmt.compile(engine, compile_kwargs={"literal_binds": True})
with engine.connect() as conn:
rows = conn.execute(text(str(compiled))).fetchall()
assert [row.id for row in rows] == [4, 5, 6]
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MongoDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector, which
pymongosql implements by sampling real documents to infer column types
-- this exercises that against an actual running instance rather than
a mocked Inspector.
"""
inspector = inspect(engine)
columns = MongoDBEngineSpec.get_columns(inspector, Table(COLLECTION))
by_name = {col["column_name"]: col for col in columns}
assert "id" in by_name
spec = MongoDBEngineSpec.get_column_spec(str(by_name["id"]["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,93 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.mssql against a real SQL Server instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
mcr.microsoft.com/mssql/server only publishes an amd64 image (SQL Server on
Linux has no ARM build), so this cannot run locally on an Apple Silicon
machine. It runs natively on GitHub Actions' x86_64 runners.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.mssql import MssqlEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.mssql")
from testcontainers.community.mssql import SqlServerContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with SqlServerContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
MssqlEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = MssqlEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = MssqlEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,95 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.oracle against a real Oracle instance (gvenzl/oracle-free),
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml.
gvenzl/oracle-free ships with its datafiles pre-baked into the image, so
once the (large-ish, ~1GB) image is pulled, container startup is fast --
under 15s measured locally. Almost all the wall-clock cost here is the
image pull itself, same as any other dialect's container.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.oracle import OracleEngineSpec
from superset.sql.parse import Table
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.oracle")
from testcontainers.community.oracle import OracleDbContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with OracleDbContainer() as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
OracleEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = OracleEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = OracleEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
@@ -0,0 +1,98 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.timescaledb against a real TimescaleDB instance,
spun up on demand via testcontainers. Run via
.github/workflows/testcontainers.yml -- these exercise real SQL execution
and dialect introspection, which mocked unit tests structurally cannot.
TimescaleDB is a genuine Postgres extension, not a fork: connects via the
plain "postgresql" dialect with psycopg2, same as vanilla Postgres, just
pointed at the timescale/timescaledb image instead of postgres:latest.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.timescaledb import TimescaleDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.postgres")
from testcontainers.community.postgres import PostgresContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with PostgresContainer("timescale/timescaledb:2.29.2-pg16") as container:
yield create_engine(container.get_connection_url())
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TimescaleDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector;
this exercises that against actual server-reported column metadata
rather than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TimescaleDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TimescaleDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,104 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.trino against a real Trino instance, spun up on
demand via testcontainers. Run via .github/workflows/testcontainers.yml.
Only Presto is covered by existing docker-compose-based integration CI;
Trino, despite sharing lineage with Presto, is not.
"""
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.trino import TrinoEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.community.trino")
from testcontainers.community.trino import TrinoContainer # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
with TrinoContainer() as container:
# TrinoContainer.get_connection_url() (testcontainers 4.15.0) returns
# the container-internal port (e.g. 8080) instead of the Docker-
# mapped host port, so the URL it builds cannot actually connect.
# Build it manually with get_exposed_port() instead. Filed upstream:
# https://github.com/testcontainers/testcontainers-python/issues
url = (
f"trino://{container.user}@{container.get_container_host_ip()}"
f":{container.get_exposed_port(container.port)}/memory/default"
)
yield create_engine(url)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. This is the exact bug class in apache/superset#42899,
where Trino emitted OFFSET before LIMIT for paginated queries -- a
dialect-compiler bug invisible to mocked tests, only catchable by
actually executing the compiled SQL.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
TrinoEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = TrinoEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = TrinoEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -0,0 +1,119 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Tests db_engine_specs.yugabytedb against a real YugabyteDB instance, spun
up on demand via testcontainers. Run via .github/workflows/testcontainers.yml.
YugabyteDB's YSQL layer is fully Postgres-wire compatible (postgresql+
psycopg2, port 5433), but the image doesn't ship a plain `psql` binary
(only its own `ysqlsh`), so testcontainers' PostgresContainer can't be
reused directly -- its built-in readiness check execs `psql`, which would
fail here. This uses a generic DockerContainer, starting the node via
`yugabyted start --background=false` and waiting for yugabyted's own final
startup message instead.
"""
import re
from collections.abc import Iterator
import pytest
from sqlalchemy import (
Column,
create_engine,
inspect,
Integer,
MetaData,
Table as SATable,
)
from sqlalchemy.engine import Engine
from superset.db_engine_specs.yugabytedb import YugabyteDBEngineSpec
from superset.sql.parse import Table
from superset.utils.core import GenericDataType
pytestmark = pytest.mark.testcontainers
from ._driver import require_driver # noqa: E402
require_driver("testcontainers.core.container")
from testcontainers.core.container import DockerContainer # noqa: E402
from testcontainers.core.wait_strategies import LogMessageWaitStrategy # noqa: E402
from ._pagination import ( # noqa: E402
assert_paginated_query_returns_correct_rows_in_order,
)
YSQL_PORT = 5433
@pytest.fixture(scope="module")
def engine() -> Iterator[Engine]:
container = DockerContainer("yugabytedb/yugabyte:latest")
container.with_exposed_ports(YSQL_PORT)
container.with_command("bin/yugabyted start --background=false")
container.waiting_for(
LogMessageWaitStrategy(
re.compile("Data placement constraint successfully verified")
)
)
with container:
host = container.get_container_host_ip()
port = container.get_exposed_port(YSQL_PORT)
# Default single-node credentials/database, per yugabyted's own
# documented quickstart defaults -- no env vars needed to set them.
yield create_engine(
f"postgresql+psycopg2://yugabyte:yugabyte@{host}:{port}/yugabyte"
)
def test_paginated_query_returns_correct_rows_in_order(engine: Engine) -> None:
"""
A plain SQLAlchemy Core LIMIT/OFFSET query, compiled and executed against
a real instance. Mocked tests cannot catch a dialect compiling this
incorrectly (see apache/superset#42899, where Trino emitted OFFSET
before LIMIT) -- only real execution can.
"""
assert_paginated_query_returns_correct_rows_in_order(engine)
def test_get_columns_maps_native_types(engine: Engine) -> None:
"""
YugabyteDBEngineSpec.get_columns wraps a real SQLAlchemy Inspector; this
exercises that against actual server-reported column metadata rather
than a mocked Inspector.
"""
metadata = MetaData()
SATable(
"pilot_types",
metadata,
Column("id", Integer, primary_key=True),
Column("amount", Integer),
)
metadata.create_all(engine)
inspector = inspect(engine)
columns = YugabyteDBEngineSpec.get_columns(inspector, Table("pilot_types"))
by_name = {col["column_name"]: col for col in columns}
assert set(by_name) == {"id", "amount"}
for col in by_name.values():
spec = YugabyteDBEngineSpec.get_column_spec(str(col["type"]))
assert spec is not None
assert spec.generic_type == GenericDataType.NUMERIC
assert isinstance(spec.sqla_type, Integer)
@@ -328,10 +328,6 @@ def test_send_chart_response_strips_guest_query_after_timing_projection(
"superset.charts.data.api.security_manager.is_guest_user",
return_value=True,
),
patch(
"superset.charts.data.api.security_manager.can_access",
return_value=False,
),
):
response = api._send_chart_response(result)
finally:
@@ -343,35 +339,6 @@ def test_send_chart_response_strips_guest_query_after_timing_projection(
assert "query" in query_payload
def test_send_chart_response_keeps_guest_query_when_permitted(
app: SupersetApp,
) -> None:
"""
A guest whose role carries "can view query on Dashboard" must receive the
generated SQL, otherwise "View query" is empty on embedded dashboards.
"""
query_payload = {"data": [{"col1": 1}], "query": "SELECT 1"}
result = _json_execution_result(query_payload)
api = ChartDataRestApi()
with (
app.test_request_context("/api/v1/chart/data"),
patch(
"superset.charts.data.api.security_manager.is_guest_user",
return_value=True,
),
patch(
"superset.charts.data.api.security_manager.can_access",
return_value=True,
) as can_access,
):
response = api._send_chart_response(result)
query = json.loads(response.get_data(as_text=True))["result"][0]
assert query["query"] == "SELECT 1"
can_access.assert_called_once_with("can_view_query", "Dashboard")
def test_send_chart_response_redacts_guest_query_error(app: SupersetApp) -> None:
result = _json_execution_result(
{
@@ -389,10 +356,6 @@ def test_send_chart_response_redacts_guest_query_error(app: SupersetApp) -> None
"superset.charts.data.api.security_manager.is_guest_user",
return_value=True,
),
patch(
"superset.charts.data.api.security_manager.can_access",
return_value=False,
),
):
response = api._send_chart_response(result)
@@ -401,42 +364,6 @@ def test_send_chart_response_redacts_guest_query_error(app: SupersetApp) -> None
assert "stacktrace" not in query
def test_send_chart_response_still_redacts_guest_errors_when_query_permitted(
app: SupersetApp,
) -> None:
"""
"can view query on Dashboard" only unlocks the generated SQL; stacktraces
and driver errors describe the deployment and stay redacted for guests.
"""
result = _json_execution_result(
{
"error": "Table mydb.myschema.mytable was not found",
"stacktrace": "Traceback ...",
"query": "SELECT 1",
},
result_type=ChartDataResultType.QUERY,
)
api = ChartDataRestApi()
with (
app.test_request_context("/api/v1/chart/data"),
patch(
"superset.charts.data.api.security_manager.is_guest_user",
return_value=True,
),
patch(
"superset.charts.data.api.security_manager.can_access",
return_value=True,
),
):
response = api._send_chart_response(result)
query = json.loads(response.get_data(as_text=True))["result"][0]
assert query["query"] == "SELECT 1"
assert query["error"] == str(GENERIC_ERROR_MESSAGE)
assert "stacktrace" not in query
def test_get_data_response_redacts_guest_query_failure(app: SupersetApp) -> None:
command = MagicMock()
command.execute.side_effect = ChartDataQueryFailedError(
@@ -473,10 +473,6 @@ def test_update_dataset_rejects_malicious_expression(
f"Expected a field-level ValidationError on '{field}'. Got: "
f"{[(type(e).__name__, getattr(e, 'field_name', None), str(e)) for e in excinfo.value._exceptions]}" # noqa: E501
)
# `messages` must stay a list even though the underlying message is a
# `LazyString`, which marshmallow does not treat as `str`/`bytes` and so
# would otherwise store bare instead of wrapping it.
assert isinstance(expression_errors[0].messages, list)
def test_update_dataset_accepts_benign_expression(mocker: MockerFixture) -> None:
@@ -137,6 +137,56 @@ def test_get_available_engine_specs_keeps_valid_third_party_dialect(
assert available[SqliteEngineSpec] == {"valid_driver"}
def test_get_available_engine_specs_restores_compiler_operators(
mocker: MockerFixture,
) -> None:
"""
A third-party ``sqlalchemy.dialects`` entry point that mutates SQLAlchemy's
shared, process-global ``compiler.OPERATORS`` mapping on import (as
``sqlalchemy-monetdb`` does, in place, rather than subclassing) must not be
allowed to leak that change into every other dialect for the rest of the
process.
Regression test: enumerating a real "monetdb" entry point here (to build the
"available databases" list) silently changed ``!=`` rendering to ``<>`` for
postgres/mysql/sqlite/etc. too, for the remainder of the process.
"""
from sqlalchemy.sql import compiler as sqla_compiler, operators
mocker.patch(
"superset.db_engine_specs.load_engine_specs",
return_value=iter([]),
)
pristine = dict(sqla_compiler.OPERATORS)
assert pristine[operators.ne] != " <> "
class MisbehavingDialect(DefaultDialect):
name = "misbehaving"
driver = "misbehaving_driver"
def load_and_mutate_globally() -> type[MisbehavingDialect]:
# Mirrors sqlalchemy-monetdb's `base.py`: grabs a reference to the
# shared dict (not a copy) and mutates it in place.
sqla_compiler.OPERATORS[operators.ne] = " <> "
return MisbehavingDialect
entry_point = mocker.MagicMock()
entry_point.name = "misbehaving"
entry_point.load.side_effect = load_and_mutate_globally
mocker.patch(
"superset.db_engine_specs.entry_points",
return_value=[entry_point],
)
try:
get_available_engine_specs()
assert sqla_compiler.OPERATORS[operators.ne] == pristine[operators.ne]
finally:
sqla_compiler.OPERATORS.clear()
sqla_compiler.OPERATORS.update(pristine)
@pytest.mark.parametrize(
"app",
[{"DBS_AVAILABLE_DENYLIST": {"databricks": {"pyhive", "pyodbc"}}}],
+1 -24
View File
@@ -15,7 +15,6 @@
# specific language governing permissions and limitations
# under the License.
from flask_appbuilder.security.sqla.models import User
from jinja2.exceptions import TemplateSyntaxError
from pytest import raises # noqa: PT013
from pytest_mock import MockerFixture
@@ -31,7 +30,7 @@ from superset.commands.exceptions import (
DatasourceNotFoundValidationError,
QueryNotFoundValidationError,
)
from superset.exceptions import SupersetSecurityException, SupersetTemplateException
from superset.exceptions import SupersetSecurityException
from superset.utils.core import DatasourceType, override_user
dataset_find_by_id = "superset.daos.dataset.DatasetDAO.find_by_id"
@@ -341,28 +340,6 @@ def test_query_has_access(mocker: MockerFixture) -> None:
)
def test_query_malformed_jinja_template(mocker: MockerFixture) -> None:
"""
``raise_for_access(query=...)`` Jinja-renders the query's SQL to resolve
the tables it touches. A malformed template must surface as a
``SupersetTemplateException``, not the raw ``jinja2`` exception.
"""
from superset.explore.utils import check_datasource_access
from superset.models.sql_lab import Query
mocker.patch(query_find_by_id, return_value=Query())
mocker.patch(
raise_for_access,
side_effect=TemplateSyntaxError("unexpected end of template", lineno=1),
)
with raises(SupersetTemplateException): # noqa: PT012
check_datasource_access(
datasource_id=1,
datasource_type=DatasourceType.QUERY,
)
def test_query_no_access(mocker: MockerFixture, client) -> None:
from superset.connectors.sqla.models import SqlaTable
from superset.explore.utils import check_datasource_access
-80
View File
@@ -105,86 +105,6 @@ def test_values_for_column(database: Database) -> None:
assert table.values_for_column("a") == [1, None]
@pytest.mark.parametrize(
"raw,expected",
[
("plain", "plain"),
("50%", "50!%"),
("a_b", "a!_b"),
("wow!", "wow!!"),
("!%_", "!!!%!_"),
],
)
def test_escape_like_pattern(raw: str, expected: str) -> None:
"""Wildcards typed by a user are data, not pattern syntax."""
from superset.models.helpers import escape_like_pattern
assert escape_like_pattern(raw) == expected
def test_build_like_predicate_is_case_insensitive_and_escaped() -> None:
import sqlalchemy as sa
from superset.models.helpers import build_like_predicate
compiled = str(
build_like_predicate(sa.column("c"), "50%").compile(
dialect=sa.dialects.registry.load("postgresql")(),
compile_kwargs={"literal_binds": True},
)
).replace("%%", "%")
assert compiled == "lower(c) LIKE '%50!%%' ESCAPE '!'"
def test_values_for_column_search(database: Database) -> None:
"""``search`` narrows the distinct-value query in the database."""
import pandas as pd
from superset.connectors.sqla.models import SqlaTable, TableColumn
table = SqlaTable(
database=database,
schema=None,
table_name="t",
columns=[TableColumn(column_name="a")],
)
with patch(
"pandas.read_sql_query",
return_value=pd.DataFrame({"column_values": ["Alice"]}),
) as read_sql_query:
assert table.values_for_column("a", search="ali") == ["Alice"]
sql = str(read_sql_query.call_args.kwargs["sql"])
assert "LIKE" in sql
assert "'%ali%'" in sql
def test_values_for_column_without_search_has_no_predicate(
database: Database,
) -> None:
"""The unsearched list must stay a plain bounded DISTINCT scan."""
import pandas as pd
from superset.connectors.sqla.models import SqlaTable, TableColumn
table = SqlaTable(
database=database,
schema=None,
table_name="t",
columns=[TableColumn(column_name="a")],
)
with patch(
"pandas.read_sql_query",
return_value=pd.DataFrame({"column_values": ["Alice"]}),
) as read_sql_query:
table.values_for_column("a")
assert "LIKE" not in str(read_sql_query.call_args.kwargs["sql"])
def test_values_for_column_passes_catalog_and_schema(
mocker: MockerFixture,
session: Session,
-93
View File
@@ -54,99 +54,6 @@ def test_memoized_func(mocker: MockerFixture) -> None:
assert result == 43
def test_memoized_func_none_cache_timeout(mocker: MockerFixture) -> None:
"""
An explicit ``cache_timeout=None`` falls back to ``CACHE_DEFAULT_TIMEOUT``.
Databases without a custom metadata cache timeout pass ``None`` explicitly, and
forwarding it to the cache backend breaks backends that require an integer.
"""
from superset.utils.cache import memoized_func
_patch_config(mocker)
cache = mocker.MagicMock()
cache.get.return_value = None
decorator = memoized_func("db:{self.id}:schema:{schema}:table_list", cache)
decorated = decorator(lambda self, schema: 42)
self = mocker.MagicMock()
self.id = 1
result = decorated(self, "public", cache_timeout=None)
assert result == 42
cache.set.assert_called_once_with("db:1:schema:public:table_list", 42, timeout=100)
def test_memoized_func_custom_cache_timeout(mocker: MockerFixture) -> None:
"""
An explicit ``cache_timeout`` takes precedence over ``CACHE_DEFAULT_TIMEOUT``.
"""
from superset.utils.cache import memoized_func
_patch_config(mocker)
cache = mocker.MagicMock()
cache.get.return_value = None
decorator = memoized_func("db:{self.id}:schema:{schema}:table_list", cache)
decorated = decorator(lambda self, schema: 42)
self = mocker.MagicMock()
self.id = 1
result = decorated(self, "public", cache_timeout=42)
assert result == 42
cache.set.assert_called_once_with("db:1:schema:public:table_list", 42, timeout=42)
def test_memoized_func_disabled_cache_timeout(mocker: MockerFixture) -> None:
"""
A timeout of -1 (``CACHE_DISABLED_TIMEOUT``) skips the cache set.
"""
from superset.utils.cache import memoized_func
_patch_config(mocker)
cache = mocker.MagicMock()
cache.get.return_value = None
decorator = memoized_func("db:{self.id}:schema:{schema}:table_list", cache)
decorated = decorator(lambda self, schema: 42)
self = mocker.MagicMock()
self.id = 1
result = decorated(self, "public", cache_timeout=-1)
assert result == 42
cache.set.assert_not_called()
def test_memoized_func_skip_cache_pops_cache_timeout(mocker: MockerFixture) -> None:
"""
``cache=False`` skips caching without touching the config or the wrapped function.
``cache_timeout`` must still be popped so it is not forwarded to the decorated
function, which does not accept it. Callers such as
``get_all_table_names_in_schema`` pass ``cache`` and ``cache_timeout`` together.
"""
from superset.utils.cache import memoized_func
mock_config = mocker.patch("superset.utils.cache.app.config", MagicMock())
cache = mocker.MagicMock()
decorator = memoized_func("db:{self.id}:schema:{schema}:table_list", cache)
decorated = decorator(lambda self, schema: 42)
self = mocker.MagicMock()
self.id = 1
result = decorated(self, "public", cache=False, cache_timeout=None)
assert result == 42
cache.get.assert_not_called()
cache.set.assert_not_called()
mock_config.__getitem__.assert_not_called()
def _make_cache_instance(mocker: MockerFixture) -> MagicMock:
"""A cache instance whose ``.cache`` is not a ``NullCache``."""
cache_instance = mocker.MagicMock()