Compare commits

...
Author SHA1 Message Date
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 686245a905 chore(deps-dev): bump oxfmt from 0.63.0 to 0.64.0 in /superset-websocket (#43537)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:17:00 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 5a4900abb1 chore(deps-dev): bump vitest from 4.1.10 to 4.1.11 in /superset-websocket (#43538)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:56 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 450e2edc2b chore(deps): bump @swc/core from 1.16.0 to 1.16.1 in /docs (#43539)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:53 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 2bc96a47a4 chore(deps): bump swagger-ui-react from 5.32.13 to 5.32.14 in /docs (#43540)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:49 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 622bc45c9e chore(deps-dev): bump @swc/core from 1.16.0 to 1.16.1 in /superset-frontend (#43541)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:45 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> e97db01482 chore(deps-dev): bump oxfmt from 0.63.0 to 0.64.0 in /superset-frontend (#43542)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:40 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 6194c852f1 chore(deps-dev): bump oxfmt from 0.63.0 to 0.64.0 in /docs (#43544)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:36 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 19eca6523f chore(deps): bump uuid from 14.0.1 to 14.0.2 in /superset-frontend (#43545)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-26 02:16:31 -07:00
Amin GhadersohiandClaude 1fd763bd29 fix(mcp): honor and validate chart filters (#43478)
Co-authored-by: Claude <noreply@anthropic.com>
2026-08-25 20:54:09 -07:00
b4f7114a09 fix(semantic-layer): warn that deleting a layer cascade-deletes its dependent views (#42845)
Co-authored-by: Mike Bridge <michael.bridge@ext.preset.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 20:47:45 -07:00
Vitor Avila 2dfe8a5bd9 fix(OAuth2): Support creating OAuth2 connections via SQLAlchemy URI (#43489) 2026-08-26 00:07:19 -03:00
62d74be0af fix(metadb): apply SUPERSET_META_DB_LIMIT after join instead of per-table (#36304) (#42598)
Co-authored-by: Claude Code <noreply@anthropic.com>
Co-authored-by: Joe Li <joe@preset.io>
2026-08-25 19:31:11 -07:00
Shivam Goel 473c318335 chore(superset-core): add __init__.py to semantic_layers (#43528) 2026-08-25 18:07:28 -07:00
Evan RusackasandSuperset Dev 61ab0cdb5d fix(ocient): update GIS test fixtures for pyocient's relocated geo types (#43496)
Co-authored-by: Superset Dev <dev@superset.apache.org>
2026-08-25 17:18:02 -07:00
e6b9205821 fix(country-map): give Alborz its own ISO code instead of reusing Tehran's (#42429)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Joe Li <joe@preset.io>
Co-authored-by: Đỗ Trọng Hải <41283691+hainenber@users.noreply.github.com>
2026-08-25 16:59:48 -07:00
0dcb2ca53a feat(maps): Add Italy regions and autonomous provinces country map (#42309)
Co-authored-by: lum4chi <francesco.lumachi@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Joe Li <joe@preset.io>
2026-08-25 15:45:31 -07:00
BexultanandBexultan Mustafin 649d6f1b41 fix(mcp): defer unknown numeric types to compile (#43131)
Co-authored-by: Bexultan Mustafin <bexultan.mustafin@ffins.kz>
2026-08-25 15:35:00 -07:00
BexultanandBexultan Mustafin 90dab7cf61 fix(mcp): accept common chart input variants (#43130)
Co-authored-by: Bexultan Mustafin <bexultan.mustafin@ffins.kz>
2026-08-25 15:22:52 -07:00
Sepuri Sai KrishnaandJoe Li 68386a53ee chore(database): remove dead extra validation exception classes (#42411)
Co-authored-by: Joe Li <joe@preset.io>
2026-08-25 15:14:16 -07:00
b89da3e9fc docs(versioning): fix post-flip doc and comment drift (#43493)
Co-authored-by: Mike Bridge <michael.bridge@ext.preset.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 13:00:25 -07:00
Mike BridgeandClaude Fable 5 88d2c2954e feat(deletion-retention): persist purge block reason codes on the audit log (#43485)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
2026-08-25 12:46:55 -07:00
Amin Ghadersohi f903e02d91 chore(deps): restore permissive marshmallow lower bound (>=3.0, <5) (#43521) 2026-08-25 14:38:44 -04:00
Maxime Beaucheminandsadpandajoe fc4d7221ec fix(explore): skip re-fetch when navigating away from /explore (#39506)
Co-authored-by: sadpandajoe <jcli38@gmail.com>
2026-08-25 11:32:25 -07:00
Chen, Ting-AnandJoe Li 3585e8235a fix(i18n): review Traditional Chinese count labels (#43063)
Co-authored-by: Joe Li <joe@preset.io>
2026-08-25 10:28:12 -07:00
Evan RusackasandSuperset Dev 34ffa37aaa chore(ci): drop inert SQLALCHEMY_WARN_20 flag from unit-test CI (#43495)
Co-authored-by: Superset Dev <dev@superset.apache.org>
2026-08-25 10:19:38 -07:00
Russlan Ramdowar 02b3e43b66 fix(number-format): preserve custom smart formatter id (#43439) 2026-08-25 10:11:52 -07:00
DmitryandDmitry Kucher f9530f31ab fix(table): keep each metric's own aggregate in the summary row by default (#43421)
Co-authored-by: Dmitry Kucher <dima@virtuman.com>
2026-08-25 10:10:02 -07:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>rusackasClaude Opus 4.8
fd3849cef1 chore(deps): bump antd from 6.6.0 to 6.6.1 in /superset-frontend (#43509)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: rusackas <evan@rusackas.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-25 05:56:18 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> ef0ad01aa5 chore(deps-dev): bump the storybook group across 1 directory with 5 updates (#43507)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 05:52:46 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> d480152735 chore(deps): bump the storybook group in /docs with 2 updates (#43505)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 02:46:30 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> cbfa94c4dd chore(deps): bump antd from 6.6.0 to 6.6.1 in /docs (#43506)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 02:46:26 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> d2aeb29223 chore(deps-dev): bump baseline-browser-mapping from 2.11.14 to 2.11.15 in /superset-frontend (#43508)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 02:46:22 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 7bad69f523 chore(deps): bump dayjs from 1.11.22 to 1.11.23 in /superset-frontend (#43510)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 02:46:17 -07:00
dependabot[bot]anddependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> 9dd8c42f3d chore(deps): bump dompurify from 3.4.12 to 3.4.13 in /superset-frontend (#43511)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-08-25 02:46:12 -07:00
6c2fef29cb feat(snowflake): Add support for OAuth 2.0 authentication (#36856)
Co-authored-by: Evan Rusackas <evan@preset.io>
Co-authored-by: Evan Rusackas <evan@rusackas.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Joe Li <joe@preset.io>
2026-08-24 21:22:14 -07:00
Evan RusackasandSuperset Dev bd7b739212 chore: remove obsolete pandas/SQLAlchemy version compat shim (#43497)
Co-authored-by: Superset Dev <dev@superset.apache.org>
2026-08-24 20:11:52 -07:00
f83fb7c0e0 fix(archived-list): use the semantic-layers-aware label for the dataset type (#43465)
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Amin Ghadersohi <amin.ghadersohi@gmail.com>
2026-08-24 20:38:29 -04:00
a9d54a0037 fix(soft-delete): card-view chart delete shows the archive dialog (#43469)
Co-authored-by: Mike Bridge <michael.bridge@ext.preset.io>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Amin Ghadersohi <amin.ghadersohi@gmail.com>
2026-08-24 20:37:52 -04:00
dependabot[bot]dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>rusackasClaude Opus 4.8
8b792ab660 chore(deps-dev): bump @swc/core from 1.15.47 to 1.16.0 in /superset-frontend (#43446)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: rusackas <evan@rusackas.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-08-24 16:52:00 -07:00
Durgaprasad M LandEvan Rusackas 107204a1fc fix(sqllab): allow SQL Lab query owners to create charts without all_datasource_access (#42479)
Co-authored-by: Evan Rusackas <evan@rusackas.com>
2026-08-24 16:51:57 -07:00
90 changed files with 5666 additions and 2564 deletions
@@ -53,12 +53,6 @@ jobs:
python-version: ${{ github.event_name == 'pull_request' && fromJSON('["current"]') || fromJSON('["current", "next"]') }}
env:
PYTHONPATH: ${{ github.workspace }}
# Promotes the SQLAlchemy 2.0 deprecation warnings already locked in as
# errors via pytest.ini's `filterwarnings` to actually run in CI, so a
# regression on those fails the build instead of relying on a
# contributor remembering to set this locally. See the migration
# battleplan: https://github.com/apache/superset/discussions/40273
SQLALCHEMY_WARN_20: "1"
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+18 -12
View File
@@ -58,6 +58,7 @@ the old counter to use the outcome-specific replacements.
- [42930](https://github.com/apache/superset/pull/42930): Dataset import data-URI fetches no longer honor an HTTP(S) proxy when `DATASET_IMPORT_ALLOW_INTERNAL_DATA_URLS` is `False` (the default): the connection is now made directly to the destination so the peer-address check validates the real target instead of a proxy's. Deployments that require an egress proxy to reach legitimate external data URLs for dataset import should set `DATASET_IMPORT_ALLOW_INTERNAL_DATA_URLS = True` or otherwise ensure those URLs resolve without one.
- [42935](https://github.com/apache/superset/pull/42935): The MCP service now refuses to start (`MCPAuthConfigError`) when `MCP_JWT_ISSUER` trusts more than one issuer and no `MCP_USER_RESOLVER` is configured, instead of only logging a warning. This was already a documented misconfiguration (the default resolver isn't issuer-scoped, so distinct trusted issuers minting the same username/email would resolve to the same Superset user); deployments trusting multiple issuers must configure an `MCP_USER_RESOLVER` that derives its identity from the token's `iss` claim before upgrading. Single-issuer deployments are unaffected.
- [42429](https://github.com/apache/superset/pull/42429): The Country Map chart's Iran GeoJSON now gives Alborz province its own ISO 3166-2 code, `IR-32`, instead of `IR-30`. `ISO` is the join key used to color/filter provinces on this chart, so any existing dataset keyed on `IR-30` for Alborz will silently stop matching after upgrading; re-key that data to `IR-32`.
- [43388](https://github.com/apache/superset/pull/43388): The MCP service now refuses to start (`MCPAuthConfigError`) if `MCP_DEV_USERNAME` and `MCP_AUTH_ENABLED = True` are both set, and separately if `MCP_AUTH_ENABLED = True` but no usable JWT key material is configured (RSA key/JWKS, or an explicit `MCP_JWT_SECRET` for HMAC) — both previously started with authentication silently weaker than configured. Deployments combining a dev-mode username with JWT auth enabled, or enabling JWT auth without key material, must pick one before upgrading: unset `MCP_DEV_USERNAME` for a real auth deployment, or unset `MCP_AUTH_ENABLED` (or configure the key material) for a dev-mode one. Response caching (`MCP_CACHE_CONFIG["enabled"] = True`) now also excludes every tool with a side effect by default, not only a partial list, so a previously-cached mutating tool call is no longer served from cache; no config change is needed to pick this up.
- [42393](https://github.com/apache/superset/pull/42393): Exported dataset YAML now carries a `uuid` for each metric and column so that custom folder assignments (which reference metrics/columns by UUID) survive an import into another workspace. This affects any export bundle that contains datasets, not just a dataset export: chart, dashboard, database and full-asset exports all embed the same dataset YAML, so a dashboard exported from this release also fails to import into an older one even though no dataset was exported directly. As with `folders` and `currency_code_column`, the affected `datasets/` files fail schema validation (`Unknown field: uuid`) when imported into Superset releases that predate this change; regenerate or hand-edit exports for older targets in mixed-version fleets.
- [42300](https://github.com/apache/superset/pull/42300): Timeseries charts (line/area/bar) with a Y-axis bound in effect — either an explicit `yAxisBounds` or one derived from `truncateYAxis` — now clamp out-of-range data points to that bound instead of letting ECharts drop the point (and the line segments around it) entirely. Any existing chart with a configured Y-axis bound and data outside it will look different after upgrading: a gap becomes a point pinned to the boundary. The clamp also rewrites the value ECharts reads for that point's tooltip and data label, so the displayed value is the bound rather than the true observation.
@@ -184,10 +185,12 @@ misrepresents the entity as unchanged.
- **Storage growth.** Capture writes shadow rows per save, so the metadata
database grows with edit volume. The `version_history.prune_old_versions`
beat task removes rows whose transaction is older than
`SUPERSET_VERSION_HISTORY_RETENTION_DAYS` (default 30). A deployment that
replaces `CELERY_CONFIG` rather than inheriting it must carry both the
`superset.tasks.version_history_retention` import and the beat entry; a
startup warning names whichever is absent.
`SUPERSET_VERSION_HISTORY_RETENTION_DAYS` (default 30).
- **Check a replaced `CELERY_CONFIG`.** Carry both the
`superset.tasks.version_history_retention` import and the
`version_history.prune_old_versions` beat entry; see
[Version-history retention (pruning)](#version-history-retention-pruning) for
the startup-warning behavior.
- **`PUT` responses change shape.** Entity updates now return populated
`old_version_uuid` / `new_version_uuid` fields and an `ETag` header, which
were null or absent while capture was off.
@@ -196,7 +199,10 @@ misrepresents the entity as unchanged.
kill-switch — not removed with the rollout toggles. Setting it to a falsy value
stops capture within a restart, without a revert-and-redeploy. Unlike the
soft-delete toggle, turning it off is a clean stop: existing version rows remain
readable and no entity state is altered.
readable and no entity state is altered. Restore is unavailable (404) while
capture is off. A full rollback also sets
`FEATURE_FLAGS = {"VERSION_HISTORY": False}` to hide the panel — capture off
with the panel left on shows an empty or stale history.
### Scheduled report execution now enforces one application deadline
@@ -658,9 +664,9 @@ ALTER TABLE tagged_object DROP CONSTRAINT <constraint_name>;
ALTER TABLE tagged_object DROP FOREIGN KEY <constraint_name>;
```
### Entity version-history infrastructure (gated off by default)
### Entity version-history infrastructure
Introduces the schema and SQLAlchemy-Continuum wiring that captures version history for charts, dashboards, and datasets, plus read-only `GET /api/v1/{chart,dashboard,dataset}/<uuid>/versions/` endpoints. This ships **inert**: a new config flag `ENABLE_VERSIONING_CAPTURE` defaults to `False`, so no save writes any version rows and the endpoints return empty. It is an operational kill-switch (a release toggle that becomes a permanent ops switch), not a feature flag — set it to `True` to enable capture once validated. The migration is additive; existing entity `PUT` responses gain `old_version_uuid` / `new_version_uuid` body fields and an `ETag` header (both null/absent when capture is off).
Introduces the schema and SQLAlchemy-Continuum wiring that captures version history for charts, dashboards, and datasets, plus read-only `GET /api/v1/{chart,dashboard,dataset}/<uuid>/versions/` endpoints. Capture is governed by the `ENABLE_VERSIONING_CAPTURE` config value — an operational kill-switch (a release toggle that became a permanent ops switch), not a feature flag; see "Version history is on by default" above for the shipped default. With capture off, no save writes version rows; the endpoints continue to serve already-captured rows read-only. The migration is additive; existing entity `PUT` responses gain `old_version_uuid` / `new_version_uuid` body fields and an `ETag` header (both null/absent when capture is off).
A few save- and import-path internals change **unconditionally** (independent of the flag), because the versioned mappers must behave correctly whether or not capture is enabled:
@@ -681,7 +687,7 @@ A read-only companion to the version-history endpoints: each entity type gains a
| `q` | string | — | Case-insensitive search over the full history, applied before pagination (so `count` reflects matches) |
| `page` / `page_size` | integer | `0` / `25` | Pagination (`page_size` clamped to 200) |
Authorization reuses the resource's `can_read` permission and per-object `raise_for_access`; related-entity rows are visibility-filtered to what the caller may see. The stream is empty unless version capture is on (`ENABLE_VERSIONING_CAPTURE`).
Authorization reuses the resource's `can_read` permission and per-object `raise_for_access`; related-entity rows are visibility-filtered to what the caller may see. The stream reflects captured history; with capture off it remains readable but stops accruing new entries.
### Version-history retention (pruning)
@@ -701,7 +707,7 @@ Purging is **live by default** (`SOFT_DELETE_PURGE_DRY_RUN=False`), so the reten
Deployments that replace the default `CELERY_CONFIG` must ensure workers register `superset.tasks.deletion_retention` and schedule the `deletion_retention.purge_soft_deleted` task themselves. The shipped Docker development config uses `imports` and includes both entries. While `SOFT_DELETE` is statically enabled, a missing beat entry logs a startup warning; when the override explicitly defines `imports`, a missing purge module is also reported.
Operators can immediately erase a specific entity for compliance (GDPR) via `superset deletion-retention force-purge --uuid <uuid>`; this applies legacy hard-delete semantics — a live chart referencing a force-purged dataset is left without a datasource until re-pointed (the chart is not modified), and it purges the named entity even when it was never soft-deleted. Every scheduled evaluation writes a provisional, content-free record to the new `purge_audit_log` table before the cascade starts. Meaningful retained outcomes survive the entity they name. Consecutive scheduled evaluations with the same blocked outcome suppress only the redundant current provisional record; completed outcomes, outcome transitions, and every force-purge attempt remain independent and immutable. The **scheduled** purge fails closed when its provisional record cannot be written, while **force-purge** proceeds even if the audit write fails — the operator is present and deletion outranks audit for a compliance erasure. Operators can monitor `deletion_retention.blocked_audit_suppressed` and `deletion_retention.blocked_audit_dedupe_fallback` to verify suppression and fail-safe fallback behavior without changing the existing blocked-workload gauge.
Operators can immediately erase a specific entity for compliance (GDPR) via `superset deletion-retention force-purge --uuid <uuid>`; this applies legacy hard-delete semantics — a live chart referencing a force-purged dataset is left without a datasource until re-pointed (the chart is not modified), and it purges the named entity even when it was never soft-deleted. Every scheduled evaluation writes a provisional, content-free record to the new `purge_audit_log` table before the cascade starts. Meaningful retained outcomes survive the entity they name. Blocked audit records carry a stable machine-readable `reason` code (`report_schedule`, `user_attribute`, or `cascade_integrity_failure` for an unexpected cascade failure caused by a database integrity constraint) so the audit table alone answers why an entity was not purged; records finalized before the column existed keep a NULL reason. Apply the migration before rolling out the new code: the audit model declares the column, so a worker on the new code with an un-migrated table fails its write-ahead write and the scheduled purge fails closed until the migration lands. During a rolling deploy, workers still on the old code write reason-less blocked rows and suppress on status alone; both effects are self-healing, since a NULL-reason record never matches a reason code and the next all-new-code run re-anchors the entity. Consecutive scheduled evaluations blocked with the same status **and reason** suppress only the redundant current provisional record — a reason change writes one new blocked record carrying the new code; completed outcomes, outcome transitions, and every force-purge attempt remain independent and immutable. Retained transition records are not automatically expired, so entities whose block reason changes repeatedly can accumulate multiple audit rows. The **scheduled** purge fails closed when its provisional record cannot be written, while **force-purge** proceeds even if the audit write fails — the operator is present and deletion outranks audit for a compliance erasure. Operators can monitor `deletion_retention.blocked_audit_suppressed` and `deletion_retention.blocked_audit_dedupe_fallback` to verify suppression and fail-safe fallback behavior without changing the existing blocked-workload gauge.
### Recently Archived view and permanent delete (purge) endpoints
@@ -897,7 +903,7 @@ The migration is transactional (all-or-nothing) and idempotent — it can be saf
### Soft delete and restore for datasets
**The soft-delete behavior in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `False`** (`@lifecycle: development`), so on a default deployment `DELETE /api/v1/dataset/<id>` continues to **hard-delete permanently** — nothing is recoverable. Enable `SOFT_DELETE` to get the behavior described below.
**The soft-delete behavior in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `True`** (`@lifecycle: testing`), so on a default deployment `DELETE /api/v1/dataset/<id>` uses the recoverable soft-delete behavior described below. Setting `SOFT_DELETE` to `False` restores legacy permanent hard-delete behavior for subsequent deletes.
**Flag-toggle caveat:** the soft-delete visibility filter is evaluated per query while the flag is on. If datasets are soft-deleted during a flag-on window and the flag is later turned **off**, those rows reappear as live datasets in all lists, lookups, and relationship loads (including charts that reference them). The `POST /<uuid>/restore` endpoint and the `dataset_deleted_state` list filter remain functional regardless of the flag, deliberately, so rows soft-deleted during a flag-on window stay discoverable and restorable after a rollback of the flag.
@@ -927,7 +933,7 @@ With the flag enabled: `DELETE /api/v1/dataset/<id>` no longer hard-deletes the
### Soft delete and restore for charts
**Everything in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `False`** (`@lifecycle: development`), so on a default deployment `DELETE /api/v1/chart/<id>` continues to **hard-delete permanently** — nothing is recoverable. Enable `SOFT_DELETE` to get the behavior described below.
**Everything in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `True`** (`@lifecycle: testing`), so on a default deployment `DELETE /api/v1/chart/<id>` uses the recoverable soft-delete behavior described below. Setting `SOFT_DELETE` to `False` restores legacy permanent hard-delete behavior for subsequent deletes.
**Flag-toggle caveat:** the soft-delete visibility filter is evaluated per query while the flag is on. If charts are soft-deleted during a flag-on window and the flag is later turned **off**, those rows reappear as live charts in all lists, lookups, and relationship loads (including dashboards that contained them). The `POST /<uuid>/restore` endpoint and the `chart_deleted_state` list filter remain functional regardless of the flag, deliberately, so rows soft-deleted during a flag-on window stay discoverable and restorable after a rollback of the flag.
@@ -951,7 +957,7 @@ With the flag enabled: `DELETE /api/v1/chart/<id>` no longer hard-deletes the ch
### Soft delete and restore for dashboards
**Everything in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `False`** (`@lifecycle: development`), so on a default deployment `DELETE /api/v1/dashboard/<id>` continues to **hard-delete permanently** — nothing is recoverable. Enable `SOFT_DELETE` to get the behavior described below.
**Everything in this section applies only when the `SOFT_DELETE` feature flag is enabled. The flag defaults to `True`** (`@lifecycle: testing`), so on a default deployment `DELETE /api/v1/dashboard/<id>` uses the recoverable soft-delete behavior described below. Setting `SOFT_DELETE` to `False` restores legacy permanent hard-delete behavior for subsequent deletes.
**Flag-toggle caveat:** the soft-delete visibility filter is evaluated per query while the flag is on. If dashboards are soft-deleted during a flag-on window and the flag is later turned **off**, those rows reappear as live dashboards in all lists and lookups (including slug lookups — if a soft-deleted dashboard's slug was reused while the flag was on, both rows become visible with the same slug). The `POST /<uuid>/restore` endpoint and the `dashboard_deleted_state` list filter remain functional regardless of the flag, deliberately, so rows soft-deleted during a flag-on window stay discoverable and restorable after a rollback of the flag.
+1
View File
@@ -86,6 +86,7 @@
"Israel",
"Italy",
"Italy (regions)",
"Italy (regions and autonomous provinces)",
"Ivory Coast",
"Japan",
"Jordan",
@@ -26,7 +26,8 @@ page and its menu entry are hidden, and deletes are permanent as before.
## Finding archived objects
Open **Recently Archived** and pick a type — **Chart**, **Dashboard**, or
**Dataset** — from the Type selector. The view shows one type at a time; each
**Dataset** (shown as **Datasource** when semantic layers are enabled) — from
the Type selector. The view shows one type at a time; each
type is read from its own list endpoint, so the same row-level access rules that
govern the normal lists apply here.
+12 -12
View File
@@ -15,29 +15,29 @@ description of what changed — "Chart renamed to Q3 Revenue", "Added filter on
'Region'" — rather than a raw diff. You can search the history and filter it
down to changes on the entity itself or on the things it depends on.
## Enabling it
Two switches are involved, and both matter.
## Enabling and disabling it
| Setting | Type | Effect |
| --- | --- | --- |
| `VERSION_HISTORY` | Feature flag | Shows the version history UI |
| `ENABLE_VERSIONING_CAPTURE` | Config value | Records versions as entities are saved |
Both default to on. To turn the feature off:
```python
# superset_config.py
FEATURE_FLAGS = {"VERSION_HISTORY": True}
ENABLE_VERSIONING_CAPTURE = True
FEATURE_FLAGS = {"VERSION_HISTORY": False}
ENABLE_VERSIONING_CAPTURE = False
```
Both default to off. They are separate because capture is the expensive half:
an operator may want to start recording history before exposing the UI, so that
there is something to show when they do.
Restart Superset and its workers for the capture change to take effect. Existing
history remains readable while capture is off, but **Restore** is unavailable
(404).
Turning the UI on without capture gives a panel that reports "No history yet"
and never fills, so enable capture first — or at the same time. History only
accrues from the moment capture is switched on; earlier edits are not
reconstructed.
Disable them together: capture off with the UI left on gives a panel that
stops filling — an empty or stale history misrepresents the entity as
unchanged. History only accrues while capture is on; edits made while it was
off are not reconstructed.
## Viewing history
+6 -6
View File
@@ -58,10 +58,10 @@
"@fontsource/inter": "^5.3.0",
"@mdx-js/react": "^3.1.1",
"@saucelabs/theme-github-codeblock": "^0.3.0",
"@storybook/addon-docs": "^10.5.8",
"@storybook/addon-docs": "^10.5.9",
"@superset-ui/core": "^0.20.4",
"@swc/core": "^1.16.0",
"antd": "^6.6.0",
"@swc/core": "^1.16.1",
"antd": "^6.6.1",
"baseline-browser-mapping": "^2.11.15",
"caniuse-lite": "^1.0.30001809",
"docusaurus-plugin-openapi-docs": "^5.2.0",
@@ -77,8 +77,8 @@
"react-table": "^7.8.0",
"remark-import-partial": "^0.0.2",
"reselect": "^5.2.0",
"storybook": "^10.5.8",
"swagger-ui-react": "^5.32.13",
"storybook": "^10.5.9",
"swagger-ui-react": "^5.32.14",
"swc-loader": "^0.2.7",
"tinycolor2": "^1.4.2",
"unist-util-visit": "^5.1.0"
@@ -94,7 +94,7 @@
"eslint": "^9.39.2",
"eslint-plugin-react": "^7.37.5",
"globals": "^17.11.0",
"oxfmt": "^0.63.0",
"oxfmt": "^0.64.0",
"typescript": "~6.0.3",
"typescript-eslint": "^8.67.0",
"webpack": "^5.109.2"
+12 -12
View File
@@ -93,12 +93,6 @@
"lifecycle": "development",
"description": "Enable semantic layers and show semantic views alongside datasets"
},
{
"name": "SOFT_DELETE",
"default": true,
"lifecycle": "development",
"description": "Temporary rollout / kill-switch gate for soft delete (off = legacy hard delete). An emergency stop, not a clean rollback: flipping ON->OFF resurrects already-soft-deleted rows. Retained through this release as the move-back lever; removed (along with its two gate points \u2014 BaseDAO.delete routing and the do_orm_execute visibility listener) once post-flip confidence is established."
},
{
"name": "TABLE_V2_TIME_COMPARISON_ENABLED",
"default": false,
@@ -110,12 +104,6 @@
"default": false,
"lifecycle": "development",
"description": "Enables the tagging system for organizing assets"
},
{
"name": "VERSION_HISTORY",
"default": true,
"lifecycle": "development",
"description": "Enables the version history panel on Explore and Dashboard pages. History only accrues while ``ENABLE_VERSIONING_CAPTURE`` is also on; with capture off the panel renders but stays empty, so the two ship with matching defaults and should be changed together."
}
],
"testing": [
@@ -233,6 +221,12 @@
"lifecycle": "testing",
"description": "Apply RLS rules to SQL Lab queries. Requires query parsing/manipulation. May break queries or allow RLS bypass. Use with care!"
},
{
"name": "SOFT_DELETE",
"default": true,
"lifecycle": "testing",
"description": "Temporary rollout / kill-switch gate for soft delete (off = legacy hard delete). An emergency stop, not a clean rollback: flipping ON->OFF resurrects already-soft-deleted rows. Retained through this release as the move-back lever; removed (along with its two gate points \u2014 BaseDAO.delete routing and the do_orm_execute visibility listener) once post-flip confidence is established."
},
{
"name": "SSH_TUNNELING",
"default": false,
@@ -245,6 +239,12 @@
"default": false,
"lifecycle": "testing",
"description": "Use analogous colors in charts"
},
{
"name": "VERSION_HISTORY",
"default": true,
"lifecycle": "testing",
"description": "Enables the version history panel on Explore and Dashboard pages. History only accrues while ``ENABLE_VERSIONING_CAPTURE`` is also on; with capture off the panel renders empty or stale history, so the two ship with matching defaults and should be changed together."
}
],
"stable": [
+207 -207
View File
@@ -3175,100 +3175,100 @@
resolved "https://registry.yarnpkg.com/@oxc-resolver/binding-win32-x64-msvc/-/binding-win32-x64-msvc-11.23.0.tgz#8b66dbfa7b796139e719063fc0e44084e80a1c15"
integrity sha512-gUGJpr+Rn6zMxm5juApV0K3U845i8t47o8k+rbO0BHbi4PoJIfSPeQmrE2dgohQm2g5k6iviNFyXCGqvmaYUpw==
"@oxfmt/binding-android-arm-eabi@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-android-arm-eabi/-/binding-android-arm-eabi-0.63.0.tgz#136176dc94fdc41e21415cc770d86f5066282e0f"
integrity sha512-YmRth4ZPGgEXcgmkhvANbC9uD67dxmSobW7DQuyt5tOBOKvPnIpk5SVHBj88E+7wMNRI2FhqaDbOhQFBix+b8A==
"@oxfmt/binding-android-arm-eabi@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-android-arm-eabi/-/binding-android-arm-eabi-0.64.0.tgz#e14e25c032f6d8a6b025eb5ee7bb606c3cbdd10e"
integrity sha512-o6uzh/jTOQeAY5TdkAeXdqv7MBRcPxiRA08zrcBtkKj5cSu/FMu0Hl7Q6Fi1KCKyCWZ6lJVjBzdsJvsKltUsGQ==
"@oxfmt/binding-android-arm64@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-android-arm64/-/binding-android-arm64-0.63.0.tgz#10bc42457179210061c801122a64304619e3bdab"
integrity sha512-icbahX8X2X3sRamOMecvdYeZXWjPDazRDIfvWfy7Ca1nc/ZDT2Y9k5Nt7s46EqFd7NQPdgk+CM3/SgIT5LPCaQ==
"@oxfmt/binding-android-arm64@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-android-arm64/-/binding-android-arm64-0.64.0.tgz#294a15b8402eedde0e0a467748e3efadf61bf523"
integrity sha512-jRGSUeeP7p3Gynw2YaCVtjBIA6ZxY6bEB/ES5i54OhqmRTyuVg7ZgstEtzgq6GOAJd+2QZ5pvf+bFfmW5Mp9cw==
"@oxfmt/binding-darwin-arm64@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-darwin-arm64/-/binding-darwin-arm64-0.63.0.tgz#5f9084d9a760a1836387f8970a7f9d614ec3d909"
integrity sha512-WV+Ze5v5gI2qoj8jpAovt8KBTW8pjEz/AiMXXjeTQS+Bmf/MmZXTS40S8xNPDszX+W8WDv2Bbk6qKrMTtUGu1A==
"@oxfmt/binding-darwin-arm64@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-darwin-arm64/-/binding-darwin-arm64-0.64.0.tgz#d55b1a5d5d97d4ccde8e4be7b63e06e4e56f2d13"
integrity sha512-JINwtU2lW7nOFSqi+H2qplipNUqah9Gc1jgGmB82kTD4UnZrZIVxCJ9qEmFiKfjNq27gYLFhrUb0to86aCwMjw==
"@oxfmt/binding-darwin-x64@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-darwin-x64/-/binding-darwin-x64-0.63.0.tgz#badd4a02218a9a62319817d5c337b30159a54a21"
integrity sha512-CJGSBdDxXOWIpoFXHpverimCvz084KA7L483rqJ44c3jDtzv6d4qOSoR/V9ywSHfV+Ks1lwIj2P49BFhunLNAA==
"@oxfmt/binding-darwin-x64@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-darwin-x64/-/binding-darwin-x64-0.64.0.tgz#1c9673270ed597ba9456d40fa0607d50e81158ea"
integrity sha512-gCmuswrgrOSajV4HCRFkVCGIruPq8bjYuPYgSE2WQB3mD6XrdyZ3JMSRZCkQ8zCxOyGWriBo6QoZ5nmMHQ1BfA==
"@oxfmt/binding-freebsd-x64@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-freebsd-x64/-/binding-freebsd-x64-0.63.0.tgz#a17261e95c8ebef1f76d8aaac746a64fdb6ba51e"
integrity sha512-BDfKY+KhL2078cgswBBFQPAYuxCy93bS/iC5frdSeSbTLcGrR6VC2hsuPTanoJmg84+wSyWl0wWC1eR+uTnkRg==
"@oxfmt/binding-freebsd-x64@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-freebsd-x64/-/binding-freebsd-x64-0.64.0.tgz#9e8f8b3a5a558043c664d43d54e441756af30c56"
integrity sha512-Ab8g7a38pT0MMImjh7anRSTve6buWBIlcXIFBYa5xl4s6UxEgKSc2xOOhbGtLwvXnEi2PsEDGoJh3oUU7xkehQ==
"@oxfmt/binding-linux-arm-gnueabihf@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.63.0.tgz#baeee34bb08e0769af878623f442e83bc0aacd7a"
integrity sha512-Ov1cQEXT4mj7cojAokWSS1eoxkoyvbDfAbxNsGIKY2o36kvdAaFzPxRN6NxFRk9fD72B8oCoTTX/NuYTUWlpsg==
"@oxfmt/binding-linux-arm-gnueabihf@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm-gnueabihf/-/binding-linux-arm-gnueabihf-0.64.0.tgz#cfe552538c9e9402ca64d7b83b1ccf02457ef391"
integrity sha512-BgvS3CoQ+Xy2deoZqEN8JVKabcCZi2RxA3yant8G9OAv9KuPJ9TCjHkqigzdHUVwErZxEP5d2bzLIEyKYyBDLg==
"@oxfmt/binding-linux-arm-musleabihf@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.63.0.tgz#e70d5697ec4b6bb5f87a3f019e01b3f956b8e44b"
integrity sha512-0LE7ro3+6L79jcMANycAZfRaC7zxr9YZ2+vEL5uMD9QlEep+rS/r1kSJsnuLl991NXJZD60euh0PC1GHrR20vw==
"@oxfmt/binding-linux-arm-musleabihf@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm-musleabihf/-/binding-linux-arm-musleabihf-0.64.0.tgz#1944e367da59e8b1770c5ba96465d0c7e640053e"
integrity sha512-QXpNxwoMj0YvnceCNZadNSden3bIcnvjn/sDp/rwZhRoZoZYGpHvtPyhGsdJz9uvT9GkaMW7SsLddurU56dt8w==
"@oxfmt/binding-linux-arm64-gnu@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.63.0.tgz#638a8ed4f3d256c50aeb6d2c19cfc65792c902e1"
integrity sha512-izPk+2Z4gjuZK32Fqh5qXoMpT/2NXzLh++ob57HiEiVSQZ1iYXu8EKMzb+K5AvWyIEXhdDIt7ADjGGtFhkT9Bw==
"@oxfmt/binding-linux-arm64-gnu@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm64-gnu/-/binding-linux-arm64-gnu-0.64.0.tgz#510386113bf6a128cf3106d612471dbd1a13b0f4"
integrity sha512-BBgH3I1ppDsI5pZ4Pdhw0ceYxwVCfbU/bZEBCeZ6caRS9x0ZabErxubP7riGUn11PXZBhe8DYdjkDKP1FlVQ5w==
"@oxfmt/binding-linux-arm64-musl@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.63.0.tgz#af5a9b787f5233f27a3360ad56235fc1b011f760"
integrity sha512-alPmbOuWXFXiSo+lOtv6X71C7SYMEDW2WVvywOvf9BwKgEhSNGhMTLeFVSjKUMCamcjbbgVdsWF8GN1uy8xshg==
"@oxfmt/binding-linux-arm64-musl@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-arm64-musl/-/binding-linux-arm64-musl-0.64.0.tgz#7235405901cb0368b659eb42b362a817fc3330a3"
integrity sha512-v19HSjC/BGXdt26qEvKZtwAHgGmQ2Agcap2kQP+KIqoRZqivVzYth3ui2dJA1i+6/fjpjga85lIOaJJjQ/bOOw==
"@oxfmt/binding-linux-ppc64-gnu@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.63.0.tgz#c1a211206134a5577e355a495989e0d733218d60"
integrity sha512-BdzCPvolJc4AWZ+YMzgUDJcDzbQWrFjYuqBHoNHNqP1aCaluQRJNs4k3vNU5IG7vTpjf9zeD73D7MFM1TecZpg==
"@oxfmt/binding-linux-ppc64-gnu@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-ppc64-gnu/-/binding-linux-ppc64-gnu-0.64.0.tgz#1f0563c530dfa634682ffa32d16830404b95a8c6"
integrity sha512-PElLnOo4xFTBZrxPhgTIj0eHqZXwEBQoNWtb7facUV170T0B0FRET0iNbb3LUeLWTybkUW+vsdyv4ihOdyXGyw==
"@oxfmt/binding-linux-riscv64-gnu@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.63.0.tgz#4863f0311e5c1b88f75ef822959b3ca4fd938937"
integrity sha512-7sIgfLzqtNKSkMGsGVyRpHwpjNezRg2XONvUOheFZs95TSZpM0JAuPpA8KrQFsWc4wPU95roX2O69JgH8igOgw==
"@oxfmt/binding-linux-riscv64-gnu@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-riscv64-gnu/-/binding-linux-riscv64-gnu-0.64.0.tgz#36f55e955c5b38b587470f181146c9a11cf8bdb1"
integrity sha512-Qzsg15n4F5CH+MorcRW4MkAEMiLzXmeG+DiDSbP/bBTqCmWOH3K9DHryNrve+JHlV0txS+B6Z9P5Xz+cmWeL+g==
"@oxfmt/binding-linux-riscv64-musl@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.63.0.tgz#ad05a017d12553e2f544743c4940adb552aa1d1c"
integrity sha512-9Tcg0y0WcVa6Mm9AgcgFMseDS+VkFJZpKZ8We9SpDY4gg5jewSwln+0sO04QLcTS1BtfDl9MwR+NfID8L7PUTg==
"@oxfmt/binding-linux-riscv64-musl@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-riscv64-musl/-/binding-linux-riscv64-musl-0.64.0.tgz#bb9c6c3860c8832fe271623eb6131ea5f5e094cd"
integrity sha512-/GZ358wnQ/Ez4UVnCcZIi56JkY0sOdZ+B108pqXKqZz3jLS59F4KEAB1Qv3fRlObrFEk+3L2vUQ/xoPx+3vjXw==
"@oxfmt/binding-linux-s390x-gnu@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.63.0.tgz#2803f539db15bc66db115888fa8f84d6531ed2b9"
integrity sha512-qWKC1pEOpx1qYhXaugPhHUeXwSfqEOk2wJH2LqVXGPV5iQYfdAZdt+d2XDiX4DTSWA2QDMUcFB+wEORh3Xn/sA==
"@oxfmt/binding-linux-s390x-gnu@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-s390x-gnu/-/binding-linux-s390x-gnu-0.64.0.tgz#7d736d923f3c7f88743f26479a49903c6dbaf818"
integrity sha512-/C9We3DXegowfLXtVCYHeNiU9azwCDr5cQkEtCVlc74vyn+lLQSPApJ1CZmxAduqeq/Oi3gQ+IVptyhCaTMtkQ==
"@oxfmt/binding-linux-x64-gnu@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.63.0.tgz#c22a06a60ae2d6b3de522095e0c50a816040a033"
integrity sha512-S9wXYOiGSqYGS4Fx/TFsY+xDd/7dE5s+rUgbA4TsHiVF9e8J3ZcKmP7dsP/7iqLI9Wz7Ic7TzEr3mdthRCTdrA==
"@oxfmt/binding-linux-x64-gnu@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-x64-gnu/-/binding-linux-x64-gnu-0.64.0.tgz#34dfe2bde9ed124324b45aae078618456e850452"
integrity sha512-91KM2CeRWscIEHlj1NsW2WSnzGeq1Ehq+39bfDowTdkn+fcvK/x4Y1RcyqT7glyBjZio0ldkeCG6Usj3v7ASog==
"@oxfmt/binding-linux-x64-musl@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-x64-musl/-/binding-linux-x64-musl-0.63.0.tgz#48d3eeaf8e3757f638cf92de5ee4858befc9c0a3"
integrity sha512-5eGyTJuMZNwBSHCivXt8Yuta6GeTYksOPXRk2MIhajiyFGQx7bjaHIwY+ZusAoFHhT157A9x6sktLjYo9D5oMQ==
"@oxfmt/binding-linux-x64-musl@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-linux-x64-musl/-/binding-linux-x64-musl-0.64.0.tgz#b5edc644409aff9715279650767d34d2fb65d59a"
integrity sha512-gw7uEk9I+7zoT1EYLra1eWArIzNcz8e3jkv+Noo2+o2T7wPvsNSQbfoa4DSfZlvn1i6mJ05RiZ4/omaXPDNhQg==
"@oxfmt/binding-openharmony-arm64@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-openharmony-arm64/-/binding-openharmony-arm64-0.63.0.tgz#02be9e140ae35ba30f52bdce27612fece4a01ab3"
integrity sha512-Rz7hx+Dv3DoW/S6pwVAyjfFXp7/trdQ1zg+vNmsdsdDNlUccugp4XNqambSuEAeP0DaG9k72AtNyfDXCEg0AGw==
"@oxfmt/binding-openharmony-arm64@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-openharmony-arm64/-/binding-openharmony-arm64-0.64.0.tgz#6b1d9c662e08bf5fbc1e9ccdb45ed28c004b90c4"
integrity sha512-HYHFf616FHSPSO07c09mjmXBfQ73wIVM3m0txOiooa5XZkGoxFd6B14PVj0LB0DXIqJ6wAO/dDR/NX/5UUaqnw==
"@oxfmt/binding-win32-arm64-msvc@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.63.0.tgz#2226eaf52b6345a2cb926499216b2486cf0dbec2"
integrity sha512-T/IuizKN9mr4Xw6YYnptkXRNdLkyIlUZ7c8zfTOBpoytZyJ1BAsMUvsMDEx0X4YvSMpaivm+DR8112rQfzC25g==
"@oxfmt/binding-win32-arm64-msvc@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-arm64-msvc/-/binding-win32-arm64-msvc-0.64.0.tgz#bc5a005e159a8f9af4168eed2e61fe477f4029db"
integrity sha512-uQjFp081IZSWD6VAofX2iO2z01awAdHmfC+NrieWIPKrT2hZKQDyq/U18M7ifC0sm0Wz8aHY/p6+FDYIzs/CrQ==
"@oxfmt/binding-win32-ia32-msvc@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.63.0.tgz#58d263bb5ecd7330c02f9dcd8cda10f66e42e74b"
integrity sha512-XjrO5FJ5Wl9vsAxtCP1G/eaeT6y1K2s9CICUHGE42cEjou32/J6S+B1KnrOAboj6E7uhJnwPbRSvznWcxNdA0g==
"@oxfmt/binding-win32-ia32-msvc@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-ia32-msvc/-/binding-win32-ia32-msvc-0.64.0.tgz#88e90b96f7b39e4b6f75178c94c52d464fa58b53"
integrity sha512-lNM6byTAQ881jugzFu8juJTbNRgsUTlswMA6pJmwi1XDvmIqnnb49lcUAs5gz94fCJLrVN+/X3s3jOKqx23WIQ==
"@oxfmt/binding-win32-x64-msvc@0.63.0":
version "0.63.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.63.0.tgz#02a166c8a8049c55d0096d1ba9d8e73f3a4d26a7"
integrity sha512-sgsHCQy432OTQH4Ikk3tZptp3GqwnhwUDuY0loBH41zyHWfMZY9v8Dy78wsnSofHejvFozZGgJgBB1A0LQRwMQ==
"@oxfmt/binding-win32-x64-msvc@0.64.0":
version "0.64.0"
resolved "https://registry.yarnpkg.com/@oxfmt/binding-win32-x64-msvc/-/binding-win32-x64-msvc-0.64.0.tgz#788c7fe26f89e57269f79e8f8a34e9b1497bc674"
integrity sha512-BtmbtL/QjMtF1a6C3CqoDluH2IfB6fJt62E+B9RFfUPtFk4Iz9PFS6+y/SzzOvSxc7aUk2Kphwg7Dh8lMbwu6g==
"@parcel/watcher-android-arm64@2.5.6":
version "2.5.6"
@@ -3789,7 +3789,7 @@
"@rc-component/util" "^1.3.0"
clsx "^2.1.1"
"@rc-component/select@~1.10.0":
"@rc-component/select@~1.10.0", "@rc-component/select@~1.10.1":
version "1.10.1"
resolved "https://registry.yarnpkg.com/@rc-component/select/-/select-1.10.1.tgz#323b2f458a637e8e752f8341094783741c613c34"
integrity sha512-H+yQsl+qED9NilQ3g6zdpsMwUgwVjrcMTkNHAWRVU/MoNCYgTbDgU+MIMgZDK+rVdd2JUfI/MkysMcZZ0cyQKw==
@@ -3824,7 +3824,7 @@
"@rc-component/util" "^1.3.0"
clsx "^2.1.1"
"@rc-component/table@~1.11.0":
"@rc-component/table@~1.11.1":
version "1.11.1"
resolved "https://registry.yarnpkg.com/@rc-component/table/-/table-1.11.1.tgz#7b5c2a7c26fd37b6a403082029b5a72fcb330a4d"
integrity sha512-OWdS6DMmeWb7bJBGqPxYZpQbzBlBiXZUu2sqo6Ii7Sjs9GeK1IsrXrWk26SL2c6KEseabswdxrRj7WUm9LdECw==
@@ -3866,7 +3866,7 @@
"@rc-component/util" "^1.7.0"
clsx "^2.1.1"
"@rc-component/tree-select@~1.16.0":
"@rc-component/tree-select@~1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@rc-component/tree-select/-/tree-select-1.16.1.tgz#dcaea96e396e98108cb29cc051840d4fbdda38cc"
integrity sha512-a1Oi6EJhqAhdOxxupdJi6fP0RPHMKn5TcfkX2+llaQ4lF4nwfH7b6SCHcnsybaa2s+pk1yZYwVyeOYkDnEBRdg==
@@ -4122,23 +4122,23 @@
resolved "https://registry.yarnpkg.com/@standard-schema/utils/-/utils-0.3.0.tgz#3d5e608f16c2390c10528e98e59aef6bf73cae7b"
integrity sha512-e7Mew686owMaPJVNNLs55PUvgz371nKgwsc4vxE49zsODpJEnxgxRo2y/OKrqueavXgZNMDVj3DdHFlaSAeU8g==
"@storybook/addon-docs@^10.5.8":
version "10.5.8"
resolved "https://registry.yarnpkg.com/@storybook/addon-docs/-/addon-docs-10.5.8.tgz#767c10c7a4cc1b625b93f869b2a2b09fc8514f2e"
integrity sha512-NlHiMKW/UvW/uL8HXFDCEVwoH3qZeGYZ/qlWax4d7H471b/T54MBq2KcB4ZrdA785FfIH3numAJdBb5jwn00Mg==
"@storybook/addon-docs@^10.5.9":
version "10.5.9"
resolved "https://registry.yarnpkg.com/@storybook/addon-docs/-/addon-docs-10.5.9.tgz#6d871977f7ad833dc142d12ee43490105dec4d07"
integrity sha512-8sFsMkZYrrdqCLdV+hnwTwDF7RaBsBPRwl4wfc8ve9Q/7Yhi5REe/Xjvd8x1yn6fBPPw9tnID9dx6Agdnr81fw==
dependencies:
"@mdx-js/react" "^3.0.0"
"@storybook/csf-plugin" "10.5.8"
"@storybook/csf-plugin" "10.5.9"
"@storybook/icons" "^2.0.2"
"@storybook/react-dom-shim" "10.5.8"
"@storybook/react-dom-shim" "10.5.9"
react "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
react-dom "^16.8.0 || ^17.0.0 || ^18.0.0 || ^19.0.0"
ts-dedent "^2.0.0"
"@storybook/csf-plugin@10.5.8":
version "10.5.8"
resolved "https://registry.yarnpkg.com/@storybook/csf-plugin/-/csf-plugin-10.5.8.tgz#c626c5bfe55d0e279b2457e5cf150a788d1fc637"
integrity sha512-/FHiMyOWWEXfwK/lM0WxmkP9GLzbSJJuzGtfeuNWSOVDnvAMbjavitxfHb5wSbWKIQo0XYC1EJ2Y7x91XNYP4w==
"@storybook/csf-plugin@10.5.9":
version "10.5.9"
resolved "https://registry.yarnpkg.com/@storybook/csf-plugin/-/csf-plugin-10.5.9.tgz#805e4c93a1704b220351d62bb0c74ce3b78c5e10"
integrity sha512-4H5QIHQVtQYCuL43GCRLGjNQhZpQg9gL03ja0DV80kO2Dn9LEt6ol87bSnSjn4VDgcAXtgTzXFvRLknfVgAAqg==
dependencies:
unplugin "^2.3.5"
@@ -4152,10 +4152,10 @@
resolved "https://registry.yarnpkg.com/@storybook/icons/-/icons-2.1.0.tgz#edfc2450a39c5e780f28c6cbc49acd7bff59b41a"
integrity sha512-Fxh9vYpX9bQqFeHRiY8h2ApeRGDzRSMLwJwNZ/AIRqnyOKHxRKL+yFe+ctEkVJmuptRE9u1Hrn8ZZNHyfDKKNg==
"@storybook/react-dom-shim@10.5.8":
version "10.5.8"
resolved "https://registry.yarnpkg.com/@storybook/react-dom-shim/-/react-dom-shim-10.5.8.tgz#40cc3e32af424baa2e4109a325dae2ede29999e2"
integrity sha512-N8D13/Xny+V3kfe1KBgsAHS0nKWXLLdgOOXS9poKdYzVwVCN+CGEGBxWX0zMMtdCptqa6/57em9coPlZMoO+bg==
"@storybook/react-dom-shim@10.5.9":
version "10.5.9"
resolved "https://registry.yarnpkg.com/@storybook/react-dom-shim/-/react-dom-shim-10.5.9.tgz#549793845bb8b966acd36002d33d7b54cc5c92d4"
integrity sha512-7qZD6CSa64p1m/zX9tG4ALcEHlEk0Bx+5++4RL3MFlRCgCFfAomXsCHTzF0RyF8cI4vl+hS7Y0ryjQchVs6UQA==
"@superset-ui/core@^0.20.4":
version "0.20.4"
@@ -4855,86 +4855,86 @@
dependencies:
apg-lite "^1.0.4"
"@swc/core-darwin-arm64@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-darwin-arm64/-/core-darwin-arm64-1.16.0.tgz#8c5a2af031c62ebcb6354aa6975bfb7eac895223"
integrity sha512-SJQPl+xG/zB8bNjC/gTg3WOmOvz7EzlQD+VShfCKFYPNr2qvb+vATUY11vYEjnMWCn6wV8H8eAtjQrVflYyX5A==
"@swc/core-darwin-arm64@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-darwin-arm64/-/core-darwin-arm64-1.16.1.tgz#f6f6983e2268888558cdbe043001d82449445def"
integrity sha512-zlJblJ8ncErD43lKdxjbUaUskJQf+LxiPXYcWXD8/8ZMV+7uuAT+CwjciLXpyZBd5Pq/S726bMpeeAwSeL1hhg==
"@swc/core-darwin-x64@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-darwin-x64/-/core-darwin-x64-1.16.0.tgz#0d2496c0429d7e8bc45b50348adf9d105bb56793"
integrity sha512-ql2JVch8V5t1i+HxiiuD4oVDI1dOku4/e3QiCkplONrm3SLitqNAP+nztHN51fSG2IgGuOwpAi3hgA+ukT5yQg==
"@swc/core-darwin-x64@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-darwin-x64/-/core-darwin-x64-1.16.1.tgz#98b61e8c7ffe9f6263a08677353ba5606f6992de"
integrity sha512-IN0BmPWb0YAh/17mmlWB/HDBtTw2MfuW4hulf/tQAgTQBRH17l+z499bNJLK6LizSjqs0P7V+jU38Zj+vJC1DA==
"@swc/core-linux-arm-gnueabihf@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.16.0.tgz#5f02a85842a04cd21f2ab9e8e67dc4b16f7024a4"
integrity sha512-PcdDBaRbe39y37h1rXVkhNy7mEU7f8b34KD761C68R23EsfMsj5oDPVddRzGdSRAvwwSfH0WSNEHgYmc/AJipg==
"@swc/core-linux-arm-gnueabihf@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm-gnueabihf/-/core-linux-arm-gnueabihf-1.16.1.tgz#afc245521cd43a65a87cdd87fe99fb9e4f4eaa58"
integrity sha512-EYgrx2YOCQ2Twz2S793kqNjPkpvYVUPzzR95bIb7by+VQcyaai4lZZ2iz/tZvcFVKSNcN3/JTKwx+aBn2ZL52A==
"@swc/core-linux-arm64-gnu@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.16.0.tgz#6264498c88c51649511c6b4af532d330d3cf0631"
integrity sha512-t21IUztHQ/COucy7Kk9eIlehmq08H/hYq7aRA6fZox3S5ddi6TxWPK6e5S/+aTCf6+Od9qQ+LIpjHMiTy737vA==
"@swc/core-linux-arm64-gnu@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm64-gnu/-/core-linux-arm64-gnu-1.16.1.tgz#c44ca749af555ef8127795de141094cd28da9714"
integrity sha512-moyKm0YZlHdHohzm1YwgAyesqnE853rO0REMfJLFAova51wF9BNi+3ZW2PeS7Vqvn6HeJuepLpAHbBdZctxpHA==
"@swc/core-linux-arm64-musl@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.16.0.tgz#7a451eba69aa9a80799b9b8b9af46bf6f49803bd"
integrity sha512-d9+iajbMB87b0umgbP+Gy3yBDSDgty4Q6H5pZ8fgTb/dOoKIwwynP4L4kvWCOFg2i49kxmAAUs1uJZh9s0E+RQ==
"@swc/core-linux-arm64-musl@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-arm64-musl/-/core-linux-arm64-musl-1.16.1.tgz#a1a3d15d5fb074c474c9a60a14488ec16124253f"
integrity sha512-kKGBO9wdapiSzuf5ZzZ2fYtlu1BNSYtIIUxvH1ir/gcelTOREEHGDCLTDFx/2Knf878nU11A40z7LxwasEFxqA==
"@swc/core-linux-ppc64-gnu@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-ppc64-gnu/-/core-linux-ppc64-gnu-1.16.0.tgz#99a7ba46a56190a52c646506e940dffe554c5d10"
integrity sha512-QRpeKGOg+B0qmo3BFU+6rL/gpoKYYJ7OFSMf5DNMafohYZ/iq2qvAH9Gcrf8NxROj3iooKOVewJ+YgahH1nSLw==
"@swc/core-linux-ppc64-gnu@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-ppc64-gnu/-/core-linux-ppc64-gnu-1.16.1.tgz#7eb33976ece5e45e63f9c9c1ab0da9405df76f7f"
integrity sha512-nZ6qahtLxC3PM54cWOQZHxt4lTCF/3J4LIoWWzz6v7A+rLs8Dx54anYQf7mH3eIi8KlNpgKci/ie8ZSqFN8O7A==
"@swc/core-linux-s390x-gnu@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-s390x-gnu/-/core-linux-s390x-gnu-1.16.0.tgz#61473e056d1dd0d4690352a875c14f41bdd9f60a"
integrity sha512-q+Vr/hmHCcRXT/WFzOJC+T6GGEEtq2iaTtmyLfxO7yzu4ckgcqSNkg9m181wfNhuMwfNBoBhOfwQCnLsGZ5F4g==
"@swc/core-linux-s390x-gnu@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-s390x-gnu/-/core-linux-s390x-gnu-1.16.1.tgz#f02f2687d2ee1c8f59430ef638c63714862c9389"
integrity sha512-4ji5PNzhYq193Z4/4xUaSoNJza6iCkDJSzhetrbB6KOYxsr+kxtQr8ePWhMJUiMt6JUWtXaZ1PYT8FhtED+nGA==
"@swc/core-linux-x64-gnu@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.16.0.tgz#008fc149a9135bca92b1e1f63037e2612c4d0fb5"
integrity sha512-DWVBc3QnpsSgKoq8N4rmZeZa5r/XrHdLkITsExN/tvTdqPtAPDPt+Ysy33OfgBlyN8lNe4xwsXWe6DXlRkJeRQ==
"@swc/core-linux-x64-gnu@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-x64-gnu/-/core-linux-x64-gnu-1.16.1.tgz#af4c571bbe07044ee0bec49ade1e53c1022d4979"
integrity sha512-VJQxqrisHV+B394IgrOu8YsIIXZgffnf5tO+yc9Z/hoUpuZEvuQTjWwlnpZdpyD+0nx6LTD1/3k646JYm43yJA==
"@swc/core-linux-x64-musl@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.16.0.tgz#4300ea0c63864dc3989ca0e956b4a5e4c666196c"
integrity sha512-6XCgDSc1HPf/5dpjvABhKHICiBcsuZyW3hQMkn8sxel0TqprkJGp+H4iaBYIUTPixhrBub2hBPtfjcZLE6yL3w==
"@swc/core-linux-x64-musl@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-linux-x64-musl/-/core-linux-x64-musl-1.16.1.tgz#113eb36a1d3bd21bbf4a48a22fad97dc1c7cc91c"
integrity sha512-r9oV1mwxxsIGcLV1IQ/tw76MW3doatKze1QFWuC+a7QqJUkhY/bKTSVk6NpKKUGm2LDsE33Va8VqSClfA7vSiQ==
"@swc/core-win32-arm64-msvc@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.16.0.tgz#1d4146b7c1aada2992692cdc72bb0b43a885136e"
integrity sha512-T/+9VVCZJ3AKEth9IP3U9AJ2YscQq+7LUqRTvfR4a2q36+Ri22oOwUizpAKOqQ42vb2Y/kOa4TOcJOfHoDIT/w==
"@swc/core-win32-arm64-msvc@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-win32-arm64-msvc/-/core-win32-arm64-msvc-1.16.1.tgz#7cc6cfde26ad7e15fe93de98033e7c1892bcf127"
integrity sha512-6huNRessoBLxWEqBm5zJXyCQ27TO7anvkdiuQ5MDO4CJni0nOXEqKtV9RllQ2TdyENKKsUMXVnIfW2hIXx/R5Q==
"@swc/core-win32-ia32-msvc@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.16.0.tgz#c5c2a60905ffa9e4647214bef75778f0c73ba0d4"
integrity sha512-Pr1lsR/PMs8ndL0UWMrW8nLZ7H7sspIxBRDdjL8f+YJ/FJNASgzfunbVVXAqj0csgIJYHPZy+OW9smjFmk1Rcg==
"@swc/core-win32-ia32-msvc@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-win32-ia32-msvc/-/core-win32-ia32-msvc-1.16.1.tgz#2330c734f4129c2064b8848fb956501788aab9a3"
integrity sha512-OVKJFUzphrGmsh+BGtcZDesx0YryV7/Yvy5XGgTqnrZfjnyfcr5uaqYQugCckdIlupc5Vs3XtDjRAj12z4ZPlw==
"@swc/core-win32-x64-msvc@1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.16.0.tgz#67dd85a90437e6fa9951cce7842f6cac3ec3f60d"
integrity sha512-ktdeYLgOQdaonvsj5tJijqgpb0wk7gfF80wCFVA0kucI1hhSUIyfcGbjo5+9sdqv38OhMnTdLoA6xbqgOgPQjw==
"@swc/core-win32-x64-msvc@1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core-win32-x64-msvc/-/core-win32-x64-msvc-1.16.1.tgz#04825a3f9e6fbe390825ff02708a5ebdd3a9841b"
integrity sha512-Bt+VIhWYCGk4urklnkkteLUOeLv1VxigwTCeB/xC6rBZxY6IIKdDwCJf6on3E3SUGsIqmQS6QqtuJQc1VxF4Aw==
"@swc/core@^1.15.40", "@swc/core@^1.16.0":
version "1.16.0"
resolved "https://registry.yarnpkg.com/@swc/core/-/core-1.16.0.tgz#79cd13789725d3e3ad0df605dc88d9e255d7ebfd"
integrity sha512-zSdvEHxBg00WhUNtW/u58hhcdR33gjtMQvOBo8F7POWJDyjRCt/miKfhidT3hCc/118RUwNnlEAmxiihFMbK4Q==
"@swc/core@^1.15.40", "@swc/core@^1.16.1":
version "1.16.1"
resolved "https://registry.yarnpkg.com/@swc/core/-/core-1.16.1.tgz#5ea7ff32f3b352c871aa47195efd4932f709a569"
integrity sha512-nUaeu91O5QZKrQdaDCHd402ogUIoNOOjpkZNq0UomWK0G6gDaGmLhvddF1/3BXf5O8aLyo6ZPY/aMDWvaJQ/hg==
dependencies:
"@swc/counter" "^0.1.3"
"@swc/types" "^0.1.28"
optionalDependencies:
"@swc/core-darwin-arm64" "1.16.0"
"@swc/core-darwin-x64" "1.16.0"
"@swc/core-linux-arm-gnueabihf" "1.16.0"
"@swc/core-linux-arm64-gnu" "1.16.0"
"@swc/core-linux-arm64-musl" "1.16.0"
"@swc/core-linux-ppc64-gnu" "1.16.0"
"@swc/core-linux-s390x-gnu" "1.16.0"
"@swc/core-linux-x64-gnu" "1.16.0"
"@swc/core-linux-x64-musl" "1.16.0"
"@swc/core-win32-arm64-msvc" "1.16.0"
"@swc/core-win32-ia32-msvc" "1.16.0"
"@swc/core-win32-x64-msvc" "1.16.0"
"@swc/core-darwin-arm64" "1.16.1"
"@swc/core-darwin-x64" "1.16.1"
"@swc/core-linux-arm-gnueabihf" "1.16.1"
"@swc/core-linux-arm64-gnu" "1.16.1"
"@swc/core-linux-arm64-musl" "1.16.1"
"@swc/core-linux-ppc64-gnu" "1.16.1"
"@swc/core-linux-s390x-gnu" "1.16.1"
"@swc/core-linux-x64-gnu" "1.16.1"
"@swc/core-linux-x64-musl" "1.16.1"
"@swc/core-win32-arm64-msvc" "1.16.1"
"@swc/core-win32-ia32-msvc" "1.16.1"
"@swc/core-win32-x64-msvc" "1.16.1"
"@swc/counter@^0.1.3":
version "0.1.3"
@@ -6181,10 +6181,10 @@ ansis@^3.2.0:
resolved "https://registry.yarnpkg.com/ansis/-/ansis-3.17.0.tgz#fa8d9c2a93fe7d1177e0c17f9eeb562a58a832d7"
integrity sha512-0qWUglt9JEqLFr3w1I1pbrChn1grhaiAR2ocX1PP/flRmxgtwTzPFFFnfIlD6aMOLQZgSuCRlidD70lvx8yhzg==
antd@^6.6.0:
version "6.6.0"
resolved "https://registry.yarnpkg.com/antd/-/antd-6.6.0.tgz#8acb84c54b36594b5c1a9084c8acb6a03b79961b"
integrity sha512-UDwWIbpmrCHB9ZQ+bPh4vQfB6DTI2ulIyoQ0Tc9xxalFblttiNGHl3ySBD9SyV/8+gUjFzfSx1+iU1Fog2i46w==
antd@^6.6.1:
version "6.6.1"
resolved "https://registry.yarnpkg.com/antd/-/antd-6.6.1.tgz#3235d76413b525b1f3287b87bdaf6ba0e7148521"
integrity sha512-QHIHYoUk9N9nJy1T9fyxWKjY0qApdTEDd/6lzqYng8Uryv9FejNmbhKvYF7obGqB+TuLXQsPVF7fOVgyzM1KrQ==
dependencies:
"@ant-design/colors" "^8.0.1"
"@ant-design/cssinjs" "^2.1.2"
@@ -6217,16 +6217,16 @@ antd@^6.6.0:
"@rc-component/rate" "~1.0.1"
"@rc-component/resize-observer" "^1.1.2"
"@rc-component/segmented" "~1.3.0"
"@rc-component/select" "~1.10.0"
"@rc-component/select" "~1.10.1"
"@rc-component/slider" "~1.1.1"
"@rc-component/steps" "~1.2.2"
"@rc-component/switch" "~1.0.3"
"@rc-component/table" "~1.11.0"
"@rc-component/table" "~1.11.1"
"@rc-component/tabs" "~1.12.0"
"@rc-component/tooltip" "~1.5.0"
"@rc-component/tour" "~2.4.0"
"@rc-component/tree" "~1.4.0"
"@rc-component/tree-select" "~1.16.0"
"@rc-component/tree-select" "~1.16.1"
"@rc-component/trigger" "^3.10.1"
"@rc-component/upload" "~1.1.1"
"@rc-component/util" "^1.12.0"
@@ -11849,10 +11849,10 @@ neotraverse@0.6.15:
resolved "https://registry.yarnpkg.com/neotraverse/-/neotraverse-0.6.15.tgz#dc4abb64700c52440f13bc53635b559862420360"
integrity sha512-HZpdkco+JeXq0G+WWpMJ4NsX3pqb5O7eR9uGz3FfoFt+LYzU8iRWp49nJtud6hsDoywM8tIrDo3gjgmOqJA8LA==
neotraverse@=0.6.18:
version "0.6.18"
resolved "https://registry.yarnpkg.com/neotraverse/-/neotraverse-0.6.18.tgz#abcb33dda2e8e713cf6321b29405e822230cdb30"
integrity sha512-Z4SmBUweYa09+o6pG+eASabEpP6QkQ70yHj351pQoEXIs8uHbaU2DWVmzBANKgflPa47A50PtB2+NgRpQvr7vA==
neotraverse@=1.0.1:
version "1.0.1"
resolved "https://registry.yarnpkg.com/neotraverse/-/neotraverse-1.0.1.tgz#7c89b43f6504ef85928c718f578c68621576d194"
integrity sha512-WmmLty1YWwJl9yZi77v2dVIV6X2kuYV8YYBI/G3LWGKdGHmHUvL1z7FW0iDvEvGAwNEoc5x1tOOOyDnf5jJw/w==
no-case@^3.0.4:
version "3.0.4"
@@ -12262,32 +12262,32 @@ oxc-resolver@^11.19.1:
"@oxc-resolver/binding-win32-arm64-msvc" "11.23.0"
"@oxc-resolver/binding-win32-x64-msvc" "11.23.0"
oxfmt@^0.63.0:
version "0.63.0"
resolved "https://registry.yarnpkg.com/oxfmt/-/oxfmt-0.63.0.tgz#c7338e6c43a68d5cf8dc61c08b617d77cb54e323"
integrity sha512-kgdDwv35wvVf6554U2Ab8Jnd0zTM+TsEQWwaB70RAjK3gICFAFGO+2Hd3Be27GMoXj3XRL9IKSNRVl7KBQL6iw==
oxfmt@^0.64.0:
version "0.64.0"
resolved "https://registry.yarnpkg.com/oxfmt/-/oxfmt-0.64.0.tgz#666a5148cdf7385007cd46e35e8ff8f94ecfd96b"
integrity sha512-XZ4GFBN/PLbXKq+0zrgpQfPKYuJlUuj+nzZJY7UpIbFMNyefNLCdN9EwViycNqnYcv0wrn0jXcQLlqJp8RCKBg==
dependencies:
tinypool "2.1.0"
optionalDependencies:
"@oxfmt/binding-android-arm-eabi" "0.63.0"
"@oxfmt/binding-android-arm64" "0.63.0"
"@oxfmt/binding-darwin-arm64" "0.63.0"
"@oxfmt/binding-darwin-x64" "0.63.0"
"@oxfmt/binding-freebsd-x64" "0.63.0"
"@oxfmt/binding-linux-arm-gnueabihf" "0.63.0"
"@oxfmt/binding-linux-arm-musleabihf" "0.63.0"
"@oxfmt/binding-linux-arm64-gnu" "0.63.0"
"@oxfmt/binding-linux-arm64-musl" "0.63.0"
"@oxfmt/binding-linux-ppc64-gnu" "0.63.0"
"@oxfmt/binding-linux-riscv64-gnu" "0.63.0"
"@oxfmt/binding-linux-riscv64-musl" "0.63.0"
"@oxfmt/binding-linux-s390x-gnu" "0.63.0"
"@oxfmt/binding-linux-x64-gnu" "0.63.0"
"@oxfmt/binding-linux-x64-musl" "0.63.0"
"@oxfmt/binding-openharmony-arm64" "0.63.0"
"@oxfmt/binding-win32-arm64-msvc" "0.63.0"
"@oxfmt/binding-win32-ia32-msvc" "0.63.0"
"@oxfmt/binding-win32-x64-msvc" "0.63.0"
"@oxfmt/binding-android-arm-eabi" "0.64.0"
"@oxfmt/binding-android-arm64" "0.64.0"
"@oxfmt/binding-darwin-arm64" "0.64.0"
"@oxfmt/binding-darwin-x64" "0.64.0"
"@oxfmt/binding-freebsd-x64" "0.64.0"
"@oxfmt/binding-linux-arm-gnueabihf" "0.64.0"
"@oxfmt/binding-linux-arm-musleabihf" "0.64.0"
"@oxfmt/binding-linux-arm64-gnu" "0.64.0"
"@oxfmt/binding-linux-arm64-musl" "0.64.0"
"@oxfmt/binding-linux-ppc64-gnu" "0.64.0"
"@oxfmt/binding-linux-riscv64-gnu" "0.64.0"
"@oxfmt/binding-linux-riscv64-musl" "0.64.0"
"@oxfmt/binding-linux-s390x-gnu" "0.64.0"
"@oxfmt/binding-linux-x64-gnu" "0.64.0"
"@oxfmt/binding-linux-x64-musl" "0.64.0"
"@oxfmt/binding-openharmony-arm64" "0.64.0"
"@oxfmt/binding-win32-arm64-msvc" "0.64.0"
"@oxfmt/binding-win32-ia32-msvc" "0.64.0"
"@oxfmt/binding-win32-x64-msvc" "0.64.0"
p-cancelable@^3.0.0:
version "3.0.0"
@@ -13583,7 +13583,7 @@ react-modal@^3.16.3:
react-lifecycles-compat "^3.0.0"
warning "^4.0.3"
react-redux@^9.2.0:
react-redux@^9.2.0, react-redux@^9.3.0:
version "9.3.0"
resolved "https://registry.yarnpkg.com/react-redux/-/react-redux-9.3.0.tgz#a30113bb6d95c0a715d54dda4308d450fca6ce09"
integrity sha512-KQopgqFo/p/fgmAs5qz6p5RWaNAzq40WAu7fJIXnQpYxFPbJYtsJPWvGeF2rOBaY/kEuV77AVsX8TsQzKm+A/g==
@@ -14783,10 +14783,10 @@ stop-iteration-iterator@^1.1.0:
es-errors "^1.3.0"
internal-slot "^1.1.0"
storybook@^10.5.8:
version "10.5.8"
resolved "https://registry.yarnpkg.com/storybook/-/storybook-10.5.8.tgz#d5f051983e6232c0a73ea02149a72c7bafb43275"
integrity sha512-rR4oFMSiWBSqI0lvsJPtcQUPj8+hzj3TkLu+Mw61Wo6YxPSb5FsLSHai0jZnuaIdKIlmu25KCfwlSQl4e1uvnA==
storybook@^10.5.9:
version "10.5.9"
resolved "https://registry.yarnpkg.com/storybook/-/storybook-10.5.9.tgz#61f476fd73785dcf09e9198ddf404b9b8c06964a"
integrity sha512-UfdMKSjEhIKr8LbqYyIE5r7vT/drL/PxN75YaouJ+UG0FssEy6cf49OdTF3kstAqVMHskc+zEqyRoiQHZXHwgA==
dependencies:
"@storybook/global" "^5.0.0"
"@storybook/icons" "^2.0.2"
@@ -15057,10 +15057,10 @@ svgo@^3.0.2, svgo@^3.2.0:
picocolors "^1.0.0"
sax "^1.5.0"
swagger-client@^3.37.8:
version "3.37.8"
resolved "https://registry.yarnpkg.com/swagger-client/-/swagger-client-3.37.8.tgz#26c24c89cbfda7459f6afb53bdfcb6d8dbe9ac82"
integrity sha512-uoKwfq+8DvWVDhoALDrEtex9f26Yi2VkvEFjsrMHd8Gl+TcApJkVXtNiE35p5JQjMsvwkvr1eLVlOFNF4GL1bQ==
swagger-client@^3.38.0:
version "3.38.0"
resolved "https://registry.yarnpkg.com/swagger-client/-/swagger-client-3.38.0.tgz#542431f02d809b49115272ff8b9e48d545b9f53c"
integrity sha512-n7aykm1BEdQ3fKePJJx63UGjYe8/5fuxFMi3qZP4OJGZvzljKvmhxNwIF/MB71sF/lop9NeWZReKvPib9CY+2g==
dependencies:
"@babel/runtime-corejs3" "^7.22.15"
"@scarf/scarf" "=1.4.0"
@@ -15074,7 +15074,7 @@ swagger-client@^3.37.8:
deepmerge "~4.3.0"
fast-json-patch "^3.0.0-1"
js-yaml "^4.2.0"
neotraverse "=0.6.18"
neotraverse "=1.0.1"
node-abort-controller "^3.1.1"
openapi-path-templating "^2.2.1"
openapi-server-url-templating "^1.3.0"
@@ -15103,10 +15103,10 @@ swagger-client@^3.37.8:
"@swagger-api/apidom-parser-adapter-openapi-yaml-3-2" "^1.12.0"
"@swagger-api/apidom-parser-adapter-yaml-1-2" "^1.12.0"
swagger-ui-react@^5.32.13:
version "5.32.13"
resolved "https://registry.yarnpkg.com/swagger-ui-react/-/swagger-ui-react-5.32.13.tgz#04c96140b0a2d4ea01ebec4d4cfc655d5ed9a500"
integrity sha512-XIDl+Ny6kE1N8wpSPiOFrjPfAevs4GR4XmV6BT6NLMikkMFIbIVocWbA8pnKYyYXQe8Rccfli5o2zDfySw0FnQ==
swagger-ui-react@^5.32.14:
version "5.32.14"
resolved "https://registry.yarnpkg.com/swagger-ui-react/-/swagger-ui-react-5.32.14.tgz#31b69b0f6910e87dbcc81886208061ca1f72e034"
integrity sha512-6LAVBeC78DplbJ7kutm/YeBYo22nPzGOca4bIZAvQG4w2eSetnYDdazaUfY0qzQUlg/H90HnYZX3rg67EmENOw==
dependencies:
"@babel/runtime-corejs3" "^7.27.1"
"@scarf/scarf" "=1.4.0"
@@ -15129,7 +15129,7 @@ swagger-ui-react@^5.32.13:
react-immutable-proptypes "2.2.0"
react-immutable-pure-component "^2.2.0"
react-inspector "^6.0.1"
react-redux "^9.2.0"
react-redux "^9.3.0"
react-syntax-highlighter "^16.0.0"
redux "^5.0.1"
redux-immutable "^4.0.0"
@@ -15137,7 +15137,7 @@ swagger-ui-react@^5.32.13:
reselect "^5.1.1"
serialize-error "^8.1.0"
sha.js "^2.4.12"
swagger-client "^3.37.8"
swagger-client "^3.38.0"
url-parse "^1.5.10"
xml "=1.0.1"
xml-but-prettier "^1.0.1"
+7 -3
View File
@@ -80,7 +80,7 @@ dependencies = [
# marshmallow 4 compatibility: see superset/marshmallow_compatibility.py for a
# Flask-AppBuilder workaround. Tracking issue:
# https://github.com/apache/superset/issues/33162
"marshmallow>=4.3.1, <5",
"marshmallow>=3.0, <5",
"marshmallow-union>=0.1.15.post1",
"msgpack>=1.2.0, <1.3",
"nh3>=0.3.5, <0.4",
@@ -218,8 +218,12 @@ motherduck = ["apache-superset[duckdb]"]
mysql = ["mysqlclient>=2.2.8, <3"]
ocient = [
# Closed-source vendor package with no public changelog; permissive
# unpinned sqlalchemy>=1.4 declared, but SQLAlchemy 2.0 support is
# unverified. Lower confidence than the other bumps in this PR.
# unpinned sqlalchemy>=1.4 declared. Verified compatible with SQLAlchemy
# 2.0 against pyocient>=3.9.0 (discussion #40273): dialect construction,
# error extraction, and GIS-type sanitization all pass under 2.0.52. Note
# pyocient 3.9.0 relocated its geo-type classes from private top-level
# names (pyocient._STPoint) to public ones under pyocient.api
# (pyocient.api.STPoint), which is unrelated to the SQLAlchemy bump.
"sqlalchemy-ocient>=3.0.0, <4",
"pyocient>=3.9.0, <4",
"shapely",
@@ -0,0 +1,18 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Semantic layer contracts for extension authors."""
+252 -418
View File
File diff suppressed because it is too large Load Diff
+13 -13
View File
@@ -158,12 +158,12 @@
"@visx/xychart": "^4.0.0",
"ag-grid-community": "36.1.0",
"ag-grid-react": "36.1.0",
"antd": "^6.6.0",
"antd": "^6.6.1",
"chrono-node": "^2.10.1",
"classnames": "^2.2.5",
"content-disposition": "^2.0.1",
"d3-scale": "^4.0.2",
"dayjs": "^1.11.22",
"dayjs": "^1.11.23",
"dom-to-image-more": "^3.10.2",
"dom-to-pdf": "^0.3.2",
"echarts": "^6.1.0",
@@ -230,7 +230,7 @@
"use-event-callback": "^0.1.0",
"use-immer": "^0.11.0",
"use-query-params": "^2.2.2",
"uuid": "^14.0.1",
"uuid": "^14.0.2",
"xlsx": "https://cdn.sheetjs.com/xlsx-0.20.3/xlsx-0.20.3.tgz",
"yargs": "^18.1.0"
},
@@ -257,14 +257,14 @@
"@istanbuljs/nyc-config-typescript": "^1.0.1",
"@playwright/test": "^1.62.1",
"@pmmmwh/react-refresh-webpack-plugin": "^0.6.2",
"@storybook/addon-docs": "10.5.8",
"@storybook/addon-links": "10.5.8",
"@storybook/react-webpack5": "10.5.8",
"@storybook/addon-docs": "10.5.9",
"@storybook/addon-links": "10.5.9",
"@storybook/react-webpack5": "10.5.9",
"@storybook/test-runner": "0.24.4",
"@svgr/webpack": "^8.1.0",
"@swc/core": "^1.15.47",
"@swc/plugin-emotion": "^14.19.0",
"@swc/plugin-transform-imports": "^12.5.0",
"@swc/core": "^1.16.1",
"@swc/plugin-emotion": "^15.0.0",
"@swc/plugin-transform-imports": "^13.0.0",
"@testing-library/dom": "^10.4.1",
"@testing-library/jest-dom": "^7.0.1",
"@testing-library/react": "^15.0.0",
@@ -295,7 +295,7 @@
"babel-loader": "^10.1.1",
"babel-plugin-dynamic-import-node": "^2.3.3",
"babel-plugin-jsx-remove-data-test-id": "^3.0.0",
"baseline-browser-mapping": "^2.11.14",
"baseline-browser-mapping": "^2.11.15",
"cheerio": "1.2.0",
"concurrently": "^10.0.5",
"copy-webpack-plugin": "^14.0.0",
@@ -312,7 +312,7 @@
"eslint-plugin-no-only-tests": "^3.4.0",
"eslint-plugin-react-prefer-function-component": "^5.0.0",
"eslint-plugin-react-you-might-not-need-an-effect": "^1.0.1",
"eslint-plugin-storybook": "10.5.8",
"eslint-plugin-storybook": "10.5.9",
"eslint-plugin-testing-library": "^7.16.2",
"eslint-plugin-theme-colors": "file:eslint-rules/eslint-plugin-theme-colors",
"fetch-mock": "^12.6.0",
@@ -331,7 +331,7 @@
"mini-css-extract-plugin": "^2.10.2",
"minimizer-webpack-plugin": "^5.6.1",
"open-cli": "^9.0.0",
"oxfmt": "^0.63.0",
"oxfmt": "^0.64.0",
"oxlint": "^1.78.0",
"po2json": "^0.4.5",
"postcss-styled-syntax": "^0.7.2",
@@ -343,7 +343,7 @@
"source-map": "^0.8.0",
"source-map-support": "^0.5.21",
"speed-measure-webpack-plugin": "^1.6.0",
"storybook": "10.5.8",
"storybook": "10.5.9",
"style-loader": "^4.0.0",
"stylelint": "^17.14.1",
"swc-loader": "^0.2.7",
@@ -17,7 +17,7 @@
* under the License.
*/
import { QueryFormMetric } from '@superset-ui/core';
import { getTotalsMetrics } from './getTotalsMetrics';
import { getTotalsMetrics, toTotalsAggregate } from './getTotalsMetrics';
const simpleMetric = (aggregate: string): QueryFormMetric =>
({
@@ -76,4 +76,31 @@ describe('getTotalsMetrics', () => {
test('returns an empty array when given no metrics', () => {
expect(getTotalsMetrics([], 'AVG')).toEqual([]);
});
test("ORIGINAL keeps each metric's own aggregate", () => {
const metrics = [
simpleMetric('COUNT_DISTINCT'),
sqlMetric(),
savedMetric(),
];
const result = getTotalsMetrics(metrics, 'ORIGINAL');
expect(result).toBe(metrics);
expect(result[0]).toEqual(
expect.objectContaining({ aggregate: 'COUNT_DISTINCT' }),
);
});
});
describe('toTotalsAggregate', () => {
test.each(['SUM', 'AVG'] as const)('passes %s through', value => {
expect(toTotalsAggregate(value)).toBe(value);
});
test.each([undefined, null, '', 'MEDIAN', 'sum'])(
'falls back to ORIGINAL for %p',
value => {
expect(toTotalsAggregate(value)).toBe('ORIGINAL');
},
);
});
@@ -18,26 +18,46 @@
*/
import { isAdhocMetricSimple, QueryFormMetric } from '@superset-ui/core';
export type TotalsAggregate = 'SUM' | 'AVG';
/**
* How the "Show summary" totals row aggregates each metric.
*
* ``ORIGINAL`` keeps every metric's own aggregation. It is the default because
* overriding is not universally valid: ``SUM`` over a ``COUNT_DISTINCT`` of a
* non-numeric column (a uuid, say) is rejected outright by the database, and
* over a numeric id column it silently produces a meaningless number.
*/
export type TotalsAggregate = 'ORIGINAL' | 'SUM' | 'AVG';
/**
* Build the metrics for a chart's "Show summary" totals query, overriding
* each Simple (adhoc) metric's aggregate function with the user-chosen
* totals aggregate. The totals query has no GROUP BY, so the database
* evaluates each metric fresh over all rows -- swapping the aggregate here
* is a correct, independent computation, not a re-aggregation of
* already-aggregated per-row values.
* Build the metrics for a chart's "Show summary" totals query.
*
* Custom-SQL metrics and saved (string) metrics pass through unchanged:
* there is no safe way to rewrite an arbitrary SQL expression's aggregate
* function without parsing it, so the totals row keeps their own native
* aggregate for those.
* With SUM or AVG, each Simple (adhoc) metric is cloned with its aggregate
* replaced. The totals query has no GROUP BY, so the database evaluates each
* metric fresh over all rows -- that swap is an independent computation, not a
* re-aggregation of already-aggregated per-row values.
*
* Custom-SQL and saved (string) metrics always pass through unchanged: there is
* no safe way to rewrite an arbitrary SQL expression's aggregate without
* parsing it, so the totals row keeps their own native aggregate.
*/
export function getTotalsMetrics(
metrics: QueryFormMetric[],
aggregate: TotalsAggregate,
): QueryFormMetric[] {
if (aggregate === 'ORIGINAL') {
return metrics;
}
return metrics.map(metric =>
isAdhocMetricSimple(metric) ? { ...metric, aggregate } : metric,
);
}
/**
* Narrow a raw ``totals_aggregate`` form-data value to a TotalsAggregate.
*
* Anything other than an explicit SUM/AVG including charts saved before the
* control existed keeps each metric's own aggregation.
*/
export function toTotalsAggregate(value: unknown): TotalsAggregate {
return value === 'SUM' || value === 'AVG' ? value : 'ORIGINAL';
}
@@ -67,7 +67,7 @@
"d3-scale": "^4.0.2",
"d3-time": "^3.1.0",
"d3-time-format": "^4.1.0",
"dayjs": "^1.11.22",
"dayjs": "^1.11.23",
"dompurify": "^3.4.13",
"fetch-retry": "^6.0.0",
"handlebars": "^4.7.9",
@@ -64,9 +64,8 @@ export default function createSmartNumberFormatter(
description,
formatFunc: value => `${getSign(value)}${formatValue(value)}`,
id:
id || signed
? NumberFormats.SMART_NUMBER_SIGNED
: NumberFormats.SMART_NUMBER,
id ??
(signed ? NumberFormats.SMART_NUMBER_SIGNED : NumberFormats.SMART_NUMBER),
label: label ?? 'Adaptive formatter',
});
}
@@ -24,6 +24,12 @@ describe('createSmartNumberFormatter(options)', () => {
const formatter = createSmartNumberFormatter();
expect(formatter).toBeInstanceOf(NumberFormatter);
});
test('uses the supplied formatter id regardless of signed option', () => {
expect(createSmartNumberFormatter({ id: 'custom' }).id).toBe('custom');
expect(
createSmartNumberFormatter({ id: 'custom-signed', signed: true }).id,
).toBe('custom-signed');
});
describe('using default options', () => {
const formatter = createSmartNumberFormatter();
test('formats 0 correctly', () => {
@@ -38,7 +38,7 @@ import {
getTotalsMetrics,
isTimeComparison,
timeCompareOperator,
TotalsAggregate,
toTotalsAggregate,
} from '@superset-ui/chart-controls';
import { isEmpty } from 'lodash-es';
import { TableChartFormData } from './types';
@@ -696,13 +696,16 @@ export const buildQueryUncached: BuildQuery<TableChartFormData> = (
formData.show_totals &&
queryMode === QueryMode.Aggregate,
);
const totalsAggregate: TotalsAggregate =
formData.totals_aggregate === 'AVG' ? 'AVG' : 'SUM';
const totalsAggregate = toTotalsAggregate(formData.totals_aggregate);
// Raw-mode summary columns have no metric of their own to preserve, so
// ORIGINAL has nothing to fall back to; sum them as before.
const rawSummaryAggregate =
totalsAggregate === 'ORIGINAL' ? 'SUM' : totalsAggregate;
const totalsMetrics =
rawSummaryColumns.length > 0
? rawSummaryColumns.map(columnName => ({
expressionType: 'SIMPLE' as const,
aggregate: totalsAggregate,
aggregate: rawSummaryAggregate,
column: { column_name: columnName },
label: columnName,
}))
@@ -503,14 +503,18 @@ const config: ControlPanelConfig = {
label: t('Summary aggregation'),
renderTrigger: true,
description: t(
'Aggregation used for the summary row, independent of each ' +
"metric's own aggregation. Only applies to simple metrics " +
'(a metric built from custom SQL keeps its own aggregation ' +
'in the summary row).',
'Aggregation used for the summary row. By default each metric ' +
'keeps its own aggregation; Sum and Average override it for ' +
'the summary row only. The override applies to simple ' +
'metrics (a metric built from custom SQL always keeps its ' +
'own aggregation). Overriding a count or a distinct count ' +
'sums the counted column instead, which fails outright on a ' +
'non-numeric column.',
),
default: 'SUM',
default: 'ORIGINAL',
clearable: false,
choices: [
['ORIGINAL', t("Each metric's own")],
['SUM', t('Sum')],
['AVG', t('Average')],
],
@@ -1561,7 +1561,7 @@ describe('plugin-chart-ag-grid-table', () => {
expect(queries[1].metrics).toEqual(['count']);
});
test('defaults aggregate-mode totals to SUM for a simple metric', () => {
test("defaults aggregate-mode totals to the metric's own aggregate", () => {
const simpleMetric = {
expressionType: 'SIMPLE' as const,
column: { column_name: 'sales' },
@@ -1580,9 +1580,29 @@ describe('plugin-chart-ag-grid-table', () => {
{ ownState: {} },
);
expect(queries[1].metrics).toEqual([
{ ...simpleMetric, aggregate: 'SUM' },
]);
expect(queries[1].metrics).toEqual([simpleMetric]);
});
test('keeps COUNT_DISTINCT in aggregate-mode totals by default', () => {
const countDistinctMetric = {
expressionType: 'SIMPLE' as const,
column: { column_name: 'contract_id' },
aggregate: 'COUNT_DISTINCT' as const,
label: 'contracts',
};
const { queries } = buildQuery(
{
viz_type: VizType.Table,
datasource: '11__table',
query_mode: QueryMode.Aggregate,
groupby: ['state'],
metrics: [countDistinctMetric],
show_totals: true,
},
{ ownState: {} },
);
expect(queries[1].metrics).toEqual([countDistinctMetric]);
});
test('overrides aggregate-mode totals to AVG for a simple metric when totals_aggregate is set', () => {
@@ -105,7 +105,7 @@
"source": [
"## Download Data\n",
"\n",
"Download datasets (_Admin 0 - Countries_ in [1:10](https://www.naturalearthdata.com/downloads/10m-cultural-vectors/), and _Admin 1 States, Provinces_ in 1:10 and [1:50](https://www.naturalearthdata.com/downloads/50m-cultural-vectors/)) from Natural Earch Data:"
"Download datasets (_Admin 0 - Countries_ in [1:10](https://www.naturalearthdata.com/downloads/10m-cultural-vectors/), and _Admin 1 \u2013 States, Provinces_ in 1:10 and [1:50](https://www.naturalearthdata.com/downloads/50m-cultural-vectors/)) from Natural Earch Data:"
]
},
{
@@ -584,7 +584,7 @@
" </tr>\n",
" </tbody>\n",
"</table>\n",
"<p>9 rows × 121 columns</p>\n",
"<p>9 rows \u00d7 121 columns</p>\n",
"</div>"
],
"text/plain": [
@@ -926,33 +926,33 @@
" <td>11.0</td>\n",
" <td>11.0</td>\n",
" <td>Q34617</td>\n",
" <td>سان بيير وميكلون</td>\n",
" <td>সাঁ পিয়ের ও মিকলোঁ</td>\n",
" <td>\u0633\u0627\u0646 \u0628\u064a\u064a\u0631 \u0648\u0645\u064a\u0643\u0644\u0648\u0646</td>\n",
" <td>\u09b8\u09be\u0981 \u09aa\u09bf\u09af\u09bc\u09c7\u09b0 \u0993 \u09ae\u09bf\u0995\u09b2\u09cb\u0981</td>\n",
" <td>Saint-Pierre und Miquelon</td>\n",
" <td>Saint Pierre and Miquelon</td>\n",
" <td>San Pedro y Miquelón</td>\n",
" <td>San Pedro y Miquel\u00f3n</td>\n",
" <td>Saint-Pierre-et-Miquelon</td>\n",
" <td>Σαιν-Πιερ και Μικελόν</td>\n",
" <td>सन्त पियर और मिकलान</td>\n",
" <td>Saint-Pierre és Miquelon</td>\n",
" <td>\u03a3\u03b1\u03b9\u03bd-\u03a0\u03b9\u03b5\u03c1 \u03ba\u03b1\u03b9 \u039c\u03b9\u03ba\u03b5\u03bb\u03cc\u03bd</td>\n",
" <td>\u0938\u0928\u094d\u0924 \u092a\u093f\u092f\u0930 \u0914\u0930 \u092e\u093f\u0915\u0932\u093e\u0928</td>\n",
" <td>Saint-Pierre \u00e9s Miquelon</td>\n",
" <td>Saint Pierre dan Miquelon</td>\n",
" <td>Saint-Pierre e Miquelon</td>\n",
" <td>サンピエール島・ミクロン島</td>\n",
" <td>생피에르 미클롱</td>\n",
" <td>\u30b5\u30f3\u30d4\u30a8\u30fc\u30eb\u5cf6\u30fb\u30df\u30af\u30ed\u30f3\u5cf6</td>\n",
" <td>\uc0dd\ud53c\uc5d0\ub974 \ubbf8\ud074\ub871</td>\n",
" <td>Saint-Pierre en Miquelon</td>\n",
" <td>Saint-Pierre i Miquelon</td>\n",
" <td>Saint-Pierre e Miquelon</td>\n",
" <td>Сен-Пьер и Микелон</td>\n",
" <td>\u0421\u0435\u043d-\u041f\u044c\u0435\u0440 \u0438 \u041c\u0438\u043a\u0435\u043b\u043e\u043d</td>\n",
" <td>Saint-Pierre och Miquelon</td>\n",
" <td>Saint Pierre ve Miquelon</td>\n",
" <td>Saint-Pierre và Miquelon</td>\n",
" <td>圣皮埃尔和密克隆</td>\n",
" <td>Saint-Pierre v\u00e0 Miquelon</td>\n",
" <td>\u5723\u76ae\u57c3\u5c14\u548c\u5bc6\u514b\u9686</td>\n",
" <td>1159315673</td>\n",
" <td>סן-פייר ומיקלון</td>\n",
" <td>Сен-П'єр і Мікелон</td>\n",
" <td>سینٹ پیئر و میکیلون</td>\n",
" <td>سن پیر و میکلن</td>\n",
" <td>聖皮埃與密克隆群島</td>\n",
" <td>\u05e1\u05df-\u05e4\u05d9\u05d9\u05e8 \u05d5\u05de\u05d9\u05e7\u05dc\u05d5\u05df</td>\n",
" <td>\u0421\u0435\u043d-\u041f'\u0454\u0440 \u0456 \u041c\u0456\u043a\u0435\u043b\u043e\u043d</td>\n",
" <td>\u0633\u06cc\u0646\u0679 \u067e\u06cc\u0626\u0631 \u0648 \u0645\u06cc\u06a9\u06cc\u0644\u0648\u0646</td>\n",
" <td>\u0633\u0646 \u067e\u06cc\u0631 \u0648 \u0645\u06cc\u06a9\u0644\u0646</td>\n",
" <td>\u8056\u76ae\u57c3\u8207\u5bc6\u514b\u9686\u7fa4\u5cf6</td>\n",
" <td>None</td>\n",
" <td>None</td>\n",
" <td>None</td>\n",
@@ -1051,33 +1051,33 @@
" <td>11.0</td>\n",
" <td>11.0</td>\n",
" <td>None</td>\n",
" <td>ميكلون ولانغليد</td>\n",
" <td>মিকুইলন-ল্যাংলেড</td>\n",
" <td>\u0645\u064a\u0643\u0644\u0648\u0646 \u0648\u0644\u0627\u0646\u063a\u0644\u064a\u062f</td>\n",
" <td>\u09ae\u09bf\u0995\u09c1\u0987\u09b2\u09a8-\u09b2\u09cd\u09af\u09be\u0982\u09b2\u09c7\u09a1</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelón-Langlade</td>\n",
" <td>Miquel\u00f3n-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Μικελόν-Λαγκλέιντ</td>\n",
" <td>मिकेलॉन-लैंगलेड</td>\n",
" <td>\u039c\u03b9\u03ba\u03b5\u03bb\u03cc\u03bd-\u039b\u03b1\u03b3\u03ba\u03bb\u03ad\u03b9\u03bd\u03c4</td>\n",
" <td>\u092e\u093f\u0915\u0947\u0932\u0949\u0928-\u0932\u0948\u0902\u0917\u0932\u0947\u0921</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>ミクロン=ラングラード</td>\n",
" <td>미클롱-랭글레이드</td>\n",
" <td>\u30df\u30af\u30ed\u30f3\uff1d\u30e9\u30f3\u30b0\u30e9\u30fc\u30c9</td>\n",
" <td>\ubbf8\ud074\ub871-\ub7ad\uae00\ub808\uc774\ub4dc</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelão-Langlade</td>\n",
" <td>Микелон-Ланглад</td>\n",
" <td>Miquel\u00e3o-Langlade</td>\n",
" <td>\u041c\u0438\u043a\u0435\u043b\u043e\u043d-\u041b\u0430\u043d\u0433\u043b\u0430\u0434</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>Miquelon-Langlade</td>\n",
" <td>密克隆-朗格拉德</td>\n",
" <td>\u5bc6\u514b\u9686-\u6717\u683c\u62c9\u5fb7</td>\n",
" <td>1159315961</td>\n",
" <td>מירה</td>\n",
" <td>Міквелон-Лангладе</td>\n",
" <td>میکیولون لینگلاڈے</td>\n",
" <td>میکوئلون-لانگلید</td>\n",
" <td>密克隆-朗格拉德</td>\n",
" <td>\u05de\u05d9\u05e8\u05d4</td>\n",
" <td>\u041c\u0456\u043a\u0432\u0435\u043b\u043e\u043d-\u041b\u0430\u043d\u0433\u043b\u0430\u0434\u0435</td>\n",
" <td>\u0645\u06cc\u06a9\u06cc\u0648\u0644\u0648\u0646 \u0644\u06cc\u0646\u06af\u0644\u0627\u0688\u06d2</td>\n",
" <td>\u0645\u06cc\u06a9\u0648\u0626\u0644\u0648\u0646-\u0644\u0627\u0646\u06af\u0644\u06cc\u062f</td>\n",
" <td>\u5bc6\u514b\u9686-\u6717\u683c\u62c9\u5fb7</td>\n",
" <td>None</td>\n",
" <td>None</td>\n",
" <td>None</td>\n",
@@ -1167,48 +1167,48 @@
"2177 PM.97501 None None 1.0 fra SB00 None \n",
"\n",
" min_label max_label min_zoom wikidataid name_ar \\\n",
"2176 11.0 11.0 11.0 Q34617 سان بيير وميكلون \n",
"2177 11.0 11.0 11.0 None ميكلون ولانغليد \n",
"2176 11.0 11.0 11.0 Q34617 \u0633\u0627\u0646 \u0628\u064a\u064a\u0631 \u0648\u0645\u064a\u0643\u0644\u0648\u0646 \n",
"2177 11.0 11.0 11.0 None \u0645\u064a\u0643\u0644\u0648\u0646 \u0648\u0644\u0627\u0646\u063a\u0644\u064a\u062f \n",
"\n",
" name_bn name_de \\\n",
"2176 সাঁ পিয়ের ও মিকলোঁ Saint-Pierre und Miquelon \n",
"2177 মিকুইলন-ল্যাংলেড Miquelon-Langlade \n",
"2176 \u09b8\u09be\u0981 \u09aa\u09bf\u09af\u09bc\u09c7\u09b0 \u0993 \u09ae\u09bf\u0995\u09b2\u09cb\u0981 Saint-Pierre und Miquelon \n",
"2177 \u09ae\u09bf\u0995\u09c1\u0987\u09b2\u09a8-\u09b2\u09cd\u09af\u09be\u0982\u09b2\u09c7\u09a1 Miquelon-Langlade \n",
"\n",
" name_en name_es \\\n",
"2176 Saint Pierre and Miquelon San Pedro y Miquelón \n",
"2177 Miquelon-Langlade Miquelón-Langlade \n",
"2176 Saint Pierre and Miquelon San Pedro y Miquel\u00f3n \n",
"2177 Miquelon-Langlade Miquel\u00f3n-Langlade \n",
"\n",
" name_fr name_el name_hi \\\n",
"2176 Saint-Pierre-et-Miquelon Σαιν-Πιερ και Μικελόν सन्त पियर और मिकलान \n",
"2177 Miquelon-Langlade Μικελόν-Λαγκλέιντ मिकेलॉन-लैंगलेड \n",
"2176 Saint-Pierre-et-Miquelon \u03a3\u03b1\u03b9\u03bd-\u03a0\u03b9\u03b5\u03c1 \u03ba\u03b1\u03b9 \u039c\u03b9\u03ba\u03b5\u03bb\u03cc\u03bd \u0938\u0928\u094d\u0924 \u092a\u093f\u092f\u0930 \u0914\u0930 \u092e\u093f\u0915\u0932\u093e\u0928 \n",
"2177 Miquelon-Langlade \u039c\u03b9\u03ba\u03b5\u03bb\u03cc\u03bd-\u039b\u03b1\u03b3\u03ba\u03bb\u03ad\u03b9\u03bd\u03c4 \u092e\u093f\u0915\u0947\u0932\u0949\u0928-\u0932\u0948\u0902\u0917\u0932\u0947\u0921 \n",
"\n",
" name_hu name_id \\\n",
"2176 Saint-Pierre és Miquelon Saint Pierre dan Miquelon \n",
"2176 Saint-Pierre \u00e9s Miquelon Saint Pierre dan Miquelon \n",
"2177 Miquelon-Langlade Miquelon-Langlade \n",
"\n",
" name_it name_ja name_ko \\\n",
"2176 Saint-Pierre e Miquelon サンピエール島・ミクロン島 생피에르 미클롱 \n",
"2177 Miquelon-Langlade ミクロン=ラングラード 미클롱-랭글레이드 \n",
"2176 Saint-Pierre e Miquelon \u30b5\u30f3\u30d4\u30a8\u30fc\u30eb\u5cf6\u30fb\u30df\u30af\u30ed\u30f3\u5cf6 \uc0dd\ud53c\uc5d0\ub974 \ubbf8\ud074\ub871 \n",
"2177 Miquelon-Langlade \u30df\u30af\u30ed\u30f3\uff1d\u30e9\u30f3\u30b0\u30e9\u30fc\u30c9 \ubbf8\ud074\ub871-\ub7ad\uae00\ub808\uc774\ub4dc \n",
"\n",
" name_nl name_pl \\\n",
"2176 Saint-Pierre en Miquelon Saint-Pierre i Miquelon \n",
"2177 Miquelon-Langlade Miquelon-Langlade \n",
"\n",
" name_pt name_ru name_sv \\\n",
"2176 Saint-Pierre e Miquelon Сен-Пьер и Микелон Saint-Pierre och Miquelon \n",
"2177 Miquelão-Langlade Микелон-Ланглад Miquelon-Langlade \n",
"2176 Saint-Pierre e Miquelon \u0421\u0435\u043d-\u041f\u044c\u0435\u0440 \u0438 \u041c\u0438\u043a\u0435\u043b\u043e\u043d Saint-Pierre och Miquelon \n",
"2177 Miquel\u00e3o-Langlade \u041c\u0438\u043a\u0435\u043b\u043e\u043d-\u041b\u0430\u043d\u0433\u043b\u0430\u0434 Miquelon-Langlade \n",
"\n",
" name_tr name_vi name_zh \\\n",
"2176 Saint Pierre ve Miquelon Saint-Pierre và Miquelon 圣皮埃尔和密克隆 \n",
"2177 Miquelon-Langlade Miquelon-Langlade 密克隆-朗格拉德 \n",
"2176 Saint Pierre ve Miquelon Saint-Pierre v\u00e0 Miquelon \u5723\u76ae\u57c3\u5c14\u548c\u5bc6\u514b\u9686 \n",
"2177 Miquelon-Langlade Miquelon-Langlade \u5bc6\u514b\u9686-\u6717\u683c\u62c9\u5fb7 \n",
"\n",
" ne_id name_he name_uk name_ur \\\n",
"2176 1159315673 סן-פייר ומיקלון Сен-П'єр і Мікелон سینٹ پیئر و میکیلون \n",
"2177 1159315961 מירה Міквелон-Лангладе میکیولون لینگلاڈے \n",
"2176 1159315673 \u05e1\u05df-\u05e4\u05d9\u05d9\u05e8 \u05d5\u05de\u05d9\u05e7\u05dc\u05d5\u05df \u0421\u0435\u043d-\u041f'\u0454\u0440 \u0456 \u041c\u0456\u043a\u0435\u043b\u043e\u043d \u0633\u06cc\u0646\u0679 \u067e\u06cc\u0626\u0631 \u0648 \u0645\u06cc\u06a9\u06cc\u0644\u0648\u0646 \n",
"2177 1159315961 \u05de\u05d9\u05e8\u05d4 \u041c\u0456\u043a\u0432\u0435\u043b\u043e\u043d-\u041b\u0430\u043d\u0433\u043b\u0430\u0434\u0435 \u0645\u06cc\u06a9\u06cc\u0648\u0644\u0648\u0646 \u0644\u06cc\u0646\u06af\u0644\u0627\u0688\u06d2 \n",
"\n",
" name_fa name_zht FCLASS_ISO FCLASS_US FCLASS_FR FCLASS_RU \\\n",
"2176 سن پیر و میکلن 聖皮埃與密克隆群島 None None None None \n",
"2177 میکوئلون-لانگلید 密克隆-朗格拉德 None None None None \n",
"2176 \u0633\u0646 \u067e\u06cc\u0631 \u0648 \u0645\u06cc\u06a9\u0644\u0646 \u8056\u76ae\u57c3\u8207\u5bc6\u514b\u9686\u7fa4\u5cf6 None None None None \n",
"2177 \u0645\u06cc\u06a9\u0648\u0626\u0644\u0648\u0646-\u0644\u0627\u0646\u06af\u0644\u06cc\u062f \u5bc6\u514b\u9686-\u6717\u683c\u62c9\u5fb7 None None None None \n",
"\n",
" FCLASS_ES FCLASS_CN FCLASS_TW FCLASS_IN FCLASS_NP FCLASS_PK FCLASS_DE \\\n",
"2176 None None None None None None None \n",
@@ -1330,7 +1330,7 @@
" 'costa rica',\n",
" 'croatia',\n",
" 'cuba',\n",
" 'curaçao',\n",
" 'cura\u00e7ao',\n",
" 'cyprus',\n",
" 'czech republic',\n",
" 'denmark',\n",
@@ -1343,7 +1343,7 @@
" 'equatorial guinea',\n",
" 'eritrea',\n",
" 'estonia',\n",
" # 'eswatini', # not sure why this doesn't work Swaziland isn't available to alias, either.\n",
" # 'eswatini', # not sure why this doesn't work \u2014 Swaziland isn't available to alias, either.\n",
" 'ethiopia',\n",
" 'falkland islands',\n",
" 'faroe islands',\n",
@@ -1443,7 +1443,7 @@
" 'portugal',\n",
" 'puerto rico',\n",
" 'qatar',\n",
" # 'réunion', # part of France, in Natural Earth data\n",
" # 'r\u00e9union', # part of France, in Natural Earth data\n",
" 'republic of serbia',\n",
" 'romania',\n",
" 'russia',\n",
@@ -1911,34 +1911,34 @@
" <td>9.0</td>\n",
" <td>1159320473</td>\n",
" <td>Q8646</td>\n",
" <td>هونغ كونغ</td>\n",
" <td>হংকং</td>\n",
" <td>\u0647\u0648\u0646\u063a \u0643\u0648\u0646\u063a</td>\n",
" <td>\u09b9\u0982\u0995\u0982</td>\n",
" <td>Hongkong</td>\n",
" <td>Hong Kong</td>\n",
" <td>Hong Kong</td>\n",
" <td>هنگ کنگ</td>\n",
" <td>\u0647\u0646\u06af \u06a9\u0646\u06af</td>\n",
" <td>Hong Kong</td>\n",
" <td>Χονγκ Κονγκ</td>\n",
" <td>הונג קונג</td>\n",
" <td>हांगकांग</td>\n",
" <td>\u03a7\u03bf\u03bd\u03b3\u03ba \u039a\u03bf\u03bd\u03b3\u03ba</td>\n",
" <td>\u05d4\u05d5\u05e0\u05d2 \u05e7\u05d5\u05e0\u05d2</td>\n",
" <td>\u0939\u093e\u0902\u0917\u0915\u093e\u0902\u0917</td>\n",
" <td>Hongkong</td>\n",
" <td>Hong Kong</td>\n",
" <td>Hong Kong</td>\n",
" <td>香港</td>\n",
" <td>홍콩</td>\n",
" <td>\u9999\u6e2f</td>\n",
" <td>\ud64d\ucf69</td>\n",
" <td>Hongkong</td>\n",
" <td>Hongkong</td>\n",
" <td>Hong Kong</td>\n",
" <td>Гонконг</td>\n",
" <td>\u0413\u043e\u043d\u043a\u043e\u043d\u0433</td>\n",
" <td>Hongkong</td>\n",
" <td>Hong Kong</td>\n",
" <td>Гонконг</td>\n",
" <td>ہانگ کانگ</td>\n",
" <td>Hồng Kông</td>\n",
" <td>香港</td>\n",
" <td>香港</td>\n",
" <td>\u0413\u043e\u043d\u043a\u043e\u043d\u0433</td>\n",
" <td>\u06c1\u0627\u0646\u06af \u06a9\u0627\u0646\u06af</td>\n",
" <td>H\u1ed3ng K\u00f4ng</td>\n",
" <td>\u9999\u6e2f</td>\n",
" <td>\u9999\u6e2f</td>\n",
" <td>MULTIPOLYGON (((114.22983 22.55581, 114.23471 ...</td>\n",
" <td>香港特别行政区</td>\n",
" <td>\u9999\u6e2f\u7279\u522b\u884c\u653f\u533a</td>\n",
" <td>CN-91</td>\n",
" </tr>\n",
" <tr>\n",
@@ -1965,34 +1965,34 @@
" <td>8.0</td>\n",
" <td>1159321335</td>\n",
" <td>Q865</td>\n",
" <td>تايوان</td>\n",
" <td>তাইওয়ান</td>\n",
" <td>\u062a\u0627\u064a\u0648\u0627\u0646</td>\n",
" <td>\u09a4\u09be\u0987\u0993\u09af\u09bc\u09be\u09a8</td>\n",
" <td>Republik China</td>\n",
" <td>Taiwan</td>\n",
" <td>República de China</td>\n",
" <td>تایوان</td>\n",
" <td>Taïwan</td>\n",
" <td>Δημοκρατία της Κίνας</td>\n",
" <td>טאיוואן</td>\n",
" <td>चीनी गणराज्य</td>\n",
" <td>Kínai Köztársaság</td>\n",
" <td>Rep\u00fablica de China</td>\n",
" <td>\u062a\u0627\u06cc\u0648\u0627\u0646</td>\n",
" <td>Ta\u00efwan</td>\n",
" <td>\u0394\u03b7\u03bc\u03bf\u03ba\u03c1\u03b1\u03c4\u03af\u03b1 \u03c4\u03b7\u03c2 \u039a\u03af\u03bd\u03b1\u03c2</td>\n",
" <td>\u05d8\u05d0\u05d9\u05d5\u05d5\u05d0\u05df</td>\n",
" <td>\u091a\u0940\u0928\u0940 \u0917\u0923\u0930\u093e\u091c\u094d\u092f</td>\n",
" <td>K\u00ednai K\u00f6zt\u00e1rsas\u00e1g</td>\n",
" <td>Taiwan</td>\n",
" <td>Taiwan</td>\n",
" <td>中華民国</td>\n",
" <td>중화민국</td>\n",
" <td>\u4e2d\u83ef\u6c11\u56fd</td>\n",
" <td>\uc911\ud654\ubbfc\uad6d</td>\n",
" <td>Taiwan</td>\n",
" <td>Republika Chińska</td>\n",
" <td>Republika Chi\u0144ska</td>\n",
" <td>Taiwan</td>\n",
" <td>Тайвань</td>\n",
" <td>\u0422\u0430\u0439\u0432\u0430\u043d\u044c</td>\n",
" <td>Taiwan</td>\n",
" <td>Çin Cumhuriyeti</td>\n",
" <td>Республіка Китай</td>\n",
" <td>تائیوان</td>\n",
" <td>Đài Loan</td>\n",
" <td>中华民国</td>\n",
" <td>中華民國</td>\n",
" <td>\u00c7in Cumhuriyeti</td>\n",
" <td>\u0420\u0435\u0441\u043f\u0443\u0431\u043b\u0456\u043a\u0430 \u041a\u0438\u0442\u0430\u0439</td>\n",
" <td>\u062a\u0627\u0626\u06cc\u0648\u0627\u0646</td>\n",
" <td>\u0110\u00e0i Loan</td>\n",
" <td>\u4e2d\u534e\u6c11\u56fd</td>\n",
" <td>\u4e2d\u83ef\u6c11\u570b</td>\n",
" <td>MULTIPOLYGON (((121.90577 24.9501, 121.83473 2...</td>\n",
" <td>中国台湾</td>\n",
" <td>\u4e2d\u56fd\u53f0\u6e7e</td>\n",
" <td>CN-71</td>\n",
" </tr>\n",
" <tr>\n",
@@ -2019,34 +2019,34 @@
" <td>9.0</td>\n",
" <td>1159320475</td>\n",
" <td>Q14773</td>\n",
" <td>ماكاو</td>\n",
" <td>মাকাও</td>\n",
" <td>\u0645\u0627\u0643\u0627\u0648</td>\n",
" <td>\u09ae\u09be\u0995\u09be\u0993</td>\n",
" <td>Macau</td>\n",
" <td>Macau</td>\n",
" <td>Macao</td>\n",
" <td>ماکائو</td>\n",
" <td>\u0645\u0627\u06a9\u0627\u0626\u0648</td>\n",
" <td>Macao</td>\n",
" <td>Μακάου</td>\n",
" <td>מקאו</td>\n",
" <td>मकाउ</td>\n",
" <td>Makaó</td>\n",
" <td>\u039c\u03b1\u03ba\u03ac\u03bf\u03c5</td>\n",
" <td>\u05de\u05e7\u05d0\u05d5</td>\n",
" <td>\u092e\u0915\u093e\u0909</td>\n",
" <td>Maka\u00f3</td>\n",
" <td>Makau</td>\n",
" <td>Macao</td>\n",
" <td>マカオ</td>\n",
" <td>마카오</td>\n",
" <td>\u30de\u30ab\u30aa</td>\n",
" <td>\ub9c8\uce74\uc624</td>\n",
" <td>Macau</td>\n",
" <td>Makau</td>\n",
" <td>Macau</td>\n",
" <td>Макао</td>\n",
" <td>\u041c\u0430\u043a\u0430\u043e</td>\n",
" <td>Macao</td>\n",
" <td>Makao</td>\n",
" <td>Аоминь</td>\n",
" <td>مکاؤ</td>\n",
" <td>\u0410\u043e\u043c\u0438\u043d\u044c</td>\n",
" <td>\u0645\u06a9\u0627\u0624</td>\n",
" <td>Ma Cao</td>\n",
" <td>澳门</td>\n",
" <td>澳門</td>\n",
" <td>\u6fb3\u95e8</td>\n",
" <td>\u6fb3\u9580</td>\n",
" <td>MULTIPOLYGON (((113.5586 22.16303, 113.56943 2...</td>\n",
" <td>澳门特别行政区</td>\n",
" <td>\u6fb3\u95e8\u7279\u522b\u884c\u653f\u533a</td>\n",
" <td>CN-92</td>\n",
" </tr>\n",
" </tbody>\n",
@@ -2070,34 +2070,34 @@
"2 4 3 MO 20070017 5 0.0 4.0 \n",
"\n",
" max_label ne_id wikidataid name_ar name_bn name_de \\\n",
"0 9.0 1159320473 Q8646 هونغ كونغ হংকং Hongkong \n",
"1 8.0 1159321335 Q865 تايوان তাইওয়ান Republik China \n",
"2 9.0 1159320475 Q14773 ماكاو মাকাও Macau \n",
"0 9.0 1159320473 Q8646 \u0647\u0648\u0646\u063a \u0643\u0648\u0646\u063a \u09b9\u0982\u0995\u0982 Hongkong \n",
"1 8.0 1159321335 Q865 \u062a\u0627\u064a\u0648\u0627\u0646 \u09a4\u09be\u0987\u0993\u09af\u09bc\u09be\u09a8 Republik China \n",
"2 9.0 1159320475 Q14773 \u0645\u0627\u0643\u0627\u0648 \u09ae\u09be\u0995\u09be\u0993 Macau \n",
"\n",
" name_en name_es name_fa name_fr name_el \\\n",
"0 Hong Kong Hong Kong هنگ کنگ Hong Kong Χονγκ Κονγκ \n",
"1 Taiwan República de China تایوان Taïwan Δημοκρατία της Κίνας \n",
"2 Macau Macao ماکائو Macao Μακάου \n",
"0 Hong Kong Hong Kong \u0647\u0646\u06af \u06a9\u0646\u06af Hong Kong \u03a7\u03bf\u03bd\u03b3\u03ba \u039a\u03bf\u03bd\u03b3\u03ba \n",
"1 Taiwan Rep\u00fablica de China \u062a\u0627\u06cc\u0648\u0627\u0646 Ta\u00efwan \u0394\u03b7\u03bc\u03bf\u03ba\u03c1\u03b1\u03c4\u03af\u03b1 \u03c4\u03b7\u03c2 \u039a\u03af\u03bd\u03b1\u03c2 \n",
"2 Macau Macao \u0645\u0627\u06a9\u0627\u0626\u0648 Macao \u039c\u03b1\u03ba\u03ac\u03bf\u03c5 \n",
"\n",
" name_he name_hi name_hu name_id name_it name_ja \\\n",
"0 הונג קונג हांगकांग Hongkong Hong Kong Hong Kong 香港 \n",
"1 טאיוואן चीनी गणराज्य Kínai Köztársaság Taiwan Taiwan 中華民国 \n",
"2 מקאו मकाउ Makaó Makau Macao マカオ \n",
"0 \u05d4\u05d5\u05e0\u05d2 \u05e7\u05d5\u05e0\u05d2 \u0939\u093e\u0902\u0917\u0915\u093e\u0902\u0917 Hongkong Hong Kong Hong Kong \u9999\u6e2f \n",
"1 \u05d8\u05d0\u05d9\u05d5\u05d5\u05d0\u05df \u091a\u0940\u0928\u0940 \u0917\u0923\u0930\u093e\u091c\u094d\u092f K\u00ednai K\u00f6zt\u00e1rsas\u00e1g Taiwan Taiwan \u4e2d\u83ef\u6c11\u56fd \n",
"2 \u05de\u05e7\u05d0\u05d5 \u092e\u0915\u093e\u0909 Maka\u00f3 Makau Macao \u30de\u30ab\u30aa \n",
"\n",
" name_ko name_nl name_pl name_pt name_ru name_sv \\\n",
"0 홍콩 Hongkong Hongkong Hong Kong Гонконг Hongkong \n",
"1 중화민국 Taiwan Republika Chińska Taiwan Тайвань Taiwan \n",
"2 마카오 Macau Makau Macau Макао Macao \n",
"0 \ud64d\ucf69 Hongkong Hongkong Hong Kong \u0413\u043e\u043d\u043a\u043e\u043d\u0433 Hongkong \n",
"1 \uc911\ud654\ubbfc\uad6d Taiwan Republika Chi\u0144ska Taiwan \u0422\u0430\u0439\u0432\u0430\u043d\u044c Taiwan \n",
"2 \ub9c8\uce74\uc624 Macau Makau Macau \u041c\u0430\u043a\u0430\u043e Macao \n",
"\n",
" name_tr name_uk name_ur name_vi name_zh_x name_zht \\\n",
"0 Hong Kong Гонконг ہانگ کانگ Hồng Kông 香港 香港 \n",
"1 Çin Cumhuriyeti Республіка Китай تائیوان Đài Loan 中华民国 中華民國 \n",
"2 Makao Аоминь مکاؤ Ma Cao 澳门 澳門 \n",
"0 Hong Kong \u0413\u043e\u043d\u043a\u043e\u043d\u0433 \u06c1\u0627\u0646\u06af \u06a9\u0627\u0646\u06af H\u1ed3ng K\u00f4ng \u9999\u6e2f \u9999\u6e2f \n",
"1 \u00c7in Cumhuriyeti \u0420\u0435\u0441\u043f\u0443\u0431\u043b\u0456\u043a\u0430 \u041a\u0438\u0442\u0430\u0439 \u062a\u0627\u0626\u06cc\u0648\u0627\u0646 \u0110\u00e0i Loan \u4e2d\u534e\u6c11\u56fd \u4e2d\u83ef\u6c11\u570b \n",
"2 Makao \u0410\u043e\u043c\u0438\u043d\u044c \u0645\u06a9\u0627\u0624 Ma Cao \u6fb3\u95e8 \u6fb3\u9580 \n",
"\n",
" geometry name_zh_y iso_3166_2 \n",
"0 MULTIPOLYGON (((114.22983 22.55581, 114.23471 ... 香港特别行政区 CN-91 \n",
"1 MULTIPOLYGON (((121.90577 24.9501, 121.83473 2... 中国台湾 CN-71 \n",
"2 MULTIPOLYGON (((113.5586 22.16303, 113.56943 2... 澳门特别行政区 CN-92 "
"0 MULTIPOLYGON (((114.22983 22.55581, 114.23471 ... \u9999\u6e2f\u7279\u522b\u884c\u653f\u533a CN-91 \n",
"1 MULTIPOLYGON (((121.90577 24.9501, 121.83473 2... \u4e2d\u56fd\u53f0\u6e7e CN-71 \n",
"2 MULTIPOLYGON (((113.5586 22.16303, 113.56943 2... \u6fb3\u95e8\u7279\u522b\u884c\u653f\u533a CN-92 "
]
},
"execution_count": 14,
@@ -2114,7 +2114,7 @@
"china_sars = china_sars.merge(pd.DataFrame(\n",
" data={\n",
" \"name_en\": [\"Taiwan\", \"Hong Kong\", \"Macau\"],\n",
" \"name_zh\": [\"中国台湾\", \"香港特别行政区\", \"澳门特别行政区\"],\n",
" \"name_zh\": [\"\u4e2d\u56fd\u53f0\u6e7e\", \"\u9999\u6e2f\u7279\u522b\u884c\u653f\u533a\", \"\u6fb3\u95e8\u7279\u522b\u884c\u653f\u533a\"],\n",
" \"iso_3166_2\": [\"CN-71\", \"CN-91\", \"CN-92\"],\n",
" },\n",
"), on=\"name_en\", how=\"left\")\n",
@@ -2252,7 +2252,7 @@
" }\n",
")[[\"geometry\", \"iso_3166_2\", \"name\"]].copy()\n",
"\n",
"# Convert MA01 MA-01\n",
"# Convert MA01 \u2192 MA-01\n",
"morocco_copy[\"iso_3166_2\"] = morocco_copy[\n",
" \"iso_3166_2\"\n",
"].str.replace(\n",
@@ -2290,7 +2290,7 @@
"source": [
"#### Finland\n",
"\n",
"- The Åland Islands (ISO country code AX) is an autonomous region of Finland, and carries the ISO-3166 code FI-01."
"- The \u00c5land Islands (ISO country code AX) is an autonomous region of Finland, and carries the ISO-3166 code FI-01."
]
},
{
@@ -2312,12 +2312,12 @@
"outputs": [],
"source": [
"finland_aland = df_admin0_10m.loc[\n",
" df_admin0_10m.name_en.isin(['Åland']),\n",
" df_admin0_10m.name_en.isin(['\u00c5land']),\n",
" [x for x in df_admin0_10m.columns if x in df.columns]\n",
"]\n",
"finland_aland = finland_aland.merge(pd.DataFrame(\n",
" data={\n",
" \"name_en\": [\"Åland\"],\n",
" \"name_en\": [\"\u00c5land\"],\n",
" \"name_fi\": [\"Ahvenanmaan maakunta\"],\n",
" \"iso_3166_2\": [\"FI-01\"],\n",
" },\n",
@@ -3197,34 +3197,34 @@
"\n",
"# Turkey city name corrections\n",
"# Fix completely wrong spellings\n",
"replace_column('name', turkey, 'Kinkkale', 'Kırıkkale')\n",
"replace_column('name', turkey, 'Kinkkale', 'K\u0131r\u0131kkale')\n",
"replace_column('name', turkey, 'Zinguldak', 'Zonguldak')\n",
"replace_column('name', turkey, 'K. Maras', 'Kahramanmaraş')\n",
"replace_column('name', turkey, 'K. Maras', 'Kahramanmara\u015f')\n",
"\n",
"# Fix missing Turkish characters\n",
"replace_column('name', turkey, 'Adiyaman', 'Adıyaman')\n",
"replace_column('name', turkey, 'Agri', 'Ağrı')\n",
"replace_column('name', turkey, 'Aydin', 'Aydın')\n",
"replace_column('name', turkey, 'Balikesir', 'Balıkesir')\n",
"replace_column('name', turkey, 'Çankiri', 'Çankırı')\n",
"replace_column('name', turkey, 'Diyarbakir', 'Diyarbakır')\n",
"replace_column('name', turkey, 'Elazig', 'Elâzığ')\n",
"replace_column('name', turkey, 'Eskisehir', 'Eskişehir')\n",
"replace_column('name', turkey, 'Gümüshane', 'Gümüşhane')\n",
"replace_column('name', turkey, 'Hakkari', 'Hakkâri')\n",
"replace_column('name', turkey, 'Istanbul', 'İstanbul')\n",
"replace_column('name', turkey, 'Izmir', 'İzmir')\n",
"replace_column('name', turkey, 'Iğdir', 'Iğdır')\n",
"replace_column('name', turkey, 'Kirklareli', 'Kırklareli')\n",
"replace_column('name', turkey, 'Kirsehir', 'Kıehir')\n",
"replace_column('name', turkey, 'Mugla', 'Muğla')\n",
"replace_column('name', turkey, 'Mus', 'Muş')\n",
"replace_column('name', turkey, 'Nevsehir', 'Nevşehir')\n",
"replace_column('name', turkey, 'Nigde', 'Niğde')\n",
"replace_column('name', turkey, 'Sanliurfa', 'Şanlıurfa')\n",
"replace_column('name', turkey, 'Sirnak', 'Şırnak')\n",
"replace_column('name', turkey, 'Tekirdag', 'Tekirdağ')\n",
"replace_column('name', turkey, 'Usak', 'Uşak')\n",
"replace_column('name', turkey, 'Adiyaman', 'Ad\u0131yaman')\n",
"replace_column('name', turkey, 'Agri', 'A\u011fr\u0131')\n",
"replace_column('name', turkey, 'Aydin', 'Ayd\u0131n')\n",
"replace_column('name', turkey, 'Balikesir', 'Bal\u0131kesir')\n",
"replace_column('name', turkey, '\u00c7ankiri', '\u00c7ank\u0131r\u0131')\n",
"replace_column('name', turkey, 'Diyarbakir', 'Diyarbak\u0131r')\n",
"replace_column('name', turkey, 'Elazig', 'El\u00e2z\u0131\u011f')\n",
"replace_column('name', turkey, 'Eskisehir', 'Eski\u015fehir')\n",
"replace_column('name', turkey, 'G\u00fcm\u00fcshane', 'G\u00fcm\u00fc\u015fhane')\n",
"replace_column('name', turkey, 'Hakkari', 'Hakk\u00e2ri')\n",
"replace_column('name', turkey, 'Istanbul', '\u0130stanbul')\n",
"replace_column('name', turkey, 'Izmir', '\u0130zmir')\n",
"replace_column('name', turkey, 'I\u011fdir', 'I\u011fd\u0131r')\n",
"replace_column('name', turkey, 'Kirklareli', 'K\u0131rklareli')\n",
"replace_column('name', turkey, 'Kirsehir', 'K\u0131r\u015fehir')\n",
"replace_column('name', turkey, 'Mugla', 'Mu\u011fla')\n",
"replace_column('name', turkey, 'Mus', 'Mu\u015f')\n",
"replace_column('name', turkey, 'Nevsehir', 'Nev\u015fehir')\n",
"replace_column('name', turkey, 'Nigde', 'Ni\u011fde')\n",
"replace_column('name', turkey, 'Sanliurfa', '\u015eanl\u0131urfa')\n",
"replace_column('name', turkey, 'Sirnak', '\u015e\u0131rnak')\n",
"replace_column('name', turkey, 'Tekirdag', 'Tekirda\u011f')\n",
"replace_column('name', turkey, 'Usak', 'U\u015fak')\n",
"turkey_copy = turkey.copy()"
]
},
@@ -3263,18 +3263,18 @@
"\n",
"# Region names corresponding to NUTS-1\n",
"\n",
"region_name_dict = {'TR1':'İstanbul',\n",
" 'TR2':'Batı Marmara',\n",
"region_name_dict = {'TR1':'\u0130stanbul',\n",
" 'TR2':'Bat\u0131 Marmara',\n",
" 'TR3':'Ege',\n",
" 'TR4':'Doğu Marmara',\n",
" 'TR5':'Batı Anadolu',\n",
" 'TR4':'Do\u011fu Marmara',\n",
" 'TR5':'Bat\u0131 Anadolu',\n",
" 'TR6':'Akdeniz',\n",
" 'TR7':'Orta Anadolu',\n",
" 'TR8':'Batı Karadeniz',\n",
" 'TR9':'Doğu Karadeniz',\n",
" 'TRA':'Kuzeydoğu Anadolu',\n",
" 'TRC':'Güneydoğu Anadolu',\n",
" 'TRB':'Ortadoğu Anadolu'\n",
" 'TR8':'Bat\u0131 Karadeniz',\n",
" 'TR9':'Do\u011fu Karadeniz',\n",
" 'TRA':'Kuzeydo\u011fu Anadolu',\n",
" 'TRC':'G\u00fcneydo\u011fu Anadolu',\n",
" 'TRB':'Ortado\u011fu Anadolu'\n",
" }\n",
"\n",
"\n",
@@ -3517,8 +3517,8 @@
"france_copy = france.copy()\n",
"reposition(france_copy, france.name=='Guadeloupe', 57.4, 25.4, 1.5, 1.5)\n",
"reposition(france_copy, france.name=='Martinique', 58.4, 27.1, 1.5, 1.5)\n",
"reposition(france_copy, france.name=='Guyane française', 52, 37.7, 0.35, 0.35)\n",
"reposition(france_copy, france.name=='La Réunion', -55, 62.8, 1.5, 1.5)\n",
"reposition(france_copy, france.name=='Guyane fran\u00e7aise', 52, 37.7, 0.35, 0.35)\n",
"reposition(france_copy, france.name=='La R\u00e9union', -55, 62.8, 1.5, 1.5)\n",
"reposition(france_copy, france.name=='Mayotte', -43, 54.3, 1.5, 1.5)\n",
"\n",
"not speed_run and france_copy.plot(figsize=(8, 8), **plot_styles)"
@@ -3669,8 +3669,8 @@
"france_overseas = france.copy()\n",
"reposition(france_overseas, france.name=='Guadeloupe', 53.2, 29, 1.5, 1.5)\n",
"reposition(france_overseas, france.name=='Martinique', 52.8, 27.5, 1.5, 1.5)\n",
"reposition(france_overseas, france.name=='Guyane française', 45, 35.5, 0.3, 0.3)\n",
"reposition(france_overseas, france.name=='La Réunion', -58.2, 60.5, 1.5, 1.5)\n",
"reposition(france_overseas, france.name=='Guyane fran\u00e7aise', 45, 35.5, 0.3, 0.3)\n",
"reposition(france_overseas, france.name=='La R\u00e9union', -58.2, 60.5, 1.5, 1.5)\n",
"reposition(france_overseas, france.name=='Mayotte', -50.5, 52.2, 2, 2)\n",
"\n",
"# Tahiti\n",
@@ -3713,7 +3713,7 @@
"france_overseas = pd.concat([france_overseas, saint_martin_data], ignore_index=True)\n",
"reposition(france_overseas, france_overseas.admin=='Saint Martin', 54.8, 30.3, 5, 5)\n",
"\n",
"# Saint Barthélémy\n",
"# Saint Barth\u00e9l\u00e9my\n",
"saint_barthelemy_data = df[(df['admin'] == 'Saint Barthelemy')]\n",
"france_overseas = pd.concat([france_overseas, saint_barthelemy_data], ignore_index=True)\n",
"reposition(france_overseas, france_overseas.admin=='Saint Barthelemy', 54.5, 30, 8, 8)\n",
@@ -3729,13 +3729,13 @@
"france_overseas = pd.concat([france_overseas, paris_and_littlecrowndpts_copy], ignore_index=True)\n",
"\n",
"# Update metadata properly\n",
"france_overseas.loc[france_overseas['name'] == 'Windward Islands', ['name', 'iso_3166_2']] = ['Polynésie française', 'FR-PF']\n",
"france_overseas.loc[france_overseas['name'] == 'Archipel des Kerguelen', ['name', 'iso_3166_2']] = ['Terres australes et antarctiques françaises', 'FR-TF']\n",
"france_overseas.loc[france_overseas['name'] == 'Windward Islands', ['name', 'iso_3166_2']] = ['Polyn\u00e9sie fran\u00e7aise', 'FR-PF']\n",
"france_overseas.loc[france_overseas['name'] == 'Archipel des Kerguelen', ['name', 'iso_3166_2']] = ['Terres australes et antarctiques fran\u00e7aises', 'FR-TF']\n",
"france_overseas.loc[france_overseas['admin'] == 'Wallis and Futuna', ['name', 'iso_3166_2']] = ['Wallis et Futuna', 'FR-WF']\n",
"france_overseas.loc[france_overseas['admin'] == 'New Caledonia', ['name', 'iso_3166_2']] = ['Nouvelle-Calédonie', 'FR-NC']\n",
"france_overseas.loc[france_overseas['admin'] == 'New Caledonia', ['name', 'iso_3166_2']] = ['Nouvelle-Cal\u00e9donie', 'FR-NC']\n",
"france_overseas.loc[france_overseas['admin'] == 'Saint Pierre and Miquelon', ['name', 'iso_3166_2']] = ['Saint-Pierre-et-Miquelon', 'FR-PM']\n",
"france_overseas.loc[france_overseas['admin'] == 'Saint Martin', ['name', 'iso_3166_2']] = ['Saint-Martin', 'FR-MF']\n",
"france_overseas.loc[france_overseas['admin'] == 'Saint Barthelemy', ['name', 'iso_3166_2']] = ['Saint-Barthélémy', 'FR-BL']\n",
"france_overseas.loc[france_overseas['admin'] == 'Saint Barthelemy', ['name', 'iso_3166_2']] = ['Saint-Barth\u00e9l\u00e9my', 'FR-BL']\n",
"\n",
"# Plot data\n",
"france_overseas = france_overseas.rename(columns={'NAME_1': 'name','ISO': 'iso_3166_2'})\n",
@@ -3821,6 +3821,51 @@
"not speed_run and italy_regions.plot(figsize=(10, 7), **plot_styles)"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "65aIalqEt1LR"
},
"source": [
"#### Italy Regions and Autonomous Provinces"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-27T19:54:28.892892Z",
"iopub.status.busy": "2026-07-27T19:54:28.892454Z",
"iopub.status.idle": "2026-07-27T19:54:31.123499Z",
"shell.execute_reply": "2026-07-27T19:54:31.122932Z"
}
},
"outputs": [],
"source": [
"trento_and_bozen = df[(df.admin == 'Italy') & (df.iso_3166_2.isin(['IT-TN', 'IT-BZ']))][['geometry','iso_3166_2','name']]\n",
"\n",
"italy_regions_and_autonomous_provinces = pd.concat([italy_regions, trento_and_bozen])\n",
"\n",
"italy_regions_and_autonomous_provinces = italy_regions_and_autonomous_provinces[italy_regions_and_autonomous_provinces['iso_3166_2'] != 'IT-32']"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"execution": {
"iopub.execute_input": "2026-07-27T19:54:28.892892Z",
"iopub.status.busy": "2026-07-27T19:54:28.892454Z",
"iopub.status.idle": "2026-07-27T19:54:31.123499Z",
"shell.execute_reply": "2026-07-27T19:54:31.122932Z"
}
},
"outputs": [],
"source": [
"not speed_run and italy_regions_and_autonomous_provinces.plot(figsize=(10, 7), **plot_styles)"
]
},
{
"cell_type": "markdown",
"metadata": {
@@ -4331,86 +4376,86 @@
"output_type": "stream",
"text": [
"Kon Tum\n",
"Đắk Nông\n",
"Đắk Lắk\n",
"\u0110\u1eafk N\u00f4ng\n",
"\u0110\u1eafk L\u1eafk\n",
"Gia Lai\n",
"Bình Phước\n",
"Tây Ninh\n",
"B\u00ecnh Ph\u01b0\u1edbc\n",
"T\u00e2y Ninh\n",
"Long An\n",
"Đồng Tháp\n",
"\u0110\u1ed3ng Th\u00e1p\n",
"An Giang\n",
"Kiên Giang\n",
"Điện Biên\n",
"Sơn La\n",
"Thanh Hóa\n",
"Ngh An\n",
"Hà Tĩnh\n",
"Quảng Bình\n",
"Quảng Trị\n",
"Thừa Thiên - Huế\n",
"Qung Nam\n",
"Hà Giang\n",
"Cao Bng\n",
"Lào Cai\n",
"Lai Châu\n",
"Lạng Sơn\n",
"Qung Ninh\n",
"Sóc Trăng\n",
"Tin Giang\n",
"Bà Rịa - Vũng Tàu\n",
"Thành phố Hồ Chí Minh\n",
"Khánh Hòa\n",
"Cà Mau\n",
"Bạc Liêu\n",
"Hu Giang\n",
"Vĩnh Long\n",
"Trà Vinh\n",
"Bến Tre\n",
"Đồng Nai\n",
"Bình Thuận\n",
"Ninh Thun\n",
"Phú Yên\n",
"Bình Định\n",
"Quảng Ngãi\n",
"Đà Nẵng\n",
"Ninh Bình\n",
"Nam Định\n",
"Thái Bình\n",
"Hải Phòng\n",
"Hòa Bình\n",
"Tuyên Quang\n",
"Yên Bái\n",
"Vĩnh Phúc\n",
"Phú Thọ\n",
"Hà Nội\n",
"Bắc Kạn\n",
"Hưng Yên\n",
"Bc Ninh\n",
"Bc Giang\n",
"Thái Nguyên\n",
"Hải Dương\n",
"Hà Nam\n",
"Bình Dương\n",
"Lâm Đồng\n",
"Cần Thơ\n"
"Ki\u00ean Giang\n",
"\u0110i\u1ec7n Bi\u00ean\n",
"S\u01a1n La\n",
"Thanh H\u00f3a\n",
"Ngh\u1ec7 An\n",
"H\u00e0 T\u0129nh\n",
"Qu\u1ea3ng B\u00ecnh\n",
"Qu\u1ea3ng Tr\u1ecb\n",
"Th\u1eeba Thi\u00ean - Hu\u1ebf\n",
"Qu\u1ea3ng Nam\n",
"H\u00e0 Giang\n",
"Cao B\u1eb1ng\n",
"L\u00e0o Cai\n",
"Lai Ch\u00e2u\n",
"L\u1ea1ng S\u01a1n\n",
"Qu\u1ea3ng Ninh\n",
"S\u00f3c Tr\u0103ng\n",
"Ti\u1ec1n Giang\n",
"B\u00e0 R\u1ecba - V\u0169ng T\u00e0u\n",
"Th\u00e0nh ph\u1ed1 H\u1ed3 Ch\u00ed Minh\n",
"Kh\u00e1nh H\u00f2a\n",
"C\u00e0 Mau\n",
"B\u1ea1c Li\u00eau\n",
"H\u1eadu Giang\n",
"V\u0129nh Long\n",
"Tr\u00e0 Vinh\n",
"B\u1ebfn Tre\n",
"\u0110\u1ed3ng Nai\n",
"B\u00ecnh Thu\u1eadn\n",
"Ninh Thu\u1eadn\n",
"Ph\u00fa Y\u00ean\n",
"B\u00ecnh \u0110\u1ecbnh\n",
"Qu\u1ea3ng Ng\u00e3i\n",
"\u0110\u00e0 N\u1eb5ng\n",
"Ninh B\u00ecnh\n",
"Nam \u0110\u1ecbnh\n",
"Th\u00e1i B\u00ecnh\n",
"H\u1ea3i Ph\u00f2ng\n",
"H\u00f2a B\u00ecnh\n",
"Tuy\u00ean Quang\n",
"Y\u00ean B\u00e1i\n",
"V\u0129nh Ph\u00fac\n",
"Ph\u00fa Th\u1ecd\n",
"H\u00e0 N\u1ed9i\n",
"B\u1eafc K\u1ea1n\n",
"H\u01b0ng Y\u00ean\n",
"B\u1eafc Ninh\n",
"B\u1eafc Giang\n",
"Th\u00e1i Nguy\u00ean\n",
"H\u1ea3i D\u01b0\u01a1ng\n",
"H\u00e0 Nam\n",
"B\u00ecnh D\u01b0\u01a1ng\n",
"L\u00e2m \u0110\u1ed3ng\n",
"C\u1ea7n Th\u01a1\n"
]
}
],
"source": [
"vietnam = df[df.admin == 'Vietnam']\n",
"vietnam_copy = vietnam.copy()\n",
"replace_column('name', vietnam_copy, 'Ðong Tháp', 'Đồng Tháp')\n",
"replace_column('name', vietnam_copy, 'Son La', 'Sơn La')\n",
"replace_column('name', vietnam_copy, 'Ha Tinh', 'Hà Tĩnh')\n",
"replace_column('name', vietnam_copy, 'Quàng Nam', 'Qung Nam')\n",
"replace_column('name', vietnam_copy, 'Lai Chau', 'Lai Châu')\n",
"replace_column('name', vietnam_copy, 'Hồ Chí Minh city', 'Thành phố Hồ Chí Minh')\n",
"replace_column('name', vietnam_copy, 'Hau Giang', 'Hu Giang')\n",
"replace_column('name', vietnam_copy, 'Ha Noi', 'Hà Nội')\n",
"replace_column('name', vietnam_copy, 'Can Tho', 'Cần Thơ')\n",
"replace_column('name', vietnam_copy, 'Đông Nam Bộ', 'Đồng Nai')\n",
"replace_column('name', vietnam_copy, 'Đông Bắc', 'Bắc Kạn')\n",
"replace_column('name', vietnam_copy, 'Đồng Bằng Sông Hồng', 'Hưng Yên')\n",
"replace_column('name', vietnam_copy, '\u00d0ong Th\u00e1p', '\u0110\u1ed3ng Th\u00e1p')\n",
"replace_column('name', vietnam_copy, 'Son La', 'S\u01a1n La')\n",
"replace_column('name', vietnam_copy, 'Ha Tinh', 'H\u00e0 T\u0129nh')\n",
"replace_column('name', vietnam_copy, 'Qu\u00e0ng Nam', 'Qu\u1ea3ng Nam')\n",
"replace_column('name', vietnam_copy, 'Lai Chau', 'Lai Ch\u00e2u')\n",
"replace_column('name', vietnam_copy, 'H\u1ed3 Ch\u00ed Minh city', 'Th\u00e0nh ph\u1ed1 H\u1ed3 Ch\u00ed Minh')\n",
"replace_column('name', vietnam_copy, 'Hau Giang', 'H\u1eadu Giang')\n",
"replace_column('name', vietnam_copy, 'Ha Noi', 'H\u00e0 N\u1ed9i')\n",
"replace_column('name', vietnam_copy, 'Can Tho', 'C\u1ea7n Th\u01a1')\n",
"replace_column('name', vietnam_copy, '\u0110\u00f4ng Nam B\u1ed9', '\u0110\u1ed3ng Nai')\n",
"replace_column('name', vietnam_copy, '\u0110\u00f4ng B\u1eafc', 'B\u1eafc K\u1ea1n')\n",
"replace_column('name', vietnam_copy, '\u0110\u1ed3ng B\u1eb1ng S\u00f4ng H\u1ed3ng', 'H\u01b0ng Y\u00ean')\n",
"for i in vietnam_copy['name']:\n",
" print(i)"
]
@@ -4454,6 +4499,7 @@
" \"turkey\": turkey_copy,\n",
" \"turkey_regions\": turkey_regions,\n",
" \"italy_regions\": italy_regions,\n",
" \"italy_regions_and_autonomous_provinces\": italy_regions_and_autonomous_provinces,\n",
" \"philippines_regions\": philippines_regions,\n",
" \"latvia\": latvia_copy,\n",
" \"netherlands\": netherlands_copy,\n",
@@ -4492,7 +4538,7 @@
"aruba has only one subdivision - removing from countries array\n",
"british indian ocean territory has only one subdivision - removing from countries array\n",
"cayman islands has only one subdivision - removing from countries array\n",
"curaçao has only one subdivision - removing from countries array\n",
"cura\u00e7ao has only one subdivision - removing from countries array\n",
"falkland islands has only one subdivision - removing from countries array\n",
"faroe islands has only one subdivision - removing from countries array\n",
"gibraltar has only one subdivision - removing from countries array\n",
@@ -4528,7 +4574,7 @@
"name": "stdout",
"output_type": "stream",
"text": [
"curaçao has only one subdivision - removing from countries array\n",
"cura\u00e7ao has only one subdivision - removing from countries array\n",
"falkland islands has only one subdivision - removing from countries array\n",
"faroe islands has only one subdivision - removing from countries array\n"
]
@@ -103,6 +103,7 @@ import iran from './countries/iran.geojson';
import israel from './countries/israel.geojson';
import italy from './countries/italy.geojson';
import italy_regions from './countries/italy_regions.geojson';
import italy_regions_and_autonomous_provinces from './countries/italy_regions_and_autonomous_provinces.geojson';
import ivory_coast from './countries/ivory_coast.geojson';
import japan from './countries/japan.geojson';
import jordan from './countries/jordan.geojson';
@@ -306,6 +307,7 @@ export const countries = {
israel,
italy,
italy_regions,
italy_regions_and_autonomous_provinces,
ivory_coast,
japan,
jordan,
@@ -430,6 +432,9 @@ export const countryOptions = Object.keys(countries).map(x => {
if (x === 'italy_regions') {
return [x, 'Italy (regions)'];
}
if (x === 'italy_regions_and_autonomous_provinces') {
return [x, 'Italy (regions and autonomous provinces)'];
}
if (x === 'france_regions') {
return [x, 'France (regions)'];
}
@@ -33,6 +33,6 @@
{ "type": "Feature", "properties": { "ISO": "IR-25", "NAME_1": "Yazd" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 53.650710076708663, 32.61286754000497 ], [ 54.70904341032508, 32.920083929729572 ], [ 54.814670038291581, 32.970700994954939 ], [ 54.908204380227914, 33.088161526533213 ], [ 54.986339146034709, 33.329955553206219 ], [ 55.040082635105477, 33.385688585459832 ], [ 55.139404737638529, 33.430362861356969 ], [ 55.217229445082808, 33.437261664194409 ], [ 55.274590285313366, 33.470644639738339 ], [ 55.359856397954388, 33.594642238948154 ], [ 55.386418085026548, 33.665464788968791 ], [ 55.375255974983247, 34.344363918859756 ], [ 55.484706658785228, 34.365163682857656 ], [ 55.741125116131172, 34.360331935770205 ], [ 55.850885858295669, 34.407254137268581 ], [ 56.210450474010145, 34.906060898691521 ], [ 56.619211053447316, 34.993678289721288 ], [ 57.016602818165495, 35.148294175835531 ], [ 57.268370395577278, 35.201314195393763 ], [ 57.37957807766611, 35.177336330707078 ], [ 57.671239861630283, 34.993936672139682 ], [ 57.705553013161023, 34.930994777786736 ], [ 57.692013788105783, 34.86686432502853 ], [ 57.595172154271495, 34.788807075386217 ], [ 57.198917271115135, 34.557735908285508 ], [ 57.13080773289056, 34.456295071360444 ], [ 57.004820591397163, 34.142567449728006 ], [ 56.9966557148893, 33.973482164370353 ], [ 57.05887413883039, 33.685256863513416 ], [ 57.105796340328823, 33.623581041032196 ], [ 57.304440545394812, 33.603737291442826 ], [ 57.559928826753946, 33.653243313528094 ], [ 57.602716913155007, 33.607716376009932 ], [ 57.642301060445561, 33.54405101214445 ], [ 57.71702518066752, 33.121699530808655 ], [ 57.782240838144162, 32.996797594033694 ], [ 58.04062300025123, 32.871068834059429 ], [ 58.147696568142067, 32.748595689139734 ], [ 58.152554151852598, 32.671882025734874 ], [ 58.10036095639299, 32.568141588163769 ], [ 58.224384393125206, 32.352547512457704 ], [ 58.222213982789071, 32.297563788138291 ], [ 58.173948195053129, 32.146048489246368 ], [ 58.040726353038735, 31.994533189455126 ], [ 58.003932732809346, 31.907200019265474 ], [ 57.901199985590836, 31.771549384496495 ], [ 57.834744093764868, 31.637268175286067 ], [ 56.761631300743886, 32.03008657533519 ], [ 56.634093865639386, 32.049181016990303 ], [ 56.566397738564774, 31.978926906851257 ], [ 56.358451776328366, 31.879036363537352 ], [ 56.287758416617578, 31.815086777932436 ], [ 55.756834750623227, 31.576264146373262 ], [ 55.712289666834636, 31.495183823874356 ], [ 55.684487746312413, 31.110917873960716 ], [ 55.514989048006157, 31.046787421202509 ], [ 55.32637006962301, 31.024773261276948 ], [ 55.116460401726158, 31.043247586207144 ], [ 54.554427525110157, 30.957774767091792 ], [ 54.466474237296211, 30.873361314273211 ], [ 54.420275506209634, 30.797913722740077 ], [ 54.398054640709063, 30.725075792014195 ], [ 54.400638462195275, 30.675466417141422 ], [ 54.515566848231458, 30.450441393055826 ], [ 54.539441359231319, 30.350550848842602 ], [ 54.603830194407919, 30.297349962131022 ], [ 54.591221144440226, 29.973106187400617 ], [ 54.616749301838809, 29.847971707528245 ], [ 54.430197381004291, 29.793220527205563 ], [ 54.227212355265976, 29.882879137362295 ], [ 54.071769646851692, 29.984268297443975 ], [ 54.043140904029485, 30.044161282317305 ], [ 54.007587518149421, 30.263191840231229 ], [ 53.964075962235825, 30.329983628841376 ], [ 53.80460249241105, 30.499068915098348 ], [ 53.638411086002804, 30.755151476559377 ], [ 53.404936964569231, 31.261502996865772 ], [ 53.276469354377184, 31.395086574985442 ], [ 53.125160760160952, 31.51394236874529 ], [ 52.870706007576189, 31.597451483998782 ], [ 52.827091098875087, 31.744419257542688 ], [ 52.824093866238911, 31.813846544482431 ], [ 52.904915806319366, 32.164600328542292 ], [ 52.883728468693846, 32.505199692911503 ], [ 53.060358513834331, 32.576694037399875 ], [ 53.164641554664001, 32.640333563742956 ], [ 53.261069777348325, 32.672011216944099 ], [ 53.334036900182753, 32.67412995043685 ], [ 53.650710076708663, 32.61286754000497 ] ] ] } },
{ "type": "Feature", "properties": { "ISO": "IR-26", "NAME_1": "Qom" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 51.788911574109818, 34.54197459605075 ], [ 51.451671176683419, 34.469730942728802 ], [ 51.158355747219957, 34.452522691469028 ], [ 51.063064406097453, 34.418235379259329 ], [ 51.013765089587196, 34.357903143914939 ], [ 50.983689405941277, 34.161610216338374 ], [ 51.004670037991843, 34.11838288036563 ], [ 51.032988723350854, 34.105799668819657 ], [ 50.802201776190998, 34.157889513290399 ], [ 50.699262323397477, 34.20488922825524 ], [ 50.441190219652867, 34.224784653788731 ], [ 50.32874230321471, 34.317879747052643 ], [ 50.30445438016551, 34.367463284403016 ], [ 50.30869184805033, 34.408520209140306 ], [ 50.264146763362419, 34.466423651730111 ], [ 50.158313429820907, 34.492106838759582 ], [ 50.060024855162908, 34.577062893038089 ], [ 50.069429966020095, 34.628739325459492 ], [ 50.162757603280738, 34.671992498954637 ], [ 50.173402947587931, 34.692042955018337 ], [ 50.152629022011752, 34.716124172492528 ], [ 50.153972609148582, 34.781494858700739 ], [ 50.21112674290481, 34.819373684547884 ], [ 50.301147088267498, 34.809374295387386 ], [ 50.388170200094635, 34.829915676067571 ], [ 50.447598097873879, 34.862058417262119 ], [ 50.57151818181859, 34.878129787859393 ], [ 50.693681268375826, 34.915956935963777 ], [ 50.723033481609889, 35.107883206244935 ], [ 50.784735141613453, 35.218419093866089 ], [ 51.072159457692806, 35.213251450893722 ], [ 51.31235151570985, 35.153952745222966 ], [ 51.882342564157966, 34.875494290429117 ], [ 51.893194614040169, 34.754002997440352 ], [ 51.866116163929803, 34.66646312077637 ], [ 51.788911574109818, 34.54197459605075 ] ] ] } },
{ "type": "Feature", "properties": { "ISO": "IR-07", "NAME_1": "Tehran" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 50.784735141613453, 35.218419093866089 ], [ 50.8712414886038, 35.4471906603207 ], [ 50.870828078353099, 35.517031358410463 ], [ 50.798687778717976, 35.604777939750079 ], [ 50.825095940725078, 35.654454591106457 ], [ 50.889640414373616, 35.686726828380415 ], [ 50.942082799381637, 35.739169213388493 ], [ 50.970321006209417, 35.799679657490174 ], [ 51.059069657138934, 35.803713687037032 ], [ 51.075205776225573, 35.916666515247471 ], [ 51.135716220327254, 35.97314292980235 ], [ 51.216396812163111, 35.997347107083328 ], [ 51.297077403998969, 35.989279047989669 ], [ 51.353553818553848, 36.017517255716768 ], [ 51.356896600397761, 36.116271267305137 ], [ 51.465003697062969, 36.064698188570503 ], [ 51.618275995141062, 36.054052843363991 ], [ 51.754391717004125, 36.010980536122815 ], [ 51.856504348396925, 35.921554469962814 ], [ 51.950245395908269, 35.799804796354238 ], [ 52.029206984015048, 35.77091767021426 ], [ 52.108168573021203, 35.767171128744565 ], [ 52.177001580758315, 35.789960435925366 ], [ 52.306812778986512, 35.917394518242418 ], [ 52.398486768949226, 35.976202298397311 ], [ 52.625863072322886, 35.931347154447622 ], [ 52.740274692622961, 35.881014309162993 ], [ 52.816342400881069, 35.86825023046373 ], [ 52.901401807947025, 35.889695950507701 ], [ 52.944810011972436, 35.881556912421559 ], [ 53.0347270036483, 35.831094875927761 ], [ 53.066249628117816, 35.718905341008679 ], [ 53.079375441123659, 35.618136298551292 ], [ 53.047232699929111, 35.528374334707735 ], [ 52.888275994941182, 35.410190335415621 ], [ 52.674955682358814, 35.336189683806822 ], [ 52.594133742278359, 35.338721829348913 ], [ 52.21968631437187, 35.414221095926791 ], [ 51.982078077840981, 35.54431651499516 ], [ 51.922443475386046, 35.54684866053725 ], [ 51.870146926239613, 35.569818833971965 ], [ 51.853403762073924, 35.555452785717478 ], [ 51.821054315304366, 35.403420721988709 ], [ 51.822501255228701, 35.315260727700377 ], [ 51.980631137916646, 35.125246487135655 ], [ 51.968022087948953, 35.063725694285324 ], [ 51.916862420364396, 34.998587551174523 ], [ 51.882342564157966, 34.875494290429117 ], [ 51.31235151570985, 35.153952745222966 ], [ 51.072159457692806, 35.213251450893722 ], [ 50.784735141613453, 35.218419093866089 ] ] ] } },
{ "type": "Feature", "properties": { "ISO": "IR-30", "NAME_1": "Alborz" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 50.696471795436992, 35.550543525163562 ], [ 50.689753858853464, 35.639246120911707 ], [ 50.650996534762271, 35.676478989813518 ], [ 50.594772576992796, 35.677047431493747 ], [ 50.581956822349468, 35.650072333271567 ], [ 50.620714146440662, 35.60635407178296 ], [ 50.610378858697345, 35.574831448212763 ], [ 50.527696568441854, 35.62911754054204 ], [ 50.285540805663629, 35.666014513558935 ], [ 50.238618605064573, 35.741022853721688 ], [ 50.2302470229817, 35.771925361466231 ], [ 50.242752720161889, 35.81083771428905 ], [ 50.297736443581982, 35.853961697474347 ], [ 50.500204706282148, 35.934266872717956 ], [ 50.534517856913567, 35.960802721368452 ], [ 50.625881789413029, 36.164769599037754 ], [ 50.47188602092308, 36.239648748890602 ], [ 50.420829706126028, 36.296389472396186 ], [ 50.449665155422565, 36.331684474958536 ], [ 50.563146599735774, 36.339177557897983 ], [ 50.966222772263109, 36.292203681354806 ], [ 51.029888137027854, 36.27179149008515 ], [ 51.086008742009824, 36.222259630477538 ], [ 51.127970005211523, 36.20745433175199 ], [ 51.291991001283634, 36.178102118517927 ], [ 51.356896600397761, 36.116271267305137 ], [ 51.353553818553848, 36.017517255716768 ], [ 51.297077403998969, 35.989279047989669 ], [ 51.216396812163111, 35.997347107083328 ], [ 51.135716220327254, 35.97314292980235 ], [ 51.075205776225573, 35.916666515247471 ], [ 51.059069657138934, 35.803713687037032 ], [ 50.970321006209417, 35.799679657490174 ], [ 50.942082799381637, 35.739169213388493 ], [ 50.889640414373616, 35.686726828380415 ], [ 50.825095940725078, 35.654454591106457 ], [ 50.798687778717976, 35.604777939750079 ], [ 50.75269575410573, 35.601341458441539 ], [ 50.724893832684188, 35.555168564877363 ], [ 50.696471795436992, 35.550543525163562 ] ] ] } }
{ "type": "Feature", "properties": { "ISO": "IR-32", "NAME_1": "Alborz" }, "geometry": { "type": "Polygon", "coordinates": [ [ [ 50.696471795436992, 35.550543525163562 ], [ 50.689753858853464, 35.639246120911707 ], [ 50.650996534762271, 35.676478989813518 ], [ 50.594772576992796, 35.677047431493747 ], [ 50.581956822349468, 35.650072333271567 ], [ 50.620714146440662, 35.60635407178296 ], [ 50.610378858697345, 35.574831448212763 ], [ 50.527696568441854, 35.62911754054204 ], [ 50.285540805663629, 35.666014513558935 ], [ 50.238618605064573, 35.741022853721688 ], [ 50.2302470229817, 35.771925361466231 ], [ 50.242752720161889, 35.81083771428905 ], [ 50.297736443581982, 35.853961697474347 ], [ 50.500204706282148, 35.934266872717956 ], [ 50.534517856913567, 35.960802721368452 ], [ 50.625881789413029, 36.164769599037754 ], [ 50.47188602092308, 36.239648748890602 ], [ 50.420829706126028, 36.296389472396186 ], [ 50.449665155422565, 36.331684474958536 ], [ 50.563146599735774, 36.339177557897983 ], [ 50.966222772263109, 36.292203681354806 ], [ 51.029888137027854, 36.27179149008515 ], [ 51.086008742009824, 36.222259630477538 ], [ 51.127970005211523, 36.20745433175199 ], [ 51.291991001283634, 36.178102118517927 ], [ 51.356896600397761, 36.116271267305137 ], [ 51.353553818553848, 36.017517255716768 ], [ 51.297077403998969, 35.989279047989669 ], [ 51.216396812163111, 35.997347107083328 ], [ 51.135716220327254, 35.97314292980235 ], [ 51.075205776225573, 35.916666515247471 ], [ 51.059069657138934, 35.803713687037032 ], [ 50.970321006209417, 35.799679657490174 ], [ 50.942082799381637, 35.739169213388493 ], [ 50.889640414373616, 35.686726828380415 ], [ 50.825095940725078, 35.654454591106457 ], [ 50.798687778717976, 35.604777939750079 ], [ 50.75269575410573, 35.601341458441539 ], [ 50.724893832684188, 35.555168564877363 ], [ 50.696471795436992, 35.550543525163562 ] ] ] } }
]
}
@@ -0,0 +1,58 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import fs from 'fs';
import path from 'path';
import { countryOptions } from '../src/countries';
type ItalyFeature = { properties: { ISO: string; NAME_1: string } };
test('countryOptions includes labeled entries for the Italy region variants', () => {
expect(countryOptions).toContainEqual(['italy_regions', 'Italy (regions)']);
expect(countryOptions).toContainEqual([
'italy_regions_and_autonomous_provinces',
'Italy (regions and autonomous provinces)',
]);
});
test('italy_regions_and_autonomous_provinces geojson has the expected shape', () => {
// jest maps `.geojson` imports to an empty object mock, so the file is
// read from disk directly to verify its actual shape.
const geojsonPath = path.join(
__dirname,
'../src/countries/italy_regions_and_autonomous_provinces.geojson',
);
const geojson = JSON.parse(fs.readFileSync(geojsonPath, 'utf-8'));
const features: ItalyFeature[] = geojson.features;
expect(features).toHaveLength(21);
features.forEach(feature => {
expect(feature.properties).toEqual(
expect.objectContaining({
ISO: expect.any(String),
NAME_1: expect.any(String),
}),
);
});
const isoCodes = features.map(feature => feature.properties.ISO);
expect(new Set(isoCodes).size).toBe(isoCodes.length);
expect(isoCodes).toContain('IT-BZ');
expect(isoCodes).toContain('IT-TN');
expect(isoCodes).not.toContain('IT-32');
});
@@ -0,0 +1,69 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import fs from 'fs';
import path from 'path';
type Feature = {
properties: {
ISO: string;
NAME_1: string;
};
};
// `.geojson` imports are mocked out to an empty object by the Jest module
// mapper (see jest.config.js), so the file is read from disk directly to
// exercise the real, committed data.
function loadIranGeoJson(): { features: Feature[] } {
const filePath = path.join(__dirname, '../src/countries/iran.geojson');
return JSON.parse(fs.readFileSync(filePath, 'utf-8'));
}
test('every Iranian province has its own distinct ISO 3166-2 code', () => {
const { features } = loadIranGeoJson();
// Pin the feature count too, so dropping a province other than
// Tehran/Alborz (which would still leave every remaining ISO code
// distinct) doesn't slip past the checks below.
expect(features.length).toBe(31);
// Sanity check: every province name in this file is unique, so a
// duplicate ISO code below can only mean two different provinces were
// mistakenly assigned the same code (as opposed to one province being
// split across multiple polygon features).
const names = features.map(feature => feature.properties.NAME_1);
expect(new Set(names).size).toBe(names.length);
const isoByName = new Map(
features.map(feature => [
feature.properties.NAME_1,
feature.properties.ISO,
]),
);
const isoCodes = features.map(feature => feature.properties.ISO);
expect(new Set(isoCodes).size).toBe(isoCodes.length);
// Tehran and Alborz were split into separate provinces in 2010, but the
// GeoJSON still assigned both the same ISO code (IR-07), which used to
// make it impossible to distinguish them on the Country Map chart. Alborz
// now uses its pre-2020 ISO 3166-2 code, IR-32.
expect(isoByName.get('Tehran')).toBe('IR-07');
expect(isoByName.get('Alborz')).toBe('IR-32');
});
@@ -34,7 +34,7 @@ import {
getTotalsMetrics,
isTimeComparison,
timeCompareOperator,
TotalsAggregate,
toTotalsAggregate,
} from '@superset-ui/chart-controls';
import { isEmpty } from 'lodash-es';
import { TableChartFormData } from './types';
@@ -349,8 +349,7 @@ export const buildQuery: BuildQuery<TableChartFormData> = (
formData.show_totals &&
queryMode === QueryMode.Aggregate
) {
const totalsAggregate: TotalsAggregate =
formData.totals_aggregate === 'AVG' ? 'AVG' : 'SUM';
const totalsAggregate = toTotalsAggregate(formData.totals_aggregate);
extraQueries.push({
...queryObject,
columns: [],
@@ -475,14 +475,18 @@ const config: ControlPanelConfig = {
type: 'SelectControl',
label: t('Summary aggregation'),
description: t(
'Aggregation used for the summary row, independent of each ' +
"metric's own aggregation. Only applies to simple metrics " +
'(a metric built from custom SQL keeps its own aggregation ' +
'in the summary row).',
'Aggregation used for the summary row. By default each metric ' +
'keeps its own aggregation; Sum and Average override it for ' +
'the summary row only. The override applies to simple ' +
'metrics (a metric built from custom SQL always keeps its ' +
'own aggregation). Overriding a count or a distinct count ' +
'sums the counted column instead, which fails outright on a ' +
'non-numeric column.',
),
default: 'SUM',
default: 'ORIGINAL',
clearable: false,
choices: [
['ORIGINAL', t("Each metric's own")],
['SUM', t('Sum')],
['AVG', t('Average')],
],
@@ -340,7 +340,7 @@ describe('plugin-chart-table', () => {
label: 'sum_sales',
};
test('defaults the totals query metric aggregate to SUM', () => {
test("defaults to each metric's own aggregate", () => {
const { queries } = buildQueryCached({
...basicFormData,
query_mode: QueryMode.Aggregate,
@@ -350,9 +350,28 @@ describe('plugin-chart-table', () => {
});
expect(queries).toHaveLength(2);
expect(queries[1].metrics).toEqual([
{ ...simpleMetric, aggregate: 'SUM' },
]);
expect(queries[1].metrics).toEqual([simpleMetric]);
});
test('keeps COUNT_DISTINCT in the summary row by default', () => {
// Overriding this to SUM sums the counted column instead of counting
// it, which is meaningless on a numeric id and is rejected outright by
// the database on a non-numeric one (e.g. a uuid).
const countDistinctMetric = {
expressionType: 'SIMPLE' as const,
column: { column_name: 'contract_id' },
aggregate: 'COUNT_DISTINCT' as const,
label: 'contracts',
};
const { queries } = buildQueryCached({
...basicFormData,
query_mode: QueryMode.Aggregate,
metrics: [countDistinctMetric],
groupby: ['category'],
show_totals: true,
});
expect(queries[1].metrics).toEqual([countDistinctMetric]);
});
test('overrides simple metric aggregate with totals_aggregate for the summary query only', () => {
@@ -18,9 +18,15 @@
*/
import { createMemoryHistory, type Update } from 'history';
import { Router } from 'react-router-dom';
import { isFeatureEnabled } from '@superset-ui/core';
import { render, screen, fireEvent } from 'spec/helpers/testing-library';
import { isFeatureEnabled, FeatureFlag } from '@superset-ui/core';
import {
render,
screen,
fireEvent,
within,
} from 'spec/helpers/testing-library';
import type Chart from 'src/types/Chart';
import type { UserWithPermissionsAndRoles } from 'src/types/bootstrapTypes';
import ChartCard from './ChartCard';
jest.mock('@superset-ui/core', () => ({
@@ -37,7 +43,18 @@ const mockChart = {
thumbnail_url: '/thumbnail.png',
} as Chart;
const renderCard = (history: ReturnType<typeof createMemoryHistory>) =>
// Admin qualifies as editor, so the card's delete entry is enabled.
const adminUser = {
userId: 1,
username: 'admin',
roles: { Admin: [] },
permissions: {},
} as unknown as UserWithPermissionsAndRoles;
const renderCard = (
history: ReturnType<typeof createMemoryHistory>,
props: Partial<React.ComponentProps<typeof ChartCard>> = {},
) =>
render(
<Router history={history}>
<ChartCard
@@ -52,6 +69,7 @@ const renderCard = (history: ReturnType<typeof createMemoryHistory>) =>
favoriteStatus={false}
showThumbnails
handleBulkChartExport={jest.fn()}
{...props}
/>
</Router>,
);
@@ -106,3 +124,44 @@ test('clicking the card outside the thumbnail navigates to the chart', () => {
expect(navigations).toEqual(['PUSH /explore/?slice_id=1']);
});
test('with soft delete on, the card delete flow shows the archive dialog', async () => {
(isFeatureEnabled as jest.Mock).mockImplementation(
flag => flag === FeatureFlag.SoftDelete,
);
renderCard(createMemoryHistory(), { user: adminUser });
fireEvent.click(screen.getByTestId('chart-card-menu'));
fireEvent.click(await screen.findByText('Archive'));
const dialog = await screen.findByRole('dialog');
expect(within(dialog).getByText('Archive Sample Chart?')).toBeInTheDocument();
// The body comes from the shared soft-delete copy module; its exact
// wording evolves there (location hint, retention clause), so pin the
// stable prefix rather than a full sentence.
expect(
within(dialog).getByText(/This chart will be moved to Recently Archived/),
).toBeInTheDocument();
expect(
within(dialog).getByRole('button', { name: 'Archive' }),
).toBeInTheDocument();
// Recoverable deletes drop the type-DELETE friction.
expect(
within(dialog).queryByTestId('delete-modal-input'),
).not.toBeInTheDocument();
});
test('with soft delete off, the card delete dialog is the permanent-delete one', async () => {
(isFeatureEnabled as jest.Mock).mockReturnValue(false);
renderCard(createMemoryHistory(), { user: adminUser });
fireEvent.click(screen.getByTestId('chart-card-menu'));
fireEvent.click(await screen.findByText('Delete'));
const dialog = await screen.findByRole('dialog');
expect(within(dialog).getByText('Please confirm')).toBeInTheDocument();
expect(
within(dialog).getByText(/Are you sure you want to delete/),
).toBeInTheDocument();
expect(within(dialog).getByTestId('delete-modal-input')).toBeInTheDocument();
});
@@ -38,6 +38,10 @@ import {
isNavigationHandledByLink,
} from 'src/views/CRUD/utils';
import { assetUrl } from 'src/utils/assetUrl';
import {
archiveConfirmDescription,
deleteActionLabel,
} from 'src/utils/softDeleteCopy';
import type { ListViewFetchDataConfig as FetchDataConfig } from 'src/components';
import { TableTab } from 'src/views/CRUD/types';
import { isUserEditorOrAdmin } from 'src/dashboard/util/permissionUtils';
@@ -159,15 +163,29 @@ export default function ChartCard({
}
if (canDelete) {
// With soft delete on, deleting archives the chart (recoverable), so the
// confirmation drops the type-DELETE friction and uses the shared archive
// copy -- matching the list view's dialog for the same action.
const softDelete = isFeatureEnabled(FeatureFlag.SoftDelete);
menuItems.push({
key: 'delete',
label: (
<ConfirmStatusChange
title={t('Please confirm')}
recoverable={softDelete}
title={
softDelete
? t('Archive %(name)s?', { name: chart.slice_name })
: t('Please confirm')
}
description={
<>
{t('Are you sure you want to delete')} <b>{chart.slice_name}</b>?
</>
softDelete ? (
<p>{archiveConfirmDescription(t('chart'))}</p>
) : (
<>
{t('Are you sure you want to delete')} <b>{chart.slice_name}</b>
?
</>
)
}
onConfirm={() =>
handleChartDelete(
@@ -204,7 +222,7 @@ export default function ChartCard({
vertical-align: text-top;
`}
/>{' '}
{t('Delete')}
{deleteActionLabel()}
</button>
</Tooltip>
)}
@@ -522,7 +522,7 @@ const ExtraOptions = ({
onChange={onInputChange}
>
{t(
'Impersonate logged in user (Presto, Trino, Drill, Hive, Databricks, and Google Sheets)',
'Impersonate logged in user (Presto, Trino, Drill, Hive, Databricks, Snowflake and Google Sheets)',
)}
</Checkbox>
<InfoTooltip
@@ -532,7 +532,10 @@ const ExtraOptions = ({
'and hive.server2.enable.doAs is enabled, will run the queries as ' +
'service account, but impersonate the currently logged on user via ' +
'hive.server2.proxy.user property. If Databricks, uses OAuth2 to ' +
'authenticate as the currently logged on user.',
'authenticate as the currently logged on user. If Snowflake or Google ' +
'Sheets, and OAuth authentication is configured for the database, will ' +
'run the queries as the currently logged on user via their own OAuth ' +
'credentials.',
)}
/>
</div>
@@ -669,11 +669,11 @@ describe('UploadDataModal Collapse Tabs', () => {
useRedux: true,
});
const generalInfoTab = screen.getByRole('tab', {
name: /expanded General information/i,
name: /General information/i,
});
expect(generalInfoTab).toHaveAttribute('aria-expanded', 'true');
const fileSettingsTab = screen.getByRole('tab', {
name: /collapsed File settings/i,
name: /File settings/i,
});
await userEvent.click(fileSettingsTab);
await waitFor(() => {
@@ -689,11 +689,11 @@ describe('UploadDataModal Collapse Tabs', () => {
useRedux: true,
});
const generalInfoTab = screen.getByRole('tab', {
name: /expanded General information/i,
name: /General information/i,
});
expect(generalInfoTab).toHaveAttribute('aria-expanded', 'true');
const fileSettingsTab = screen.getByRole('tab', {
name: /collapsed File settings/i,
name: /File settings/i,
});
await userEvent.click(fileSettingsTab);
await waitFor(() => {
@@ -709,11 +709,11 @@ describe('UploadDataModal Collapse Tabs', () => {
useRedux: true,
});
const generalInfoTab = screen.getByRole('tab', {
name: /expanded General information/i,
name: /General information/i,
});
expect(generalInfoTab).toHaveAttribute('aria-expanded', 'true');
const fileSettingsTab = screen.getByRole('tab', {
name: /collapsed File settings/i,
name: /File settings/i,
});
await userEvent.click(fileSettingsTab);
await waitFor(() => {
@@ -25,8 +25,10 @@ import {
fireEvent,
userEvent,
waitFor,
within,
selectOption,
} from 'spec/helpers/testing-library';
import { FeatureFlag, isFeatureEnabled } from '@superset-ui/core';
import { MemoryRouter } from 'react-router-dom';
import { QueryParamProvider } from 'use-query-params';
import { ReactRouter5Adapter } from 'use-query-params/adapters/react-router-5';
@@ -88,6 +90,15 @@ const mockCharts = [
// list so `_info` requests resolve to it rather than the broader list glob.
// withToasts injects the toast callbacks as props; the harness renders no
// toast container, so the spy is the only way to pin what the user is told.
// The type label for the dataset concept is flag-aware (SEMANTIC_LAYERS →
// "Datasource"); mock the flag reader so tests can exercise both states. The
// default (false for every flag) matches the real test environment, where no
// bootstrap flags are set.
jest.mock('@superset-ui/core', () => ({
...jest.requireActual('@superset-ui/core'),
isFeatureEnabled: jest.fn(() => false),
}));
const mockAddDangerToast = jest.fn();
jest.mock('src/components/MessageToasts/withToasts', () => ({
__esModule: true,
@@ -144,6 +155,13 @@ beforeEach(() => {
mockAddDangerToast.mockClear();
});
afterEach(() => {
// The flag mock is shared module state; restore the environment default so a
// flag-flipping test that dies mid-body (e.g. by Jest timeout) cannot leak
// SEMANTIC_LAYERS into whichever test runs next.
(isFeatureEnabled as jest.Mock).mockImplementation(() => false);
});
test('renders archived rows with Name and Type columns', async () => {
mockRoutes();
renderArchivedList();
@@ -573,3 +591,57 @@ test('a viewer who can read none of the types gets an empty state, not three 403
// No list fetch was ever issued.
expect(fetchMock.callHistory.calls(/chart\/\?q/)).toHaveLength(0);
});
test('labels the dataset type "Datasource" when semantic layers is enabled', async () => {
(isFeatureEnabled as jest.Mock).mockImplementation(
(flag: FeatureFlag) => flag === FeatureFlag.SemanticLayers,
);
mockRoutes();
renderArchivedList();
await screen.findByText('Deleted Chart One');
userEvent.click(screen.getByRole('combobox', { name: 'Type' }));
expect(
await screen.findByRole('option', { name: 'Datasource' }),
).toBeInTheDocument();
expect(
screen.queryByRole('option', { name: 'Dataset' }),
).not.toBeInTheDocument();
// Selecting the renamed option still drives the dataset resource —
// the underlying type value is flag-independent.
await selectOption('Datasource', 'Type');
await screen.findByText('deleted_table_one');
expect(
fetchMock.callHistory.calls(datasetListEndpoint).length,
).toBeGreaterThan(0);
// Pin the Type COLUMN cell, not just the Select's own rendered value.
const datasetRow = screen.getByText('deleted_table_one').closest('tr');
expect(
within(datasetRow as HTMLElement).getByText('Datasource'),
).toBeInTheDocument();
});
test('labels the dataset type "Dataset" when semantic layers is disabled', async () => {
mockRoutes();
renderArchivedList();
await screen.findByText('Deleted Chart One');
userEvent.click(screen.getByRole('combobox', { name: 'Type' }));
expect(
await screen.findByRole('option', { name: 'Dataset' }),
).toBeInTheDocument();
expect(
screen.queryByRole('option', { name: 'Datasource' }),
).not.toBeInTheDocument();
await selectOption('Dataset', 'Type');
await screen.findByText('deleted_table_one');
expect(
fetchMock.callHistory.calls(datasetListEndpoint).length,
).toBeGreaterThan(0);
const datasetRow = screen.getByText('deleted_table_one').closest('tr');
expect(
within(datasetRow as HTMLElement).getByText('Dataset'),
).toBeInTheDocument();
});
@@ -37,6 +37,7 @@ import {
type ListViewFilters,
} from 'src/components';
import SubMenu from 'src/features/home/SubMenu';
import { datasetLabel } from 'src/features/semanticLayers/label';
import withToasts from 'src/components/MessageToasts/withToasts';
import { recoveredToast } from 'src/utils/softDeleteCopy';
import { findPermission } from 'src/utils/findPermission';
@@ -82,10 +83,12 @@ const EmptyStateRow = styled.div`
`}
`;
const TYPE_LABELS: Record<ArchivedType, string> = {
chart: t('Chart'),
dashboard: t('Dashboard'),
dataset: t('Dataset'),
// Getters, not strings: the dataset label follows the SEMANTIC_LAYERS flag
// ("Dataset" / "Datasource"), read at render time via the shared naming module.
const TYPE_LABELS: Record<ArchivedType, () => string> = {
chart: () => t('Chart'),
dashboard: () => t('Dashboard'),
dataset: datasetLabel,
};
interface ToastProps {
@@ -166,7 +169,7 @@ function ArchivedListBody({
refreshData,
} = useListViewResource<ArchivedItem>(
config.resource,
TYPE_LABELS[type],
TYPE_LABELS[type](),
addDangerToast,
true,
[],
@@ -247,7 +250,7 @@ function ArchivedListBody({
name => {
const { text, options } = recoveredToast(
name,
TYPE_LABELS[type],
TYPE_LABELS[type](),
item.url ?? item.explore_url,
);
addSuccessToast(text, options);
@@ -306,7 +309,7 @@ function ArchivedListBody({
id: config.nameField,
},
{
Cell: () => TYPE_LABELS[type],
Cell: () => TYPE_LABELS[type](),
Header: t('Type'),
id: 'type',
disableSortBy: true,
@@ -539,7 +542,7 @@ function ArchivedList({ addDangerToast, addSuccessToast }: ToastProps) {
onChange={handleTypeChange}
options={availableTypes.map(option => ({
value: option,
label: TYPE_LABELS[option],
label: TYPE_LABELS[option](),
}))}
/>
</TypeSelectRow>
@@ -223,7 +223,7 @@ describe('ChartPage', () => {
window.history.pushState(
{},
'',
`/?${URL_PARAMS.dashboardPageId.name}=${dashboardPageId}`,
`/explore/?${URL_PARAMS.dashboardPageId.name}=${dashboardPageId}`,
);
const { getByTestId } = render(<ChartPage />, {
useRouter: true,
@@ -261,13 +261,13 @@ describe('ChartPage', () => {
window.history.pushState(
{},
'',
`/?${URL_PARAMS.dashboardPageId.name}=${dashboardPageId}`,
`/explore/?${URL_PARAMS.dashboardPageId.name}=${dashboardPageId}`,
);
const { getByTestId } = render(
<>
<Link
to={{
pathname: '/',
pathname: '/explore/',
search: `?${URL_PARAMS.dashboardPageId.name}=${dashboardPageId}`,
state: { saveAction: 'overwrite' },
}}
@@ -324,7 +324,7 @@ describe('ChartPage', () => {
});
render(
<>
<Link to="/?slice_id=99">Navigate away</Link>
<Link to="/explore/?slice_id=99">Navigate away</Link>
<ChartPage />
</>,
{
@@ -382,7 +382,7 @@ describe('ChartPage', () => {
<>
<Link
to={{
pathname: '/',
pathname: '/explore/',
search: `?${URL_PARAMS.sliceId.name}=${formData.slice_id}`,
state: toChartStateHistoryState({
...formData,
@@ -392,7 +392,7 @@ describe('ChartPage', () => {
>
Change the chart
</Link>
<Link to="/?slice_id=99">Navigate away</Link>
<Link to="/explore/?slice_id=99">Navigate away</Link>
<ChartPage />
</>,
{ useRouter: true, useRedux: true, useDnd: true },
@@ -433,14 +433,14 @@ describe('ChartPage', () => {
<>
<Link
to={{
pathname: '/',
pathname: '/explore/',
search: `?${URL_PARAMS.sliceId.name}=99`,
state: toChartStateHistoryState({ ...formData, slice_id: 99 }),
}}
>
Another chart
</Link>
<Link to="/?slice_id=100">Navigate away</Link>
<Link to="/explore/?slice_id=100">Navigate away</Link>
<ChartPage />
</>,
{ useRouter: true, useRedux: true, useDnd: true },
@@ -477,14 +477,14 @@ describe('ChartPage', () => {
<>
<Link
to={{
pathname: '/',
pathname: '/explore/',
search: `?${URL_PARAMS.sliceId.name}=${formData.slice_id}`,
state: toChartStateHistoryState(formData),
}}
>
Change the chart
</Link>
<Link to="/?slice_id=99">Navigate away</Link>
<Link to="/explore/?slice_id=99">Navigate away</Link>
<ChartPage />
</>,
{ useRouter: true, useRedux: true, useDnd: true, store },
@@ -507,6 +507,32 @@ describe('ChartPage', () => {
window.history.back();
await waitFor(() => expect(loads()).toBe(1));
});
test('does not re-fetch explore data when navigating to a dashboard', async () => {
const exploreApiRoute = 'glob:*/api/v1/explore/*';
const exploreFormData = getExploreFormData({
viz_type: VizType.Table,
show_cell_bars: true,
});
fetchMock.get(exploreApiRoute, {
result: { dataset: { id: 1 }, form_data: exploreFormData },
});
render(
<>
<Link to="/dashboard/5/">Go to dashboard</Link>
<ChartPage />
</>,
{ useRouter: true, useRedux: true, useDnd: true },
);
await waitFor(() =>
expect(fetchMock.callHistory.calls(exploreApiRoute).length).toBe(1),
);
fireEvent.click(screen.getByText('Go to dashboard'));
await new Promise(resolve => setTimeout(resolve, 0));
expect(fetchMock.callHistory.calls(exploreApiRoute).length).toBe(1);
});
});
test('does not show error toast when request is aborted on unmount', async () => {
@@ -559,7 +585,7 @@ describe('ChartPage', () => {
render(
<>
<Link to="/?slice_id=99">Navigate</Link>
<Link to="/explore/?slice_id=99">Navigate</Link>
<ChartPage />
</>,
{
@@ -56,6 +56,11 @@ const isValidResult = (rv: JsonObject): boolean =>
const hasDatasetId = (rv: JsonObject): boolean =>
isDefined(rv?.result?.dataset?.id);
const EXPLORE_ROUTE_PREFIX = '/explore/';
const isExploreRoute = (pathname: string): boolean =>
pathname.startsWith(EXPLORE_ROUTE_PREFIX);
const fetchExploreData = async (
exploreUrlParams: URLSearchParams,
signal?: AbortSignal,
@@ -312,6 +317,8 @@ export default function ExplorePage() {
// Other REPLACE: ignored (URL sync from updateHistory).
// Entries holding a chart state of the loaded chart are skipped: Explore
// pushed them itself, and ExploreViewContainer restores a popped one in place.
// Navigations that leave Explore must not trigger a re-fetch while the page
// is unmounting, as the destination's URL params are not chart params.
useEffect(() => {
const unlisten = history.listen((loc: Location, action: Action) => {
const saveAction = (loc.state as Record<string, unknown>)?.saveAction as
@@ -326,6 +333,9 @@ export default function ExplorePage() {
return;
}
}
if (!isExploreRoute(loc.pathname)) {
return;
}
if (action === 'PUSH' || action === 'POP') {
setIsLoaded(false);
loadExploreData(loc, saveAction);
@@ -0,0 +1,425 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import fetchMock from 'fetch-mock';
import rison from 'rison';
import { configureStore } from '@reduxjs/toolkit';
import {
act,
fireEvent,
render,
screen,
waitFor,
within,
} from 'spec/helpers/testing-library';
import userEvent from '@testing-library/user-event';
import DatabaseList from 'src/pages/DatabaseList';
/**
* Deleting a semantic layer cascade-deletes its semantic views (SC-108418).
* These tests pin the delete confirmation's cascade warning: the dependent
* views are counted and named before the user confirms, and a failed lookup
* still opens the modal with an uncounted warning rather than blocking.
*/
const SL_UUID = '6a000000-0000-4000-8000-000000000001';
const SL_UUID_B = '6b000000-0000-4000-8000-000000000002';
const semanticLayerRow = {
source_type: 'semantic_layer',
uuid: SL_UUID,
database_name: 'Demo Semantic Layer',
backend: 'Demo',
sl_type: 'demo',
description: null,
allow_run_async: null,
allow_dml: null,
allow_file_upload: null,
expose_in_sqllab: null,
changed_on_delta_humanized: 'a day ago',
changed_by: null,
};
const semanticLayerRowB = {
...semanticLayerRow,
uuid: SL_UUID_B,
database_name: 'Second Semantic Layer',
};
const CONNECTIONS_ROUTE = 'glob:*/api/v1/semantic_layer/connections/*';
const DATASOURCE_ROUTE = 'glob:*/api/v1/datasource/?*';
const DELETE_ROUTE = `glob:*/api/v1/semantic_layer/${SL_UUID}`;
const mockUser = {
userId: 1,
firstName: 'Admin',
lastName: 'User',
roles: { Admin: [['can_write', 'Database']] },
permissions: {},
isActive: true,
email: 'admin@example.com',
createdOn: '2026-01-01T00:00:00',
};
const dependentView = (id: number, name: string) => ({
id,
table_name: name,
kind: 'semantic_view',
source_type: 'semantic_layer',
});
const setupMocks = ({
dependents,
dependentsError = false,
rows = [semanticLayerRow],
}: {
dependents: { id: number; table_name: string }[];
dependentsError?: boolean;
rows?: (typeof semanticLayerRow)[];
}) => {
fetchMock.clearHistory().removeRoutes();
fetchMock.get('glob:*/api/v1/database/_info*', {
permissions: ['can_read', 'can_write', 'can_export'],
});
fetchMock.get('glob:*/api/v1/database/?q=*', { result: [], count: 0 });
fetchMock.get('glob:*/api/v1/database/related/*', { result: [], count: 0 });
fetchMock.get(CONNECTIONS_ROUTE, {
result: rows,
count: rows.length,
});
if (dependentsError) {
fetchMock.get(DATASOURCE_ROUTE, 500, { name: DATASOURCE_ROUTE });
} else {
fetchMock.get(
DATASOURCE_ROUTE,
{ result: dependents, count: dependents.length },
{ name: DATASOURCE_ROUTE },
);
}
fetchMock.delete(DELETE_ROUTE, {});
};
const renderDatabaseList = () => {
const store = configureStore({
reducer: {
user: (state = mockUser) => state,
common: (
state = {
conf: {
CSV_EXTENSIONS: ['csv'],
EXCEL_EXTENSIONS: ['xls'],
COLUMNAR_EXTENSIONS: ['parquet'],
ALLOWED_EXTENSIONS: ['csv', 'xls', 'parquet'],
SYNC_DB_PERMISSIONS_IN_ASYNC_MODE: false,
},
},
) => state,
},
middleware: getDefaultMiddleware =>
getDefaultMiddleware({ serializableCheck: false, immutableCheck: false }),
});
return render(<DatabaseList user={mockUser} />, {
store,
useQueryParams: true,
useRouter: true,
});
};
const openDeleteModal = async () => {
const deleteButton = await screen.findByTestId('Delete');
await userEvent.click(deleteButton);
return screen.findByRole('dialog');
};
beforeEach(() => {
window.featureFlags = { SEMANTIC_LAYERS: true } as never;
});
afterEach(() => {
window.featureFlags = {} as never;
fetchMock.clearHistory();
fetchMock.removeRoutes();
});
test('delete confirmation warns about cascade-deleting dependent views by count and name', async () => {
setupMocks({
dependents: [
dependentView(1, 'marketing'),
dependentView(2, 'sales'),
dependentView(3, 'orders'),
],
});
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'This will also permanently delete its 3 semantic views. Charts built on those views will stop working.',
),
).toBeInTheDocument();
expect(
within(dialog).getByText('Affected semantic views'),
).toBeInTheDocument();
expect(within(dialog).getByText('marketing')).toBeInTheDocument();
expect(within(dialog).getByText('sales')).toBeInTheDocument();
expect(within(dialog).getByText('orders')).toBeInTheDocument();
// The dependent lookup must target this layer's views.
const lookupCalls = fetchMock.callHistory.calls(DATASOURCE_ROUTE);
expect(lookupCalls).toHaveLength(1);
const q = new URL(lookupCalls[0].url).searchParams.get('q') as string;
expect(rison.decode(q)).toMatchObject({
filters: [{ col: 'semantic_layer_uuid', opr: 'eq', value: SL_UUID }],
});
});
test('a single dependent view is announced in the singular', async () => {
setupMocks({ dependents: [dependentView(1, 'marketing')] });
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'This will also permanently delete its 1 semantic view. Charts built on that view will stop working.',
),
).toBeInTheDocument();
});
test('a genuinely empty layer says so instead of warning about nonexistent views', async () => {
// A successful count === 0 is always genuinely empty, never
// access-filtering: a layer is only reachable when its perm is granted,
// and the dependent-view count ORs on that same perm.
setupMocks({ dependents: [] });
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'This semantic layer has no dependent semantic views.',
),
).toBeInTheDocument();
expect(
within(dialog).queryByText(
/charts built on those views will stop working/i,
),
).not.toBeInTheDocument();
expect(
within(dialog).queryByText('Affected semantic views'),
).not.toBeInTheDocument();
});
test('a counted response with an empty name page keeps the count but omits the list', async () => {
setupMocks({ dependents: [] });
fetchMock.removeRoutes({ names: [DATASOURCE_ROUTE] });
fetchMock.get(
DATASOURCE_ROUTE,
{ result: [], count: 3 },
{ name: DATASOURCE_ROUTE },
);
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'This will also permanently delete its 3 semantic views. Charts built on those views will stop working.',
),
).toBeInTheDocument();
expect(
within(dialog).queryByText('Affected semantic views'),
).not.toBeInTheDocument();
});
test('a failed dependent lookup still opens the modal with an uncounted warning', async () => {
setupMocks({ dependents: [], dependentsError: true });
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'Deleting this semantic layer also permanently deletes any semantic views it contains, and charts built on those views will stop working. The affected views could not be listed.',
),
).toBeInTheDocument();
});
test('the overflow footer reports dependent views beyond the listed page', async () => {
// The lookup pages at 10 names; the count is the full total.
setupMocks({ dependents: [] });
fetchMock.removeRoutes({ names: [DATASOURCE_ROUTE] });
fetchMock.get(
DATASOURCE_ROUTE,
{
result: Array.from({ length: 10 }, (_, i) =>
dependentView(i + 1, `view_${i + 1}`),
),
count: 12,
},
{ name: DATASOURCE_ROUTE },
);
renderDatabaseList();
const dialog = await openDeleteModal();
expect(
within(dialog).getByText(
'This will also permanently delete its 12 semantic views. Charts built on those views will stop working.',
),
).toBeInTheDocument();
expect(within(dialog).getByText('view_10')).toBeInTheDocument();
expect(within(dialog).getByText('... and 2 others')).toBeInTheDocument();
});
test('the overflow footer uses the singular for one unlisted view', async () => {
setupMocks({ dependents: [] });
fetchMock.removeRoutes({ names: [DATASOURCE_ROUTE] });
fetchMock.get(
DATASOURCE_ROUTE,
{
result: Array.from({ length: 10 }, (_, i) =>
dependentView(i + 1, `view_${i + 1}`),
),
count: 11,
},
{ name: DATASOURCE_ROUTE },
);
renderDatabaseList();
const dialog = await openDeleteModal();
expect(within(dialog).getByText('... and 1 other')).toBeInTheDocument();
});
test('a pending lookup disables repeated delete requests and shows progress', async () => {
setupMocks({ dependents: [dependentView(1, 'marketing')] });
fetchMock.removeRoutes({ names: [DATASOURCE_ROUTE] });
let releaseLookup: () => void = () => {};
const lookupGate = new Promise<void>(resolve => {
releaseLookup = resolve;
});
fetchMock.get(
DATASOURCE_ROUTE,
async () => {
await lookupGate;
return { result: [dependentView(1, 'marketing')], count: 1 };
},
{ name: DATASOURCE_ROUTE },
);
renderDatabaseList();
const deleteButton = await screen.findByTestId('Delete');
await userEvent.click(deleteButton);
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
await waitFor(() => {
expect(screen.getByTestId('Delete')).toHaveAttribute(
'aria-disabled',
'true',
);
});
expect(screen.getByTestId('Delete')).toHaveAccessibleName(
'Loading dependent semantic views',
);
await userEvent.click(screen.getByTestId('Delete'));
expect(fetchMock.callHistory.calls(DATASOURCE_ROUTE)).toHaveLength(1);
releaseLookup();
expect(await screen.findByRole('dialog')).toBeInTheDocument();
});
test("a stale lookup resolving late cannot replace a newer row's modal", async () => {
// Click Delete on layer A (its lookup hangs), then on layer B (resolves
// immediately). When A's lookup finally resolves, the generation guard must
// drop it: the modal keeps showing B's preview.
setupMocks({
dependents: [],
rows: [semanticLayerRow, semanticLayerRowB],
});
fetchMock.removeRoutes({ names: [DATASOURCE_ROUTE] });
let releaseFirstLookup: () => void = () => {};
const firstLookupGate = new Promise<void>(resolve => {
releaseFirstLookup = resolve;
});
fetchMock.get(
DATASOURCE_ROUTE,
async ({ url }) => {
// The layer uuid rides in the rison-encoded `q` filter and its
// characters survive URL encoding, so a substring check is enough to
// tell the two lookups apart.
if (url.includes(SL_UUID)) {
await firstLookupGate;
return { result: [dependentView(1, 'stale_view')], count: 1 };
}
return { result: [dependentView(2, 'fresh_view')], count: 1 };
},
{ name: DATASOURCE_ROUTE },
);
renderDatabaseList();
const deleteButtons = await screen.findAllByTestId('Delete');
expect(deleteButtons).toHaveLength(2);
// fireEvent, not userEvent: userEvent's hover step re-renders the row
// (tooltip) and detaches the pressed node mid-sequence when the table has
// multiple rows, so its click never reaches the handler.
fireEvent.click(deleteButtons[0]);
await waitFor(() => {
expect(screen.getAllByTestId('Delete')[0]).toHaveAttribute(
'aria-disabled',
'true',
);
});
expect(screen.queryByRole('dialog')).not.toBeInTheDocument();
fireEvent.click(screen.getAllByTestId('Delete')[1]);
const dialog = await screen.findByRole('dialog');
expect(within(dialog).getByText('fresh_view')).toBeInTheDocument();
await act(async () => {
releaseFirstLookup();
await new Promise(resolve => {
setTimeout(resolve, 0);
});
});
expect(
within(screen.getByRole('dialog')).getByText('fresh_view'),
).toBeInTheDocument();
expect(screen.queryByText('stale_view')).not.toBeInTheDocument();
});
test('confirming the modal deletes the semantic layer', async () => {
setupMocks({ dependents: [dependentView(1, 'marketing')] });
renderDatabaseList();
const dialog = await openDeleteModal();
await userEvent.type(
within(dialog).getByTestId('delete-modal-input'),
'DELETE',
);
await userEvent.click(within(dialog).getByRole('button', { name: 'Delete' }));
await waitFor(() => {
expect(fetchMock.callHistory.calls(DELETE_ROUTE)).toHaveLength(1);
});
});
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import { t } from '@apache-superset/core/translation';
import { t, tn } from '@apache-superset/core/translation';
import {
getExtensionsRegistry,
SupersetClient,
@@ -24,7 +24,7 @@ import {
FeatureFlag,
} from '@superset-ui/core';
import { css, useTheme } from '@apache-superset/core/theme';
import { useState, useMemo, useEffect, useCallback } from 'react';
import { useState, useMemo, useEffect, useCallback, useRef } from 'react';
import type { CellProps } from 'react-table';
import rison from 'rison';
import { useSelector } from 'react-redux';
@@ -101,6 +101,91 @@ interface DatabaseDeleteObject extends DatabaseObject {
dashboards: any;
sqllab_tab_count: number;
}
/** How many dependent semantic views the delete confirmation lists by name. */
const MAX_DEPENDENT_VIEWS_LISTED = 10;
type SemanticLayerDeletePreview =
| { status: 'loading'; item: ConnectionItem }
| {
status: 'loaded';
item: ConnectionItem;
dependentViewCount: number;
dependentViewNames: string[];
}
| { status: 'failed'; item: ConnectionItem };
type ResolvedSemanticLayerDeletePreview = Exclude<
SemanticLayerDeletePreview,
{ status: 'loading' }
>;
function SemanticLayerCascadeWarning({
preview,
}: {
preview: ResolvedSemanticLayerDeletePreview;
}) {
if (preview.status === 'failed') {
return (
<p>
{t(
'Deleting this semantic layer also permanently deletes any semantic views it contains, and charts built on those views will stop working. The affected views could not be listed.',
)}
</p>
);
}
// A reachable layer always has all of its views counted (the layer's perm
// and its views' perms travel together), so zero means genuinely empty —
// never access-filtered. An honest empty message keeps the destructive
// warning credible for the layers where it matters.
if (preview.dependentViewCount === 0) {
return <p>{t('This semantic layer has no dependent semantic views.')}</p>;
}
const listedViewCount = preview.dependentViewNames.length;
const overflowViewCount = preview.dependentViewCount - listedViewCount;
return (
<>
<p>
{tn(
'This will also permanently delete its %s semantic view. Charts built on that view will stop working.',
'This will also permanently delete its %s semantic views. Charts built on those views will stop working.',
preview.dependentViewCount,
preview.dependentViewCount,
)}
</p>
{listedViewCount > 0 && (
<>
<h4>{t('Affected semantic views')}</h4>
<List
split={false}
size="small"
dataSource={preview.dependentViewNames}
renderItem={(name: string, index: number) => (
<List.Item key={`${index}-${name}`} compact>
<List.Item.Meta avatar={<span></span>} title={name} />
</List.Item>
)}
footer={
overflowViewCount > 0 && (
<div>
{tn(
'... and %s other',
'... and %s others',
overflowViewCount,
overflowViewCount,
)}
</div>
)
}
/>
</>
)}
</>
);
}
interface DatabaseListProps {
addDangerToast: (msg: string) => void;
addSuccessToast: (msg: string) => void;
@@ -257,8 +342,8 @@ function DatabaseList({
const [slCurrentlyEditing, setSlCurrentlyEditing] = useState<string | null>(
null,
);
const [slCurrentlyDeleting, setSlCurrentlyDeleting] =
useState<ConnectionItem | null>(null);
const [slDeletePreview, setSlDeletePreview] =
useState<SemanticLayerDeletePreview | null>(null);
const [allowUploads, setAllowUploads] = useState<boolean>(false);
const isAdmin = isUserAdmin(fullUser);
@@ -304,6 +389,43 @@ function DatabaseList({
[],
);
// Deleting a semantic layer cascade-deletes its semantic views, so the
// confirmation must say what else is about to be destroyed. If the lookup
// fails the modal still opens, with an uncounted warning: the count is an
// aid, not a gate on deleting. The generation counter drops stale
// resolutions -- without it a slow lookup could reopen a modal the user
// already dismissed, or replace a newer row's modal with an older one.
const slDeleteLookupRef = useRef(0);
const openSemanticLayerDeleteModal = useCallback((item: ConnectionItem) => {
slDeleteLookupRef.current += 1;
const lookupId = slDeleteLookupRef.current;
setSlDeletePreview({ status: 'loading', item });
return SupersetClient.get({
endpoint: `/api/v1/datasource/?q=${rison.encode_uri({
filters: [{ col: 'semantic_layer_uuid', opr: 'eq', value: item.uuid }],
order_column: 'table_name',
order_direction: 'asc',
page: 0,
page_size: MAX_DEPENDENT_VIEWS_LISTED,
})}`,
})
.then(({ json = {} }) => {
if (slDeleteLookupRef.current !== lookupId) return;
setSlDeletePreview({
status: 'loaded',
item,
dependentViewCount: json.count ?? 0,
dependentViewNames: (json.result ?? []).map(
(view: { table_name: string }) => view.table_name,
),
});
})
.catch(() => {
if (slDeleteLookupRef.current !== lookupId) return;
setSlDeletePreview({ status: 'failed', item });
});
}, []);
function handleDatabaseDelete(database: DatabaseObject) {
const { id, database_name: dbName } = database;
SupersetClient.delete({
@@ -566,7 +688,7 @@ function DatabaseList({
() => {
refreshData();
addSuccessToast(t('Deleted: %s', item.database_name));
setSlCurrentlyDeleting(null);
setSlDeletePreview(null);
},
createErrorHandler(errMsg =>
addDangerToast(
@@ -677,15 +799,29 @@ function DatabaseList({
if (isSemanticLayer) {
if (!canEdit && !canDelete) return null;
const isLoadingDependents =
slDeletePreview?.status === 'loading' &&
slDeletePreview.item.uuid === original.uuid;
return (
<div className="actions">
{canDelete && (
<ActionButton
label={t('Delete')}
tooltip={t('Delete')}
tooltip={
isLoadingDependents
? t('Loading dependent semantic views')
: t('Delete')
}
placement="bottom"
icon={<Icons.DeleteOutlined iconSize="l" />}
onClick={() => setSlCurrentlyDeleting(original)}
icon={
isLoadingDependents ? (
<Icons.LoadingOutlined iconSize="l" spin />
) : (
<Icons.DeleteOutlined iconSize="l" />
)
}
disabled={isLoadingDependents}
onClick={() => openSemanticLayerDeleteModal(original)}
/>
)}
{canEdit && (
@@ -781,6 +917,8 @@ function DatabaseList({
handleDatabaseExport,
handleDatabasePermSync,
openDatabaseDeleteModal,
openSemanticLayerDeleteModal,
slDeletePreview,
],
);
@@ -932,20 +1070,21 @@ function DatabaseList({
addSuccessToast={addSuccessToast}
semanticLayerUuid={slCurrentlyEditing ?? undefined}
/>
{slCurrentlyDeleting && (
{slDeletePreview && slDeletePreview.status !== 'loading' && (
<DeleteModal
description={
<p>
{t('Are you sure you want to delete')}{' '}
<b>{slCurrentlyDeleting.database_name}</b>?
</p>
<>
<p>
{t('Are you sure you want to delete')}{' '}
<b>{slDeletePreview.item.database_name}</b>?
</p>
<SemanticLayerCascadeWarning preview={slDeletePreview} />
</>
}
onConfirm={() => {
if (slCurrentlyDeleting) {
handleSemanticLayerDelete(slCurrentlyDeleting);
}
handleSemanticLayerDelete(slDeletePreview.item);
}}
onHide={() => setSlCurrentlyDeleting(null)}
onHide={() => setSlDeletePreview(null)}
open
title={
<ModalTitleWithIcon
@@ -182,7 +182,7 @@ test('With sql role - renders all panels on the page on page load', async () =>
test('With sql role - renders distinct recent activities', async () => {
await renderWelcome();
const recentPanel = screen.getByRole('button', { name: 'collapsed Recents' });
const recentPanel = screen.getByRole('button', { name: 'Recents' });
userEvent.click(recentPanel);
await waitFor(() =>
expect(
+282 -344
View File
File diff suppressed because it is too large Load Diff
+2 -2
View File
@@ -37,11 +37,11 @@
"eslint": "^10.8.1",
"eslint-config-prettier": "^10.1.8",
"globals": "^17.11.0",
"oxfmt": "^0.63.0",
"oxfmt": "^0.64.0",
"tscw-config": "^1.1.2",
"typescript": "^6.0.3",
"typescript-eslint": "^8.67.0",
"vitest": "^4.1.10"
"vitest": "^4.1.11"
},
"engines": {
"node": "^24.16.0",
-10
View File
@@ -28,16 +28,6 @@ from werkzeug.local import LocalProxy
# form.
flask_appbuilder.Model.__allow_unmapped__ = True
# pandas >= 2.2 advertises a minimum SQLAlchemy of 2.0 and silently ignores
# older installations, breaking DataFrame.to_sql / read_sql with SQLAlchemy
# 1.4 engines. Its SQL layer still works with 1.4, so restore support until
# Superset itself requires SQLAlchemy >= 2. Must run before any pandas SQL IO.
from superset.utils.pandas_sqlalchemy_compat import ( # noqa: E402
restore_pandas_sqlalchemy_support,
)
restore_pandas_sqlalchemy_support()
from superset.app import create_app # noqa: E402, F401
from superset.extensions import ( # noqa: E402
appbuilder, # noqa: F401
-36
View File
@@ -52,42 +52,6 @@ class DatabaseRequiredFieldValidationError(ValidationError):
)
class DatabaseExtraJSONValidationError(ValidationError):
"""
Marshmallow validation error for database encrypted extra must be a valid JSON
"""
def __init__(self, json_error: str = "") -> None:
super().__init__(
[
_(
"Field cannot be decoded by JSON. %(json_error)s",
json_error=json_error,
)
],
field_name="extra",
)
class DatabaseExtraValidationError(ValidationError):
"""
Marshmallow validation error for database encrypted extra must be a valid JSON
"""
def __init__(self, key: str = "") -> None:
super().__init__(
[
_(
"The metadata_params in Extra field "
"is not configured correctly. The key "
"%{key}s is invalid.",
key=key,
)
],
field_name="extra",
)
class DatabaseConnectionSyncPermissionsError(CommandException):
status = 500
message = _("Unable to sync permissions for this database connection.")
+99 -23
View File
@@ -24,8 +24,10 @@ purge rolls back. The record is written ``pending`` *before* the purge and
flipped to ``confirmed`` *after* it commits, so a crash leaves at most a
``pending`` row, never a missing one. ``pending`` rows are reconciled on the
next run. Completed records are immutable. Consecutive scheduled evaluations
that remain blocked may discard only their current redundant provisional row;
force-purge and other meaningful outcomes are retained independently.
that remain blocked for the same reason may discard only their current
redundant provisional row a reason change retains one new row carrying the
new code; force-purge and other meaningful outcomes are retained
independently.
The dedicated ``purge_audit_log`` table is content-free (no name or PII; only
action, actor, UTC time, entity type, UUID, and affected referrers) and is never
@@ -87,6 +89,10 @@ class _AuditRecoverySnapshot:
entity_type: str
entity_uuid: str | None
created_on: datetime
# Sourced from the finalization call's reason argument, never from the
# row: pending rows are reason-less by design, so reading the row here
# would silently record NULL.
reason: str | None
def _utc_now() -> datetime:
@@ -136,8 +142,18 @@ def write_ahead(
session.close()
def finalize(record_id: UUID | None, status: str, **details: Any) -> None:
"""Finalize a pending attempt on the dedicated audit session."""
def finalize(
record_id: UUID | None,
status: str,
*,
reason: str | None = None,
**details: Any,
) -> None:
"""Finalize a pending attempt on the dedicated audit session.
``reason`` is persisted only for blocked outcomes; the audit records a
cause for a purge that did not happen, never for one that did.
"""
if record_id is None:
return
session = _dedicated_session()
@@ -148,6 +164,8 @@ def finalize(record_id: UUID | None, status: str, **details: Any) -> None:
referrers = details.get("affected_referrers")
if referrers:
values["affected_referrers"] = ",".join(referrers)
if reason is not None and status == STATUS_BLOCKED:
values["reason"] = reason
removed_dashboard_slices = details.get("removed_dashboard_slices")
if removed_dashboard_slices is not None:
values["removed_dashboard_slices"] = removed_dashboard_slices
@@ -192,12 +210,21 @@ def fail(record_id: UUID | None) -> None:
finalize(record_id, STATUS_FAILED)
def block(record_id: UUID | None) -> None:
"""Mark an attempt blocked by ordinary deletion policy."""
finalize(record_id, STATUS_BLOCKED)
def block(record_id: UUID | None, reason: str | None) -> None:
"""Mark an attempt blocked by ordinary deletion policy.
``reason`` is a stable machine code from the closed ``REASON_*``
vocabulary in :mod:`superset.commands.deletion_retention.purge_policy`.
It is required by signature (every blocked outcome has a classified
cause); ``None`` is tolerated defensively so a threading gap can never
block a purge, and leaves the persisted reason NULL.
"""
finalize(record_id, STATUS_BLOCKED, reason=reason)
def _capture_recovery_snapshot(record: PurgeAuditLog) -> _AuditRecoverySnapshot:
def _capture_recovery_snapshot(
record: PurgeAuditLog, reason: str | None
) -> _AuditRecoverySnapshot:
"""Capture the content-free fields needed for fail-safe recovery."""
return _AuditRecoverySnapshot(
id=cast(UUID, record.id),
@@ -205,26 +232,37 @@ def _capture_recovery_snapshot(record: PurgeAuditLog) -> _AuditRecoverySnapshot:
entity_type=str(record.entity_type),
entity_uuid=record.entity_uuid,
created_on=cast(datetime, record.created_on),
reason=reason,
)
def _retention_predecessor(
session: Session, current: PurgeAuditLog
) -> PurgeAuditLog | None:
"""Return the latest row that could unambiguously precede ``current``."""
"""Return the latest row that could unambiguously precede ``current``.
The latest same-entity retention row by ``created_on`` deliberately not
bounded by ``current.created_on``. If another visible row has a later
timestamp, it surfaces here so the caller's strictly-older check retains
the current row. Timestamps provide database ordering for this predicate,
not causal ordering across workers.
"""
predecessor: PurgeAuditLog | None = session.execute(
sa.select(PurgeAuditLog)
.where(PurgeAuditLog.entity_uuid == current.entity_uuid)
.where(PurgeAuditLog.entity_type == current.entity_type)
.where(PurgeAuditLog.trigger == TRIGGER_RETENTION)
.where(PurgeAuditLog.created_on <= current.created_on)
.where(PurgeAuditLog.id != current.id)
.order_by(PurgeAuditLog.created_on.desc())
.limit(1)
).scalar_one_or_none()
if predecessor is None:
return predecessor
tied_mixed_status_exists: bool = session.execute(
# A timestamp-tied row differing in status OR reason makes the
# predecessor ambiguous. ``is_distinct_from`` keeps the reason
# comparison NULL-safe on all supported dialects (reason is nullable;
# status is not).
tied_mixed_exists: bool = session.execute(
sa.select(
sa.exists().where(
PurgeAuditLog.entity_uuid == current.entity_uuid,
@@ -232,23 +270,43 @@ def _retention_predecessor(
PurgeAuditLog.trigger == TRIGGER_RETENTION,
PurgeAuditLog.created_on == predecessor.created_on,
PurgeAuditLog.id != current.id,
PurgeAuditLog.status != predecessor.status,
sa.or_(
PurgeAuditLog.status != predecessor.status,
PurgeAuditLog.reason.is_distinct_from(predecessor.reason),
),
)
)
).scalar_one()
if tied_mixed_status_exists:
if tied_mixed_exists:
return None
return predecessor
def _suppress_redundant_block(
session: Session, current: PurgeAuditLog, predecessor: PurgeAuditLog | None
session: Session,
current: PurgeAuditLog,
predecessor: PurgeAuditLog | None,
reason: str | None,
) -> bool:
"""Delete only a pending row with a strictly older blocked predecessor."""
"""Delete a pending row only against a strictly older same-reason block."""
if not reason:
# Fail safe on a threading gap: a missing current code must retain
# the row (and be visible), never silently revive the status-only
# predicate.
logger.warning(
"deletion_retention: blocked audit row %s has no reason code; "
"refusing suppression",
current.id,
)
return False
if (
predecessor is None
or predecessor.created_on >= current.created_on
or predecessor.status != STATUS_BLOCKED
# A reason-less (pre-feature) predecessor never matches: the first
# post-upgrade block of a long-blocked entity is retained once and
# becomes the new suppression anchor.
or predecessor.reason != reason
):
return False
deleted_rows: int | None = session.execute(
@@ -264,7 +322,7 @@ def _suppress_redundant_block(
return deleted_rows == 1
def _retain_blocked(session: Session, record_id: UUID) -> None:
def _retain_blocked(session: Session, record_id: UUID, reason: str | None) -> None:
"""Conditionally retain the current provisional row as blocked."""
session.execute(
sa.update(PurgeAuditLog.__table__)
@@ -272,7 +330,7 @@ def _retain_blocked(session: Session, record_id: UUID) -> None:
PurgeAuditLog.__table__.c.id == record_id,
PurgeAuditLog.__table__.c.status == STATUS_PENDING,
)
.values(status=STATUS_BLOCKED, removed_dashboard_slices=0)
.values(status=STATUS_BLOCKED, removed_dashboard_slices=0, reason=reason)
)
@@ -285,7 +343,11 @@ def _recover_retention_blocked(
current: PurgeAuditLog | None = recovery_session.get(PurgeAuditLog, record_id)
if current is not None:
if current.status == STATUS_PENDING:
_retain_blocked(recovery_session, record_id)
_retain_blocked(
recovery_session,
record_id,
snapshot.reason if snapshot else None,
)
recovery_session.commit()
return "fallback"
if snapshot is None:
@@ -300,6 +362,7 @@ def _recover_retention_blocked(
entity_uuid=snapshot.entity_uuid,
removed_dashboard_slices=0,
created_on=snapshot.created_on,
reason=snapshot.reason,
)
)
recovery_session.commit()
@@ -319,9 +382,14 @@ def _recover_retention_blocked(
def finalize_retention_blocked(
record_id: UUID | None,
record_id: UUID | None, reason: str | None
) -> RetentionBlockedDisposition:
"""Finalize a scheduled blocker, suppressing only proven redundant evidence."""
"""Finalize a scheduled blocker, suppressing only proven redundant evidence.
``reason`` is the stable machine code for the block (see
:func:`block`); it is persisted on retained rows and captured in the
snapshot used by the crash-recovery path.
"""
if record_id is None:
return "fallback"
session: Session = _dedicated_session()
@@ -330,15 +398,17 @@ def finalize_retention_blocked(
current: PurgeAuditLog | None = session.get(PurgeAuditLog, record_id)
if current is None:
return "fallback"
snapshot = _capture_recovery_snapshot(current)
snapshot = _capture_recovery_snapshot(current, reason)
if current.status != STATUS_PENDING or current.trigger != TRIGGER_RETENTION:
return "retained"
predecessor: PurgeAuditLog | None = None
if current.entity_uuid is not None:
predecessor = _retention_predecessor(session, current)
suppressed: bool = _suppress_redundant_block(session, current, predecessor)
suppressed: bool = _suppress_redundant_block(
session, current, predecessor, reason
)
if not suppressed:
_retain_blocked(session, record_id)
_retain_blocked(session, record_id, reason)
session.commit()
return "suppressed" if suppressed else "retained"
except SQLAlchemyError:
@@ -383,6 +453,12 @@ def reconcile_pending(stale_before: datetime | None = None) -> dict[str, int]:
``confirmed``. A surviving or unresolvable entity means the attempt did
not durably purge it and is finalized as failed; normal selection may
retry.
An attempt that had already decided ``blocked`` when its worker died is
indistinguishable here from any other stalled attempt, so it reconciles
as failed with no reason: pending rows carry no reason by design, and
inventing one would assert evidence this process never witnessed. The
next scheduled attempt re-anchors the entity with its real code.
"""
cutoff = stale_before or _utc_now() - _PENDING_STALE_AFTER
reconciled = absent = failed = 0
@@ -183,7 +183,7 @@ class ForcePurgeCommand:
removed_dashboard_slices=result.removed_dashboard_slices,
)
elif result.blocked_reason is not None:
audit.block(record_id)
audit.block(record_id, result.blocker.code if result.blocker else None)
else:
audit.fail(record_id)
if result.purged:
@@ -53,9 +53,11 @@ from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import Session
from superset.commands.deletion_retention.purge_policy import (
BlockerReason,
get_purge_policy,
PurgeBlockedError,
PurgeEntityPolicy,
REASON_CASCADE_INTEGRITY_FAILURE,
)
logger: logging.Logger = logging.getLogger(__name__)
@@ -144,7 +146,17 @@ class CascadeResult:
dangling_chart_uuids: list[str] = field(default_factory=list)
removed_dashboard_slices: int = 0
version_rows_removed: int = 0
blocked_reason: str | None = None
blocker: BlockerReason | None = None
@property
def blocked_reason(self) -> str | None:
"""Return the operator-facing blocker phrase, if the purge was blocked."""
return self.blocker.phrase if self.blocker else None
@property
def blocked_reason_code(self) -> str | None:
"""Return the stable audit code, if the purge was blocked."""
return self.blocker.code if self.blocker else None
class PurgeRaceLostError(Exception):
@@ -267,20 +279,21 @@ def cascade_hard_delete(
purged=False,
entity_type=entity_type,
entity_uuid=uuid,
blocked_reason=str(ex),
blocker=ex.reason,
)
except IntegrityError as ex:
# Not a policy decision: a restrictive FK the cascade did not handle.
# Not a policy decision: a database integrity constraint failed.
# Two audiences, two messages. The curated reason goes to the caller
# (and from there into a user toast), because raw driver text carries
# the failing SQL and bind parameters. The constraint detail goes to
# the log at WARNING, because an entity permanently unpurgeable via an
# unknown FK is a cascade-coverage bug someone has to be able to
# diagnose -- reported at INFO as a policy block, it read as intended
# behaviour.
# the log at WARNING, because an entity permanently unpurgeable after
# an integrity failure represents a cascade defect someone has to be
# able to diagnose. The stable audit code identifies this as an unexpected
# cascade failure rather than intended policy behavior without
# claiming which kind of constraint the database reported.
logger.warning(
"deletion_retention: %s id=%s purge failed on a restrictive "
"foreign key the cascade does not handle: %s",
"deletion_retention: %s id=%s purge failed on a database "
"integrity constraint: %s",
entity_type,
entity_id,
ex,
@@ -289,7 +302,10 @@ def cascade_hard_delete(
purged=False,
entity_type=entity_type,
entity_uuid=uuid,
blocked_reason="blocked by database references",
blocker=BlockerReason(
REASON_CASCADE_INTEGRITY_FAILURE,
"cascade blocked by a database integrity constraint",
),
)
return CascadeResult(
@@ -24,7 +24,7 @@ from dataclasses import dataclass
from enum import Enum
from functools import lru_cache
from types import MappingProxyType
from typing import Any, cast
from typing import Any, cast, NamedTuple
import sqlalchemy as sa
from sqlalchemy.orm import Mapper, Session
@@ -36,10 +36,44 @@ from superset.utils.sqlalchemy_events import (
logger: logging.Logger = logging.getLogger(__name__)
# Stable machine-readable reason codes persisted on purge audit records.
# The values are frozen identifiers pinned by a golden-set test: they equal
# the related-table names at introduction by coincidence, never by derivation,
# so a physical table rename changes only the blocker mapping's key and
# leaves the persisted code untouched — audit history and the suppression
# predicate compare these literals.
REASON_REPORT_SCHEDULE: str = "report_schedule"
REASON_USER_ATTRIBUTE: str = "user_attribute"
REASON_CASCADE_INTEGRITY_FAILURE: str = "cascade_integrity_failure"
ALL_REASON_CODES: frozenset[str] = frozenset(
{
REASON_REPORT_SCHEDULE,
REASON_USER_ATTRIBUTE,
REASON_CASCADE_INTEGRITY_FAILURE,
}
)
class BlockerReason(NamedTuple):
"""One blocker's persisted audit code paired with its operator phrase."""
code: str
phrase: str
class PurgeBlockedError(Exception):
"""Raised when ordinary deletion policy forbids purging an entity."""
def __init__(self, reason: BlockerReason) -> None:
super().__init__(reason.phrase)
self.reason: BlockerReason = reason
@property
def reason_code(self) -> str:
"""Return the stable machine-readable blocker code."""
return self.reason.code
class DependencyClassification(str, Enum):
"""Describe how purge treats a persistence dependency."""
@@ -106,11 +140,21 @@ class DependencyPolicy:
key: DependencyKey
classification: DependencyClassification
phase: ExecutionPhase | None = None
blocked_reason: str | None = None
blocker: BlockerReason | None = None
optional_listener: bool = False
listener_action: ListenerAction | None = None
version_column: str | None = None
@property
def blocked_reason(self) -> str | None:
"""Return the operator-facing blocker phrase, if this policy blocks."""
return self.blocker.phrase if self.blocker else None
@property
def blocked_reason_code(self) -> str | None:
"""Return the stable audit code, if this policy blocks."""
return self.blocker.code if self.blocker else None
@dataclass(frozen=True)
class PurgeEntityPolicy:
@@ -416,7 +460,7 @@ def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
keys: tuple[DependencyKey, ...],
classifications: tuple[DependencyClassification, ...],
synthetic: tuple[DependencyPolicy, ...],
blocked_reasons: Mapping[str, str] = MappingProxyType({}),
blocked_reasons: Mapping[str, BlockerReason] = MappingProxyType({}),
version_columns: Mapping[str, str] = MappingProxyType({}),
) -> tuple[DependencyPolicy, ...]:
phases: dict[DependencyClassification, ExecutionPhase | None] = {
@@ -428,15 +472,22 @@ def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
}
if len(keys) != len(classifications):
raise ValueError("Every dependency key requires one classification")
def declare(
key: DependencyKey, classification: DependencyClassification
) -> DependencyPolicy:
blocker: BlockerReason | None = blocked_reasons.get(key.related_table)
return DependencyPolicy(
key,
classification,
phases[classification],
blocker=blocker,
version_column=version_columns.get(key.related_table),
)
return (
tuple(
DependencyPolicy(
key,
classification,
phases[classification],
blocked_reason=blocked_reasons.get(key.related_table),
version_column=version_columns.get(key.related_table),
)
declare(key, classification)
for key, classification in zip(keys, classifications, strict=True)
)
+ synthetic
@@ -539,7 +590,12 @@ def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
DependencyClassification.PRESERVE,
),
(tag_cleanup, chart_membership_versions),
{"report_schedule": "associated alerts or reports exist"},
# Keyed by related table; the audit code is declared, not derived.
{
"report_schedule": BlockerReason(
REASON_REPORT_SCHEDULE, "associated alerts or reports exist"
)
},
{"slices_version": "id"},
),
validate=validate_deletion_allowed,
@@ -687,10 +743,16 @@ def purge_policy_registry() -> Mapping[type[Any], PurgeEntityPolicy]:
DependencyClassification.PRESERVE,
),
(tag_cleanup, dashboard_membership_versions),
# Keyed by related table; the audit code is declared, not derived.
# Declaration order is part of the audit contract: the first
# matching blocker's code is the one recorded.
{
"report_schedule": "associated alerts or reports exist",
"user_attribute": (
"a user has this dashboard set as their welcome page"
"report_schedule": BlockerReason(
REASON_REPORT_SCHEDULE, "associated alerts or reports exist"
),
"user_attribute": BlockerReason(
REASON_USER_ATTRIBUTE,
"a user has this dashboard set as their welcome page",
),
},
{"dashboards_version": "id"},
@@ -893,9 +955,9 @@ def validate_deletion_allowed(
if session.execute(
sa.select(sa.literal(1)).select_from(table).where(*predicates).limit(1)
).first():
if dependency.blocked_reason is None:
if dependency.blocker is None:
raise RuntimeError(f"Missing blocker reason for {key.describe()}")
raise PurgeBlockedError(dependency.blocked_reason)
raise PurgeBlockedError(dependency.blocker)
def count_dashboard_slices(
+2 -2
View File
@@ -138,8 +138,8 @@ class BaseRestoreVersionCommand(BaseCommand):
# With capture off, Continuum's write listeners are detached: a
# revert would mutate the live entity with NO new version row —
# a destructive, untracked write. The whole restore surface is
# therefore inert under the kill-switch, matching the read-side
# convention (404, indistinguishable from "no such version").
# therefore inert under the kill-switch (404, indistinguishable from
# "no such version"). Existing history remains readable.
if not capture_enabled():
raise self.not_found_exc()
entity = find_active_by_uuid(self.model_cls, self._uuid)
+7 -15
View File
@@ -708,7 +708,7 @@ DEFAULT_FEATURE_FLAGS: dict[str, bool] = {
# the move-back lever; removed (along with its two gate points —
# BaseDAO.delete routing and the do_orm_execute visibility listener) once
# post-flip confidence is established.
# @lifecycle: development
# @lifecycle: testing
"SOFT_DELETE": True,
# Enable semantic layers and show semantic views alongside datasets
# @lifecycle: development
@@ -742,9 +742,9 @@ DEFAULT_FEATURE_FLAGS: dict[str, bool] = {
"TAGGING_SYSTEM": False,
# Enables the version history panel on Explore and Dashboard pages.
# History only accrues while ``ENABLE_VERSIONING_CAPTURE`` is also on;
# with capture off the panel renders but stays empty, so the two ship
# with matching defaults and should be changed together.
# @lifecycle: development
# with capture off the panel renders empty or stale history, so the two
# ship with matching defaults and should be changed together.
# @lifecycle: testing
"VERSION_HISTORY": True,
# =================================================================
# IN TESTING
@@ -1694,17 +1694,9 @@ DATETIME_FORMAT_DETECTION_SAMPLE_SIZE = 1000
# The limit for the Superset Meta DB when the feature flag ENABLE_SUPERSET_META_DB is on
SUPERSET_META_DB_LIMIT: int | None = 1000
# Master switch for entity-version-history capture. Capture is enabled by
# default, so saves write shadow rows and a ``version_transaction`` /
# ``version_changes`` record. Set this to a falsy value in
# ``superset_config.py`` (or via the environment variable of the same name) to
# disable the before-flush listeners while keeping the /versions/ endpoints
# available read-only.
# Capture ships on. It is an operational escape hatch — set the environment
# variable to a falsy value when a versioning-induced regression needs a
# 30-second recovery instead of revert-and-redeploy — not a feature flag,
# and it remains permanently as the kill-switch rather than being removed
# with the rollout toggles.
# Master switch for entity-version-history capture. A falsy value disables
# version writes while keeping existing history available read-only through the
# ``/versions/`` endpoints; Restore is unavailable while capture is disabled.
ENABLE_VERSIONING_CAPTURE: bool = utils.parse_boolean_string(
os.environ.get("ENABLE_VERSIONING_CAPTURE", "true")
)
+4
View File
@@ -1316,6 +1316,10 @@ class DatabaseRestApi(BaseSupersetModelRestApi):
try:
TestConnectionDatabaseCommand(item).run()
return self.response(200, message="OK")
except OAuth2RedirectError:
# OAuth2 connections pass, so they can be saved. A user later
# can then store an OAuth2 token.
return self.response(200, message="OK")
except (
SSHTunnelingNotEnabledError,
SSHTunnelDatabasePortError,
+187 -2
View File
@@ -20,21 +20,23 @@ import logging
import re
from datetime import datetime
from re import Pattern
from typing import Any, Callable, Optional, TYPE_CHECKING, TypedDict
from typing import Any, Callable, cast, Optional, TYPE_CHECKING, TypedDict
from urllib import parse
from apispec import APISpec
from apispec.ext.marshmallow import MarshmallowPlugin
from cryptography.hazmat.backends import default_backend
from cryptography.hazmat.primitives import serialization
from flask import current_app as app
from flask import current_app as app, has_request_context
from flask_babel import gettext as __
from marshmallow import fields, Schema
from sqlalchemy import text, types
from sqlalchemy.engine.reflection import Inspector
from sqlalchemy.engine.url import URL
from sqlalchemy.exc import DatabaseError as SqlalchemyDatabaseError
from sqlalchemy.sql.elements import ColumnElement
from superset import is_feature_enabled, security_manager
from superset.constants import TimeGrain
from superset.databases.utils import make_url_safe
from superset.db_engine_specs.base import (
@@ -44,13 +46,63 @@ from superset.db_engine_specs.base import (
)
from superset.db_engine_specs.postgres import PostgresBaseEngineSpec
from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
from superset.exceptions import OAuth2TokenRefreshError
from superset.models.sql_lab import Query
from superset.superset_typing import (
OAuth2ClientConfig,
OAuth2State,
)
from superset.utils import json
from superset.utils.core import get_user_agent, QuerySource
from superset.utils.oauth2 import encode_oauth2_state, generate_code_challenge
if TYPE_CHECKING:
from superset.models.core import Database
try:
from snowflake.connector.errors import DatabaseError
except ImportError:
# Use a distinct sentinel type when snowflake is not installed to avoid
# matching unrelated exception types (using `Exception` would be too broad).
class _SnowflakeDatabaseError(Exception):
"""Sentinel type to stand in for snowflake.connector.errors.DatabaseError."""
pass
DatabaseError = _SnowflakeDatabaseError
class CustomSnowflakeAuthErrorMeta(type):
"""
Metaclass whose ``__instancecheck__`` matches Snowflake's invalid/expired
OAuth access-token error, so ``CustomSnowflakeAuthError`` can be used as the
``oauth2_exception`` that triggers the OAuth2 re-auth dance.
This is only honored via ``isinstance()`` (the path used by
``BaseEngineSpec.needs_oauth2()``); ``except`` clauses do not call
``__instancecheck__``, so it must not be relied on for exception catching.
"""
def __instancecheck__(cls, instance: object) -> bool:
"""
Match Snowflake's invalid/expired OAuth token error, whether it arrives
wrapped by SQLAlchemy (e.g. ``Engine``-based execution) or as the raw
DBAPI exception ``BaseEngineSpec.execute()`` runs against a bare
cursor and never wraps it, so both shapes must be handled here.
"""
orig: object = instance
if isinstance(instance, SqlalchemyDatabaseError):
orig = cast(SqlalchemyDatabaseError, instance).orig
return isinstance(orig, DatabaseError) and "Invalid OAuth access token" in str(
orig
)
class CustomSnowflakeAuthError(DatabaseError, metaclass=CustomSnowflakeAuthErrorMeta):
"""Snowflake OAuth error type matched via the metaclass above (see note there)."""
# Regular expressions to catch custom errors
OBJECT_DOES_NOT_EXIST_REGEX = re.compile(
r"Object (?P<object>.*?) does not exist or not authorized."
@@ -160,6 +212,7 @@ class SnowflakeEngineSpec(PostgresBaseEngineSpec):
encrypted_extra_sensitive_fields = {
"$.auth_params.privatekey_body": "Private Key Body",
"$.auth_params.privatekey_pass": "Private Key Password",
"$.oauth2_client_info.secret": "OAuth2 Client Secret",
}
_time_grain_expressions = {
@@ -198,6 +251,126 @@ class SnowflakeEngineSpec(PostgresBaseEngineSpec):
),
}
# OAuth 2.0 support
supports_oauth2: bool = True
# `CustomSnowflakeAuthError` is only matched via `isinstance()` (see the
# metaclass docstring above), so it's paired with `OAuth2TokenRefreshError`
# (a real subclass) to keep `refresh_oauth2_token`'s `except` clause working.
oauth2_exception: type[Exception] | tuple[type[Exception], ...] = (
CustomSnowflakeAuthError,
OAuth2TokenRefreshError,
)
@classmethod
def is_oauth2_enabled(cls) -> bool:
"""
Return whether OAuth2 authentication is enabled.
"""
# When alerts or reports connect to the database in the background,
# OAuth2 authentication fails; therefore, OAuth2 authentication is disabled
# for background execution.
if not has_request_context():
return False
return (
cls.supports_oauth2
and cls.engine_name in app.config["DATABASE_OAUTH2_CLIENTS"]
)
@classmethod
def get_oauth2_config(cls) -> OAuth2ClientConfig | None:
"""
Build the DB engine spec level OAuth2 client config.
"""
if not cls.is_oauth2_enabled():
return None
return super().get_oauth2_config()
@classmethod
def impersonate_user(
cls,
database: Database,
username: str | None,
user_token: str | None,
url: URL,
engine_kwargs: dict[str, Any],
) -> tuple[URL, dict[str, Any]]:
"""
Modify URL and/or engine kwargs to impersonate a different user.
"""
connect_args: dict[str, Any] = engine_kwargs.setdefault("connect_args", {})
# When test_connection is executed (i.e., when validate_default_parameters is
# set to True in connect_args), authentication via OAuth is not performed.
#
# ``database.is_oauth2_enabled()`` returns True for a database-level OAuth2
# client (``encrypted_extra.oauth2_client_info``) regardless of request
# context, unlike the app-config-based check in ``is_oauth2_enabled()``
# above. Background executions (alerts/reports) have no per-user token, so
# ``has_request_context()`` must be checked explicitly here too, or OAuth
# gets switched on with no token to send.
if (
not connect_args.get("validate_default_parameters", False)
and has_request_context()
and database.is_oauth2_enabled()
):
url = url.update_query_dict({"authenticator": "oauth"})
connect_args["authenticator"] = "oauth"
if user_token:
if username is not None:
if is_feature_enabled("IMPERSONATE_WITH_EMAIL_PREFIX"):
# ``Database._get_sqla_engine()`` has already looked
# up the login and substituted the email prefix into
# ``username`` before calling this method when this
# flag is on. Looking it up again here as if it were
# still the login would fail whenever the two differ,
# leaving the default/service-account username paired
# with this user's OAuth token. Use it as given.
url = url.set(username=username)
else:
user = security_manager.find_user(username=username)
if user and user.email:
url = url.set(username=user.email)
url = url.update_query_dict({"token": user_token})
return url, engine_kwargs
@classmethod
def get_oauth2_authorization_uri(
cls,
config: OAuth2ClientConfig,
state: OAuth2State,
code_verifier: str | None = None, # pylint: disable=unused-argument
) -> str:
"""
Return URI for initial OAuth2 request.
"""
uri = config["authorization_request_uri"]
# When calling the Snowflake OAuth authorization endpoint for a custom client,
# specify only the query parameters documented in the URL below.
# Adding unsupported parameters
# (e.g., `prompt` as used in BaseEngineSpec.get_oauth2_authorization_uri)
# will cause an error.
# https://docs.snowflake.com/user-guide/oauth-custom#query-parameters
params: dict[str, str] = {
"scope": config["scope"],
"response_type": "code",
"state": encode_oauth2_state(state),
"redirect_uri": config["redirect_uri"],
"client_id": config["id"],
}
# Add PKCE parameters (RFC 7636) if code_verifier is provided
if code_verifier:
params["code_challenge"] = generate_code_challenge(code_verifier)
params["code_challenge_method"] = "S256"
return parse.urljoin(uri, "?" + parse.urlencode(params))
@staticmethod
def get_extra_params(
database: Database, source: QuerySource | None = None
@@ -448,6 +621,18 @@ class SnowflakeEngineSpec(PostgresBaseEngineSpec):
database: "Database",
params: dict[str, Any],
) -> None:
# To use OAuth authentication, a database connection must first be created using
# another authenticator (typically key-pair authentication)
# with “Impersonate logged in user” enabled.
# Key-pair authentication is used for connection tests,
# while OAuth authentication is used when executing actual queries,
# such as in SQL Lab or dashboards.
# Therefore, when using OAuth authentication, the key-pair authentication
# settings are not loaded, and the connection is established using OAuth only.
connect_args: dict[str, Any] = params.get("connect_args") or {}
if connect_args.get("authenticator") == "oauth":
return
if not database.encrypted_extra:
return
try:
+84 -2
View File
@@ -37,6 +37,7 @@ joins and unions are done in memory, using the SQLite engine.
from __future__ import annotations
import contextvars
import datetime
import decimal
import operator
@@ -68,7 +69,34 @@ from sqlalchemy.exc import NoSuchTableError
from sqlalchemy.sql import Select, select
from superset import db, feature_flag_manager, security_manager
from superset.sql.parse import Table
from superset.sql.parse import count_referenced_tables, Table
def _count_referenced_tables(statement: str) -> int:
"""
Count the distinct `superset://` virtual tables a statement references,
so ``get_data`` can tell whether it's being asked for a standalone table
or for one side of a multi-table statement (see ``get_data`` for why this
matters). Shillelagh calls `SupersetShillelaghAdapter.get_data` once per
underlying table, independently of any other table referenced by the
same statement, so it has no way on its own to tell the two cases apart.
Uses the real SQL parser rather than pattern-matching on quoted
identifiers, since a naive `"db.table"`-shaped regex also matches
dotted, double-quoted column aliases (e.g. `AS "metric.value"`) that
have nothing to do with table references, and would misclassify a
single-table statement as multi-table.
"""
return count_referenced_tables(statement, "sqlite")
# `SupersetAPSWDialect.on_connect` populates `_executing_multi_table_query` for
# the duration of a statement so that `get_data` can tell whether it's being
# asked for a standalone table or for one side of a multi-table query (see
# `get_data` for why this matters).
_executing_multi_table_query: contextvars.ContextVar[bool] = contextvars.ContextVar(
"_executing_multi_table_query", default=False
)
# pylint: disable=abstract-method
@@ -119,6 +147,49 @@ class SupersetAPSWDialect(APSWDialect):
},
)
def on_connect(self) -> Callable[[Any], None]:
"""
Wrap cursor creation on every new DBAPI connection so ``execute`` tracks
whether the statement it's about to run references more than one
`superset://` virtual table, no matter how that statement reaches the
cursor.
SQLAlchemy's `do_execute*` hooks only fire for statements executed
through a SQLAlchemy `Connection` (the ORM/Core path). SQL Lab, the
primary way users query these tables, instead pulls a raw DBAPI cursor
via `engine.raw_connection()` and calls `cursor.execute()` on it
directly, bypassing those hooks entirely -- which would leave
`_executing_multi_table_query` permanently `False` for that path, and
`get_data` back to silently truncating one side of a join (see
`get_data` and #36304). Patching the cursor factory here, at the point
a new physical connection is established, catches every path, since
each one ultimately calls `execute()` on a cursor obtained from this
same connection.
"""
def setup(dbapi_connection: Any) -> None:
original_cursor = dbapi_connection.cursor
def cursor(*args: Any, **kwargs: Any) -> Any:
raw_cursor = original_cursor(*args, **kwargs)
original_execute = raw_cursor.execute
def execute(operation: str, parameters: Any = None) -> Any:
token = _executing_multi_table_query.set(
_count_referenced_tables(operation) > 1
)
try:
return original_execute(operation, parameters)
finally:
_executing_multi_table_query.reset(token)
raw_cursor.execute = execute
return raw_cursor
dbapi_connection.cursor = cursor
return setup
F = TypeVar("F", bound=Callable[..., Any])
@@ -409,7 +480,18 @@ class SupersetShillelaghAdapter(Adapter):
"""
app_limit: int | None = current_app.config["SUPERSET_META_DB_LIMIT"]
if limit is None:
limit = app_limit
# Shillelagh calls `get_data` once per table, independently of any
# other table referenced by the same statement, so a value of `None`
# here doesn't necessarily mean this table is the whole query -- it
# can equally mean this table is one side of a join (or other
# multi-table statement). Applying the app-wide default in that case
# would silently truncate this table before the in-memory join runs,
# dropping rows that have a genuine match on the other side with no
# error (see #36304). Only fall back to the default for statements
# that reference a single table, where truncating it can't hide
# otherwise-valid matches.
if app_limit is not None and not _executing_multi_table_query.get():
limit = app_limit
elif app_limit is not None:
limit = min(limit, app_limit)
+3 -10
View File
@@ -791,16 +791,9 @@ class SupersetAppInitializer: # pylint: disable=too-many-public-methods
Must be called after all versioned model classes have been imported so
that VERSIONED_MODELS can be populated and configure_mappers() has run.
``ENABLE_VERSIONING_CAPTURE`` (ships default ``False``) gates the two
before-flush listener registrations. The flag is operational, not
feature: with it off the infrastructure is inert (no save writes
shadow rows); flipping it on activates capture. The switch also lets
an operator who observes a versioning-induced regression (e.g. a
save-path slowdown attributable to the change-record listener)
disable capture in ``superset_config.py`` and restart workers a
30-second recovery instead of revert-and-redeploy. Shadow tables
already created by the migration stay; they just stop accumulating
new rows.
``ENABLE_VERSIONING_CAPTURE`` gates the baseline and change-record
listener registrations. When disabled, initialization also detaches
SQLAlchemy-Continuum's write listeners.
The fallback here is ``False`` so that any app-factory path that
does not load ``superset.config`` (some test factories, embedded
@@ -199,6 +199,8 @@ def apply_form_data_filters_to_query(
query["where"] = where
if having := form_data.get("having"):
query["having"] = having
if extras := form_data.get("extras"):
query["extras"] = {**(query.get("extras") or {}), **extras}
def _join_sql_clause(existing_clause: str, additional_clause: str) -> str:
@@ -256,6 +258,9 @@ def merge_form_data_filters_into_query(
else:
query[clause] = additional_clause
if extras := form_data.get("extras"):
query["extras"] = {**(query.get("extras") or {}), **extras}
def merge_extra_form_data_filters_into_query(
query: dict[str, Any],
@@ -665,6 +665,8 @@ def add_legend_config(form_data: Dict[str, Any], config: XYChartConfig) -> None:
# Canonical form_data key is camelCase; the echarts plugins read
# `legendOrientation` directly off form_data.
form_data["legendOrientation"] = config.legend.position
if config.legend_orientation:
form_data["legendOrientation"] = config.legend_orientation
def add_color_scheme(form_data: Dict[str, Any], color_scheme: str | None) -> None:
@@ -1415,6 +1417,8 @@ def map_filter_operator(op: str) -> str:
"NOT LIKE": "NOT LIKE",
"IN": "IN",
"NOT IN": "NOT IN",
"IS NULL": "IS NULL",
"IS NOT NULL": "IS NOT NULL",
}
return operator_map.get(op, op)
@@ -19,7 +19,6 @@
from __future__ import annotations
import re
from collections.abc import Mapping
from typing import Any, ClassVar
@@ -29,7 +28,10 @@ from superset.mcp_service.chart.chart_utils import (
)
from superset.mcp_service.chart.plugin import BaseChartPlugin
from superset.mcp_service.chart.schemas import ColumnRef, HistogramChartConfig
from superset.mcp_service.chart.validation.dataset_validator import DatasetValidator
from superset.mcp_service.chart.validation.dataset_validator import (
DatasetValidator,
is_numeric_column,
)
from superset.mcp_service.common.error_schemas import ChartGenerationError
@@ -116,27 +118,13 @@ class HistogramChartPlugin(BaseChartPlugin):
# Column existence is validated separately; don't double-report.
return None
def _is_numeric(col: dict[str, Any]) -> bool:
if col.get("is_numeric", False):
return True
# Backends report many spellings (BIGINT, SMALLINT, REAL, NUMBER,
# DOUBLE PRECISION); match numeric tokens at word boundaries so
# INTERVAL/POINT (which merely contain "INT") stay non-numeric.
type_upper = str(col.get("type", "")).upper()
return bool(
re.search(
r"\b(?:TINY|SMALL|MEDIUM|BIG)?INT(?:EGER)?\b"
r"|\bFLOAT\b|\bDOUBLE\b|\bDECIMAL\b"
r"|\bNUMERIC\b|\bREAL\b|\bNUMBER\b",
type_upper,
)
)
if _is_numeric(col_info):
if is_numeric_column(col_info):
return None
numeric_columns = sorted(
col["name"] for col in dataset_context.available_columns if _is_numeric(col)
col["name"]
for col in dataset_context.available_columns
if is_numeric_column(col)
)
return ChartGenerationError(
error_type="non_numeric_histogram_column",
+33 -5
View File
@@ -706,7 +706,7 @@ class ColumnRef(UnknownFieldCheckMixin):
None,
min_length=1,
max_length=255,
validation_alias=AliasChoices("name", "column_name"),
validation_alias=AliasChoices("name", "column_name", "column"),
)
label: str | None = Field(None, max_length=500)
dtype: str | None = None
@@ -884,17 +884,32 @@ class FilterConfig(UnknownFieldCheckMixin):
"NOT LIKE",
"IN",
"NOT IN",
"IS NULL",
"IS NOT NULL",
] = Field(
...,
description="LIKE/ILIKE use % wildcards. IN/NOT IN take a list.",
description=(
"LIKE/ILIKE use % wildcards. IN/NOT IN take a list. "
"IS NULL/IS NOT NULL omit value."
),
validation_alias=AliasChoices("op", "operator", "opr"),
)
value: str | int | float | bool | list[str | int | float | bool] = Field(
...,
description="For IN/NOT IN, provide a list.",
value: str | int | float | bool | list[str | int | float | bool] | None = Field(
None,
description="For IN/NOT IN, provide a list. Omit for null operators.",
validation_alias=AliasChoices("value", "val"),
)
@model_validator(mode="after")
def validate_value(self) -> "FilterConfig":
"""Null checks have no comparator; every other operator requires one."""
if self.op in {"IS NULL", "IS NOT NULL"}:
if self.value is not None:
raise ValueError(f"Filter operator {self.op!r} must not have 'value'.")
elif self.value is None:
raise ValueError(f"Filter operator {self.op!r} requires 'value'.")
return self
@field_validator("column")
@classmethod
def sanitize_column(cls, v: str) -> str:
@@ -1662,6 +1677,10 @@ class XYChartConfig(BaseChartConfig):
None,
validation_alias=AliasChoices("legend", "show_legend"),
)
legend_orientation: LEGEND_POSITION_LITERAL | None = Field(
None,
description="Legend placement around the chart",
)
x_axis_time_format: str | None = Field(
None,
description=(
@@ -2810,6 +2829,15 @@ class GetChartSqlRequest(BaseModel):
"Can be used alone (without identifier) for unsaved charts."
),
)
extra_form_data: dict[str, Any] | None = Field(
default=None,
description=(
"Extra form data to merge into the chart query before rendering SQL, "
"typically from dashboard native filters. Same format accepted by "
"get_chart_data. Format: "
'{"filters": [{"col": "country", "op": "IN", "val": ["US"]}]}'
),
)
@model_validator(mode="after")
def validate_identifier_or_form_data_key(self) -> "GetChartSqlRequest":
@@ -62,6 +62,42 @@ from superset.utils.core import GenericDataType
logger = logging.getLogger(__name__)
def _requested_filter_columns(extra_form_data: dict[str, Any] | None) -> set[str]:
"""Return simple column names explicitly requested through extra form data."""
if not extra_form_data:
return set()
columns: set[str] = set()
for filter_ in extra_form_data.get("filters", []):
if isinstance(filter_, dict) and isinstance(column := filter_.get("col"), str):
columns.add(column)
for filter_ in extra_form_data.get("adhoc_filters", []):
if (
isinstance(filter_, dict)
and filter_.get("expressionType") == "SIMPLE"
and isinstance(column := filter_.get("subject"), str)
):
columns.add(column)
return columns
def _rejected_requested_filter_columns(
result: Any, extra_form_data: dict[str, Any] | None
) -> list[str]:
"""Find request filters rejected by datasource query construction."""
if not isinstance(result, dict):
return []
requested = _requested_filter_columns(extra_form_data)
rejected = {
column
for query in result.get("queries", [])
for column in query.get("rejected_filter_columns", [])
if isinstance(column, str)
}
return sorted(requested & rejected)
_GENERIC_TYPE_MAP: dict[int, str] = {
GenericDataType.NUMERIC: "numeric",
GenericDataType.STRING: "string",
@@ -685,6 +721,19 @@ async def get_chart_data( # noqa: C901
command.validate()
result = command.run()
if rejected := _rejected_requested_filter_columns(
result, request.extra_form_data
):
rejected_columns = ", ".join(rejected)
await ctx.warning(
"Requested filters reference unknown dataset columns: %s"
% rejected_columns
)
return ChartError(
error=f"Unknown dataset column(s) in filters: {rejected_columns}",
error_type="ValidationError",
)
# Handle empty query results for certain chart types
if not result or ("queries" not in result) or len(result["queries"]) == 0:
await ctx.warning(
@@ -979,7 +1028,7 @@ async def get_chart_data( # noqa: C901
)
async def _query_from_form_data(
async def _query_from_form_data( # noqa: C901
form_data: Dict[str, Any],
request: GetChartDataRequest,
ctx: Context,
@@ -1034,6 +1083,19 @@ async def _query_from_form_data(
command.validate()
result = command.run()
if rejected := _rejected_requested_filter_columns(
result, request.extra_form_data
):
rejected_columns = ", ".join(rejected)
await ctx.warning(
"Requested filters reference unknown dataset columns: %s"
% rejected_columns
)
return ChartError(
error=f"Unknown dataset column(s) in filters: {rejected_columns}",
error_type="ValidationError",
)
if not result or "queries" not in result or len(result["queries"]) == 0:
logger.warning(
"get_chart_data: empty query results for unsaved chart "
@@ -23,11 +23,13 @@ import logging
from typing import Any, TYPE_CHECKING
from fastmcp import Context
from marshmallow import ValidationError as MarshmallowValidationError
from superset_core.mcp.decorators import tool, ToolAnnotations
if TYPE_CHECKING:
from superset.models.slice import Slice
from superset.charts.data.form_data import set_query_context_form_data
from superset.commands.exceptions import CommandException
from superset.commands.explore.form_data.parameters import CommandParameters
from superset.exceptions import SupersetException, SupersetSecurityException
@@ -35,6 +37,8 @@ from superset.extensions import event_logger
from superset.mcp_service.chart.chart_helpers import (
build_query_context_from_form_data,
extract_x_axis_col,
merge_extra_form_data_filters_into_query,
resolve_form_data_datasource,
resolve_groupby,
resolve_metrics,
resolve_metrics_and_groupby,
@@ -90,6 +94,7 @@ def _extract_x_axis_col(form_data: dict[str, Any]) -> str | None:
def _build_query_context_from_form_data(
form_data: dict[str, Any],
chart: "Slice | None" = None,
extra_form_data: dict[str, Any] | None = None,
) -> Any:
"""Build a QueryContext from form_data with result_type=QUERY.
@@ -101,6 +106,7 @@ def _build_query_context_from_form_data(
return build_query_context_from_form_data(
form_data,
chart=chart,
extra_form_data=extra_form_data,
result_type=ChartDataResultType.QUERY,
force=False,
)
@@ -150,6 +156,7 @@ def _resolve_effective_form_data(
def _sql_from_saved_query_context(
chart: "Slice",
extra_form_data: dict[str, Any] | None = None,
) -> ChartSql | ChartError | None:
"""Try to extract SQL from a chart's saved query_context.
@@ -168,8 +175,63 @@ def _sql_from_saved_query_context(
qc_json["result_type"] = ChartDataResultType.QUERY
qc_json["force"] = False
query_context = ChartDataQueryContextSchema().load(qc_json)
if extra_form_data:
# Resolve the pieces of the saved context the merge depends on first.
# Failures here mean the context itself is stale, not that the
# request's filters are bad, so the caller should rebuild it from
# form_data rather than surfacing a validation error.
try:
datasource_id = qc_json["datasource"]["id"]
datasource_type = qc_json["datasource"]["type"]
queries = qc_json.get("queries", [])
if not isinstance(queries, list):
raise TypeError("queries must be a list")
except (AttributeError, KeyError, TypeError) as ex:
logger.warning(
"Saved query context is unusable for chart %s; "
"falling back to form_data: %s",
chart.id,
ex,
)
return None
try:
for query in queries:
merge_extra_form_data_filters_into_query(
query,
extra_form_data,
datasource_id,
datasource_type,
)
except (AttributeError, KeyError, TypeError) as ex:
return ChartError(
error=f"Invalid extra_form_data filter: {ex}",
error_type="ValidationError",
)
try:
query_context = ChartDataQueryContextSchema().load(qc_json)
except MarshmallowValidationError as ex:
# A saved query context can become stale as schemas evolve. Let the
# caller rebuild it from form_data; malformed request filters will
# still produce a ValidationError from that fallback path.
logger.warning(
"Saved query context validation failed for chart %s; "
"falling back to form_data: %s",
chart.id,
ex,
)
return None
query_context.result_type = ChartDataResultType.QUERY
# ChartDataDatasourceSchema only requires "id", so fall back to the
# chart's own datasource rather than raising on a context that the
# schema itself considers valid.
datasource_json = qc_json.get("datasource") or {}
set_query_context_form_data(
query_context,
datasource_json.get("id", chart.datasource_id),
datasource_json.get("type", chart.datasource_type),
)
command = ChartDataCommand(query_context)
command.validate()
@@ -235,11 +297,26 @@ def _resolve_datasource_name(
def _sql_from_form_data(
form_data: dict[str, Any],
chart: "Slice | None",
extra_form_data: dict[str, Any] | None = None,
) -> ChartSql | ChartError:
"""Build SQL from form_data (fallback path)."""
from superset.commands.chart.data.get_data_command import ChartDataCommand
query_context = _build_query_context_from_form_data(form_data, chart)
try:
_, datasource_type = resolve_form_data_datasource(form_data, chart)
query_context = _build_query_context_from_form_data(
form_data, chart, extra_form_data=extra_form_data
)
except (AttributeError, KeyError, TypeError, MarshmallowValidationError) as ex:
return ChartError(
error=f"Invalid chart query data: {ex}",
error_type="ValidationError",
)
set_query_context_form_data(
query_context,
query_context.datasource.id,
datasource_type,
)
command = ChartDataCommand(query_context)
command.validate()
result = command.run()
@@ -335,6 +412,8 @@ async def get_chart_sql(
Supports:
- Numeric ID or UUID lookup
- form_data_key: get SQL for unsaved chart state from Explore view
- extra_form_data: preview SQL with dashboard-filter-style predicates merged
in, same format accepted by get_chart_data
Example usage:
```json
@@ -382,7 +461,9 @@ async def _handle_chart_sql_request(
# Handle unsaved chart (form_data_key only, no identifier)
if not request.identifier and request.form_data_key:
return await _handle_unsaved_chart_sql(request.form_data_key, ctx)
return await _handle_unsaved_chart_sql(
request.form_data_key, ctx, request.extra_form_data
)
# Find the chart by identifier
if request.identifier is None:
@@ -425,7 +506,7 @@ async def _handle_chart_sql_request(
# Try saved query_context first (faster, more accurate)
with event_logger.log_context(action="mcp.get_chart_sql.build_query"):
if not using_unsaved_state:
saved_result = _sql_from_saved_query_context(chart)
saved_result = _sql_from_saved_query_context(chart, request.extra_form_data)
if saved_result is not None:
return saved_result
await ctx.warning(
@@ -435,7 +516,9 @@ async def _handle_chart_sql_request(
# Fallback: build query context from form_data
try:
return _sql_from_form_data(effective_form_data, chart)
return _sql_from_form_data(
effective_form_data, chart, request.extra_form_data
)
except (SupersetException, CommandException, ValueError) as e:
await ctx.warning("Failed to build SQL from form_data: %s" % str(e))
return ChartError(
@@ -445,7 +528,9 @@ async def _handle_chart_sql_request(
async def _handle_unsaved_chart_sql(
form_data_key: str, ctx: Context
form_data_key: str,
ctx: Context,
extra_form_data: dict[str, Any] | None = None,
) -> ChartSql | ChartError:
"""Handle SQL retrieval for unsaved charts (form_data_key only)."""
from superset.utils import json as utils_json
@@ -476,7 +561,9 @@ async def _handle_unsaved_chart_sql(
)
try:
return _sql_from_form_data(form_data, chart=None)
return _sql_from_form_data(
form_data, chart=None, extra_form_data=extra_form_data
)
except (SupersetException, CommandException, ValueError) as e:
await ctx.warning("Failed to generate SQL from form_data: %s" % str(e))
return ChartError(
@@ -22,6 +22,8 @@ Validates that referenced columns exist in the dataset schema.
import difflib
import logging
import re
from collections.abc import Mapping
from typing import Any, Dict, List, Tuple, TypeVar
from superset.mcp_service.chart.schemas import (
@@ -38,6 +40,18 @@ _C = TypeVar("_C", bound=ChartConfig)
logger = logging.getLogger(__name__)
_NUMERIC_TYPE_PATTERN = re.compile(
r"\b(?:(?:TINY|SMALL|MEDIUM|BIG)?INT(?:EGER)?|INT[248]|FLOAT[48]?|"
r"DOUBLE(?:\s+PRECISION)?|DECIMAL|NUMERIC|REAL|NUMBER|(?:SMALL)?MONEY)\b"
)
def is_numeric_column(column: Mapping[str, Any]) -> bool:
"""Return whether dataset metadata identifies a numeric SQL column."""
if column.get("is_numeric", False):
return True
return bool(_NUMERIC_TYPE_PATTERN.search(str(column.get("type") or "").upper()))
def is_dataset_column_temporal(
column: Any, column_name: str, db_engine_spec: Any
@@ -702,11 +716,11 @@ class DatasetValidator:
"STDDEV",
"VAR",
]
type_name = str(col_info.get("type") or "").strip().upper()
if (
col_ref.aggregate in numeric_aggs
and not col_info.get("is_numeric", False)
and col_info.get("type", "").upper()
not in ["INTEGER", "FLOAT", "DOUBLE", "DECIMAL", "NUMERIC"]
and type_name not in {"", "UNKNOWN"}
and not is_numeric_column(col_info)
):
from superset.mcp_service.utils.error_builder import ( # noqa: E501
ChartErrorBuilder,
@@ -0,0 +1,60 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Add a reason to purge_audit_log.
Adds a nullable ``reason`` column to ``purge_audit_log`` holding a stable
machine code identifying which policy rule blocked a purge (or the
cascade-integrity failure class). Written at finalization for
blocked outcomes only; NULL for confirmed, failed, non-blocked, and
pre-existing rows. No backfill: the information was never captured for
historical records, and readers treat the column as optional.
Apply this migration before deploying the code that depends on it. The
audit model declares the column, so a worker running the new code against
the un-migrated table cannot write its write-ahead record; the scheduled
purge then fails closed (nothing is purged unaudited) and logs a
write-ahead warning every run until the migration lands.
The downgrade discards every recorded block reason -- the rows survive and
revert to reason-less, exactly like pre-feature history.
Revision ID: 39097d124752
Revises: 1072de5ed955
Create Date: 2026-08-24 12:00:00.000000
"""
import sqlalchemy as sa
from superset.migrations.shared.utils import add_columns, drop_columns
# revision identifiers, used by Alembic.
revision: str = "39097d124752"
down_revision: str = "1072de5ed955"
def upgrade() -> None:
"""Add the nullable ``reason`` column to ``purge_audit_log``."""
add_columns(
"purge_audit_log",
sa.Column("reason", sa.String(64), nullable=True),
)
def downgrade() -> None:
"""Drop the ``reason`` column from ``purge_audit_log``."""
drop_columns("purge_audit_log", "reason")
+6
View File
@@ -72,6 +72,12 @@ class PurgeAuditLog(Model):
# Comma-joined UUIDs of charts left dangling / dashboards that lost a join
# row (force-purge visibility). Free text, content-free.
affected_referrers = Column(Text, nullable=True)
# Stable machine code identifying which rule blocked the purge (or the
# cascade-integrity failure class). Written at finalization for
# blocked outcomes only; NULL for confirmed, failed, non-blocked, and
# pre-feature rows. Vocabulary: REASON_* constants in
# superset.commands.deletion_retention.purge_policy.
reason: Column[str] = Column(String(64), nullable=True)
removed_dashboard_slices = Column(Integer, nullable=False, default=0)
created_on = Column(
DateTime()
+12
View File
@@ -5618,6 +5618,18 @@ class SupersetSecurityManager( # pylint: disable=too-many-public-methods
editor_subject_ids = set(get_extra_editor_subject_ids(resource))
if hasattr(resource, "editors"):
editor_subject_ids.update(s.id for s in resource.editors)
# Fallback ONLY for Query and SavedQuery models that use 'user_id'
from superset.models.sql_lab import Query, SavedQuery
from superset.subjects.utils import get_user_subject
if (
isinstance(resource, (Query, SavedQuery))
and getattr(resource, "user_id", None) is not None
):
if subject := get_user_subject(resource.user_id):
editor_subject_ids.add(subject.id)
return bool(subject_ids & editor_subject_ids)
def is_viewer(self, resource: Model) -> bool:
+133 -45
View File
@@ -22,7 +22,6 @@ import enum
import logging
import re
import urllib.parse
from collections.abc import Iterable
from dataclasses import dataclass
from typing import Any, Generic, Optional, TYPE_CHECKING, TypeVar
@@ -2152,12 +2151,47 @@ class SQLScript:
return len(self.statements) == 1 and self.statements[0].is_select()
def extract_tables_from_statement(
def _find_show_statement_tables(statement: exp.Show) -> set[Table]:
"""
Build the table references for a ``SHOW`` statement.
Structured metadata statements (`SHOW CREATE TABLE foo.bar`,
`SHOW COLUMNS FROM foo`, ...) reference their target via dedicated
args rather than query sources, so build the table references
explicitly. Statements with no extractable target (e.g.
`SHOW TABLES FROM some_schema`) yield an empty set and are treated
as unparseable for authorization purposes (see
`SQLScript.has_unparseable_statement`).
``SHOW`` statements reference a single metadata target, never a join, so
(unlike ``_find_table_sources``) there is no distinct occurrence-counting
variant of this helper: the deduplicated set is always the right count.
"""
show_tables = {
Table(
source.name,
source.db if source.db != "" else None,
source.catalog if source.catalog != "" else None,
)
for source in statement.find_all(exp.Table)
}
if target := statement.args.get("target"):
db = statement.args.get("db")
show_tables.add(
Table(
target.name if isinstance(target, exp.Expression) else str(target),
db.name if isinstance(db, exp.Expression) else db,
)
)
return show_tables
def _find_table_sources(
statement: exp.Expression,
dialect: Dialects | None,
) -> set[Table]:
) -> list[exp.Table]:
"""
Extract all table references in a single statement.
Find every table reference (occurrence, not deduplicated) in a statement.
Please note that this is not trivial; consider the following queries:
@@ -2165,60 +2199,45 @@ def extract_tables_from_statement(
SHOW PARTITIONS FROM some_table;
WITH masked_name AS (SELECT * FROM some_table) SELECT * FROM masked_name;
See the unit tests for other tricky cases.
See the unit tests for other tricky cases. Note that `exp.Show` statements
are not handled here: see `_find_show_statement_tables`.
"""
sources: Iterable[exp.Table]
if isinstance(statement, exp.Describe):
# A `DESCRIBE` query has no sources in sqlglot, so we need to explicitly
# query for all tables.
sources = statement.find_all(exp.Table)
elif isinstance(statement, exp.Command):
return list(statement.find_all(exp.Table))
if isinstance(statement, exp.Command):
# Commands, like `SHOW COLUMNS FROM foo`, have to be converted into a
# `SELECT` statetement in order to extract tables.
literal = statement.find(exp.Literal)
if not literal:
return set()
return []
pseudo_sql = f"SELECT {literal.this}"
try:
_check_script_length(pseudo_sql, None)
pseudo_query = sqlglot.parse_one(pseudo_sql, dialect=dialect)
except (ParseError, SupersetParseError):
return set()
sources = pseudo_query.find_all(exp.Table)
elif isinstance(statement, exp.Show):
# Structured metadata statements (`SHOW CREATE TABLE foo.bar`,
# `SHOW COLUMNS FROM foo`, ...) reference their target via dedicated
# args rather than query sources, so build the table references
# explicitly. Statements with no extractable target (e.g.
# `SHOW TABLES FROM some_schema`) yield an empty set and are treated
# as unparseable for authorization purposes (see
# `SQLScript.has_unparseable_statement`).
show_tables = {
Table(
source.name,
source.db if source.db != "" else None,
source.catalog if source.catalog != "" else None,
)
for source in statement.find_all(exp.Table)
}
if target := statement.args.get("target"):
db = statement.args.get("db")
show_tables.add(
Table(
target.name if isinstance(target, exp.Expression) else str(target),
db.name if isinstance(db, exp.Expression) else db,
)
)
return show_tables
else:
sources = [
source
for scope in traverse_scope(statement)
for source in scope.sources.values()
if isinstance(source, exp.Table) and not is_cte(source, scope)
]
return []
return list(pseudo_query.find_all(exp.Table))
return [
source
for scope in traverse_scope(statement)
for source in scope.sources.values()
if isinstance(source, exp.Table) and not is_cte(source, scope)
]
def extract_tables_from_statement(
statement: exp.Expression,
dialect: Dialects | None,
) -> set[Table]:
"""
Extract all distinct table references in a single statement.
"""
if isinstance(statement, exp.Show):
return _find_show_statement_tables(statement)
return {
Table(
@@ -2226,10 +2245,79 @@ def extract_tables_from_statement(
source.db if source.db != "" else None,
source.catalog if source.catalog != "" else None,
)
for source in sources
for source in _find_table_sources(statement, dialect)
}
def count_referenced_tables(statement: str, dialect: Dialects | str | None) -> int:
"""
Count the table references in a raw SQL string.
This counts occurrences, not distinct tables, so a self-join referencing
the same physical table twice (via two aliases) is still counted as 2 -
callers use this count to decide whether a statement is a join, and a
self-join needs the same treatment as a join across different tables.
A CTE that's referenced more than once (e.g. self-joined) is weighted the
same way: each reference to it counts its own underlying tables again,
since a CTE is inlined at every place it's used (see
``_count_weighted_table_references``).
Falls back to a conservative count of 1 (i.e. "not multi-table") if the
statement can't be parsed, since callers gating multi-table-only behavior
on this count should default to treating an unparseable statement as a
single table.
"""
try:
_check_script_length(statement, str(dialect) if dialect else None)
parsed = sqlglot.parse_one(statement, dialect=dialect)
if isinstance(parsed, exp.Show):
return len(_find_show_statement_tables(parsed))
if isinstance(parsed, (exp.Describe, exp.Command)):
# Neither has join semantics for a per-table row cap to interact
# with, so the plain (unweighted) extraction already used for
# permissioning is fine here too.
return len(_find_table_sources(parsed, dialect))
return _count_weighted_table_references(parsed)
except Exception: # pylint: disable=broad-except
return 1
def _count_weighted_table_references(statement: exp.Expression) -> int:
"""
Count table references the way callers gating multi-table-only behavior
need: weighting each CTE by how many times it's actually referenced,
not by how many distinct tables its own definition reads.
``_find_table_sources`` (used for permissioning) intentionally counts a
CTE's underlying tables exactly once regardless of how many times the
CTE is referenced downstream, since permission checks only care about
the *set* of tables read. But a CTE that wraps a single virtual table
and is then self-joined N ways is inlined at each of those N places, so
it triggers N separate reads of that table -- one per join side -- and
must count as N here too. Otherwise a per-table row cap (see
``SUPERSET_META_DB_LIMIT`` and #36304) looks safe to apply and silently
truncates one side of the self-join away before the join runs.
"""
def resolve(scope: Scope, seen: frozenset[int]) -> list[exp.Table]:
if id(scope) in seen:
return [] # guards a WITH RECURSIVE self-reference from looping forever
seen = seen | {id(scope)}
tables: list[exp.Table] = []
for _, source in scope.selected_sources.values():
if isinstance(source, exp.Table) and not is_cte(source, scope):
tables.append(source)
elif isinstance(source, Scope) and source.scope_type == ScopeType.CTE:
tables.extend(resolve(source, seen))
return tables
return sum(
len(resolve(scope, frozenset()))
for scope in traverse_scope(statement)
if scope.scope_type != ScopeType.CTE
)
def is_cte(source: exp.Table, scope: Scope) -> bool:
"""
Does this reference resolve to a CTE rather than to a real table?
+17 -12
View File
@@ -32,6 +32,7 @@ import logging
from collections.abc import Iterator
from datetime import datetime, timedelta
from typing import Any, cast
from uuid import UUID
import sqlalchemy as sa
from flask import current_app
@@ -45,6 +46,7 @@ from superset.commands.deletion_retention.purge_cascade import (
entity_uuid,
suppress_purge_association_versions,
)
from superset.commands.deletion_retention.purge_policy import BlockerReason
from superset.commands.deletion_retention.window import resolve_retention_window
from superset.extensions import celery_app, feature_flag_manager, stats_logger_manager
from superset.models.helpers import (
@@ -205,6 +207,19 @@ def _purge_model(
return purged, would, failures, blocked
def _finalize_blocked(record_id: UUID | None, blocker: BlockerReason) -> None:
"""Finalize a blocked retention outcome and count suppression metrics."""
disposition: audit.RetentionBlockedDisposition = audit.finalize_retention_blocked(
record_id, blocker.code
)
if disposition == "suppressed":
stats_logger_manager.instance.incr(f"{_METRIC_PREFIX}.blocked_audit_suppressed")
elif disposition == "fallback":
stats_logger_manager.instance.incr(
f"{_METRIC_PREFIX}.blocked_audit_dedupe_fallback"
)
def _purge_one(
model: type[SoftDeleteMixin], entity_id: int, cutoff: datetime
) -> CascadeResult | None:
@@ -279,18 +294,8 @@ def _purge_one(
affected_referrers=result.dangling_chart_uuids,
removed_dashboard_slices=result.removed_dashboard_slices,
)
elif result.blocked_reason is not None:
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
)
if disposition == "suppressed":
stats_logger_manager.instance.incr(
f"{_METRIC_PREFIX}.blocked_audit_suppressed"
)
elif disposition == "fallback":
stats_logger_manager.instance.incr(
f"{_METRIC_PREFIX}.blocked_audit_dedupe_fallback"
)
elif result.blocker is not None:
_finalize_blocked(record_id, result.blocker)
else:
audit.fail(record_id)
return result
@@ -379,37 +379,37 @@ msgstr "虛擬"
msgid "%s aggregates(s)"
msgstr "%s 聚合"
#, fuzzy, python-format
#, python-format
msgid "%s column"
msgid_plural "%s columns"
msgstr[0] "%s "
msgstr[0] "%s 個欄位"
#, python-format
msgid "%s column(s)"
msgstr "%s "
msgstr "%s 個欄位"
#, fuzzy, python-format
#, python-format
msgid "%s day ago"
msgid_plural "%s days ago"
msgstr[0] "1前"
msgstr[0] "%s 天前"
#, fuzzy, python-format
#, python-format
msgid "%s hr ago"
msgid_plural "%s hr ago"
msgstr[0] "%s "
msgstr[0] "%s 小時前"
#, fuzzy, python-format
#, python-format
msgid "%s imported"
msgstr "數據集已導入"
msgstr "已匯入 %s"
#, fuzzy, python-format
#, python-format
msgid "%s item"
msgid_plural "%s items"
msgstr[0] "%s 個項"
msgstr[0] "%s 個項"
#, fuzzy, python-format
#, python-format
msgid "%s item(s)"
msgstr "%s 個項"
msgstr "%s 個項"
# Machine-translated via backfill_po.py (claude-sonnet-4-6) [refs: ar, cs, de,
# es, fa, fr, ja, lv, mi, nl, pl, pt_BR, ru, sk, sl, sr, sr_Latn, tr, uk]
@@ -419,15 +419,15 @@ msgid ""
"all selected objects."
msgstr "%s 個項目無法標記,因為您對所有選取的物件沒有編輯權限。"
#, fuzzy, python-format
#, python-format
msgid "%s metric"
msgid_plural "%s metrics"
msgstr[0] "排序指標"
msgstr[0] "%s 個指標"
#, fuzzy, python-format
#, python-format
msgid "%s min ago"
msgid_plural "%s min ago"
msgstr[0] ""
msgstr[0] "%s 分鐘前"
#, python-format
msgid ""
@@ -451,47 +451,47 @@ msgstr[0] "%s 個選項"
msgid "%s option(s)"
msgstr "%s 個選項"
#, fuzzy, python-format
#, python-format
msgid "%s out of %s column"
msgid_plural "%s out of %s columns"
msgstr[0] "自定義列"
msgstr[0] "已選取 %s%s 個欄位"
#, fuzzy, python-format
#, python-format
msgid "%s out of %s metric"
msgid_plural "%s out of %s metrics"
msgstr[0] "排序指標"
msgstr[0] "已選取 %s%s 個指標"
#, fuzzy, python-format
#, python-format
msgid "%s out of %s selected"
msgstr "%s 已選定"
msgstr "已選取 %s%s 個項目"
#, fuzzy, python-format
#, python-format
msgid "%s recipients"
msgstr "%s 最近"
msgstr "%s 收件者"
#, fuzzy, python-format
#, python-format
msgid "%s record..."
msgid_plural "%s records..."
msgstr[0] "%s 異常"
msgstr[0] "%s 筆記錄..."
#, fuzzy, python-format
#, python-format
msgid "%s row"
msgid_plural "%s rows"
msgstr[0] "%s "
msgstr[0] "%s "
#, fuzzy, python-format
#, python-format
msgid "%s s ago"
msgid_plural "%s s ago"
msgstr[0] "30 天之前"
msgstr[0] "%s 秒前"
#, python-format
msgid "%s saved metric(s)"
msgstr "%s 保存的指標"
#, fuzzy, python-format
#, python-format
msgid "%s second"
msgid_plural "%s seconds"
msgstr[0] "5 秒"
msgstr[0] "%s 秒"
# Machine-translated via backfill_po.py (claude-sonnet-4-6) [refs: de, es, sr,
# sr_Latn]
@@ -511,13 +511,13 @@ msgstr "%s 個語意檢視新增失敗"
msgid "%s semantic view(s) failed to add: %s"
msgstr "%s 個語意檢視新增失敗:%s"
#, fuzzy, python-format
#, python-format
msgid "%s tab selected"
msgstr "%s 已選定"
msgstr "已選取「%s」分頁"
#, fuzzy, python-format
#, python-format
msgid "%s updated"
msgstr "上次更新 %s"
msgstr "更新 %s"
#, python-format
msgid "%s%s"
@@ -677,13 +677,11 @@ msgstr "每年年初的頻率"
msgid "10 minute"
msgstr "10 分鐘"
#, fuzzy
msgid "10 seconds"
msgstr "30 秒"
msgstr "10 秒"
#, fuzzy
msgid "10/90 percentiles"
msgstr "9/91 百分位"
msgstr "10/90 百分位"
#. do-not-translate
msgid "10000"
@@ -696,9 +694,8 @@ msgstr "週"
msgid "104 weeks ago"
msgstr "104 週之前"
#, fuzzy
msgid "12 hours"
msgstr "1 小時"
msgstr "12 小時"
msgid "15 minute"
msgstr "15 分鐘"
@@ -761,9 +758,8 @@ msgstr "2/98 百分位"
msgid "22"
msgstr "22"
#, fuzzy
msgid "24 hours"
msgstr "6 小時"
msgstr "24 小時"
#, fuzzy
msgid "28 days"
@@ -831,9 +827,8 @@ msgstr "5 秒"
msgid "5 seconds"
msgstr "5 秒"
#, fuzzy
msgid "5/95 percentiles"
msgstr "9/91 百分位"
msgstr "5/95 百分位"
#, fuzzy
msgid "52 weeks"
@@ -1,80 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Compatibility shim letting pandas >= 2.2 use SQLAlchemy 1.4 engines.
pandas 2.2 raised its advertised minimum SQLAlchemy version to 2.0 as a
support-policy change. When an older SQLAlchemy is installed, pandas does not
fail loudly: ``pandas.io.sql`` silently pretends SQLAlchemy is absent, treats
Engine/Connection arguments as raw DBAPI connections, and falls back to its
sqlite-only code path, breaking every ``DataFrame.to_sql`` / ``read_sql``
call site (dataset uploads, example data loading, annotation queries, filter
values).
The pandas SQL layer itself still works with SQLAlchemy 1.4 because it only
uses the API subset common to SQLAlchemy 1.4 and 2.x. Lowering the advertised
minimum back to the pandas 2.1 value restores the working behavior.
This module is obsolete once Superset requires SQLAlchemy >= 2; at that point
the patch becomes a no-op and the module (and its call site in
``superset/__init__.py``) can be deleted.
"""
import logging
import sqlalchemy
from packaging.version import Version
logger = logging.getLogger(__name__)
# The last pandas release line to support SQLAlchemy 1.4 (pandas 2.1)
# required at least this version.
_SQLALCHEMY_MINIMUM = "1.4.16"
def restore_pandas_sqlalchemy_support() -> None:
"""Lower pandas' advertised SQLAlchemy minimum so 1.4 engines work.
Only applies when the installed SQLAlchemy predates 2.0 and pandas
advertises a 2.x minimum; in every other combination this is a no-op.
Safe to call multiple times.
"""
if Version(sqlalchemy.__version__) >= Version("2.0.0"):
# pandas supports SQLAlchemy 2.x natively; nothing to patch.
return
try:
from pandas.compat import _optional
except ImportError:
# The private module moved in a newer pandas; SQL IO with a pre-2.0
# SQLAlchemy will misbehave, so make the situation diagnosable.
logger.warning(
"Could not adjust pandas' minimum SQLAlchemy version; "
"DataFrame.to_sql/read_sql may not accept SQLAlchemy %s engines",
sqlalchemy.__version__,
)
return
advertised = _optional.VERSIONS.get("sqlalchemy")
if advertised and Version(advertised) > Version(_SQLALCHEMY_MINIMUM):
_optional.VERSIONS["sqlalchemy"] = _SQLALCHEMY_MINIMUM
logger.debug(
"Lowered pandas' minimum SQLAlchemy version from %s to %s so "
"pandas SQL IO keeps working with the installed SQLAlchemy %s",
advertised,
_SQLALCHEMY_MINIMUM,
sqlalchemy.__version__,
)
@@ -2403,6 +2403,54 @@ class TestDatabaseApi(SupersetTestCase):
assert rv.status_code == 200
assert rv.headers["Content-Type"] == "application/json; charset=utf-8"
@with_config({"PREVENT_UNSAFE_DB_CONNECTIONS": False})
def test_test_connection_oauth2(self):
"""
Database API: Test test connection flow with a connection authenticated via
OAuth2.
The test would always raise ``OAuth2RedirectError``, and we can't start the
OAuth2 dance before the connection is saved, so it should return a 200 status.
"""
self.login(ADMIN_USERNAME)
example_db = get_example_database()
masked_encrypted_extra = json.dumps(
{
"oauth2_client_info": {
"id": "client_id",
"secret": "client_secret",
"scope": "some-scope",
"authorization_request_uri": "https://example.org/authorize",
"token_request_uri": "https://example.org/token",
}
}
)
data = {
"database_name": "examples",
"masked_encrypted_extra": masked_encrypted_extra,
"impersonate_user": True,
"sqlalchemy_uri": example_db.safe_sqlalchemy_uri(),
"server_cert": None,
}
url = "api/v1/database/test_connection/"
with (
mock.patch(
"superset.commands.database.test_connection.ping",
side_effect=Exception("Unauthorized"),
),
mock.patch.object(
example_db.db_engine_spec,
"needs_oauth2",
return_value=True,
),
):
rv = self.post_assert_metric(url, data, "test_connection")
assert rv.status_code == 200
assert rv.headers["Content-Type"] == "application/json; charset=utf-8"
assert json.loads(rv.data.decode("utf-8")) == {"message": "OK"}
def test_test_connection_failed(self):
"""
Database API: Test test connection failed
@@ -29,6 +29,10 @@ from sqlalchemy.orm import Session
from superset import db
from superset.commands.deletion_retention import audit
from superset.commands.deletion_retention.audit import PurgeAuditLog
from superset.commands.deletion_retention.purge_policy import (
REASON_CASCADE_INTEGRITY_FAILURE,
REASON_REPORT_SCHEDULE,
)
from superset.models.slice import Slice
from superset.tasks.deletion_retention import _purge_impl
@@ -75,6 +79,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
assert row.trigger == audit.TRIGGER_RETENTION
assert row.actor == audit.ACTOR_SYSTEM
assert row.confirmed_on is not None
assert row.reason is None
assert isinstance(row.id, UUID)
def test_known_failure_finalizes_audit_row(self) -> None:
@@ -97,6 +102,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
row = db.session.query(PurgeAuditLog).filter_by(entity_uuid=chart_uuid).one()
assert row.status == audit.STATUS_FAILED
assert row.confirmed_on is None
assert row.reason is None
def test_reconcile_confirms_pending_after_entity_commit(self) -> None:
"""A crash after entity commit is reconciled to confirmed."""
@@ -172,6 +178,9 @@ class TestPurgeAudit(DeletionRetentionTestBase):
row = db.session.get(PurgeAuditLog, record_id)
assert row.status == audit.STATUS_TARGET_ABSENT
assert row.removed_dashboard_slices == 0
# The reconcile crash window is the documented reason-losing path:
# finalized rows here never carry a fabricated code.
assert row.reason is None
def test_blocked_attempt_does_not_keep_the_intended_removal_count(self) -> None:
"""The write-ahead row records what the purge INTENDED to remove;
@@ -184,7 +193,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
entity_uuid="00000000-0000-0000-0000-00000000cafe",
removed_dashboard_slices=7,
)
audit.block(record_id)
audit.block(record_id, REASON_REPORT_SCHEDULE)
row = db.session.query(PurgeAuditLog).filter_by(id=record_id).one()
assert row.status == audit.STATUS_BLOCKED
@@ -194,20 +203,56 @@ class TestPurgeAudit(DeletionRetentionTestBase):
record_id: UUID = self._write_retention_record(entity_uuid="first-block")
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(record_id)
assert disposition == "retained"
assert record.status == audit.STATUS_BLOCKED
assert record.reason == REASON_REPORT_SCHEDULE
def test_reason_is_persisted_only_for_blocked_outcomes(self) -> None:
"""A reason offered for a non-blocked outcome is refused, not stored.
The audit records a cause for a purge that did not happen; a
confirmed or failed row asserting a blocker would misreport its own
outcome.
"""
for status in (
audit.STATUS_CONFIRMED,
audit.STATUS_FAILED,
audit.STATUS_TARGET_ABSENT,
):
record_id: UUID = self._write_retention_record(
entity_uuid=f"non-blocked-{status}"
)
audit.finalize(record_id, status, reason=REASON_REPORT_SCHEDULE)
record: PurgeAuditLog = self._get_audit_record(record_id)
db.session.refresh(record)
assert record.status == status
assert record.reason is None
def test_finalized_reason_is_immutable(self) -> None:
"""A second finalization attempt never rewrites the recorded reason."""
record_id: UUID = self._write_retention_record(entity_uuid="reason-immutable")
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
audit.block(record_id, "some_other_code")
audit.finalize_retention_blocked(record_id, "some_other_code")
record: PurgeAuditLog = self._get_audit_record(record_id)
db.session.refresh(record)
assert record.status == audit.STATUS_BLOCKED
assert record.reason == REASON_REPORT_SCHEDULE
def test_repeated_retention_block_suppresses_current_provisional(self) -> None:
first_id: UUID = self._write_retention_record(entity_uuid="repeat-block")
audit.finalize_retention_blocked(first_id)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
second_id: UUID = self._write_retention_record(entity_uuid="repeat-block")
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(second_id)
audit.finalize_retention_blocked(second_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "suppressed"
@@ -215,18 +260,160 @@ class TestPurgeAudit(DeletionRetentionTestBase):
assert first.status == audit.STATUS_BLOCKED
assert db.session.get(PurgeAuditLog, second_id) is None
def test_reason_change_breaks_suppression_exactly_once(self) -> None:
"""A reason change writes one new blocked row, then re-suppresses.
The suppression predicate keys on status AND reason: same-reason
nights suppress; the night the reason changes is retained with the
new code and becomes the new anchor.
"""
entity: str = "reason-change"
first_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
== "retained"
)
second_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(second_id, REASON_REPORT_SCHEDULE)
== "suppressed"
)
changed_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(
changed_id, REASON_CASCADE_INTEGRITY_FAILURE
)
== "retained"
)
repeat_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(
repeat_id, REASON_CASCADE_INTEGRITY_FAILURE
)
== "suppressed"
)
rows: list[PurgeAuditLog] = (
db.session.query(PurgeAuditLog).filter_by(entity_uuid=entity).all()
)
assert {row.reason for row in rows} == {
REASON_REPORT_SCHEDULE,
REASON_CASCADE_INTEGRITY_FAILURE,
}
assert len(rows) == 2
assert all(row.status == audit.STATUS_BLOCKED for row in rows)
def test_mixed_reason_timestamp_tie_is_ambiguous_and_retains(self) -> None:
"""Tied predecessors differing only in reason refuse suppression."""
timestamp: datetime = datetime.utcnow()
first_id: UUID = self._write_retention_record(
entity_uuid="mixed-reason-tie", created_on=timestamp
)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
second_id: UUID = self._write_retention_record(
entity_uuid="mixed-reason-tie", created_on=timestamp
)
audit.finalize_retention_blocked(second_id, REASON_CASCADE_INTEGRITY_FAILURE)
current_id: UUID = self._write_retention_record(
entity_uuid="mixed-reason-tie", created_on=timestamp + timedelta(seconds=1)
)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "retained"
current: PurgeAuditLog = self._get_audit_record(current_id)
assert current.status == audit.STATUS_BLOCKED
def test_null_reason_historical_predecessor_never_suppresses(self) -> None:
"""The first post-upgrade block of a long-blocked entity is retained.
Pre-feature blocked rows carry NULL; NULL never matches a current
code, so the entity anchors once with its code and same-code nights
suppress against the new anchor.
"""
entity: str = "null-historical"
prior_id: UUID = self._write_retention_record(entity_uuid=entity)
audit.finalize(prior_id, audit.STATUS_BLOCKED)
prior: PurgeAuditLog = self._get_audit_record(prior_id)
assert prior.reason is None
current_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
== "retained"
)
current: PurgeAuditLog = self._get_audit_record(current_id)
assert current.reason == REASON_REPORT_SCHEDULE
repeat_id: UUID = self._write_retention_record(entity_uuid=entity)
assert (
audit.finalize_retention_blocked(repeat_id, REASON_REPORT_SCHEDULE)
== "suppressed"
)
def test_none_current_code_never_suppresses_and_warns(self) -> None:
"""A missing current code fails safe: retained, with a warning."""
entity: str = "none-current-code"
first_id: UUID = self._write_retention_record(entity_uuid=entity)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
current_id: UUID = self._write_retention_record(entity_uuid=entity)
with patch(
"superset.commands.deletion_retention.audit.logger.warning"
) as warning:
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id, None)
)
assert disposition == "retained"
assert warning.called
current: PurgeAuditLog = self._get_audit_record(current_id)
assert current.status == audit.STATUS_BLOCKED
assert current.reason is None
def test_predecessor_is_the_latest_row_overall(self) -> None:
"""A newer same-entity row forbids suppressing against an older one.
With rows timestamped both before and after the current attempt, the
later-timestamped row is selected, fails the strictly-older check, and
causes retention. This verifies timestamp ordering, not causal order
across workers.
"""
timestamp: datetime = datetime.utcnow()
older_id: UUID = self._write_retention_record(
entity_uuid="latest-overall", created_on=timestamp - timedelta(seconds=1)
)
audit.finalize_retention_blocked(older_id, REASON_REPORT_SCHEDULE)
newer_id: UUID = self._write_retention_record(
entity_uuid="latest-overall", created_on=timestamp + timedelta(seconds=1)
)
audit.finalize(newer_id, audit.STATUS_BLOCKED, reason=REASON_REPORT_SCHEDULE)
current_id: UUID = self._write_retention_record(
entity_uuid="latest-overall", created_on=timestamp
)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "retained"
current: PurgeAuditLog = self._get_audit_record(current_id)
assert current.status == audit.STATUS_BLOCKED
def test_equal_timestamp_is_ambiguous_and_retains_current(self) -> None:
timestamp: datetime = datetime.utcnow()
first_id: UUID = self._write_retention_record(
entity_uuid="equal-time", created_on=timestamp
)
audit.finalize_retention_blocked(first_id)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
second_id: UUID = self._write_retention_record(
entity_uuid="equal-time", created_on=timestamp
)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(second_id)
audit.finalize_retention_blocked(second_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "retained"
@@ -257,7 +444,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
session.close()
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
assert predecessor is None
@@ -273,10 +460,10 @@ class TestPurgeAudit(DeletionRetentionTestBase):
newer_id: UUID = self._write_retention_record(
entity_uuid="overlap", created_on=current_time + timedelta(seconds=1)
)
audit.finalize_retention_blocked(newer_id)
audit.finalize_retention_blocked(newer_id, REASON_REPORT_SCHEDULE)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "retained"
@@ -291,7 +478,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
assert disposition == "retained"
@@ -300,7 +487,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
null_id: UUID = self._write_retention_record(entity_uuid=None)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(null_id)
audit.finalize_retention_blocked(null_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(null_id)
@@ -312,13 +499,13 @@ class TestPurgeAudit(DeletionRetentionTestBase):
chart_id: UUID = self._write_retention_record(
entity_uuid="shared-type", entity_type="slices"
)
audit.finalize_retention_blocked(chart_id)
audit.finalize_retention_blocked(chart_id, REASON_REPORT_SCHEDULE)
dashboard_id: UUID = self._write_retention_record(
entity_uuid="shared-type", entity_type="dashboards"
)
dashboard_disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(dashboard_id)
audit.finalize_retention_blocked(dashboard_id, REASON_REPORT_SCHEDULE)
)
assert dashboard_disposition == "retained"
@@ -328,7 +515,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
audit.fail(record_id)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(record_id)
@@ -343,7 +530,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
side_effect=audit.SQLAlchemyError("lookup failed"),
):
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(record_id)
@@ -352,7 +539,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
def test_suppression_delete_failure_recovers_blocked_evidence(self) -> None:
first_id: UUID = self._write_retention_record(entity_uuid="delete-failure")
audit.finalize_retention_blocked(first_id)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
current_id: UUID = self._write_retention_record(entity_uuid="delete-failure")
with patch(
@@ -360,7 +547,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
side_effect=audit.SQLAlchemyError("delete failed"),
):
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(current_id)
@@ -383,16 +570,18 @@ class TestPurgeAudit(DeletionRetentionTestBase):
),
):
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(record_id)
assert disposition == "fallback"
assert record.status == audit.STATUS_BLOCKED
# The recovery retain branch carries the argument-sourced snapshot reason.
assert record.reason == REASON_REPORT_SCHEDULE
def test_uncertain_suppression_commit_recreates_absent_evidence(self) -> None:
first_id: UUID = self._write_retention_record(entity_uuid="absent-current")
audit.finalize_retention_blocked(first_id)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
current_id: UUID = self._write_retention_record(entity_uuid="absent-current")
primary_session: Session = audit._dedicated_session()
recovery_session: Session = audit._dedicated_session()
@@ -410,12 +599,16 @@ class TestPurgeAudit(DeletionRetentionTestBase):
),
):
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(current_id)
assert disposition == "fallback"
assert record.status == audit.STATUS_BLOCKED
# The recovery re-insert branch sources the reason from the snapshot
# (populated from the call argument, never from the reason-less
# pending row).
assert record.reason == REASON_REPORT_SCHEDULE
def test_failed_fallback_leaves_pending_evidence_for_reconciliation(self) -> None:
record_id: UUID = self._write_retention_record(entity_uuid="fallback-failure")
@@ -438,7 +631,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
),
):
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(record_id)
audit.finalize_retention_blocked(record_id, REASON_REPORT_SCHEDULE)
)
record: PurgeAuditLog = self._get_audit_record(record_id)
@@ -464,31 +657,34 @@ class TestPurgeAudit(DeletionRetentionTestBase):
entity_type="slices",
entity_uuid="indeterminate-rowcount",
created_on=timestamp - timedelta(seconds=1),
reason=REASON_REPORT_SCHEDULE,
)
result: MagicMock = MagicMock(rowcount=-1)
session: MagicMock = MagicMock()
session.execute.return_value = result
with pytest.raises(audit.SQLAlchemyError, match="indeterminate"):
audit._suppress_redundant_block(session, current, predecessor)
audit._suppress_redundant_block(
session, current, predecessor, REASON_REPORT_SCHEDULE
)
def test_overlap_duplicates_do_not_cause_unbounded_sequential_growth(self) -> None:
timestamp: datetime = datetime.utcnow()
first_id: UUID = self._write_retention_record(
entity_uuid="bounded-overlap", created_on=timestamp
)
audit.finalize_retention_blocked(first_id)
audit.finalize_retention_blocked(first_id, REASON_REPORT_SCHEDULE)
overlap_id: UUID = self._write_retention_record(
entity_uuid="bounded-overlap", created_on=timestamp
)
audit.finalize_retention_blocked(overlap_id)
audit.finalize_retention_blocked(overlap_id, REASON_REPORT_SCHEDULE)
later_id: UUID = self._write_retention_record(
entity_uuid="bounded-overlap",
created_on=timestamp + timedelta(seconds=1),
)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(later_id)
audit.finalize_retention_blocked(later_id, REASON_REPORT_SCHEDULE)
)
retained_count: int = (
@@ -516,7 +712,7 @@ class TestPurgeAudit(DeletionRetentionTestBase):
current_id: UUID = self._write_retention_record(entity_uuid=entity_uuid)
disposition: audit.RetentionBlockedDisposition = (
audit.finalize_retention_blocked(current_id)
audit.finalize_retention_blocked(current_id, REASON_REPORT_SCHEDULE)
)
current: PurgeAuditLog = self._get_audit_record(current_id)
@@ -140,6 +140,7 @@ class TestForcePurge(DeletionRetentionTestBase):
assert self.exists(Slice, chart_id)
row = db.session.query(PurgeAuditLog).filter_by(entity_uuid=chart_uuid).one()
assert row.status == "blocked"
assert row.reason == "report_schedule"
log_info.assert_called_once_with(
"force_purge: blocked %s uuid=%s reason=%s",
"chart",
@@ -24,6 +24,7 @@ guarantee under FK enforcement OFF, and the version-tables-absent no-op.
from __future__ import annotations
from collections.abc import Callable
from dataclasses import replace
from datetime import datetime, timedelta
from typing import Any
@@ -45,6 +46,9 @@ from superset.commands.deletion_retention.purge_cascade import (
from superset.commands.deletion_retention.purge_policy import (
get_purge_policy,
PurgeEntityPolicy,
REASON_CASCADE_INTEGRITY_FAILURE,
REASON_REPORT_SCHEDULE,
REASON_USER_ATTRIBUTE,
)
from superset.connectors.sqla.models import (
RLSFilterTables,
@@ -286,6 +290,7 @@ class TestSoftDeletePurge(DeletionRetentionTestBase):
.one()
)
assert row.status == audit.STATUS_BLOCKED
assert row.reason == REASON_REPORT_SCHEDULE
def test_repeated_report_blocker_preserves_counts_and_suppresses_noise(
self,
@@ -768,6 +773,7 @@ class TestExplicitBlockerGuards(DeletionRetentionTestBase):
assert result.purged is False
assert result.blocked_reason is not None
assert "welcome page" in result.blocked_reason
assert result.blocked_reason_code == REASON_USER_ATTRIBUTE
assert self.exists(Dashboard, dashboard_id)
finally:
self._restore_welcome(attribute, created, previous)
@@ -812,10 +818,120 @@ class TestExplicitBlockerGuards(DeletionRetentionTestBase):
db.session.commit()
assert result.purged is False
assert result.blocked_reason == "blocked by database references"
assert (
result.blocked_reason
== "cascade blocked by a database integrity constraint"
)
assert "SQL:" not in result.blocked_reason
assert result.blocked_reason_code == REASON_CASCADE_INTEGRITY_FAILURE
assert self.exists(Slice, chart_id)
def test_three_way_distinction_is_readable_from_the_audit_alone(self) -> None:
"""Report, welcome, and database-integrity blocks write distinct codes.
The audit table is the durable record: each of the three
non-completing outcomes must be identifiable from its row alone,
with no SQL fragments and the integrity case keeping blocked status.
"""
chart: Slice = self.make_chart("threeway_report")
report: ReportSchedule = ReportSchedule(
type="Report",
name="retention_it_threeway",
crontab="0 0 * * *",
chart=chart,
)
db.session.add(report)
db.session.commit()
chart_uuid: str = str(chart.uuid)
self.soft_delete(chart, days_ago=90)
dashboard: Dashboard = self.make_dashboard("threeway_welcome")
dashboard_uuid: str = str(dashboard.uuid)
self.soft_delete(dashboard, days_ago=90)
attribute: UserAttribute
created: bool
previous: int | None
attribute, created, previous = self._set_welcome(dashboard.id)
fk_chart: Slice = self.make_chart("threeway_fk")
fk_uuid: str = str(fk_chart.uuid)
self.soft_delete(fk_chart, days_ago=90)
real_get_policy: Callable[[type[Any]], PurgeEntityPolicy] = get_purge_policy
def fail_fk_chart_cleanup(
session: Session, policy: PurgeEntityPolicy, entity_id: int
) -> None:
if entity_id == fk_chart.id:
raise IntegrityError("FOREIGN KEY constraint failed", None, Exception())
real_get_policy(Slice).delete_associations(session, policy, entity_id)
def patched_policy(model: type[Any]) -> PurgeEntityPolicy:
policy: PurgeEntityPolicy = real_get_policy(model)
if model is Slice:
return replace(policy, delete_associations=fail_fk_chart_cleanup)
return policy
try:
with patch(
"superset.commands.deletion_retention.purge_cascade.get_purge_policy",
side_effect=patched_policy,
):
_purge(window=30)
rows: dict[str, audit.PurgeAuditLog] = {
uuid: db.session.query(audit.PurgeAuditLog)
.filter_by(entity_uuid=uuid)
.one()
for uuid in (chart_uuid, dashboard_uuid, fk_uuid)
}
assert rows[chart_uuid].reason == REASON_REPORT_SCHEDULE
assert rows[dashboard_uuid].reason == REASON_USER_ATTRIBUTE
assert rows[fk_uuid].reason == REASON_CASCADE_INTEGRITY_FAILURE
assert len({row.reason for row in rows.values()}) == 3
for row in rows.values():
assert row.status == audit.STATUS_BLOCKED
assert "SQL" not in row.reason
assert "?" not in row.reason
finally:
self._restore_welcome(attribute, created, previous)
def test_first_declared_blocker_wins_in_the_audit_record(self) -> None:
"""A dashboard blocked by both rules records the first-declared code.
Declaration order is part of the audit contract: report_schedule is
declared before user_attribute, so a dashboard that is both
report-referenced and someone's welcome page records
REASON_REPORT_SCHEDULE.
"""
dashboard: Dashboard = self.make_dashboard("firstmatch")
report: ReportSchedule = ReportSchedule(
type="Report",
name="retention_it_firstmatch",
crontab="0 0 * * *",
dashboard=dashboard,
)
db.session.add(report)
db.session.commit()
dashboard_uuid: str = str(dashboard.uuid)
self.soft_delete(dashboard, days_ago=90)
attribute: UserAttribute
created: bool
previous: int | None
attribute, created, previous = self._set_welcome(dashboard.id)
try:
_purge(window=30)
row: audit.PurgeAuditLog = (
db.session.query(audit.PurgeAuditLog)
.filter_by(entity_uuid=dashboard_uuid)
.one()
)
assert row.status == audit.STATUS_BLOCKED
assert row.reason == REASON_REPORT_SCHEDULE
finally:
self._restore_welcome(attribute, created, previous)
def test_policy_action_failure_rolls_back_prior_phases(self) -> None:
"""A later policy-action failure restores earlier association cleanup."""
chart: Slice = self.make_chart("action_rollback")
@@ -605,9 +605,6 @@ class TestSavedQueryApi(SupersetTestCase):
db.session.query(SavedQuery).filter(SavedQuery.label == "label1").all()[0]
)
self.login(ADMIN_USERNAME)
# Freeze relative to the persisted timestamp so database-specific
# timestamp precision cannot make the humanized value age into the
# next bucket while the request is being handled.
with freeze_time(saved_query.changed_on):
uri = f"api/v1/saved_query/{saved_query.id}"
rv = self.get_assert_metric(uri, "get")
@@ -0,0 +1,182 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Golden-set tests pinning the purge-audit reason-code vocabulary."""
from __future__ import annotations
from dataclasses import replace
from unittest.mock import MagicMock
import pytest
from superset.commands.deletion_retention.purge_policy import (
ALL_REASON_CODES,
DependencyClassification,
DependencyPolicy,
get_purge_policy,
purge_policy_registry,
PurgeBlockedError,
PurgeEntityPolicy,
REASON_CASCADE_INTEGRITY_FAILURE,
REASON_REPORT_SCHEDULE,
REASON_USER_ATTRIBUTE,
validate_deletion_allowed,
)
def test_reason_code_literals_are_frozen() -> None:
"""The persisted code values are frozen identifiers.
Audit history and the suppression predicate compare these exact strings;
a physical table rename or constant refactor must not re-mint them. If
this test fails, the fix is to restore the literal, never to update the
expectation.
"""
assert REASON_REPORT_SCHEDULE == "report_schedule"
assert REASON_USER_ATTRIBUTE == "user_attribute"
assert REASON_CASCADE_INTEGRITY_FAILURE == "cascade_integrity_failure"
assert ALL_REASON_CODES == {
"report_schedule",
"user_attribute",
"cascade_integrity_failure",
}
def test_reason_codes_are_distinct_and_column_sized() -> None:
"""Codes are mutually distinct and fit the String(64) audit column."""
codes: list[str] = [
REASON_REPORT_SCHEDULE,
REASON_USER_ATTRIBUTE,
REASON_CASCADE_INTEGRITY_FAILURE,
]
assert len(set(codes)) == len(codes)
assert all(0 < len(code) <= 64 for code in ALL_REASON_CODES)
def test_every_declared_blocker_code_is_in_the_closed_set() -> None:
"""Each blocker declared in the registry carries a code from ALL_REASON_CODES."""
blocker_codes: set[str] = set()
for policy in purge_policy_registry().values():
for dependency in policy.dependencies:
if dependency.classification is DependencyClassification.BLOCK:
assert dependency.blocker is not None, (
f"blocker {dependency.key.describe()} has no reason code"
)
blocker_codes.add(dependency.blocker.code)
assert blocker_codes <= ALL_REASON_CODES
assert blocker_codes == {REASON_REPORT_SCHEDULE, REASON_USER_ATTRIBUTE}
def test_cascade_integrity_failure_code_is_reserved_for_the_cascade() -> None:
"""No declared policy blocker may claim the cascade-failure code."""
for policy in purge_policy_registry().values():
for dependency in policy.dependencies:
assert (
dependency.blocker is None
or dependency.blocker.code != REASON_CASCADE_INTEGRITY_FAILURE
)
def _session_matching_blockers(*matches: bool) -> MagicMock:
"""A mock session whose Nth blocker query reports a match iff matches[N].
Deliberately positional: which blocker matches first is the audit
contract under test, so these cases are coupled to the order (and the
count) of the queries ``validate_deletion_allowed`` issues.
"""
session: MagicMock = MagicMock()
session.execute.side_effect = [
MagicMock(first=MagicMock(return_value=(1,) if match else None))
for match in matches
]
return session
def test_report_block_raises_with_the_report_schedule_code() -> None:
"""A chart blocked by a report reference carries REASON_REPORT_SCHEDULE."""
# avoid app-init regression: superset.models.* evaluates
# encrypted_field_factory at class-definition time, which fails
# in a partial-collection unit run with no Flask app active.
from superset.models.slice import Slice
info: pytest.ExceptionInfo[PurgeBlockedError]
with pytest.raises(PurgeBlockedError) as info:
validate_deletion_allowed(
_session_matching_blockers(True), get_purge_policy(Slice), 1
)
assert info.value.reason_code == REASON_REPORT_SCHEDULE
assert str(info.value) == "associated alerts or reports exist"
def test_welcome_dashboard_block_raises_with_the_user_attribute_code() -> None:
"""A welcome-page block carries a code distinct from the report code."""
# avoid app-init regression: superset.models.* evaluates
# encrypted_field_factory at class-definition time, which fails
# in a partial-collection unit run with no Flask app active.
from superset.models.dashboard import Dashboard
info: pytest.ExceptionInfo[PurgeBlockedError]
with pytest.raises(PurgeBlockedError) as info:
validate_deletion_allowed(
_session_matching_blockers(False, True), get_purge_policy(Dashboard), 1
)
assert info.value.reason_code == REASON_USER_ATTRIBUTE
def test_reason_code_survives_a_related_table_rename() -> None:
"""A renamed table keeps the blocker's declared code.
The code is declared on the blocker, never derived from the physical
table name, so a schema rename changes only which table the blocker
looks at persisted audit history and the suppression predicate keep
comparing the same literal.
"""
# avoid app-init regression: superset.models.* evaluates
# encrypted_field_factory at class-definition time, which fails
# in a partial-collection unit run with no Flask app active.
from superset.models.slice import Slice
policy: PurgeEntityPolicy = get_purge_policy(Slice)
renamed: tuple[DependencyPolicy, ...] = tuple(
replace(dependency, key=replace(dependency.key, related_table="reports_v2"))
if dependency.classification is DependencyClassification.BLOCK
else dependency
for dependency in policy.dependencies
)
blocker: DependencyPolicy = next(
dependency
for dependency in renamed
if dependency.classification is DependencyClassification.BLOCK
)
assert blocker.key.related_table == "reports_v2"
assert blocker.blocker is not None
assert blocker.blocker.code == REASON_REPORT_SCHEDULE
def test_first_declared_blocker_wins_when_several_match() -> None:
"""A dashboard matching both blockers records the first-declared code."""
# avoid app-init regression: superset.models.* evaluates
# encrypted_field_factory at class-definition time, which fails
# in a partial-collection unit run with no Flask app active.
from superset.models.dashboard import Dashboard
info: pytest.ExceptionInfo[PurgeBlockedError]
with pytest.raises(PurgeBlockedError) as info:
validate_deletion_allowed(
_session_matching_blockers(True, True), get_purge_policy(Dashboard), 1
)
assert info.value.reason_code == REASON_REPORT_SCHEDULE
@@ -68,11 +68,10 @@ def test_dashboard_import_with_overwrite_replaces_charts(
}
ImportDashboardsCommand._import(initial_configs, overwrite=True)
# Commit between imports, as production does: ``run()`` carries
# ``@transaction()``, so two imports are two transactions. Calling the
# private ``_import`` twice without committing puts both in one Continuum
# transaction, where adding and removing the same association collides on
# ``dashboard_slices_version``'s (dashboard_id, slice_id, transaction_id)
# key — an artifact of the test's shortcut, not a reachable state.
# ``@transaction()``, so two imports are two transactions. Without this the
# add and the remove of one association share a Continuum transaction and
# collide on ``dashboard_slices_version``'s composite key — an artifact of
# the test's shortcut, not a reachable production state.
db.session.commit()
# Verify initial state: 2 charts associated with the dashboard
@@ -557,6 +557,44 @@ def test_extra_validator_accepts_catalog_cache_timeout() -> None:
assert extra["metadata_cache_timeout"]["catalog_cache_timeout"] == 600
def test_extra_validator_interpolates_invalid_metadata_params_key() -> None:
"""
The message names the offending key. It is built with a lazy translated
string, so a malformed placeholder would only fail once the message is
rendered; asserting on the rendered text pins the interpolation.
"""
from superset.databases.schemas import DatabasePostSchema
schema = DatabasePostSchema()
payload = {
"database_name": "test_db",
"extra": json.dumps({"metadata_params": {"not_a_metadata_arg": 1}}),
}
with pytest.raises(ValidationError) as exc_info:
schema.load(payload)
message = str(exc_info.value)
assert "not_a_metadata_arg" in message
assert "%(" not in message
def test_extra_validator_interpolates_json_decode_error() -> None:
"""
As above, for the message raised when ``extra`` is not decodable JSON.
"""
from superset.databases.schemas import DatabasePostSchema
schema = DatabasePostSchema()
payload = {"database_name": "test_db", "extra": "{not json"}
with pytest.raises(ValidationError) as exc_info:
schema.load(payload)
message = str(exc_info.value)
# Assert on the interpolated value rather than the surrounding wording. The
# value comes from json.JSONDecodeError, which is not translated, so this
# stays valid under any locale.
assert "line 1 column" in message
assert "%(" not in message
def test_cache_timeout_rejects_values_below_minus_one() -> None:
"""
Test that cache_timeout rejects values less than -1.
@@ -715,12 +715,33 @@ def _patch_bq_fetch_deps(
mocker: MockerFixture, max_mb: int = 200
) -> tuple[mock.MagicMock, mock.MagicMock]:
"""Helper to patch Flask g and current_app for BigQuery fetch_data tests."""
flask_g = mocker.patch("superset.db_engine_specs.bigquery.g")
app = mocker.patch("superset.db_engine_specs.bigquery.current_app")
# `new_callable=mock.MagicMock` is pinned explicitly rather than relying on
# ``mocker.patch``'s auto-detection of the mock class. That detection
# inspects whatever object currently sits at the patched attribute, so if
# an earlier test in the same session ever leaves an ``AsyncMock`` there
# (e.g. an improperly torn-down patch), every subsequent patch of the same
# attribute -- even ones created fresh here -- would also become an
# ``AsyncMock``, since ``AsyncMock`` classifies its own non-dunder child
# attributes as ``AsyncMock`` too. Pinning the callable sidesteps that
# self-perpetuating class inference entirely.
flask_g = mocker.patch(
"superset.db_engine_specs.bigquery.g", new_callable=mock.MagicMock
)
app = mocker.patch(
"superset.db_engine_specs.bigquery.current_app", new_callable=mock.MagicMock
)
# Make current_app truthy and .config.get() return a plain int
app.__bool__ = mock.Mock(return_value=True)
app.config = mock.MagicMock()
app.config.get = mock.Mock(return_value=max_mb)
# ``fetch_data`` only records ``g.bq_memory_limited*`` when
# ``has_request_context()`` is true. Outside of a real Flask request
# (as in these unit tests) that's always false, so without patching it
# the assignments never happen and the mocked ``g`` attributes stay
# unset child mocks instead of the expected booleans/counts.
mocker.patch(
"superset.db_engine_specs.bigquery.has_request_context", return_value=True
)
return flask_g, app
@@ -229,7 +229,12 @@ def _generate_gis_type_sanitization_test_cases() -> list[
if not ocient_is_installed():
return []
from pyocient import _STLinestring, _STPoint, _STPolygon, TypeCodes
from pyocient import TypeCodes
from pyocient.api import (
STLinestring as _STLinestring,
STPoint as _STPoint,
STPolygon as _STPolygon,
)
return [
(
@@ -296,7 +301,7 @@ def _generate_gis_type_sanitization_test_cases() -> list[
(
"empty_polygon",
TypeCodes.ST_POLYGON,
_STPolygon(exterior=[], holes=[]),
_STPolygon(exterior=[], holes=[], fullFlag=False),
{
"geometry": None,
"properties": {},
@@ -311,6 +316,7 @@ def _generate_gis_type_sanitization_test_cases() -> list[
_STPoint(long=t[0], lat=t[1]) for t in [(1, 0), (1, 1), (1, 0)]
],
holes=[],
fullFlag=False,
),
{
"geometry": {
@@ -332,6 +338,7 @@ def _generate_gis_type_sanitization_test_cases() -> list[
[_STPoint(long=t[0], lat=t[1]) for t in [(2, 0), (2, 1), (2, 0)]],
[_STPoint(long=t[0], lat=t[1]) for t in [(3, 0), (3, 1), (3, 0)]],
],
fullFlag=False,
),
{
"geometry": {
@@ -352,6 +359,7 @@ def _generate_gis_type_sanitization_test_cases() -> list[
_STPolygon(
exterior=[_STPoint(long=t[0], lat=t[1]) for t in [(1, 0)]],
holes=[],
fullFlag=False,
),
{
"geometry": {
@@ -368,6 +376,7 @@ def _generate_gis_type_sanitization_test_cases() -> list[
_STPolygon(
exterior=[_STPoint(long=t[0], lat=t[1]) for t in [(1, 0), (0, 1)]],
holes=[],
fullFlag=False,
),
{
"geometry": {
@@ -400,7 +409,7 @@ def test_gis_type_sanitization(
@pytest.mark.skipif(not ocient_is_installed(), reason="requires ocient dependencies")
def test_point_list_to_wkt() -> None:
from pyocient import _STPoint
from pyocient.api import STPoint as _STPoint
wkt = _point_list_to_wkt(
[_STPoint(long=t[0], lat=t[1]) for t in [(2, 0), (2, 1), (2, 0)]]
@@ -23,9 +23,11 @@ from unittest import mock
import pytest
from pytest_mock import MockerFixture
from sqlalchemy.engine.url import make_url
from sqlalchemy.engine.url import make_url, URL
from superset.app import SupersetApp
from superset.errors import ErrorLevel, SupersetError, SupersetErrorType
from superset.superset_typing import OAuth2ClientConfig
from superset.utils import json
from tests.unit_tests.db_engine_specs.utils import assert_convert_dttm
from tests.unit_tests.fixtures.common import dttm # noqa: F401
@@ -350,6 +352,30 @@ def test_mask_encrypted_extra() -> None:
)
def test_mask_encrypted_extra_oauth2_client_secret() -> None:
"""
The database-level OAuth2 client secret must be masked in
``masked_encrypted_extra``, matching the other engine specs supporting
the same ``oauth2_client_info`` path (gsheets, trino) -- otherwise a
database editor can read it back unmasked.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
config = json.dumps(
{
"auth_method": "oauth2",
"oauth2_client_info": {"id": "client-id", "secret": "my-secret"},
}
)
assert SnowflakeEngineSpec.mask_encrypted_extra(config) == json.dumps(
{
"auth_method": "oauth2",
"oauth2_client_info": {"id": "client-id", "secret": "XXXXXXXXXX"},
}
)
def test_mask_encrypted_extra_no_fields() -> None:
"""
Test that the private key is masked when the database is edited.
@@ -462,3 +488,278 @@ def test_unmask_encrypted_extra() -> None:
},
}
)
@pytest.fixture
def oauth2_config() -> OAuth2ClientConfig:
"""
Config for Snowflake OAuth2.
"""
return {
"id": "snowflake-oauth2-client-id",
"secret": "snowflake-oauth2-client-secret",
"scope": "refresh_token",
"redirect_uri": "http://localhost:8088/api/v1/database/oauth2/",
"authorization_request_uri": "https://snowflake.oauth2.example/oauth/authorize",
"token_request_uri": "https://snowflake.oauth2.example/oauth/token-request",
"request_content_type": "data",
}
def test_get_oauth2_token(
mocker: MockerFixture,
oauth2_config: OAuth2ClientConfig,
) -> None:
"""
Test `get_oauth2_token`.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
requests: mock.MagicMock = mocker.patch("superset.db_engine_specs.base.requests")
requests.post().json.return_value = {
"access_token": "access-token",
"expires_in": 3600,
"scope": "scope",
"token_type": "Bearer",
"refresh_token": "refresh-token",
}
assert SnowflakeEngineSpec.get_oauth2_token(oauth2_config, "code") == {
"access_token": "access-token",
"expires_in": 3600,
"scope": "scope",
"token_type": "Bearer",
"refresh_token": "refresh-token",
}
requests.post.assert_called_with(
"https://snowflake.oauth2.example/oauth/token-request",
data={
"code": "code",
"client_id": "snowflake-oauth2-client-id",
"client_secret": "snowflake-oauth2-client-secret",
"redirect_uri": "http://localhost:8088/api/v1/database/oauth2/",
"grant_type": "authorization_code",
},
timeout=30.0,
)
def test_impersonate_user(app: SupersetApp, mocker: MockerFixture) -> None:
"""
Test that Snowflake supports user impersonation.
Impersonation only applies within a request context (see
``test_impersonate_user_outside_request_context`` below for the
background-execution case), so these assertions run inside one.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
from superset.models.core import Database
database: Database = Database(sqlalchemy_uri="snowflake://abc")
mocker.patch(
"superset.db_engine_specs.snowflake.SnowflakeEngineSpec.is_oauth2_enabled",
return_value=True,
)
with app.test_request_context("/some/place/"):
assert SnowflakeEngineSpec.impersonate_user(
database=database,
username=None,
user_token=None,
url=make_url("snowflake://user:pass@account/database_name/default"),
engine_kwargs={
"connect_args": {
"validate_default_parameters": True,
},
},
) == (
make_url("snowflake://user:pass@account/database_name/default"),
{"connect_args": {"validate_default_parameters": True}},
)
assert SnowflakeEngineSpec.impersonate_user(
database=database,
username=None,
user_token=None,
url=make_url("snowflake://user:pass@account/database_name/default"),
engine_kwargs={},
) == (
make_url(
"snowflake://user:pass@account/database_name/default?authenticator=oauth"
),
{"connect_args": {"authenticator": "oauth"}},
)
mocker.patch(
"superset.db_engine_specs.snowflake.is_feature_enabled",
return_value=True,
)
mocker.patch(
"superset.security_manager.find_user",
return_value=mocker.MagicMock(email="impersonated_user@example.com"),
)
assert SnowflakeEngineSpec.impersonate_user(
database=database,
username="impersonated_user",
user_token="test_token", # noqa: S106
url=make_url("snowflake://user:pass@account/database_name/default"),
engine_kwargs={},
) == (
make_url(
"snowflake://impersonated_user:pass@account/database_name/default?authenticator=oauth&token=test_token"
),
{"connect_args": {"authenticator": "oauth"}},
)
def test_impersonate_user_email_prefix_uses_username_directly(
app: SupersetApp, mocker: MockerFixture
) -> None:
"""
With IMPERSONATE_WITH_EMAIL_PREFIX enabled, ``Database._get_sqla_engine()``
has already substituted the email prefix for the login username before
calling ``impersonate_user`` -- the value it passes in is no longer a
lookupable login. Re-looking it up as a username (the pre-fix behavior)
fails whenever the login differs from the prefix, silently leaving the
default/service-account username paired with the impersonated user's
OAuth token instead of failing loudly. The fixed code must use the given
value directly and must not call ``find_user`` at all in this branch.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
from superset.models.core import Database
database: Database = Database(sqlalchemy_uri="snowflake://abc")
mocker.patch(
"superset.db_engine_specs.snowflake.SnowflakeEngineSpec.is_oauth2_enabled",
return_value=True,
)
mocker.patch(
"superset.db_engine_specs.snowflake.is_feature_enabled",
return_value=True,
)
find_user = mocker.patch("superset.security_manager.find_user")
with app.test_request_context("/some/place/"):
# "jdoe" is the email prefix Database._get_sqla_engine() already
# derived; the login it derived it from ("jdoe123", say) is gone by
# this point and must not be re-derived here.
result = SnowflakeEngineSpec.impersonate_user(
database=database,
username="jdoe",
user_token="test_token", # noqa: S106
url=make_url("snowflake://user:pass@account/database_name/default"),
engine_kwargs={},
)
assert result == (
make_url(
"snowflake://jdoe:pass@account/database_name/default?authenticator=oauth&token=test_token"
),
{"connect_args": {"authenticator": "oauth"}},
)
find_user.assert_not_called()
def test_impersonate_user_outside_request_context(mocker: MockerFixture) -> None:
"""
Background executions (alerts/reports) have no per-user token, so OAuth
impersonation must not engage outside a request context even when
``database.is_oauth2_enabled()`` returns True because of a
database-level OAuth2 client config, which (unlike the app-config-based
check) isn't itself request-context-aware.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
from superset.models.core import Database
database: Database = Database(sqlalchemy_uri="snowflake://abc")
mocker.patch.object(Database, "is_oauth2_enabled", return_value=True)
url: URL = make_url("snowflake://user:pass@account/database_name/default")
assert SnowflakeEngineSpec.impersonate_user(
database=database,
username=None,
user_token="test_token", # noqa: S106
url=url,
engine_kwargs={},
) == (url, {"connect_args": {}})
def test_custom_snowflake_auth_error_matches_raw_dbapi_exception() -> None:
"""
`BaseEngineSpec.execute()` runs against a bare DBAPI cursor, so the
exception it sees is the raw Snowflake error, never wrapped by
SQLAlchemy. `CustomSnowflakeAuthError` must still recognize it so the
OAuth2 re-auth dance triggers for SQL Lab queries.
"""
from superset.db_engine_specs.snowflake import (
CustomSnowflakeAuthError,
DatabaseError,
)
raw_error: Exception = DatabaseError("250001: Invalid OAuth access token.")
assert isinstance(raw_error, CustomSnowflakeAuthError)
def test_custom_snowflake_auth_error_matches_sqlalchemy_wrapped_exception() -> None:
"""
Some call sites execute through SQLAlchemy's `Engine`, which wraps the
original DBAPI exception in `sqlalchemy.exc.DatabaseError.orig`.
`CustomSnowflakeAuthError` must keep matching this shape too.
"""
from sqlalchemy.exc import DatabaseError as SqlalchemyDatabaseError
from superset.db_engine_specs.snowflake import (
CustomSnowflakeAuthError,
DatabaseError,
)
wrapped_error: SqlalchemyDatabaseError = SqlalchemyDatabaseError(
statement="SELECT 1",
params=None,
orig=DatabaseError("250001: Invalid OAuth access token."),
)
assert isinstance(wrapped_error, CustomSnowflakeAuthError)
def test_custom_snowflake_auth_error_does_not_match_unrelated_errors() -> None:
"""
Other Snowflake DB errors, and non-Snowflake exceptions, must not be
mistaken for an expired OAuth token.
"""
from superset.db_engine_specs.snowflake import (
CustomSnowflakeAuthError,
DatabaseError,
)
assert not isinstance(
DatabaseError("Object FOO does not exist."), CustomSnowflakeAuthError
)
assert not isinstance(
ValueError("Invalid OAuth access token."), CustomSnowflakeAuthError
)
def test_snowflake_oauth2_exception_catches_refresh_token_error() -> None:
"""
`refresh_oauth2_token()` catches failures from the (unoverridden) base
`get_oauth2_fresh_token()` with `except db_engine_spec.oauth2_exception`.
That base method raises `OAuth2TokenRefreshError`, which isn't related to
`CustomSnowflakeAuthError` by real subclassing, so `oauth2_exception` must
include it directly -- an `except` clause never triggers the metaclass's
`__instancecheck__`, unlike `isinstance()`.
"""
from superset.db_engine_specs.snowflake import SnowflakeEngineSpec
from superset.exceptions import OAuth2TokenRefreshError
try:
raise OAuth2TokenRefreshError("refresh token revoked")
except SnowflakeEngineSpec.oauth2_exception:
pass
else:
pytest.fail(
"OAuth2TokenRefreshError must be caught by "
"SnowflakeEngineSpec.oauth2_exception"
)
@@ -22,6 +22,7 @@ from collections.abc import Iterator
from typing import TYPE_CHECKING
import pytest
from flask import current_app
from pytest_mock import MockerFixture
from sqlalchemy import text
from sqlalchemy.engine import create_engine
@@ -239,6 +240,410 @@ def test_superset_joins(
assert list(results) == [(10, "ten"), (20, "twenty")]
@pytest.mark.parametrize(
("statement", "expected"),
[
# A single table reference is not a multi-table statement...
('SELECT * FROM "database1.table1"', 1),
# ...even when it has a dotted, double-quoted column alias, which a
# naive `"[^"]*\.[^"]*"`-shaped regex would also match, misidentifying
# a single-table statement as multi-table and silently skipping
# SUPERSET_META_DB_LIMIT for it.
(
'SELECT COUNT(id) AS "metric.value" FROM "database1.table1"',
1,
),
(
'SELECT t1.b, t2.b FROM "database1.table1" AS t1 '
'JOIN "database2.table2" AS t2 ON t1.a = t2.a',
2,
),
(
'SELECT * FROM "database1.table1", "database2.table2" WHERE t1.a = t2.a',
2,
),
# Statements the parser can't handle fall back to the safe default
# (treat as single-table, so the app-wide limit still applies).
("this is not valid sql (((", 1),
],
)
def test_count_referenced_tables(statement: str, expected: int) -> None:
"""
Regression for a review comment on #42598/#36304: the multi-table
detection used to gate SUPERSET_META_DB_LIMIT must count actual table
references via the real SQL parser, not pattern-match dotted quoted
identifiers, which also matches dotted column aliases.
"""
from superset.extensions.metadb import _count_referenced_tables
assert _count_referenced_tables(statement) == expected
@pytest.fixture
def table1_large(session: Session, database1: "Database") -> Iterator[None]:
with database1.get_sqla_engine() as engine:
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE table1_large (a INTEGER NOT NULL PRIMARY KEY, "
"b INTEGER)"
)
)
conn.execute(
text("INSERT INTO table1_large (a, b) VALUES (1, 10), (2, 20), (3, 30)")
)
db.session.commit()
yield
with engine.begin() as conn:
conn.execute(text("DROP TABLE table1_large"))
db.session.commit()
@pytest.fixture
def table2_late_match(session: Session, database2: "Database") -> Iterator[None]:
with database2.get_sqla_engine() as engine:
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE table2_late_match (a INTEGER NOT NULL PRIMARY KEY, "
"b TEXT)"
)
)
conn.execute(
text("INSERT INTO table2_late_match (a, b) VALUES (3, 'thirty')")
)
db.session.commit()
yield
with engine.begin() as conn:
conn.execute(text("DROP TABLE table2_late_match"))
db.session.commit()
@pytest.fixture
def table2_multi_late_match(session: Session, database2: "Database") -> Iterator[None]:
with database2.get_sqla_engine() as engine:
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE table2_multi_late_match "
"(a INTEGER NOT NULL PRIMARY KEY, b TEXT)"
)
)
conn.execute(
text(
"INSERT INTO table2_multi_late_match (a, b) "
"VALUES (2, 'twenty'), (3, 'thirty')"
)
)
db.session.commit()
yield
with engine.begin() as conn:
conn.execute(text("DROP TABLE table2_multi_late_match"))
db.session.commit()
@pytest.fixture
def table2_fanout_match(session: Session, database2: "Database") -> Iterator[None]:
with database2.get_sqla_engine() as engine:
with engine.begin() as conn:
conn.execute(
text(
"CREATE TABLE table2_fanout_match "
"(id INTEGER NOT NULL PRIMARY KEY, a INTEGER, b TEXT)"
)
)
# `a` is deliberately not unique (unlike table2_late_match, where
# it's the primary key): a single outer row matching on `a=3`
# fans out into two inner rows here, so reading the match
# requires pulling more than one row through the cursor per
# outer probe, instead of a single unique-index lookup.
conn.execute(
text(
"INSERT INTO table2_fanout_match (a, b) "
"VALUES (3, 'thirty-x'), (3, 'thirty-y')"
)
)
db.session.commit()
yield
with engine.begin() as conn:
conn.execute(text("DROP TABLE table2_fanout_match"))
db.session.commit()
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_superset_joins_with_limit_drops_fanout_matches(
mocker: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
app_context: None,
table1_large: None,
table2_fanout_match: None,
) -> None:
"""
Coverage note from review of #42598: the other join regression tests
match on a primary key on both sides, so each inner lookup returns at
most one row. Here `table1_large`'s single match (a=3) fans out into two
rows in `table2_fanout_match`, so satisfying it means pulling more than
one row through the cursor for a single outer probe, rather than a single
unique-index lookup.
"""
monkeypatch.setitem(current_app.config, "DB_SQLA_URI_VALIDATOR", None)
monkeypatch.setitem(current_app.config, "SUPERSET_META_DB_LIMIT", 2)
monkeypatch.setitem(current_app.config, "DATABASE_OAUTH2_CLIENTS", {})
monkeypatch.setitem(current_app.config, "SQLALCHEMY_CUSTOM_PASSWORD_STORE", None)
mocker.patch(
"superset.extensions.metadb.security_manager.raise_for_access",
return_value=None,
)
from flask import g
g.user = mocker.MagicMock()
g.user.is_anonymous = False
try:
engine = create_engine("superset://", future=True)
except Exception as e:
pytest.skip(f"Superset dialect not available: {e}")
with engine.connect() as conn:
results = conn.execute(
text("""
SELECT t1.b, t2.b
FROM "database1.table1_large" AS t1
JOIN "database2.table2_fanout_match" AS t2
ON t1.a = t2.a
ORDER BY t2.b
""")
)
assert list(results) == [(30, "thirty-x"), (30, "thirty-y")]
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_superset_joins_with_limit_multiple_late_matches(
mocker: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
app_context: None,
table1_large: None,
table2_multi_late_match: None,
) -> None:
"""
Diagnostic probe raised in review of #42598: does the per-table
``SUPERSET_META_DB_LIMIT`` skip (keyed on the
``_executing_multi_table_query`` ContextVar) hold for every row of a
multi-row join result, or only the first? ``table1_large`` has two
genuine matches in ``table2_multi_late_match`` (a=2 and a=3), both of
which fall past SUPERSET_META_DB_LIMIT=2 in table1_large's own row
order for a naive per-table truncation.
"""
monkeypatch.setitem(current_app.config, "DB_SQLA_URI_VALIDATOR", None)
monkeypatch.setitem(current_app.config, "SUPERSET_META_DB_LIMIT", 2)
monkeypatch.setitem(current_app.config, "DATABASE_OAUTH2_CLIENTS", {})
monkeypatch.setitem(current_app.config, "SQLALCHEMY_CUSTOM_PASSWORD_STORE", None)
mocker.patch(
"superset.extensions.metadb.security_manager.raise_for_access",
return_value=None,
)
from flask import g
g.user = mocker.MagicMock()
g.user.is_anonymous = False
try:
engine = create_engine("superset://", future=True)
except Exception as e:
pytest.skip(f"Superset dialect not available: {e}")
with engine.connect() as conn:
results = conn.execute(
text("""
SELECT t1.b, t2.b
FROM "database1.table1_large" AS t1
JOIN "database2.table2_multi_late_match" AS t2
ON t1.a = t2.a
ORDER BY t1.b
""")
)
assert list(results) == [(20, "twenty"), (30, "thirty")]
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_superset_joins_with_limit_drops_matches(
mocker: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
app_context: None,
table1_large: None,
table2_late_match: None,
) -> None:
"""
Regression for #36304: SUPERSET_META_DB_LIMIT is applied to each
underlying table independently, before the in-memory join runs. A row
that has a genuine match on the other side of the join but falls past
the per-table limit is silently dropped from the join result, with no
error or truncation warning.
"""
# Use monkeypatch (rather than the `@with_config` decorator) so the
# config overrides are guaranteed to be undone even though this test is
# expected to fail its assertion until the underlying bug is fixed.
# `@with_config` only restores the original values after the wrapped
# test function returns normally, so an assertion failure here would
# otherwise leak SUPERSET_META_DB_LIMIT=2 into later tests.
monkeypatch.setitem(current_app.config, "DB_SQLA_URI_VALIDATOR", None)
monkeypatch.setitem(current_app.config, "SUPERSET_META_DB_LIMIT", 2)
monkeypatch.setitem(current_app.config, "DATABASE_OAUTH2_CLIENTS", {})
monkeypatch.setitem(current_app.config, "SQLALCHEMY_CUSTOM_PASSWORD_STORE", None)
mocker.patch(
"superset.extensions.metadb.security_manager.raise_for_access",
return_value=None,
)
from flask import g
g.user = mocker.MagicMock()
g.user.is_anonymous = False
try:
engine = create_engine("superset://", future=True)
except Exception as e:
# Skip test if superset:// dialect can't be loaded (common in Docker)
pytest.skip(f"Superset dialect not available: {e}")
with engine.connect() as conn:
results = conn.execute(
text("""
SELECT t1.b, t2.b
FROM "database1.table1_large" AS t1
JOIN "database2.table2_late_match" AS t2
ON t1.a = t2.a
""")
)
# table2_late_match's only row (a=3) has a genuine match in
# table1_large (a=3, b=30), but SUPERSET_META_DB_LIMIT=2 truncates
# table1_large to its first two rows (a=1, a=2) before the join
# runs, so the join comes back empty instead of finding the match.
assert list(results) == [(30, "thirty")]
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_superset_comma_join_with_limit_drops_matches(
mocker: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
app_context: None,
table1_large: None,
table2_late_match: None,
) -> None:
"""
Regression for #36304: an implicit comma join (``FROM a, b WHERE ...``)
references two tables just like an explicit ``JOIN``, but doesn't contain
the literal `JOIN` keyword. Multi-table detection has to catch this shape
too, or the per-table limit still gets applied and silently drops matches.
"""
# See test_superset_joins_with_limit_drops_matches for why monkeypatch is
# used here instead of `@with_config`.
monkeypatch.setitem(current_app.config, "DB_SQLA_URI_VALIDATOR", None)
monkeypatch.setitem(current_app.config, "SUPERSET_META_DB_LIMIT", 2)
monkeypatch.setitem(current_app.config, "DATABASE_OAUTH2_CLIENTS", {})
monkeypatch.setitem(current_app.config, "SQLALCHEMY_CUSTOM_PASSWORD_STORE", None)
mocker.patch(
"superset.extensions.metadb.security_manager.raise_for_access",
return_value=None,
)
from flask import g
g.user = mocker.MagicMock()
g.user.is_anonymous = False
try:
engine = create_engine("superset://", future=True)
except Exception as e:
# Skip test if superset:// dialect can't be loaded (common in Docker)
pytest.skip(f"Superset dialect not available: {e}")
with engine.connect() as conn:
results = conn.execute(
text("""
SELECT t1.b, t2.b
FROM "database1.table1_large" AS t1, "database2.table2_late_match" AS t2
WHERE t1.a = t2.a
""")
)
# Same scenario as test_superset_joins_with_limit_drops_matches, but
# using a comma join instead of the `JOIN` keyword.
assert list(results) == [(30, "thirty")]
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_superset_joins_via_raw_cursor_drops_matches(
mocker: MockerFixture,
monkeypatch: pytest.MonkeyPatch,
app_context: None,
table1_large: None,
table2_late_match: None,
) -> None:
"""
Regression for #36304: SQL Lab executes statements through a raw DBAPI
cursor (``engine.raw_connection().cursor()``), not through SQLAlchemy's
``Connection.execute()``. That path never reaches
``SupersetAPSWDialect.do_execute*``, so a fix keyed only on those hooks
leaves the per-table ``SUPERSET_META_DB_LIMIT`` skip blind to exactly the
statements SQL Lab runs, and the same join match SQL Lab users see would
still be silently dropped even though
``test_superset_joins_with_limit_drops_matches`` (which goes through
``Connection.execute()``) passes.
"""
monkeypatch.setitem(current_app.config, "DB_SQLA_URI_VALIDATOR", None)
monkeypatch.setitem(current_app.config, "SUPERSET_META_DB_LIMIT", 2)
monkeypatch.setitem(current_app.config, "DATABASE_OAUTH2_CLIENTS", {})
monkeypatch.setitem(current_app.config, "SQLALCHEMY_CUSTOM_PASSWORD_STORE", None)
mocker.patch(
"superset.extensions.metadb.security_manager.raise_for_access",
return_value=None,
)
from flask import g
g.user = mocker.MagicMock()
g.user.is_anonymous = False
try:
engine = create_engine("superset://", future=True)
except Exception as e:
# Skip test if superset:// dialect can't be loaded (common in Docker)
pytest.skip(f"Superset dialect not available: {e}")
raw_connection = engine.raw_connection()
try:
cursor = raw_connection.cursor()
cursor.execute(
"""
SELECT t1.b, t2.b
FROM "database1.table1_large" AS t1
JOIN "database2.table2_late_match" AS t2
ON t1.a = t2.a
"""
)
# Same scenario as test_superset_joins_with_limit_drops_matches, but
# executed the way SQL Lab actually runs queries: a raw DBAPI cursor
# obtained from `engine.raw_connection()`, bypassing `do_execute*`.
assert list(cursor) == [(30, "thirty")]
finally:
raw_connection.close()
@with_feature_flags(ENABLE_SUPERSET_META_DB=True)
def test_dml(
mocker: MockerFixture,
@@ -258,6 +258,19 @@ def test_merge_form_data_filters_into_query_applies_regular_overrides():
assert query["having"] == "(SUM(num) > 10) AND (COUNT(*) > 1)"
def test_filter_helpers_copy_relative_time_extras():
"""Relative time anchors reach both saved and freshly built queries."""
extras = {"relative_start": "now", "relative_end": "today"}
fresh_query = {"extras": {"where": "country = 'US'"}}
apply_form_data_filters_to_query(fresh_query, {"extras": extras})
assert fresh_query["extras"] == {"where": "country = 'US'", **extras}
saved_query = {"extras": {"having": "COUNT(*) > 1"}}
merge_form_data_filters_into_query(saved_query, {"extras": extras})
assert saved_query["extras"] == {"having": "COUNT(*) > 1", **extras}
def test_merge_extra_form_data_filters_into_query_adds_only_extra_predicates(
monkeypatch,
):
@@ -479,6 +479,25 @@ class TestMapTableConfig:
assert result["row_limit"] == 500
def test_map_table_config_supports_null_filter(self) -> None:
config = TableChartConfig(
chart_type="table",
columns=[ColumnRef(name="optional_value")],
filters=[FilterConfig(column="optional_value", op="IS NOT NULL")],
)
result = map_table_config(config)
assert result["adhoc_filters"] == [
{
"clause": "WHERE",
"expressionType": "SIMPLE",
"subject": "optional_value",
"operator": "IS NOT NULL",
"comparator": None,
}
]
def test_map_table_config_default_row_limit(self) -> None:
"""Test that default row_limit is mapped to form_data."""
config = TableChartConfig(
@@ -686,6 +705,23 @@ class TestMapXYConfig:
assert result["show_legend"] is False
assert result["legendOrientation"] == "top"
def test_map_xy_config_with_legend_orientation(self) -> None:
config = XYChartConfig.model_validate(
{
"chart_type": "xy",
"x": {"name": "date"},
"y": [{"name": "revenue", "aggregate": "SUM"}],
"show_legend": True,
"legend_orientation": "bottom",
}
)
result = map_xy_config(config)
assert config.legend is not None
assert config.legend.show is True
assert result["legendOrientation"] == "bottom"
def test_map_xy_config_with_color_scheme(self) -> None:
"""color_scheme propagates to form_data when set."""
config = XYChartConfig(
@@ -188,6 +188,13 @@ class TestGenerateChart:
for i, f in enumerate(filters):
assert f.op == operators[i]
null_filter = FilterConfig(column="optional_value", op="IS NOT NULL")
assert null_filter.value is None
with pytest.raises(ValueError, match="must not have 'value'"):
FilterConfig(column="optional_value", op="IS NULL", value="unexpected")
with pytest.raises(ValueError, match="requires 'value'"):
FilterConfig(column="optional_value", op="=")
@pytest.mark.asyncio
async def test_generate_chart_response_structure(self):
"""Test the expected response structure for chart generation."""
@@ -306,6 +313,10 @@ class TestGenerateChart:
assert col2.aggregate == "SUM"
assert col2.label == "Total Sales"
aliased = ColumnRef.model_validate({"column": "sales", "aggregate": "AVG"})
assert aliased.name == "sales"
assert aliased.aggregate == "AVG"
# All supported aggregations
aggs = ["SUM", "AVG", "COUNT", "MIN", "MAX", "COUNT_DISTINCT"]
for agg in aggs:
@@ -41,10 +41,52 @@ from superset.mcp_service.chart.tool.get_chart_data import (
_MAX_RECOMMENDATIONS,
_query_from_form_data,
_recommend_visualizations,
_rejected_requested_filter_columns,
_requested_filter_columns,
)
from superset.utils import json
from superset.utils.core import GenericDataType
def test_requested_filter_columns_supports_both_payload_shapes() -> None:
assert _requested_filter_columns(
{
"filters": [{"col": "country", "op": "==", "val": "USA"}],
"adhoc_filters": [
{
"expressionType": "SIMPLE",
"subject": "city",
"operator": "==",
"comparator": "New York",
},
{"expressionType": "SQL", "sqlExpression": "revenue > 0"},
],
}
) == {"country", "city"}
def test_rejected_requested_filter_columns_ignores_saved_chart_filters() -> None:
result = {
"queries": [
{"rejected_filter_columns": ["missing_request", "stale_saved_filter"]}
]
}
assert _rejected_requested_filter_columns(
result,
{
"adhoc_filters": [
{
"expressionType": "SIMPLE",
"subject": "missing_request",
"operator": "==",
"comparator": "value",
}
]
},
) == ["missing_request"]
def _collect_groupby_extras(
form_data: dict[str, Any],
groupby_columns: list[str],
@@ -1349,6 +1391,203 @@ class TestChartLookupEagerLoading:
assert _extract_metrics_load_path(query_options[0]) == ["table", "metrics"]
class TestSavedChartExtraFormDataFilters:
"""Regression tests: extra_form_data filters passed alongside a saved
chart identifier must reach the executed query, not just the cached
form_data / unsaved-chart path already covered elsewhere.
A chart with a saved query_context is the common case (any chart that
has been opened and saved through Explore), so this is the primary path
exercised when a caller passes extra_form_data with a chart identifier.
"""
def _chart(self) -> SimpleNamespace:
from superset.utils import json as utils_json
return SimpleNamespace(
id=9,
slice_name="Sales",
viz_type="table",
datasource_id=1,
datasource_type="table",
query_context=utils_json.dumps(
{
"datasource": {"id": 1, "type": "table"},
"queries": [
{
"columns": ["country"],
"metrics": ["count"],
"filters": [],
"row_limit": 100,
}
],
"result_format": "json",
"result_type": "full",
}
),
params=None,
)
async def _run(
self,
extra_form_data: dict[str, Any],
mcp_server: Any,
rejected_filter_columns: list[str] | None = None,
) -> tuple[Any, Any]:
from unittest.mock import patch
from fastmcp import Client
module = importlib.import_module(
"superset.mcp_service.chart.tool.get_chart_data"
)
captured: dict[str, Any] = {}
def fake_load(self: Any, data: dict[str, Any]) -> Any:
captured["loaded_query_context_json"] = data
# Mirror the QueryContext/QueryObject surface the tool relies on
# (set_query_context_form_data serializes every query object).
queries = [
SimpleNamespace(
filter=query.get("filters", []),
time_range=query.get("time_range"),
to_dict=lambda query=query: dict(query),
)
for query in data.get("queries", [])
]
return SimpleNamespace(queries=queries, form_data=data.get("form_data", {}))
class _Command:
def __init__(self, query_context: Any) -> None: ...
def validate(self) -> None: ...
def run(self) -> dict[str, Any]:
return {
"queries": [
{
"data": [{"country": "USA"}],
"colnames": ["country"],
"rowcount": 1,
"rejected_filter_columns": rejected_filter_columns or [],
}
]
}
with (
patch.object(
module, "find_chart_by_identifier", return_value=self._chart()
),
patch.object(
module,
"validate_chart_dataset",
return_value=SimpleNamespace(is_valid=True, warnings=[], error=None),
),
patch(
"superset.commands.chart.data.get_data_command.ChartDataCommand",
_Command,
),
patch(
"superset.charts.schemas.ChartDataQueryContextSchema.load",
fake_load,
),
):
async with Client(mcp_server) as client:
tool_result = await client.call_tool(
"get_chart_data",
{
"request": {
"identifier": "9",
"extra_form_data": extra_form_data,
}
},
)
return captured["loaded_query_context_json"], tool_result
@pytest.mark.asyncio
async def test_filters_key_reaches_executed_query(
self, mcp_server: Any, mock_auth: Any
) -> None:
"""extra_form_data using the native 'filters' format is applied."""
loaded, _ = await self._run(
{"filters": [{"col": "country", "op": "==", "val": "USA"}]}, mcp_server
)
filters = loaded["queries"][0].get("filters", [])
assert {"col": "country", "op": "==", "val": "USA"} in filters
@pytest.mark.asyncio
async def test_adhoc_filters_key_reaches_executed_query(
self, mcp_server: Any, mock_auth: Any
) -> None:
"""extra_form_data using the 'adhoc_filters' format is also applied."""
loaded, _ = await self._run(
{
"adhoc_filters": [
{
"clause": "WHERE",
"expressionType": "SIMPLE",
"subject": "country",
"operator": "==",
"comparator": "USA",
}
]
},
mcp_server,
)
filters = loaded["queries"][0].get("filters", [])
assert {"col": "country", "op": "==", "val": "USA"} in filters
@pytest.mark.asyncio
async def test_temporal_range_filter_reaches_executed_query(
self, mcp_server: Any, mock_auth: Any
) -> None:
"""A TEMPORAL_RANGE filter narrows the query, not just simple filters."""
loaded, _ = await self._run(
{
"filters": [
{
"col": "order_date",
"op": "TEMPORAL_RANGE",
"val": "2024-01-01 : 2024-02-01",
}
]
},
mcp_server,
)
filters = loaded["queries"][0].get("filters", [])
assert {
"col": "order_date",
"op": "TEMPORAL_RANGE",
"val": "2024-01-01 : 2024-02-01",
} in filters
@pytest.mark.asyncio
async def test_unknown_adhoc_filter_column_returns_validation_error(
self, mcp_server: Any, mock_auth: Any
) -> None:
"""A rejected request filter must not return plausible unfiltered data."""
_, result = await self._run(
{
"adhoc_filters": [
{
"clause": "WHERE",
"expressionType": "SIMPLE",
"subject": "does_not_exist",
"operator": "==",
"comparator": "value",
}
]
},
mcp_server,
rejected_filter_columns=["does_not_exist"],
)
data = json.loads(result.content[0].text)
assert data["error_type"] == "ValidationError"
assert "does_not_exist" in data["error"]
assert "USA" not in result.content[0].text
class TestOAuthErrorRouting:
"""Query-time OAuth errors must reach the dedicated OAuth handlers.
@@ -86,6 +86,26 @@ class TestGetChartSqlRequestSchema:
with pytest.raises(ValueError, match="At least one of"):
GetChartSqlRequest()
def test_extra_form_data_defaults_to_none(self):
"""extra_form_data is optional and defaults to None."""
request = GetChartSqlRequest(identifier=123)
assert request.extra_form_data is None
def test_extra_form_data_is_accepted(self):
"""extra_form_data is a real field, not silently dropped.
Regression test: previously GetChartSqlRequest had no extra_form_data
field at all, so callers passing filters got no error and no effect
get_chart_sql always rendered the chart's unfiltered baseline SQL.
"""
request = GetChartSqlRequest(
identifier=123,
extra_form_data={"filters": [{"col": "country", "op": "==", "val": "USA"}]},
)
assert request.extra_form_data == {
"filters": [{"col": "country", "op": "==", "val": "USA"}]
}
class TestExtractSqlFromResult:
"""Tests for the _extract_sql_from_result helper."""
@@ -463,6 +483,46 @@ class TestBuildQueryContextFromFormData:
assert queries[0]["metrics"] == ["sum_revenue"]
assert queries[0]["columns"] == ["product"]
@patch("superset.common.query_context_factory.QueryContextFactory")
@patch("superset.daos.datasource.DatasourceDAO.get_datasource")
def test_extra_form_data_merged_into_query(self, mock_get_ds, mock_factory_cls):
"""extra_form_data (e.g. dashboard-style filters) reaches the rendered
query, not just chart.params.
Regression test: _build_query_context_from_form_data previously never
forwarded its caller's extra_form_data to build_query_context_from_form_data,
so get_chart_sql could not preview SQL with request-supplied filters applied.
"""
mock_ds = Mock()
mock_ds.database.db_engine_spec.engine = "postgresql"
mock_get_ds.return_value = mock_ds
mock_factory = Mock()
mock_factory.create.return_value = Mock()
mock_factory_cls.return_value = mock_factory
form_data = {
"datasource_id": 1,
"datasource_type": "table",
"metrics": ["count"],
"groupby": ["country"],
}
extra_form_data = {"filters": [{"col": "country", "op": "==", "val": "USA"}]}
with patch(
"superset.common.chart_data.ChartDataResultType"
) as mock_result_type:
mock_result_type.QUERY = "QUERY"
_build_query_context_from_form_data(
form_data, chart=None, extra_form_data=extra_form_data
)
call_kwargs = mock_factory.create.call_args[1]
queries = call_kwargs["queries"]
assert len(queries) == 1
filters = queries[0].get("filters", [])
assert {"col": "country", "op": "==", "val": "USA"} in filters
class TestExtractXAxisCol:
"""Tests for the _extract_x_axis_col helper."""
@@ -990,6 +1050,146 @@ class TestResolveDatasourceName:
assert result == "combined_dataset"
def _run_sql_from_saved_query_context(extra_form_data, datasource=None):
"""Call _sql_from_saved_query_context with schema.load/ChartDataCommand
stubbed, and return the raw query_context_json dict that was handed to
ChartDataQueryContextSchema.load."""
from superset.mcp_service.chart.tool.get_chart_sql import (
_sql_from_saved_query_context,
)
from superset.utils import json as _json
chart = Mock()
chart.id = 10
chart.slice_name = "Sales"
chart.datasource_name = "sales"
chart.datasource_id = 7
chart.datasource_type = "query"
chart.query_context = _json.dumps(
{
"datasource": (
{"id": 1, "type": "table"} if datasource is None else datasource
),
"queries": [{"columns": ["country"], "metrics": ["count"], "filters": []}],
}
)
captured = {}
def fake_load(self, data):
captured["query_context_json"] = data
fake_qc = Mock()
fake_qc.result_type = None
return fake_qc
class _Command:
def __init__(self, query_context):
pass
def validate(self):
pass
def run(self):
return {"queries": [{"query": "SELECT * FROM sales", "language": "sql"}]}
with (
patch(
"superset.charts.schemas.ChartDataQueryContextSchema.load",
fake_load,
),
patch(
"superset.commands.chart.data.get_data_command.ChartDataCommand",
_Command,
),
patch(
"superset.mcp_service.chart.tool.get_chart_sql.set_query_context_form_data"
) as mock_set_form_data,
):
_sql_from_saved_query_context(chart, extra_form_data=extra_form_data)
captured["query_context_json"]["_set_form_data_args"] = mock_set_form_data.call_args
return captured["query_context_json"]
class TestSqlFromSavedQueryContextExtraFormData:
"""Regression tests: extra_form_data must reach the query built from a
chart's saved query_context, not just the request-supplied form_data."""
def test_real_column_filter_via_filters_key(self):
"""A filter on a real column, using the native 'filters' format,
ends up in the query handed to ChartDataQueryContextSchema.load."""
query_context_json = _run_sql_from_saved_query_context(
extra_form_data={"filters": [{"col": "country", "op": "==", "val": "USA"}]}
)
filters = query_context_json["queries"][0].get("filters", [])
assert {"col": "country", "op": "==", "val": "USA"} in filters
assert query_context_json["_set_form_data_args"].args[1:] == (1, "table")
def test_real_column_filter_via_adhoc_filters_key(self):
"""A filter on a real column, using the Explore 'adhoc_filters'
format, ends up in the query handed to
ChartDataQueryContextSchema.load too."""
query_context_json = _run_sql_from_saved_query_context(
extra_form_data={
"adhoc_filters": [
{
"clause": "WHERE",
"expressionType": "SIMPLE",
"subject": "country",
"operator": "==",
"comparator": "USA",
}
]
}
)
filters = query_context_json["queries"][0].get("filters", [])
assert {"col": "country", "op": "==", "val": "USA"} in filters
def test_no_extra_form_data_leaves_query_unchanged(self):
"""Without extra_form_data, the saved query_context is used as-is."""
query_context_json = _run_sql_from_saved_query_context(extra_form_data=None)
assert query_context_json["queries"][0]["filters"] == []
def test_datasource_without_type_falls_back_to_the_chart(self):
"""ChartDataDatasourceSchema only requires 'id', so a saved context
that omits 'type' is valid and must still render SQL."""
query_context_json = _run_sql_from_saved_query_context(
extra_form_data=None, datasource={"id": 1}
)
# id comes from the saved context; the missing type comes from the chart
assert query_context_json["_set_form_data_args"].args[1:] == (1, "query")
def test_schema_validation_failure_uses_form_data_fallback(self, caplog):
"""A stale saved query context must not prevent form_data fallback."""
from marshmallow import ValidationError
from superset.mcp_service.chart.tool.get_chart_sql import (
_sql_from_saved_query_context,
)
from superset.utils import json as _json
chart = Mock(
id=42,
query_context=_json.dumps(
{
"datasource": {"id": 1, "type": "table"},
"queries": [{}],
}
),
)
with patch(
"superset.charts.schemas.ChartDataQueryContextSchema.load",
side_effect=ValidationError("stale query context"),
):
assert _sql_from_saved_query_context(chart) is None
assert "stale query context" in caplog.text
class TestGetChartSqlTool:
"""Integration-style tests for the get_chart_sql MCP tool via Client."""
@@ -1094,6 +1294,61 @@ class TestGetChartSqlTool:
assert "SELECT COUNT(*) FROM sales" in data["sql"]
assert data["chart_id"] == 10
@patch.object(_get_chart_sql_mod, "_sql_from_form_data")
@patch.object(_get_chart_sql_mod, "_sql_from_saved_query_context")
@patch.object(_get_chart_sql_mod, "_resolve_effective_form_data")
@patch.object(_get_chart_sql_mod, "validate_chart_dataset")
@patch.object(_get_chart_sql_mod, "_find_chart_by_identifier")
@pytest.mark.asyncio
async def test_extra_form_data_reaches_saved_query_context_builder(
self,
mock_find,
mock_validate,
mock_resolve,
mock_saved_qc,
mock_form_data_sql,
mcp_server,
):
"""Regression test: request.extra_form_data must be forwarded to the
saved-query_context SQL builder, not silently dropped by the request
schema or ignored on the way to the builder call."""
from fastmcp import Client
from superset.mcp_service.chart.chart_utils import (
DatasetValidationResult,
)
mock_chart = Mock()
mock_chart.id = 11
mock_chart.slice_name = "Sales Chart"
mock_chart.viz_type = "table"
mock_find.return_value = mock_chart
mock_validate.return_value = DatasetValidationResult(
is_valid=True, dataset_id=1, dataset_name="ds", warnings=[]
)
mock_resolve.return_value = ({"metrics": ["count"]}, False)
mock_saved_qc.return_value = ChartSql(
chart_id=11,
chart_name="Sales Chart",
sql="SELECT COUNT(*) FROM sales WHERE country = 'USA'",
language="sql",
datasource_name="sales",
)
extra_form_data = {"filters": [{"col": "country", "op": "==", "val": "USA"}]}
async with Client(mcp_server) as client:
result = await client.call_tool(
"get_chart_sql",
{"request": {"identifier": 11, "extra_form_data": extra_form_data}},
)
data = result.structured_content.get("result", result.structured_content)
assert "WHERE country = 'USA'" in data["sql"]
mock_saved_qc.assert_called_once_with(mock_chart, extra_form_data)
@patch.object(_get_chart_sql_mod, "_sql_from_form_data")
@patch.object(_get_chart_sql_mod, "_sql_from_saved_query_context")
@patch.object(_get_chart_sql_mod, "_resolve_effective_form_data")
@@ -1177,3 +1432,123 @@ class TestGetChartSqlTool:
data = result.structured_content.get("result", result.structured_content)
assert data["error_type"] == "DatasetNotAccessible"
assert "Access denied" in data["error"]
@patch.object(_get_chart_sql_mod, "_sql_from_form_data")
@patch.object(_get_chart_sql_mod, "_get_cached_form_data")
@pytest.mark.asyncio
async def test_unsaved_chart_extra_form_data_reaches_sql_builder(
self, mock_cached, mock_form_data_sql, mcp_server
):
"""Regression test: extra_form_data must reach the SQL builder on the
form_data_key-only (unsaved chart) path too, not just the saved-chart
paths."""
from fastmcp import Client
from superset.utils import json as _json
cached_form_data = {"datasource_id": 1, "datasource_type": "table"}
mock_cached.return_value = _json.dumps(cached_form_data)
mock_form_data_sql.return_value = ChartSql(
chart_id=0,
chart_name=None,
sql="SELECT * FROM sales WHERE country = 'USA'",
language="sql",
datasource_name="sales",
)
extra_form_data = {"filters": [{"col": "country", "op": "==", "val": "USA"}]}
async with Client(mcp_server) as client:
result = await client.call_tool(
"get_chart_sql",
{
"request": {
"form_data_key": "cached-key",
"extra_form_data": extra_form_data,
}
},
)
data = result.structured_content.get("result", result.structured_content)
assert "WHERE country = 'USA'" in data["sql"]
mock_form_data_sql.assert_called_once_with(
cached_form_data, chart=None, extra_form_data=extra_form_data
)
@patch.object(_get_chart_sql_mod, "validate_chart_dataset")
@patch.object(_get_chart_sql_mod, "_find_chart_by_identifier")
@pytest.mark.asyncio
async def test_malformed_extra_form_data_filter_returns_clean_error(
self, mock_find, mock_validate, mcp_server
):
"""A malformed extra_form_data filter (missing 'op') must return a
structured ChartError, not crash with an unhandled KeyError.
Regression test: merge_extra_form_data_filters_into_query normalizes
filters via simple_filter_to_adhoc, which raises KeyError on a filter
entry missing "col" or "op". That KeyError previously propagated out
of get_chart_sql uncaught.
"""
from fastmcp import Client
from superset.mcp_service.chart.chart_utils import DatasetValidationResult
from superset.utils import json as _json
mock_chart = Mock()
mock_chart.id = 40
mock_chart.slice_name = "Sales"
mock_chart.viz_type = "table"
mock_chart.query_context = _json.dumps(
{
"datasource": {"id": 1, "type": "table"},
"queries": [
{"columns": ["country"], "metrics": ["count"], "filters": []}
],
}
)
mock_find.return_value = mock_chart
mock_validate.return_value = DatasetValidationResult(
is_valid=True, dataset_id=1, dataset_name="ds", warnings=[]
)
async with Client(mcp_server) as client:
result = await client.call_tool(
"get_chart_sql",
{
"request": {
"identifier": 40,
# missing "op" — malformed filter entry
"extra_form_data": {"filters": [{"col": "country"}]},
}
},
)
data = result.structured_content.get("result", result.structured_content)
assert data["error_type"] == "ValidationError"
# The saved query_context path names the offending input directly
# instead of deferring to the form_data fallback's generic message.
assert "Invalid extra_form_data filter" in data["error"]
def test_stale_query_context_falls_back_instead_of_erroring(self):
"""A saved query_context missing "datasource" is stale, not bad input.
Filter merging needs the datasource id/type, so it cannot run. That must
hand control back to the caller (return None) so the SQL is rebuilt from
the chart's form_data — not surface a filter ValidationError.
"""
from superset.utils import json as _json
mock_chart = Mock()
mock_chart.id = 41
mock_chart.query_context = _json.dumps(
{"queries": [{"columns": ["country"], "filters": []}]}
)
result = _get_chart_sql_mod._sql_from_saved_query_context(
mock_chart,
{"filters": [{"col": "country", "op": "==", "val": "USA"}]},
)
assert result is None
@@ -0,0 +1,74 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import pytest
from superset.mcp_service.chart.schemas import ColumnRef
from superset.mcp_service.chart.validation.dataset_validator import DatasetValidator
from superset.mcp_service.common.error_schemas import (
ChartGenerationError,
DatasetContext,
)
def _validate_sum(sql_type: str) -> list[ChartGenerationError]:
context = DatasetContext(
id=69,
table_name="virtual_metrics",
schema=None,
database_name="database",
available_columns=[
{"name": "computed_total", "type": sql_type, "is_numeric": False}
],
available_metrics=[],
)
return DatasetValidator._validate_aggregations(
[ColumnRef(name="computed_total", aggregate="SUM")], context
)
@pytest.mark.parametrize(
"sql_type",
[
"BIGINT",
"SMALLINT",
"TINYINT",
"REAL",
"NUMBER",
"DOUBLE PRECISION",
"INT8",
"FLOAT8",
"DECIMAL(10, 2)",
"MONEY",
"SMALLMONEY",
],
)
def test_numeric_type_spelling_is_accepted(sql_type: str) -> None:
assert _validate_sum(sql_type) == []
@pytest.mark.parametrize("sql_type", ["", "UNKNOWN"])
def test_unknown_type_is_deferred_to_compile_check(sql_type: str) -> None:
assert _validate_sum(sql_type) == []
@pytest.mark.parametrize("sql_type", ["VARCHAR", "INTERVAL", "POINT"])
def test_non_numeric_type_is_rejected_for_numeric_aggregation(
sql_type: str,
) -> None:
assert _validate_sum(sql_type)[0].error_type == "invalid_aggregation"
+105 -666
View File
@@ -36,7 +36,6 @@ from superset.extensions import appbuilder
from superset.models.slice import Slice
from superset.security.manager import (
_collect_sortable_identifiers,
_sql_filters_modified,
freeze_value,
query_context_modified,
SupersetSecurityManager,
@@ -3794,681 +3793,121 @@ def test_validate_guest_token_resources_accepts_embedded_int_id(
)
# ---------------------------------------------------------------------------
# _sql_filters_modified block custom SQL injection by guest users
# ---------------------------------------------------------------------------
def test_sql_filters_extras_where_injected_blocked(
mocker: MockerFixture,
) -> None:
"""Injecting extras.where when the chart has no SQL filters is blocked."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"metrics": ["count"]}
query = QueryObject(extras={"where": "1=1"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_extras_having_injected_blocked(
mocker: MockerFixture,
) -> None:
"""Injecting extras.having when the chart has no SQL filters is blocked."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(extras={"having": "COUNT(*) > 0"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_extras_where_replay_allowed(
mocker: MockerFixture,
) -> None:
"""Replaying the chart's own SQL WHERE filter is allowed."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "region = 'EMEA'",
"clause": "WHERE",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
# freeform_where_having wraps each clause in parens
query = QueryObject(extras={"where": "(region = 'EMEA')"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_extras_having_replay_allowed(
mocker: MockerFixture,
) -> None:
"""Replaying the chart's own SQL HAVING filter is allowed."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "SUM(sales) > 100",
"clause": "HAVING",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
query = QueryObject(extras={"having": "(SUM(sales) > 100)"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_adhoc_sql_filter_injected_blocked(
mocker: MockerFixture,
) -> None:
"""Injecting a new SQL adhoc filter not on the stored chart is blocked."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject()
query_context.queries = [query]
injected_filter = {
"expressionType": "SQL",
"sqlExpression": "1=1",
"clause": "WHERE",
}
form_data: dict[str, Any] = {"slice_id": 1, "adhoc_filters": [injected_filter]}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_adhoc_sql_filter_replay_allowed(
mocker: MockerFixture,
) -> None:
"""Replaying the exact stored SQL adhoc filter is allowed."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "region = 'EMEA'",
"clause": "WHERE",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
query = QueryObject()
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1, "adhoc_filters": [sql_filter]}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_empty_extras_always_allowed(
mocker: MockerFixture,
) -> None:
"""No SQL in extras is always allowed, even when the chart has SQL filters."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "region = 'EMEA'",
"clause": "WHERE",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
query = QueryObject()
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_from_stored_qc_allowed(
mocker: MockerFixture,
) -> None:
"""extras.where from stored query_context is allowed."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
stored_qc = {
"queries": [{"extras": {"where": "(col > 5)"}}],
}
query = QueryObject(extras={"where": "(col > 5)"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, stored_qc)
def test_sql_filters_multi_query_stored_predicate_allowed(
mocker: MockerFixture,
) -> None:
"""Multiple queries replaying predicates from the stored chart are allowed.
The allowed set is global across all stored queries per-query pinning is
intentionally not applied because there is no stable identity linking a
request query to a stored query, and all queries share the same
chart/datasource so predicates only restrict rows, never expand access.
def test_is_editor_query_owner(mocker: MockerFixture, app_context: None) -> None:
"""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
stored_qc = {
"queries": [
{"extras": {"where": "(region = 'EMEA')"}},
{"extras": {"where": "(status = 'active')"}},
],
}
# Both request queries use predicates from the stored chart.
query_context.queries = [
QueryObject(extras={"where": "(region = 'EMEA')"}),
QueryObject(extras={"where": "(status = 'active')"}),
]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, stored_qc)
def test_sql_filters_multi_query_novel_predicate_blocked(
mocker: MockerFixture,
) -> None:
"""A novel predicate on any query is blocked even when others are valid."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
stored_qc = {
"queries": [{"extras": {"where": "(region = 'EMEA')"}}],
}
query_context.queries = [
QueryObject(extras={"where": "(region = 'EMEA')"}),
QueryObject(extras={"where": "(1=1)"}), # not stored
]
form_data: dict[str, Any] = {"slice_id": 1}
assert _sql_filters_modified(query_context, form_data, stored_chart, stored_qc)
def test_sql_filters_different_sql_blocked(
mocker: MockerFixture,
) -> None:
"""Modified SQL (appending extra predicates) is blocked."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "col > 5",
"clause": "WHERE",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
# Attacker appends extra predicate
query = QueryObject(
extras={"where": "(col > 5) AND (1=1)"},
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_simple_filters_not_blocked(
mocker: MockerFixture,
) -> None:
"""SIMPLE structured filters (from dashboard native filters) are not blocked."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(
filters=[{"col": "country", "op": "==", "val": "US"}],
)
query_context.queries = [query]
simple_adhoc_filter = {
"expressionType": "SIMPLE",
"subject": "country",
"operator": "==",
"comparator": "US",
"clause": "WHERE",
}
form_data: dict[str, Any] = {
"slice_id": 1,
"adhoc_filters": [simple_adhoc_filter],
}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_structured_filter_adhoc_col_blocked(
mocker: MockerFixture,
) -> None:
"""Structured filter with an adhoc SQL column in ``col`` is blocked.
``ChartDataFilterSchema.col`` is ``fields.Raw``, so an attacker can pass
an adhoc column dict that reaches ``adhoc_column_to_sqla`` and executes
arbitrary SQL in the WHERE clause.
Test that a Query owner is considered an editor via Subject resolution.
"""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
from superset.models.sql_lab import Query
adhoc_col: Any = {
"expressionType": "SQL",
"sqlExpression": "1; DROP TABLE users--",
"label": "x",
}
query = QueryObject(
filters=[{"col": adhoc_col, "op": "!=", "val": "z"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_structured_filter_stored_adhoc_col_allowed(
mocker: MockerFixture,
) -> None:
"""Cross-filter with an adhoc SQL column matching a stored chart dimension
is allowed."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {
"columns": [
{"sqlExpression": "YEAR(order_date)", "label": "order_year"},
],
}
adhoc_col: Any = {
"sqlExpression": "YEAR(order_date)",
"label": "order_year",
}
query = QueryObject(
filters=[{"col": adhoc_col, "op": "==", "val": "2024"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_cross_filter_adhoc_col_from_sibling_chart_allowed(
mocker: MockerFixture,
) -> None:
"""Cross-filter with an adhoc SQL column from a sibling chart on the same
dashboard is allowed."""
from superset.models.dashboard import Dashboard
# Target chart (chart B) has no custom SQL columns.
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.id = 2
stored_chart.params_dict = {"metrics": ["count"]}
# Source chart (chart A) has the custom SQL dimension.
sibling_chart = mocker.MagicMock()
sibling_chart.id = 1
sibling_chart.params_dict = {
"columns": [
{"sqlExpression": "YEAR(order_date)", "label": "order_year"},
],
}
# Dashboard contains both charts.
dashboard = mocker.MagicMock(spec=Dashboard)
dashboard.slices = [sibling_chart, stored_chart]
mocker.patch("superset.db.session.query")
db_query = mocker.patch("superset.db.session.query").return_value
db_query.filter.return_value.one_or_none.return_value = dashboard
mocker.patch(
"superset.security_manager.has_guest_access",
return_value=True,
)
adhoc_col: Any = {
"sqlExpression": "YEAR(order_date)",
"label": "order_year",
}
query = QueryObject(
filters=[{"col": adhoc_col, "op": "==", "val": "2024"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 2, "dashboardId": 10}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_cross_filter_rejected_for_unauthorized_dashboard(
mocker: MockerFixture,
) -> None:
"""Cross-filter lookup must not use a dashboard the guest has no access to."""
from superset.models.dashboard import Dashboard
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.id = 2
stored_chart.params_dict = {}
sibling_chart = mocker.MagicMock()
sibling_chart.id = 1
sibling_chart.params_dict = {
"columns": [{"sqlExpression": "YEAR(order_date)", "label": "order_year"}],
}
dashboard = mocker.MagicMock(spec=Dashboard)
dashboard.slices = [sibling_chart, stored_chart]
mocker.patch("superset.db.session.query")
db_query = mocker.patch("superset.db.session.query").return_value
db_query.filter.return_value.one_or_none.return_value = dashboard
mocker.patch(
"superset.security_manager.has_guest_access",
return_value=False,
)
adhoc_col: Any = {"sqlExpression": "YEAR(order_date)", "label": "order_year"}
query = QueryObject(
filters=[{"col": adhoc_col, "op": "==", "val": "2024"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 2, "dashboardId": 999}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_cross_filter_rejected_when_chart_not_on_dashboard(
mocker: MockerFixture,
) -> None:
"""Cross-filter lookup must verify the target chart belongs to the dashboard."""
from superset.models.dashboard import Dashboard
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.id = 99 # not on the dashboard
stored_chart.params_dict = {}
sibling_chart = mocker.MagicMock()
sibling_chart.id = 1
sibling_chart.params_dict = {
"columns": [{"sqlExpression": "YEAR(order_date)", "label": "order_year"}],
}
dashboard = mocker.MagicMock(spec=Dashboard)
dashboard.slices = [sibling_chart] # stored_chart not here
mocker.patch("superset.db.session.query")
db_query = mocker.patch("superset.db.session.query").return_value
db_query.filter.return_value.one_or_none.return_value = dashboard
mocker.patch(
"superset.security_manager.has_guest_access",
return_value=True,
)
adhoc_col: Any = {"sqlExpression": "YEAR(order_date)", "label": "order_year"}
query = QueryObject(
filters=[{"col": adhoc_col, "op": "==", "val": "2024"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 99, "dashboardId": 10}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_sibling_expressions_cannot_inject_where_having(
mocker: MockerFixture,
) -> None:
"""Sibling chart column expressions must not legitimize novel WHERE/HAVING."""
from superset.models.dashboard import Dashboard
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.id = 2
stored_chart.params_dict = {}
# Sibling has a column expression that an attacker tries to use as WHERE.
sibling_chart = mocker.MagicMock()
sibling_chart.id = 1
sibling_chart.params_dict = {
"columns": [
{"sqlExpression": "(SELECT secret FROM users LIMIT 1)", "label": "x"},
],
}
dashboard = mocker.MagicMock(spec=Dashboard)
dashboard.slices = [sibling_chart, stored_chart]
mocker.patch("superset.db.session.query")
db_query = mocker.patch("superset.db.session.query").return_value
db_query.filter.return_value.one_or_none.return_value = dashboard
# Attacker injects the sibling expression into extras.where.
query = QueryObject(
extras={"where": "(SELECT secret FROM users LIMIT 1)"},
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 2, "dashboardId": 10}
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_collect_allowed_sql_includes_scalar_column_params(
mocker: MockerFixture,
) -> None:
"""Scalar column params like x_axis contribute their sqlExpression."""
from superset.security.manager import _collect_allowed_sql
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {
"x_axis": {"sqlExpression": "DATE_TRUNC('month', ts)", "label": "m"},
"groupby": [{"sqlExpression": "UPPER(country)", "label": "c"}],
}
_, col_allowed = _collect_allowed_sql(stored_chart, None)
assert "DATE_TRUNC('month', ts)" in col_allowed
assert "UPPER(country)" in col_allowed
def test_sql_filters_structured_filter_string_col_allowed(
mocker: MockerFixture,
) -> None:
"""Structured filter with a plain string column is allowed."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(
filters=[{"col": "status", "op": "==", "val": "active"}],
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_empty_filter_sentinel_allowed(
mocker: MockerFixture,
) -> None:
"""The ``(1 = 0)`` sentinel from a required-but-empty native filter is allowed."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(extras={"where": "(1 = 0)"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_double_sentinel_allowed(
mocker: MockerFixture,
) -> None:
"""Two required-but-empty filters compose ``(1 = 0) AND (1 = 0)``."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(extras={"where": "(1 = 0) AND (1 = 0)"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_stored_clause_plus_sentinel_allowed(
mocker: MockerFixture,
) -> None:
"""A stored SQL filter composed with the empty-filter sentinel is allowed."""
sql_filter = {
"expressionType": "SQL",
"sqlExpression": "region = 'EMEA'",
"clause": "WHERE",
}
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {"adhoc_filters": [sql_filter]}
query = QueryObject(
extras={"where": "(region = 'EMEA') AND (1 = 0)"},
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_non_dict_adhoc_filter_skipped(
mocker: MockerFixture,
) -> None:
"""Non-dict items in adhoc_filters are skipped, not 500."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject()
query_context.queries = [query]
form_data: dict[str, Any] = {
"slice_id": 1,
"adhoc_filters": ["not_a_dict", 42, None],
}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_raise_for_access_guest_user_sql_filter_injection_blocked(
mocker: MockerFixture,
app_context: None,
stored_metrics: list[AdhocMetric],
) -> None:
"""Guest user injecting SQL via extras.where is rejected by raise_for_access."""
sm = SupersetSecurityManager(appbuilder)
mocker.patch.object(sm, "is_guest_user", return_value=True)
mocker.patch.object(sm, "can_access", return_value=True)
query_context = mocker.MagicMock()
query_context.slice_.id = 42
query_context.slice_.query_context = None
query_context.slice_.params_dict = {"metrics": stored_metrics}
query_context.form_data = {"slice_id": 42, "metrics": stored_metrics}
query_context.queries = [
QueryObject(
metrics=stored_metrics, # type: ignore
extras={"where": "1=1 UNION SELECT password FROM users"},
)
]
with pytest.raises(SupersetSecurityException):
sm.raise_for_access(query_context=query_context)
def test_sql_filters_cache_replay_skips_check(
mocker: MockerFixture,
) -> None:
"""Cache-replay requests skip the SQL filter check."""
query_context = mocker.MagicMock()
query_context._from_cache_replay = True
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
query = QueryObject(extras={"where": "(injected SQL)"})
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
assert not _sql_filters_modified(query_context, form_data, stored_chart, None)
def test_sql_filters_column_expression_cannot_become_where(
mocker: MockerFixture,
) -> None:
"""A chart's column sqlExpression must not be injectable as extras.where."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {
"columns": [
{
"sqlExpression": "(SELECT secret FROM users LIMIT 1)",
"label": "x",
},
],
}
query = QueryObject(
extras={"where": "((SELECT secret FROM users LIMIT 1))"},
mocker.patch.object(sm, "is_admin", return_value=False)
mocker.patch(
"superset.security.manager.get_user_id",
return_value=100,
)
mocker.patch(
"superset.subjects.utils.get_user_subject_ids",
return_value={1000},
)
mocker.patch(
"superset.security.manager.get_extra_editor_subject_ids",
return_value=set(),
)
query_context.queries = [query]
form_data: dict[str, Any] = {"slice_id": 1}
subject_user_100 = mocker.MagicMock(id=1000)
subject_user_200 = mocker.MagicMock(id=2000)
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
def mock_get_user_subject(uid: int):
if uid == 100:
return subject_user_100
if uid == 200:
return subject_user_200
return None
mocker.patch(
"superset.subjects.utils.get_user_subject",
side_effect=mock_get_user_subject,
)
query = Query(user_id=100)
assert sm.is_editor(query) is True
other_query = Query(user_id=200)
assert sm.is_editor(other_query) is False
def test_sql_filters_unbalanced_parens_rejected(
mocker: MockerFixture,
def test_is_editor_saved_query_owner(mocker: MockerFixture, app_context: None) -> None:
"""
Test that a SavedQuery owner is considered an editor via Subject resolution.
"""
from superset.models.sql_lab import SavedQuery
sm = SupersetSecurityManager(appbuilder)
mocker.patch.object(sm, "is_admin", return_value=False)
mocker.patch(
"superset.security.manager.get_user_id",
return_value=100,
)
mocker.patch(
"superset.subjects.utils.get_user_subject_ids",
return_value={1000},
)
mocker.patch(
"superset.security.manager.get_extra_editor_subject_ids",
return_value=set(),
)
subject_user_100 = mocker.MagicMock(id=1000)
subject_user_200 = mocker.MagicMock(id=2000)
def mock_get_user_subject(uid: int):
if uid == 100:
return subject_user_100
if uid == 200:
return subject_user_200
return None
mocker.patch(
"superset.subjects.utils.get_user_subject",
side_effect=mock_get_user_subject,
)
saved_query = SavedQuery(user_id=100)
assert sm.is_editor(saved_query) is True
other_saved_query = SavedQuery(user_id=200)
assert sm.is_editor(other_saved_query) is False
def test_is_editor_other_model_with_user_id_not_editor(
mocker: MockerFixture, app_context: None
) -> None:
"""Unbalanced parens in extras.where are rejected (403, not 500)."""
query_context = mocker.MagicMock()
stored_chart = mocker.MagicMock()
stored_chart.params_dict = {}
"""
Test that a model with user_id that is NOT Query or SavedQuery
does NOT receive the fallback and is not considered an editor.
"""
from superset.models.sql_lab import TabState
query = QueryObject(extras={"where": "(a) AND (b"})
query_context.queries = [query]
sm = SupersetSecurityManager(appbuilder)
mocker.patch.object(sm, "is_admin", return_value=False)
mocker.patch(
"superset.security.manager.get_user_id",
return_value=100,
)
mocker.patch(
"superset.subjects.utils.get_user_subject_ids",
return_value={1000},
)
mocker.patch(
"superset.security.manager.get_extra_editor_subject_ids",
return_value=set(),
)
form_data: dict[str, Any] = {"slice_id": 1}
subject_user_100 = mocker.MagicMock(id=1000)
mocker.patch(
"superset.subjects.utils.get_user_subject",
return_value=subject_user_100,
)
assert _sql_filters_modified(query_context, form_data, stored_chart, None)
tab_state = TabState(user_id=100)
assert sm.is_editor(tab_state) is False
+166
View File
@@ -28,7 +28,9 @@ from superset.exceptions import QueryClauseValidationException, SupersetParseErr
from superset.jinja_context import JinjaTemplateProcessor
from superset.sql.parse import (
_check_script_length,
_count_weighted_table_references,
BaseSQLStatement,
count_referenced_tables,
CTASMethod,
extract_tables_from_statement,
has_aggregate,
@@ -234,6 +236,170 @@ def test_extract_tables_from_sql() -> None:
) == {Table("forbidden_table")}
def test_count_referenced_tables() -> None:
"""
Test that ``count_referenced_tables`` counts table reference occurrences
(not distinct tables), ignoring dotted quoted aliases, and falls back to
1 for unparseable SQL.
"""
assert count_referenced_tables('SELECT * FROM "db.table1"', Dialects.SQLITE) == 1
assert (
count_referenced_tables(
'SELECT COUNT(id) AS "metric.value" FROM "db.table1"', Dialects.SQLITE
)
== 1
)
assert (
count_referenced_tables(
'SELECT t1.b, t2.b FROM "db.table1" AS t1 '
'JOIN "db.table2" AS t2 ON t1.a = t2.a',
Dialects.SQLITE,
)
== 2
)
assert count_referenced_tables("this is not valid sql (((", Dialects.SQLITE) == 1
assert count_referenced_tables("SHOW CREATE TABLE s1.t1", "mysql") == 1
def test_count_referenced_tables_self_join() -> None:
"""
A self-join references the same physical table twice via two aliases;
it must still count as 2 (a join), not 1 (deduplicated to a single
table), or the caller's multi-table detection would incorrectly treat
it as single-table.
"""
assert (
count_referenced_tables(
'SELECT l.a, r.a FROM "db.table1" AS l JOIN "db.table1" AS r ON l.a = r.a',
Dialects.SQLITE,
)
== 2
)
def test_count_referenced_tables_cte_self_join() -> None:
"""
A CTE that reads a single virtual table and is then self-joined must
count as 2, matching the direct self-join case, since the CTE is
inlined at each of its two consumption sites and triggers a read of
that table for both sides of the join.
"""
assert (
count_referenced_tables(
'WITH cte AS (SELECT a FROM "db.table1") '
"SELECT l.a, r.a FROM cte AS l JOIN cte AS r ON l.a = r.a",
Dialects.SQLITE,
)
== 2
)
# A CTE used exactly once, with no join, still counts as a single table.
assert (
count_referenced_tables(
'WITH cte AS (SELECT a FROM "db.table1") SELECT a FROM cte',
Dialects.SQLITE,
)
== 1
)
# A CTE joined against a distinct real table also counts as 2.
assert (
count_referenced_tables(
'WITH cte AS (SELECT a FROM "db.table1") '
'SELECT l.a, r.a FROM cte AS l JOIN "db.table2" AS r ON l.a = r.a',
Dialects.SQLITE,
)
== 2
)
# Nested CTEs: a CTE built on top of another CTE, then self-joined,
# still weights the base CTE's own table by the self-join count.
assert (
count_referenced_tables(
'WITH base AS (SELECT a FROM "db.table1"), derived AS (SELECT a FROM base) '
"SELECT l.a, r.a FROM derived AS l JOIN derived AS r ON l.a = r.a",
Dialects.SQLITE,
)
== 2
)
def test_count_referenced_tables_describe() -> None:
"""
``DESCRIBE`` (and other ``exp.Describe``/``exp.Command`` statements) has
no join semantics for a per-table row cap to interact with, so it takes
the plain unweighted table-extraction path rather than
``_count_weighted_table_references``.
"""
assert count_referenced_tables("DESCRIBE table1", Dialects.SQLITE) == 1
def test_count_referenced_tables_derived_subqueries() -> None:
"""
Two distinct derived (non-CTE) subqueries joined together must each
resolve their own tables directly, without recursing as if they were
CTE sources -- covering the branch in ``_count_weighted_table_references``
where a selected source is a ``Scope`` but not a CTE.
"""
assert (
count_referenced_tables(
'SELECT l.a, r.a FROM (SELECT a FROM "db.table1") AS l '
'JOIN (SELECT a FROM "db.table1") AS r ON l.a = r.a',
Dialects.SQLITE,
)
== 2
)
def test_count_weighted_table_references_self_referential_scope_guard(
mocker: MockerFixture,
) -> None:
"""
``_count_weighted_table_references`` must not recurse forever on a
self-referential ``Scope`` graph, the shape a ``WITH RECURSIVE`` CTE
could in principle produce if sqlglot ever resolved its own
self-reference to the same ``Scope`` object instead of a bare
``exp.Table``. The ``seen`` guard must catch the repeat visit and treat
it as contributing no further table reads.
"""
from sqlglot.optimizer.scope import Scope, ScopeType # noqa: PLC0415
cte_scope = Scope.__new__(Scope)
cte_scope.scope_type = ScopeType.CTE
# The CTE's own body references itself.
cte_scope._selected_sources = {"t": (None, cte_scope)} # noqa: SLF001
root_scope = Scope.__new__(Scope)
root_scope.scope_type = ScopeType.ROOT
root_scope._selected_sources = {"t": (None, cte_scope)} # noqa: SLF001
mocker.patch(
"superset.sql.parse.traverse_scope",
return_value=[cte_scope, root_scope],
)
assert _count_weighted_table_references(mocker.MagicMock()) == 0
def test_count_referenced_tables_respects_parse_length_cap(
mocker: MockerFixture,
) -> None:
"""
``count_referenced_tables`` must not bypass ``SQL_MAX_PARSE_LENGTH``: an
oversized statement should fail the length check before reaching
sqlglot, and fall back to the conservative single-table count. The
statement references two tables so that bypassing the guard (and
reaching sqlglot) would produce a different, detectable result.
"""
mocker.patch("superset.config.SQL_MAX_PARSE_LENGTH", 100)
mocker.patch("superset.sql.parse.has_app_context", return_value=False)
padding = "1, " * 50
statement = (
'SELECT * FROM "db.table1" AS t1 ' # noqa: S608
'JOIN "db.table2" AS t2 ON t1.a = t2.a '
f"WHERE t1.a IN ({padding}1)"
)
assert len(statement.encode("utf-8")) > 100
assert count_referenced_tables(statement, Dialects.SQLITE) == 1
def test_extract_tables_subselect() -> None:
"""
Test that tables inside subselects are parsed correctly.
+75 -68
View File
@@ -465,86 +465,93 @@ def test_get_sql_results_oauth2(mocker: MockerFixture, app) -> None:
"""
Test that `get_sql_results` works with OAuth2.
"""
# Pushed/popped manually (rather than via a ``with`` block) so the
# ``finally`` below still pops it if an assertion fails, preventing the
# request context from leaking into later tests in the same session.
app_context = app.test_request_context()
app_context.push()
mocker.patch(
"superset.db_engine_specs.base.uuid4",
return_value=UUID("fb11f528-6eba-4a8a-837e-6b0d39ee9187"),
)
mocker.patch(
"superset.db_engine_specs.base.generate_code_verifier",
return_value="xkBPVZoFChVcy3VZ2l5u7d0FZPTU-olO7HtsAOok2IUGigyoZ62tG_oldy2xg9_HdqPKrWUmKZLmU-CUqz_SQ",
)
mocker.patch("superset.daos.key_value.KeyValueDAO.delete_expired_entries")
mocker.patch("superset.daos.key_value.KeyValueDAO.create_entry")
mocker.patch("superset.db_engine_specs.base.db.session.commit")
try:
mocker.patch(
"superset.db_engine_specs.base.uuid4",
return_value=UUID("fb11f528-6eba-4a8a-837e-6b0d39ee9187"),
)
mocker.patch(
"superset.db_engine_specs.base.generate_code_verifier",
return_value="xkBPVZoFChVcy3VZ2l5u7d0FZPTU-olO7HtsAOok2IUGigyoZ62tG_oldy2xg9_HdqPKrWUmKZLmU-CUqz_SQ",
)
mocker.patch("superset.daos.key_value.KeyValueDAO.delete_expired_entries")
mocker.patch("superset.daos.key_value.KeyValueDAO.create_entry")
mocker.patch("superset.db_engine_specs.base.db.session.commit")
g = mocker.patch("superset.db_engine_specs.base.g")
g.user = mocker.MagicMock()
g.user.id = 42
g = mocker.patch("superset.db_engine_specs.base.g")
g.user = mocker.MagicMock()
g.user.id = 42
database = Database(
id=1,
database_name="my_db",
sqlalchemy_uri="sqlite://",
encrypted_extra=json.dumps(oauth2_client_info),
)
database.db_engine_spec.oauth2_exception = OAuth2Error
get_sqla_engine = mocker.patch.object(database, "get_sqla_engine")
get_sqla_engine().__enter__().raw_connection.side_effect = OAuth2Error(
"OAuth2 required"
)
database = Database(
id=1,
database_name="my_db",
sqlalchemy_uri="sqlite://",
encrypted_extra=json.dumps(oauth2_client_info),
)
database.db_engine_spec.oauth2_exception = OAuth2Error
get_sqla_engine = mocker.patch.object(database, "get_sqla_engine")
get_sqla_engine().__enter__().raw_connection.side_effect = OAuth2Error(
"OAuth2 required"
)
# `limit` and `select_as_cta_used` must match the real `Query` model's
# defaults (nullable Integer -> None, Boolean default=False) so that
# `apply_limit` -- called unconditionally before the mocked OAuth2 error
# is ever reached -- doesn't try to compare an unconfigured MagicMock
# against an int.
query = mocker.MagicMock(
select_as_cta=False,
select_as_cta_used=False,
limit=None,
database=database,
)
mocker.patch("superset.sql_lab.get_query", return_value=query)
# `limit` and `select_as_cta_used` must match the real `Query` model's
# defaults (nullable Integer -> None, Boolean default=False) so that
# `apply_limit` -- called unconditionally before the mocked OAuth2 error
# is ever reached -- doesn't try to compare an unconfigured MagicMock
# against an int.
query = mocker.MagicMock(
select_as_cta=False,
select_as_cta_used=False,
limit=None,
database=database,
)
mocker.patch("superset.sql_lab.get_query", return_value=query)
payload = get_sql_results(query_id=1, rendered_query="SELECT 1")
assert payload["status"] == QueryStatus.FAILED
assert payload["error"] == "You don't have permission to access the data."
assert len(payload["errors"]) == 1
payload = get_sql_results(query_id=1, rendered_query="SELECT 1")
assert payload["status"] == QueryStatus.FAILED
assert payload["error"] == "You don't have permission to access the data."
assert len(payload["errors"]) == 1
error = payload["errors"][0]
assert error["message"] == "You don't have permission to access the data."
assert error["error_type"] == SupersetErrorType.OAUTH2_REDIRECT
assert error["level"] == ErrorLevel.WARNING
assert error["extra"]["tab_id"] == "fb11f528-6eba-4a8a-837e-6b0d39ee9187"
assert (
error["extra"]["redirect_uri"] == "http://example.com/api/v1/database/oauth2/"
)
error = payload["errors"][0]
assert error["message"] == "You don't have permission to access the data."
assert error["error_type"] == SupersetErrorType.OAUTH2_REDIRECT
assert error["level"] == ErrorLevel.WARNING
assert error["extra"]["tab_id"] == "fb11f528-6eba-4a8a-837e-6b0d39ee9187"
assert (
error["extra"]["redirect_uri"]
== "http://example.com/api/v1/database/oauth2/"
)
# Parse the OAuth2 authorization URL and verify components individually,
# since the JWT state and PKCE code_challenge are computed deterministically
# from mocked inputs but their exact encoding depends on library internals.
url = urlparse(error["extra"]["url"])
assert url.scheme == "https"
assert url.netloc == "abcd1234.snowflakecomputing.com"
assert url.path == "/oauth/authorize"
# Parse the OAuth2 authorization URL and verify components individually,
# since the JWT state and PKCE code_challenge are computed deterministically
# from mocked inputs but their exact encoding depends on library internals.
url = urlparse(error["extra"]["url"])
assert url.scheme == "https"
assert url.netloc == "abcd1234.snowflakecomputing.com"
assert url.path == "/oauth/authorize"
params = parse_qs(url.query)
assert params["scope"] == ["refresh_token session:role:USERADMIN"]
assert params["response_type"] == ["code"]
assert params["redirect_uri"] == ["http://example.com/api/v1/database/oauth2/"]
assert params["client_id"] == ["my_client_id"]
assert params["code_challenge_method"] == ["S256"]
params = parse_qs(url.query)
assert params["scope"] == ["refresh_token session:role:USERADMIN"]
assert params["response_type"] == ["code"]
assert params["redirect_uri"] == ["http://example.com/api/v1/database/oauth2/"]
assert params["client_id"] == ["my_client_id"]
assert params["code_challenge_method"] == ["S256"]
# Verify PKCE code_challenge matches the mocked code_verifier
from superset.utils.oauth2 import generate_code_challenge
# Verify PKCE code_challenge matches the mocked code_verifier
from superset.utils.oauth2 import generate_code_challenge
expected_code_challenge = generate_code_challenge(
"xkBPVZoFChVcy3VZ2l5u7d0FZPTU-olO7HtsAOok2IUGigyoZ62tG_oldy2xg9_HdqPKrWUmKZLmU-CUqz_SQ"
)
assert params["code_challenge"] == [expected_code_challenge]
expected_code_challenge = generate_code_challenge(
"xkBPVZoFChVcy3VZ2l5u7d0FZPTU-olO7HtsAOok2IUGigyoZ62tG_oldy2xg9_HdqPKrWUmKZLmU-CUqz_SQ"
)
assert params["code_challenge"] == [expected_code_challenge]
finally:
app_context.pop()
def test_apply_rls(mocker: MockerFixture) -> None:
@@ -1,67 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
import pandas as pd
from sqlalchemy import create_engine, types
from superset.utils.pandas_sqlalchemy_compat import (
restore_pandas_sqlalchemy_support,
)
def test_to_sql_accepts_sqlalchemy_engine_and_dtypes() -> None:
"""
``DataFrame.to_sql`` must accept a SQLAlchemy engine plus SQLAlchemy
``dtype`` objects regardless of the installed pandas/SQLAlchemy combo.
This is the exact call shape used by dataset uploads
(``BaseEngineSpec.df_to_sql``), example data loading, and the test data
loaders; it breaks when pandas silently rejects the installed SQLAlchemy
as too old (pandas >= 2.2 with SQLAlchemy 1.x) and no compat shim is
applied.
"""
restore_pandas_sqlalchemy_support()
engine = create_engine("sqlite://")
df = pd.DataFrame(
{
"name": ["a", "b"],
"num": [1, 2],
"ds": pd.to_datetime(["2021-01-01", "2021-01-02"]),
}
)
df.to_sql(
"birth_names",
engine,
index=False,
dtype={"ds": types.DateTime(), "name": types.String(255)},
method="multi",
chunksize=100,
)
df.to_sql("birth_names", engine, index=False, if_exists="replace")
result = pd.read_sql_query("SELECT name, num FROM birth_names", engine)
assert result["name"].tolist() == ["a", "b"]
assert result["num"].tolist() == [1, 2]
def test_restore_pandas_sqlalchemy_support_is_idempotent() -> None:
from pandas.compat import _optional
restore_pandas_sqlalchemy_support()
first = _optional.VERSIONS.get("sqlalchemy")
restore_pandas_sqlalchemy_support()
assert _optional.VERSIONS.get("sqlalchemy") == first