Compare commits

...

13 Commits

Author SHA1 Message Date
Evan Rusackas
014d6acd9a docs(installation): fix PyPI install Python version and OS deps
The PyPI install page claimed Python 3.12 (Ubuntu 24.04 default) was
unsupported and steered users to a deadsnakes 3.11 workaround. pyproject.toml
declares support for 3.10-3.12 (requires-python >=3.10), so the claim was
wrong and the workaround unnecessary.

- Drop the "3.12 not supported" claim and the deadsnakes/python3.11 steps
- Collapse the redundant Ubuntu blocks into one (tested 20.04/22.04/24.04),
  removing the EOL "before 20.04" block with its py2-era package names
- Point at pyproject.toml for supported versions (consistent with macOS section)
- Align OS deps with the repo Dockerfile: add python3-venv (needed by
  `python3 -m venv`), pkg-config (mysqlclient build), and libpq-dev (Postgres)
- Fix stale py2 package names in the yum block (python-devel ->
  python3-devel, etc.)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 18:11:24 -07:00
Hans Yu
883b7a286d refactor: update SQLAlchemy select() syntax to 2.0 (#40276) 2026-06-17 17:50:32 -07:00
Evan Rusackas
d9d8b2bcc0 chore(ci): correct action ref version comments (zizmor) (#41160)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:42:14 -07:00
Evan Rusackas
9da54eff84 chore(ci): set least-privilege workflow permissions (zizmor) (#41161)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:41:47 -07:00
dependabot[bot]
fb2b9fa8ff chore(deps): bump cryptography from 46.0.7 to 48.0.1 (#41010)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Evan <evan@preset.io>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 15:01:20 -07:00
Dante R. Giuliano
31797005db docs(INTHEWILD): adding Tech Solution (#37178)
Co-authored-by: Evan Rusackas <evan@preset.io>
Co-authored-by: Joe Li <joe@preset.io>
2026-06-17 14:59:15 -07:00
Evan Rusackas
ca2d340db3 fix(security): validate dynamic method dispatch in asyncEvent (#41163)
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
2026-06-17 14:58:34 -07:00
jesperct
ef82da8458 fix(charts): apply datetime format to unaggregated temporal columns (#41060) 2026-06-17 14:56:09 -07:00
Jean Massucatto
fee1cf9f08 chore(sqllab): remove dead TableElement component (#41029) 2026-06-17 14:54:41 -07:00
jesperct
2d2a8f3ab0 fix(plugin-chart-handlebars): follow the app theme in Customize code editors (#40952) 2026-06-17 14:52:52 -07:00
dependabot[bot]
a19093e65a chore(deps-dev): bump webpack-dev-server from 5.2.4 to 5.2.5 in /superset-frontend (#41168)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-17 12:55:44 -07:00
dependabot[bot]
b72a0a53c0 chore(deps): bump webpack-dev-server from 5.2.4 to 5.2.5 in /docs (#41169)
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2026-06-17 12:55:40 -07:00
Thomas Bernhard
512b6f43c1 chore(embedded sdk): bump sdk version number (#40991) 2026-06-17 12:47:41 -07:00
37 changed files with 618 additions and 1192 deletions

View File

@@ -37,7 +37,7 @@ jobs:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
distribution: "temurin"
java-version: "11"

View File

@@ -23,7 +23,7 @@ jobs:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
distribution: "temurin"
java-version: "11"

View File

@@ -71,7 +71,7 @@ jobs:
node-version-file: "./docs/.nvmrc"
- name: Setup Python
uses: ./.github/actions/setup-backend/
- uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5
- uses: actions/setup-java@be666c2fcd27ec809703dec50e508c2fdc7f6654 # v5.2.0
with:
distribution: "zulu"
java-version: "21"

View File

@@ -1,6 +1,11 @@
# Python integration tests
name: Python-Integration
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:

View File

@@ -1,6 +1,11 @@
# Python unit tests
name: Python-Unit
# Least-privilege default for GITHUB_TOKEN. Jobs that need more (e.g. OIDC for
# codecov uploads) opt in via their own job-level `permissions:` block.
permissions:
contents: read
on:
push:
branches:

View File

@@ -396,6 +396,10 @@ categories:
url: https://www.techaudit.info
contributors: ["@ETselikov"]
- name: Tech Solution
url: https://www.tech-solution.com.ar/
contributors: ["@danteGiuliano", "@LeandroVallejos", "@McJaben", "@xJeree", "@zeo-return-null"]
- name: Tenable
url: https://www.tenable.com
contributors: ["@dflionis"]

View File

@@ -22,31 +22,24 @@ level dependencies.
**Debian and Ubuntu**
Ubuntu **24.04** uses python 3.12 per default, which currently is not supported by Superset. You need to add a second python installation of 3.11 and install the required additional dependencies.
```bash
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.11 python3.11-dev python3.11-venv build-essential libssl-dev libffi-dev libsasl2-dev libldap2-dev default-libmysqlclient-dev
```
In Ubuntu **20.04 and 22.04** the following command will ensure that the required dependencies are installed:
The following command will ensure that the required dependencies are installed (tested on Ubuntu 20.04, 22.04, and 24.04):
```bash
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev python3-pip libsasl2-dev libldap2-dev default-libmysqlclient-dev
sudo apt-get install build-essential libssl-dev libffi-dev python3-dev python3-pip python3-venv libsasl2-dev libldap2-dev libpq-dev default-libmysqlclient-dev pkg-config
```
In Ubuntu **before 20.04** the following command will ensure that the required dependencies are installed:
```bash
sudo apt-get install build-essential libssl-dev libffi-dev python-dev python-pip libsasl2-dev libldap2-dev default-libmysqlclient-dev
```
Refer to the
[pyproject.toml](https://github.com/apache/superset/blob/master/pyproject.toml) file for the list of
Python versions officially supported by Superset, and install a matching `python3` interpreter for
your distribution. The `libpq-dev` package is only needed if you intend to connect to (or use) a
PostgreSQL database; you can omit it otherwise.
**Fedora and RHEL-derivative Linux distributions**
Install the following packages using the `yum` package manager:
```bash
sudo yum install gcc gcc-c++ libffi-devel python-devel python-pip python-wheel openssl-devel cyrus-sasl-devel openldap-devel
sudo yum install gcc gcc-c++ libffi-devel python3-devel python3-pip python3-wheel openssl-devel cyrus-sasl-devel openldap-devel
```
In more recent versions of CentOS and Fedora, you may need to install a slightly different set of packages using `dnf`:

View File

@@ -15006,9 +15006,9 @@ webpack-dev-middleware@^7.4.2:
schema-utils "^4.0.0"
webpack-dev-server@^5.2.2:
version "5.2.4"
resolved "https://registry.yarnpkg.com/webpack-dev-server/-/webpack-dev-server-5.2.4.tgz#6e6306ce59848ed322c235e48b326632b1eed6d6"
integrity sha512-GqDPGZN9bRqKBTkp4aWkobDDHMsrXKoGSdOH56smIri8qR0JG8gfL8/v/f/OZR3/OKXjG8uwJbFVhKm/FNU/UA==
version "5.2.5"
resolved "https://registry.yarnpkg.com/webpack-dev-server/-/webpack-dev-server-5.2.5.tgz#648fceaac6a5736b0935e5c1e55d6aa1d0626119"
integrity sha512-4wZtCquSuv9CKX8oybo+mqxtxZqWz47uM1Ch94lxowBztOhWCbhqvRbfC/mODOwxgV2brY+JGZpHq58/SuVFYg==
dependencies:
"@types/bonjour" "^3.5.13"
"@types/connect-history-api-fallback" "^1.5.4"

View File

@@ -45,7 +45,7 @@ dependencies = [
"flask-cors>=6.0.0, <7.0",
"croniter>=6.2.2",
"cron-descriptor",
"cryptography>=42.0.4, <47.0.0",
"cryptography>=48.0.0, <49.0.0",
"deprecation>=2.1.0, <2.2.0",
"flask>=2.2.5, <4.0.0",
"flask-appbuilder>=5.2.1, <6.0.0",

View File

@@ -18,5 +18,30 @@
testpaths =
tests
python_files = *_test.py test_*.py *_tests.py *viz/utils.py
addopts = -p no:warnings
# `-p no:warnings` temporarily disabled in favor of more finely tuned `filterwarnings`.
#addopts = -p no:warnings
asyncio_mode = auto
# `ignore` is effectively equivalent to `-p no:warnings`.
# Always print RemovedIn20Warning when SQLALCHEMY_WARN_20=1.
# Additionally, raise errors for refactored RemovedIn20Warning cases to prevent regression.
filterwarnings =
ignore
always::sqlalchemy.exc.RemovedIn20Warning
# error:Passing a string to Connection.execute\(\) is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:"Query" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SavedQuery" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SqlaTable" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"SqlMetric" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"TableColumn" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:"TaggedObject" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning
# error:The ``as_declarative\(\)`` function is now available:sqlalchemy.exc.RemovedIn20Warning
# error:The autoload parameter is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The connection.execute\(\) method:sqlalchemy.exc.RemovedIn20Warning
# error:The current statement is being autocommitted using implicit autocommit:sqlalchemy.exc.RemovedIn20Warning
# error:The `database` package is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The ``declarative_base\(\)`` function is now available:sqlalchemy.exc.RemovedIn20Warning
# error:The Engine.execute\(\) method is considered legacy:sqlalchemy.exc.RemovedIn20Warning
error:The legacy calling style of select\(\) is deprecated:sqlalchemy.exc.RemovedIn20Warning
# error:The "whens" argument to case:sqlalchemy.exc.RemovedIn20Warning
# error:"User" object is being merged into a Session:sqlalchemy.exc.RemovedIn20Warning

View File

@@ -26,7 +26,7 @@ filelock>=3.20.3,<4.0.0
brotli>=1.2.0,<2.0.0
numexpr>=2.9.0
# Security: CVE-2026-34073 (MEDIUM) - Improper Certificate Validation
cryptography>=46.0.7,<47.0.0
cryptography>=48.0.0,<49.0.0
# Security: Snyk - XSS vulnerability in Mako templates
mako>=1.3.11,<2.0.0
# Security: CVE-2024-52338 (CRITICAL) - Deserialization of untrusted data in IPC/Parquet readers

View File

@@ -86,7 +86,7 @@ cron-descriptor==1.4.5
# via apache-superset (pyproject.toml)
croniter==6.2.2
# via apache-superset (pyproject.toml)
cryptography==46.0.7
cryptography==48.0.1
# via
# -r requirements/base.in
# apache-superset (pyproject.toml)
@@ -323,7 +323,7 @@ pyjwt==2.12.0
# redis
pynacl==1.6.2
# via paramiko
pyopenssl==26.0.0
pyopenssl==26.2.0
# via
# -r requirements/base.in
# shillelagh
@@ -344,7 +344,6 @@ python-dotenv==1.2.2
# via apache-superset (pyproject.toml)
pytz==2025.2
# via
# croniter
# flask-babel
# pandas
pyxlsb==1.0.10

View File

@@ -178,7 +178,7 @@ croniter==6.2.2
# via
# -c requirements/base-constraint.txt
# apache-superset
cryptography==46.0.7
cryptography==48.0.1
# via
# -c requirements/base-constraint.txt
# apache-superset
@@ -780,7 +780,7 @@ pynacl==1.6.2
# via
# -c requirements/base-constraint.txt
# paramiko
pyopenssl==26.0.0
pyopenssl==26.2.0
# via
# -c requirements/base-constraint.txt
# shillelagh
@@ -841,7 +841,6 @@ python-multipart==0.0.29
pytz==2025.2
# via
# -c requirements/base-constraint.txt
# croniter
# flask-babel
# pandas
# trino

View File

@@ -1,6 +1,6 @@
{
"name": "@superset-ui/embedded-sdk",
"version": "0.3.0",
"version": "0.4.0",
"description": "SDK for embedding resources from Superset into your own application",
"access": "public",
"keywords": [

View File

@@ -287,7 +287,7 @@
"webpack": "^5.107.2",
"webpack-bundle-analyzer": "^5.3.0",
"webpack-cli": "^7.0.3",
"webpack-dev-server": "^5.2.4",
"webpack-dev-server": "^5.2.5",
"webpack-manifest-plugin": "^6.0.1",
"webpack-sources": "^3.5.0",
"webpack-visualizer-plugin2": "^2.0.0"
@@ -26127,21 +26127,6 @@
}
}
},
"node_modules/jsdom/node_modules/@noble/hashes": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/@noble/hashes/-/hashes-2.2.0.tgz",
"integrity": "sha512-IYqDGiTXab6FniAgnSdZwgWbomxpy9FtYvLKs7wCUs2a8RkITG+DFGO1DM9cr+E3/RgADRpFjrKVaJ1z6sjtEg==",
"dev": true,
"license": "MIT",
"optional": true,
"peer": true,
"engines": {
"node": ">= 20.19.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/jsdom/node_modules/css-tree": {
"version": "3.2.1",
"resolved": "https://registry.npmjs.org/css-tree/-/css-tree-3.2.1.tgz",
@@ -42790,9 +42775,9 @@
}
},
"node_modules/webpack-dev-server": {
"version": "5.2.4",
"resolved": "https://registry.npmjs.org/webpack-dev-server/-/webpack-dev-server-5.2.4.tgz",
"integrity": "sha512-GqDPGZN9bRqKBTkp4aWkobDDHMsrXKoGSdOH56smIri8qR0JG8gfL8/v/f/OZR3/OKXjG8uwJbFVhKm/FNU/UA==",
"version": "5.2.5",
"resolved": "https://registry.npmjs.org/webpack-dev-server/-/webpack-dev-server-5.2.5.tgz",
"integrity": "sha512-4wZtCquSuv9CKX8oybo+mqxtxZqWz47uM1Ch94lxowBztOhWCbhqvRbfC/mODOwxgV2brY+JGZpHq58/SuVFYg==",
"dev": true,
"license": "MIT",
"dependencies": {
@@ -43264,21 +43249,6 @@
}
}
},
"node_modules/whatwg-url/node_modules/@noble/hashes": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/@noble/hashes/-/hashes-2.2.0.tgz",
"integrity": "sha512-IYqDGiTXab6FniAgnSdZwgWbomxpy9FtYvLKs7wCUs2a8RkITG+DFGO1DM9cr+E3/RgADRpFjrKVaJ1z6sjtEg==",
"dev": true,
"license": "MIT",
"optional": true,
"peer": true,
"engines": {
"node": ">= 20.19.0"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/whatwg-url/node_modules/webidl-conversions": {
"version": "8.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-8.0.1.tgz",
@@ -44399,7 +44369,7 @@
"cross-env": "^10.1.0",
"fs-extra": "^11.3.5",
"jest": "^30.4.2",
"yeoman-test": "^11.5.2"
"yeoman-test": "^11.5.3"
},
"engines": {
"node": ">= 18.0.0",

View File

@@ -370,7 +370,7 @@
"webpack": "^5.107.2",
"webpack-bundle-analyzer": "^5.3.0",
"webpack-cli": "^7.0.3",
"webpack-dev-server": "^5.2.4",
"webpack-dev-server": "^5.2.5",
"webpack-manifest-plugin": "^6.0.1",
"webpack-sources": "^3.5.0",
"webpack-visualizer-plugin2": "^2.0.0"

View File

@@ -23,7 +23,7 @@ import {
} from '@superset-ui/chart-controls';
import { t } from '@apache-superset/core/translation';
import { validateNonEmpty } from '@superset-ui/core';
import { useTheme } from '@apache-superset/core/theme';
import { useTheme, useThemeMode } from '@apache-superset/core/theme';
import { InfoTooltip } from '@superset-ui/core/components';
import { CodeEditor } from '../../components/CodeEditor/CodeEditor';
import { ControlHeader } from '../../components/ControlHeader/controlHeader';
@@ -37,6 +37,7 @@ const HandlebarsTemplateControl = (
props: CustomControlConfig<HandlebarsCustomControlProps>,
) => {
const theme = useTheme();
const isDarkMode = useThemeMode();
const val = String(
props?.value ? props?.value : props?.default ? props?.default : '',
);
@@ -65,7 +66,7 @@ const HandlebarsTemplateControl = (
</div>
</ControlHeader>
<CodeEditor
theme="dark"
theme={isDarkMode ? 'dark' : 'light'}
value={val}
onChange={source => {
debounceFunc(props.onChange, source || '');

View File

@@ -22,7 +22,7 @@ import {
sharedControls,
} from '@superset-ui/chart-controls';
import { t } from '@apache-superset/core/translation';
import { useTheme } from '@apache-superset/core/theme';
import { useTheme, useThemeMode } from '@apache-superset/core/theme';
import { InfoTooltip } from '@superset-ui/core/components';
import { CodeEditor } from '../../components/CodeEditor/CodeEditor';
import { ControlHeader } from '../../components/ControlHeader/controlHeader';
@@ -35,6 +35,7 @@ interface StyleCustomControlProps {
const StyleControl = (props: CustomControlConfig<StyleCustomControlProps>) => {
const theme = useTheme();
const isDarkMode = useThemeMode();
const htmlSanitization = props.htmlSanitization ?? true;
const defaultValue = props?.value
@@ -63,7 +64,7 @@ const StyleControl = (props: CustomControlConfig<StyleCustomControlProps>) => {
</div>
</ControlHeader>
<CodeEditor
theme="dark"
theme={isDarkMode ? 'dark' : 'light'}
mode="css"
value={props.value}
defaultValue={defaultValue}

View File

@@ -0,0 +1,77 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { ComponentType } from 'react';
import { CustomControlItem } from '@superset-ui/chart-controls';
import { render } from 'spec/helpers/testing-library';
import { useThemeMode } from '@apache-superset/core/theme';
import { handlebarsTemplateControlSetItem } from '../../src/plugin/controls/handlebarTemplate';
import { styleControlSetItem } from '../../src/plugin/controls/style';
const mockCodeEditor = jest.fn((_props: { theme?: string }) => null);
jest.mock('../../src/components/CodeEditor/CodeEditor', () => ({
CodeEditor: (props: { theme?: string }) => mockCodeEditor(props),
}));
jest.mock('@apache-superset/core/theme', () => ({
...jest.requireActual('@apache-superset/core/theme'),
useThemeMode: jest.fn(),
}));
const HandlebarsTemplateControl = (
handlebarsTemplateControlSetItem as CustomControlItem
).config.type as ComponentType<{ value: string; onChange: () => void }>;
const StyleControl = (styleControlSetItem as CustomControlItem).config
.type as ComponentType<{ value: string; onChange: () => void }>;
const mockedUseThemeMode = useThemeMode as jest.Mock;
afterEach(() => jest.clearAllMocks());
test('Handlebars Template editor uses the light Ace theme in a light UI', () => {
mockedUseThemeMode.mockReturnValue(false);
render(<HandlebarsTemplateControl value="x" onChange={jest.fn()} />);
expect(mockCodeEditor).toHaveBeenCalledWith(
expect.objectContaining({ theme: 'light' }),
);
});
test('Handlebars Template editor uses the dark Ace theme in a dark UI', () => {
mockedUseThemeMode.mockReturnValue(true);
render(<HandlebarsTemplateControl value="x" onChange={jest.fn()} />);
expect(mockCodeEditor).toHaveBeenCalledWith(
expect.objectContaining({ theme: 'dark' }),
);
});
test('CSS Styles editor uses the light Ace theme in a light UI', () => {
mockedUseThemeMode.mockReturnValue(false);
render(<StyleControl value="x" onChange={jest.fn()} />);
expect(mockCodeEditor).toHaveBeenCalledWith(
expect.objectContaining({ theme: 'light' }),
);
});
test('CSS Styles editor uses the dark Ace theme in a dark UI', () => {
mockedUseThemeMode.mockReturnValue(true);
render(<StyleControl value="x" onChange={jest.fn()} />);
expect(mockCodeEditor).toHaveBeenCalledWith(
expect.objectContaining({ theme: 'dark' }),
);
});

View File

@@ -1,464 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { isValidElement } from 'react';
import fetchMock from 'fetch-mock';
import { FeatureFlag, isFeatureEnabled } from '@superset-ui/core';
import TableElement, { Column } from 'src/SqlLab/components/TableElement';
import { table, initialState } from 'src/SqlLab/fixtures';
import { render, waitFor, fireEvent } from 'spec/helpers/testing-library';
import * as sqlLabActions from 'src/SqlLab/actions/sqlLab';
import { QueryEditor } from 'src/SqlLab/types';
jest.mock('@superset-ui/core', () => ({
...jest.requireActual('@superset-ui/core'),
isFeatureEnabled: jest.fn(),
}));
const mockedIsFeatureEnabled = isFeatureEnabled as jest.Mock;
jest.mock('@superset-ui/core/components/Loading', () => ({
Loading: () => <div data-test="mock-loading" />,
}));
jest.mock('@superset-ui/core/components/IconTooltip', () => ({
IconTooltip: ({
onClick,
tooltip,
}: {
onClick: () => void;
tooltip: string;
}) => (
<button type="button" data-test="mock-icon-tooltip" onClick={onClick}>
{tooltip}
</button>
),
}));
jest.mock(
'src/SqlLab/components/ColumnElement',
() =>
({ column }: { column: Column }) => (
<div data-test="mock-column-element">{column.name}</div>
),
);
const getTableMetadataEndpoint =
/\/api\/v1\/database\/\d+\/table_metadata\/(?:\?.*)?$/;
const getExtraTableMetadataEndpoint =
/\/api\/v1\/database\/\d+\/table_metadata\/extra\/(?:\?.*)?$/;
const updateTableSchemaExpandedEndpoint = 'glob:*/tableschemaview/*/expanded';
const updateTableSchemaEndpoint = 'glob:*/tableschemaview/';
beforeEach(() => {
fetchMock.get(getTableMetadataEndpoint, table);
fetchMock.get(getExtraTableMetadataEndpoint, {});
fetchMock.post(updateTableSchemaExpandedEndpoint, {});
fetchMock.post(updateTableSchemaEndpoint, {});
});
afterEach(() => fetchMock.clearHistory().removeRoutes());
const mockedProps = {
table: {
...table,
initialized: true,
},
activeKey: [table.id],
};
const createStateWithQueryEditor = (queryEditor: Partial<QueryEditor>) => ({
...initialState,
sqlLab: {
...initialState.sqlLab,
queryEditors: [queryEditor],
},
});
const setupSyncTableTest = () => {
const spy = jest.spyOn(sqlLabActions, 'syncTable');
mockedIsFeatureEnabled.mockImplementation(
featureFlag => featureFlag === FeatureFlag.SqllabBackendPersistence,
);
fetchMock.removeRoute(updateTableSchemaEndpoint);
fetchMock.post(
updateTableSchemaEndpoint,
{ id: 100 },
{ name: updateTableSchemaEndpoint },
);
return spy;
};
test('renders', () => {
expect(isValidElement(<TableElement table={table} />)).toBe(true);
});
test('renders with props', () => {
expect(isValidElement(<TableElement {...mockedProps} />)).toBe(true);
});
test('has 4 IconTooltip elements', async () => {
const { getAllByTestId } = render(<TableElement {...mockedProps} />, {
useRedux: true,
initialState,
});
await waitFor(() =>
expect(getAllByTestId('mock-icon-tooltip')).toHaveLength(6),
);
});
test('has 14 columns', async () => {
const { getAllByTestId } = render(<TableElement {...mockedProps} />, {
useRedux: true,
initialState,
});
await waitFor(() =>
expect(getAllByTestId('mock-column-element')).toHaveLength(14),
);
});
test('fades table', async () => {
const { getAllByTestId } = render(<TableElement {...mockedProps} />, {
useRedux: true,
initialState,
});
await waitFor(() =>
expect(getAllByTestId('mock-icon-tooltip')).toHaveLength(6),
);
const style = window.getComputedStyle(getAllByTestId('fade')[0]);
expect(style.opacity).toBe('0');
fireEvent.mouseEnter(getAllByTestId('table-element-header-container')[0]);
await waitFor(() =>
expect(window.getComputedStyle(getAllByTestId('fade')[0]).opacity).toBe(
'1',
),
);
});
test('sorts columns', async () => {
const { getAllByTestId, getByText } = render(
<TableElement {...mockedProps} />,
{
useRedux: true,
initialState,
},
);
await waitFor(() =>
expect(getAllByTestId('mock-icon-tooltip')).toHaveLength(6),
);
expect(
getAllByTestId('mock-column-element').map(el => el.textContent),
).toEqual(table.columns.map(col => col.name));
fireEvent.click(getByText('Sort columns alphabetically'));
const sorted = table.columns.map(col => col.name).sort();
expect(
getAllByTestId('mock-column-element').map(el => el.textContent),
).toEqual(sorted);
expect(getAllByTestId('mock-column-element')[0]).toHaveTextContent('active');
});
test('removes the table', async () => {
const updateTableSchemaEndpoint = 'glob:*/tableschemaview/*';
fetchMock.delete(updateTableSchemaEndpoint, {});
mockedIsFeatureEnabled.mockImplementation(
featureFlag => featureFlag === FeatureFlag.SqllabBackendPersistence,
);
const { getAllByTestId, getByText } = render(
<TableElement {...mockedProps} />,
{
useRedux: true,
initialState,
},
);
await waitFor(() =>
expect(getAllByTestId('mock-icon-tooltip')).toHaveLength(6),
);
expect(fetchMock.callHistory.calls(updateTableSchemaEndpoint)).toHaveLength(
0,
);
fireEvent.click(getByText('Remove table preview'));
await waitFor(() =>
expect(fetchMock.callHistory.calls(updateTableSchemaEndpoint)).toHaveLength(
1,
),
);
mockedIsFeatureEnabled.mockClear();
});
test('fetches table metadata when expanded', async () => {
render(<TableElement {...mockedProps} />, {
useRedux: true,
initialState,
});
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(0);
expect(
fetchMock.callHistory.calls(getExtraTableMetadataEndpoint),
).toHaveLength(0);
await waitFor(() =>
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(
1,
),
);
expect(
fetchMock.callHistory.calls(updateTableSchemaExpandedEndpoint),
).toHaveLength(0);
expect(
fetchMock.callHistory.calls(getExtraTableMetadataEndpoint),
).toHaveLength(1);
});
test('refreshes table metadata when triggered', async () => {
const { getAllByTestId, getByText } = render(
<TableElement {...mockedProps} />,
{
useRedux: true,
initialState,
},
);
await waitFor(() =>
expect(getAllByTestId('mock-icon-tooltip')).toHaveLength(6),
);
expect(fetchMock.callHistory.calls(updateTableSchemaEndpoint)).toHaveLength(
0,
);
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(1);
fireEvent.click(getByText('Refresh table schema'));
await waitFor(() =>
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(
2,
),
);
await waitFor(() =>
expect(fetchMock.callHistory.calls(updateTableSchemaEndpoint)).toHaveLength(
1,
),
);
});
test('calls syncTable with valid backend ID when query editor has tabViewId', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: false,
queryEditorId: 'temp-id-123',
};
const state = createStateWithQueryEditor({
id: 'temp-id-123',
tabViewId: '42',
inLocalStorage: false,
name: 'Test Editor',
});
render(<TableElement table={testTable} activeKey={[testTable.id]} />, {
useRedux: true,
initialState: state,
});
await waitFor(() => {
expect(syncTableSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.any(Object),
'42', // finalQueryEditorId
);
});
syncTableSpy.mockRestore();
});
test('does not call syncTable when query editor is in localStorage', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: false,
queryEditorId: 'local-id',
};
const state = createStateWithQueryEditor({
id: 'local-id',
tabViewId: undefined,
inLocalStorage: true,
name: 'Local Editor',
});
render(<TableElement table={testTable} activeKey={[testTable.id]} />, {
useRedux: true,
initialState: state,
});
await waitFor(() => {
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(
1,
);
});
await new Promise(resolve => setTimeout(resolve, 100));
expect(syncTableSpy).not.toHaveBeenCalled();
syncTableSpy.mockRestore();
});
test('does not call syncTable with non-numeric queryEditorId', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: false,
queryEditorId: 'not-a-number',
};
const state = createStateWithQueryEditor({
id: 'not-a-number',
tabViewId: 'also-not-a-number',
inLocalStorage: false,
name: 'Invalid Editor',
});
render(<TableElement table={testTable} activeKey={[testTable.id]} />, {
useRedux: true,
initialState: state,
});
await waitFor(() => {
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(
1,
);
});
await new Promise(resolve => setTimeout(resolve, 100));
expect(syncTableSpy).not.toHaveBeenCalled();
syncTableSpy.mockRestore();
});
test('does not call syncTable for already initialized tables', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: true, // Already initialized
queryEditorId: '789',
};
const state = createStateWithQueryEditor({
id: '789',
tabViewId: '789',
inLocalStorage: false,
name: 'Initialized Editor',
});
render(<TableElement table={testTable} activeKey={[testTable.id]} />, {
useRedux: true,
initialState: state,
});
await waitFor(() => {
expect(fetchMock.callHistory.calls(getTableMetadataEndpoint)).toHaveLength(
1,
);
});
await new Promise(resolve => setTimeout(resolve, 100));
expect(syncTableSpy).not.toHaveBeenCalled();
syncTableSpy.mockRestore();
});
test('calls syncTable after query editor is migrated from localStorage', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: false,
queryEditorId: 'temp-editor-id',
};
// Start with editor in localStorage
const localState = createStateWithQueryEditor({
id: 'temp-editor-id',
tabViewId: undefined,
inLocalStorage: true,
name: 'Temp Editor',
});
const { rerender } = render(
<TableElement table={testTable} activeKey={[testTable.id]} />,
{
useRedux: true,
initialState: localState,
},
);
await new Promise(resolve => setTimeout(resolve, 100));
expect(syncTableSpy).not.toHaveBeenCalled();
const migratedState = createStateWithQueryEditor({
id: 'temp-editor-id',
tabViewId: '999',
inLocalStorage: false,
name: 'Temp Editor',
});
rerender(<TableElement table={testTable} activeKey={[testTable.id]} />);
const { unmount } = render(
<TableElement table={testTable} activeKey={[testTable.id]} />,
{
useRedux: true,
initialState: migratedState,
},
);
await waitFor(() => {
expect(syncTableSpy).toHaveBeenCalledWith(
expect.any(Object),
expect.any(Object),
'999',
);
});
unmount();
syncTableSpy.mockRestore();
});
test('passes numeric queryEditorId validation', async () => {
const syncTableSpy = setupSyncTableTest();
const testTable = {
...table,
initialized: false,
queryEditorId: 'editor-123',
};
const state = createStateWithQueryEditor({
id: 'editor-123',
tabViewId: '456',
inLocalStorage: false,
name: 'Valid Editor',
});
render(<TableElement table={testTable} activeKey={[testTable.id]} />, {
useRedux: true,
initialState: state,
});
await waitFor(() => {
expect(syncTableSpy).toHaveBeenCalled();
const [, , finalQueryEditorId] = syncTableSpy.mock.calls[0];
// Verify it's a valid numeric string
expect(Number.isNaN(Number(finalQueryEditorId))).toBe(false);
expect(typeof finalQueryEditorId).toBe('string');
expect(finalQueryEditorId).toMatch(/^\d+$/);
});
syncTableSpy.mockRestore();
});

View File

@@ -1,420 +0,0 @@
/**
* Licensed to the Apache Software Foundation (ASF) under one
* or more contributor license agreements. See the NOTICE file
* distributed with this work for additional information
* regarding copyright ownership. The ASF licenses this file
* to you under the Apache License, Version 2.0 (the
* "License"); you may not use this file except in compliance
* with the License. You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing,
* software distributed under the License is distributed on an
* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
* KIND, either express or implied. See the License for the
* specific language governing permissions and limitations
* under the License.
*/
import { useState, useRef, useEffect } from 'react';
import { useSelector } from 'react-redux';
import { useAppDispatch } from 'src/SqlLab/hooks/useAppDispatch';
import type { QueryEditor, SqlLabRootState, Table } from 'src/SqlLab/types';
import {
ButtonGroup,
Card,
Collapse,
Tooltip,
Flex,
IconTooltip,
Loading,
ModalTrigger,
type CollapseProps,
} from '@superset-ui/core/components';
import { CopyToClipboard } from 'src/components';
import { t } from '@apache-superset/core/translation';
import { styled, useTheme } from '@apache-superset/core/theme';
import { debounce } from 'lodash';
import {
removeDataPreview,
removeTables,
addDangerToast,
syncTable,
} from 'src/SqlLab/actions/sqlLab';
import {
tableApiUtil,
useTableExtendedMetadataQuery,
useTableMetadataQuery,
} from 'src/hooks/apiResources';
import useEffectEvent from 'src/hooks/useEffectEvent';
import { ActionType } from 'src/types/Action';
import { Icons } from '@superset-ui/core/components/Icons';
import { Space } from '@superset-ui/core/components/Space';
import ColumnElement, { ColumnKeyTypeType } from '../ColumnElement';
import ShowSQL from '../ShowSQL';
export interface Column {
name: string;
keys?: { type: ColumnKeyTypeType }[];
type: string;
}
export interface TableElementProps extends CollapseProps {
table: Table;
}
const StyledSpan = styled.span`
cursor: pointer;
`;
const Fade = styled.div`
transition: all ${({ theme }) => theme.motionDurationMid};
opacity: ${(props: { hovered: boolean }) => (props.hovered ? 1 : 0)};
`;
const TableElement = ({ table, ...props }: TableElementProps) => {
const { dbId, catalog, schema, name, expanded, id } = table;
const theme = useTheme();
const dispatch = useAppDispatch();
const {
currentData: tableMetadata,
isSuccess: isMetadataSuccess,
isFetching: isMetadataFetching,
isError: hasMetadataError,
} = useTableMetadataQuery(
{
dbId,
catalog,
schema,
table: name,
},
{ skip: !expanded },
);
const {
currentData: tableExtendedMetadata,
isSuccess: isExtraMetadataSuccess,
isLoading: isExtraMetadataLoading,
isError: hasExtendedMetadataError,
} = useTableExtendedMetadataQuery(
{
dbId,
catalog,
schema,
table: name,
},
{ skip: !expanded },
);
const tableData = {
...tableMetadata,
...tableExtendedMetadata,
};
const queryEditors = useSelector<SqlLabRootState, QueryEditor[]>(
state => state.sqlLab.queryEditors,
);
const currentTable = { ...tableData, ...table };
const { queryEditorId } = currentTable;
const queryEditor = queryEditors.find(
qe => qe.id === queryEditorId || qe.tabViewId === queryEditorId,
);
const currentQueryEditorId = queryEditor?.tabViewId || queryEditorId;
useEffect(() => {
if (hasMetadataError || hasExtendedMetadataError) {
dispatch(
addDangerToast(t('An error occurred while fetching table metadata')),
);
}
}, [hasMetadataError, hasExtendedMetadataError, dispatch]);
// TODO: migrate syncTable logic by SIP-93
const syncTableMetadata = useEffectEvent(() => {
const { initialized } = table;
// if not a valid number, wait for backend to assign one
const hasFinalQueryEditorId =
currentQueryEditorId &&
!Number.isNaN(Number(currentQueryEditorId)) &&
currentTable.queryEditorId !== currentQueryEditorId;
if (!initialized && hasFinalQueryEditorId) {
dispatch(syncTable(currentTable, tableData, currentQueryEditorId));
}
});
useEffect(() => {
if (isMetadataSuccess && isExtraMetadataSuccess) {
syncTableMetadata();
}
}, [
isMetadataSuccess,
isExtraMetadataSuccess,
currentQueryEditorId,
syncTableMetadata,
]);
const [sortColumns, setSortColumns] = useState(false);
const [hovered, setHovered] = useState(false);
const tableNameRef = useRef<HTMLInputElement>(null);
const setHover = (hovered: boolean) => {
debounce(() => setHovered(hovered), 100)();
};
const removeTable = () => {
dispatch(removeDataPreview(table));
dispatch(removeTables([table]));
};
const toggleSortColumns = () => {
setSortColumns(prevState => !prevState);
};
const refreshTableMetadata = () => {
dispatch(
tableApiUtil.invalidateTags([{ type: 'TableMetadatas', id: name }]),
);
dispatch(syncTable(table, tableData, table.queryEditorId));
};
const renderWell = () => {
let partitions;
let metadata;
if (tableData.partitions) {
let partitionQuery;
let partitionClipBoard;
if (tableData.partitions.partitionQuery) {
({ partitionQuery } = tableData.partitions);
const tt = t('Copy partition query to clipboard');
partitionClipBoard = (
<CopyToClipboard
text={partitionQuery}
shouldShowText={false}
tooltipText={tt}
copyNode={<Icons.CopyOutlined iconSize="s" />}
/>
);
}
const latest = Object.entries(tableData.partitions?.latest || [])
.map(([key, value]) => `${key}=${value}`)
.join('/');
partitions = (
<div>
<small>
{t('latest partition:')} {latest}
</small>{' '}
{partitionClipBoard}
</div>
);
}
if (tableData.metadata) {
metadata = Object.entries(tableData.metadata).map(([key, value]) => (
<div>
<small>
<strong>{key}:</strong> {value}
</small>
</div>
));
if (!metadata?.length) {
// hide metadata card view
return null;
}
}
if (!partitions) {
// hide partition card view
return null;
}
return (
<Card size="small">
{partitions}
{metadata}
</Card>
);
};
const renderControls = () => {
let keyLink;
const KEYS_FOR_TABLE_TEXT = t('Keys for table');
if (tableData?.indexes?.length) {
keyLink = (
<ModalTrigger
modalTitle={`${KEYS_FOR_TABLE_TEXT} ${name}`}
modalBody={tableData.indexes.map((ix, i) => (
<pre key={i}>{JSON.stringify(ix, null, ' ')}</pre>
))}
triggerNode={
<IconTooltip
className="pull-left"
tooltip={t('View keys & indexes (%s)', tableData.indexes.length)}
>
<Icons.TableOutlined
iconSize="m"
iconColor={theme.colorPrimary}
/>
</IconTooltip>
}
/>
);
}
return (
<Flex style={{ height: 22 }} align="center">
{isMetadataFetching || isExtraMetadataLoading ? (
<Loading position="inline" />
) : (
<Fade
data-test="fade"
hovered={hovered}
onClick={e => e.stopPropagation()}
>
<ButtonGroup>
<Space size="small">
<IconTooltip
className="pull-left pointer"
onClick={refreshTableMetadata}
tooltip={t('Refresh table schema')}
>
<Icons.SyncOutlined
iconSize="m"
iconColor={theme.colorIcon}
/>
</IconTooltip>
{keyLink}
<IconTooltip
onClick={toggleSortColumns}
tooltip={
sortColumns
? t('Original table column order')
: t('Sort columns alphabetically')
}
>
<Icons.SortAscendingOutlined
iconSize="m"
aria-hidden
iconColor={
sortColumns ? theme.colorIcon : theme.colorTextDisabled
}
/>
</IconTooltip>
{tableData.selectStar && (
<CopyToClipboard
copyNode={
<IconTooltip
aria-label={t('Copy')}
tooltip={t('Copy SELECT statement to the clipboard')}
>
<Icons.CopyOutlined
iconSize="m"
aria-hidden
iconColor={theme.colorIcon}
/>
</IconTooltip>
}
text={tableData.selectStar}
shouldShowText={false}
/>
)}
{tableData.view && (
<ShowSQL
sql={tableData.view}
tooltipText={t('Show CREATE VIEW statement')}
title={t('CREATE VIEW statement')}
/>
)}
<IconTooltip
className=" table-remove pull-left pointer"
onClick={removeTable}
tooltip={t('Remove table preview')}
>
<Icons.CloseOutlined
iconSize="m"
aria-hidden
iconColor={theme.colorIcon}
/>
</IconTooltip>
</Space>
</ButtonGroup>
</Fade>
)}
</Flex>
);
};
const renderHeader = () => {
const element: HTMLInputElement | null = tableNameRef.current;
let trigger = [] as ActionType[];
if (element && element.offsetWidth < element.scrollWidth) {
trigger = ['hover'];
}
return (
<div
data-test="table-element-header-container"
className="clearfix header-container"
>
<Tooltip
id="copy-to-clipboard-tooltip"
style={{ cursor: 'pointer' }}
title={name}
trigger={trigger}
>
<StyledSpan
data-test="collapse"
ref={tableNameRef}
className="table-name"
>
<strong>{name}</strong>
</StyledSpan>
</Tooltip>
</div>
);
};
const renderBody = () => {
let cols;
if (tableData.columns) {
cols = tableData.columns.slice();
if (sortColumns) {
cols.sort((a: Column, b: Column) => {
const colA = a.name.toUpperCase();
const colB = b.name.toUpperCase();
return colA < colB ? -1 : colA > colB ? 1 : 0;
});
}
}
const metadata = (
<div data-test="table-element" css={{ paddingTop: 6 }}>
{renderWell()}
<div>
{cols?.map(col => (
<ColumnElement column={col} key={col.name} />
))}
</div>
</div>
);
return metadata;
};
return (
<Collapse
activeKey={props.activeKey}
expandIconPosition="end"
onChange={props.onChange}
ghost
items={[
{
key: id,
label: renderHeader(),
children: renderBody(),
extra: renderControls(),
onMouseEnter: () => setHover(true),
onMouseLeave: () => setHover(false),
},
]}
/>
);
};
export default TableElement;

View File

@@ -144,7 +144,11 @@ export const processEvents = async (events: AsyncEvent[]) => {
events.forEach((asyncEvent: AsyncEvent) => {
const jobId = asyncEvent.job_id;
const listener = listenersByJobId.get(jobId);
if (listener) {
// `jobId` originates from server/WebSocket payloads, so the listener is
// resolved exclusively through a Map (never plain-object property access,
// which would expose the prototype chain), and we confirm the retrieved
// value is a registered function before dispatching the event to it.
if (typeof listener === 'function') {
listener(asyncEvent);
retriesByJobId.delete(jobId);
} else {

View File

@@ -362,7 +362,7 @@ def safe_insert_dashboard_chart_relationships(
# Get existing relationships only for dashboards being updated
dashboard_ids = {dashboard_id for dashboard_id, _ in dashboard_chart_ids}
existing_relationships = db.session.execute(
select([dashboard_slices.c.dashboard_id, dashboard_slices.c.slice_id]).where(
select(dashboard_slices.c.dashboard_id, dashboard_slices.c.slice_id).where(
dashboard_slices.c.dashboard_id.in_(dashboard_ids)
)
).fetchall()

View File

@@ -32,11 +32,9 @@ def add_types_to_charts(
charts = (
select(
[
tag.c.id.label("tag_id"),
slices.c.id.label("object_id"),
literal(ObjectType.chart.name).label("object_type"),
]
tag.c.id.label("tag_id"),
slices.c.id.label("object_id"),
literal(ObjectType.chart.name).label("object_type"),
)
.select_from(
join(
@@ -64,11 +62,9 @@ def add_types_to_dashboards(
dashboards = (
select(
[
tag.c.id.label("tag_id"),
dashboard_table.c.id.label("object_id"),
literal(ObjectType.dashboard.name).label("object_type"),
]
tag.c.id.label("tag_id"),
dashboard_table.c.id.label("object_id"),
literal(ObjectType.dashboard.name).label("object_type"),
)
.select_from(
join(
@@ -96,11 +92,9 @@ def add_types_to_saved_queries(
saved_queries = (
select(
[
tag.c.id.label("tag_id"),
saved_query.c.id.label("object_id"),
literal(ObjectType.query.name).label("object_type"),
]
tag.c.id.label("tag_id"),
saved_query.c.id.label("object_id"),
literal(ObjectType.query.name).label("object_type"),
)
.select_from(
join(
@@ -128,11 +122,9 @@ def add_types_to_datasets(
datasets = (
select(
[
tag.c.id.label("tag_id"),
tables.c.id.label("object_id"),
literal(ObjectType.dataset.name).label("object_type"),
]
tag.c.id.label("tag_id"),
tables.c.id.label("object_id"),
literal(ObjectType.dataset.name).label("object_type"),
)
.select_from(
join(
@@ -238,11 +230,9 @@ def add_owners_to_charts(
charts = (
select(
[
tag.c.id.label("tag_id"),
slices.c.id.label("object_id"),
literal(ObjectType.chart.name).label("object_type"),
]
tag.c.id.label("tag_id"),
slices.c.id.label("object_id"),
literal(ObjectType.chart.name).label("object_type"),
)
.select_from(
join(
@@ -274,11 +264,9 @@ def add_owners_to_dashboards(
dashboards = (
select(
[
tag.c.id.label("tag_id"),
dashboard_table.c.id.label("object_id"),
literal(ObjectType.dashboard.name).label("object_type"),
]
tag.c.id.label("tag_id"),
dashboard_table.c.id.label("object_id"),
literal(ObjectType.dashboard.name).label("object_type"),
)
.select_from(
join(
@@ -310,11 +298,9 @@ def add_owners_to_saved_queries(
saved_queries = (
select(
[
tag.c.id.label("tag_id"),
saved_query.c.id.label("object_id"),
literal(ObjectType.query.name).label("object_type"),
]
tag.c.id.label("tag_id"),
saved_query.c.id.label("object_id"),
literal(ObjectType.query.name).label("object_type"),
)
.select_from(
join(
@@ -346,11 +332,9 @@ def add_owners_to_datasets(
datasets = (
select(
[
tag.c.id.label("tag_id"),
tables.c.id.label("object_id"),
literal(ObjectType.dataset.name).label("object_type"),
]
tag.c.id.label("tag_id"),
tables.c.id.label("object_id"),
literal(ObjectType.dataset.name).label("object_type"),
)
.select_from(
join(
@@ -440,7 +424,7 @@ def add_owners(metadata: MetaData) -> None:
columns = ["tag_id", "object_id", "object_type"]
# create a custom tag for each user
ids = select([users.c.id])
ids = select(users.c.id)
insert = tag.insert()
for (id_,) in db.session.execute(ids):
with contextlib.suppress(IntegrityError): # already exists
@@ -478,18 +462,16 @@ def add_favorites(metadata: MetaData) -> None:
columns = ["tag_id", "object_id", "object_type"]
# create a custom tag for each user
ids = select([users.c.id])
ids = select(users.c.id)
insert = tag.insert()
for (id_,) in db.session.execute(ids):
with contextlib.suppress(IntegrityError): # already exists
db.session.execute(insert, name=f"favorited_by:{id_}", type=TagType.type)
favstars = (
select(
[
tag.c.id.label("tag_id"),
favstar.c.obj_id.label("object_id"),
func.lower(favstar.c.class_name).label("object_type"),
]
tag.c.id.label("tag_id"),
favstar.c.obj_id.label("object_id"),
func.lower(favstar.c.class_name).label("object_type"),
)
.select_from(
join(

View File

@@ -1791,11 +1791,9 @@ class SqlaTable(
# for those we fall back to LIMIT 1.
tbl, _unused_cte = self.get_from_clause(template_processor)
if self.db_engine_spec.type_probe_needs_row:
qry = sa.select([sqla_column]).limit(1).select_from(tbl)
qry = sa.select(sqla_column).limit(1).select_from(tbl)
else:
qry = (
sa.select([sqla_column]).where(sa.false()).select_from(tbl)
)
qry = sa.select(sqla_column).where(sa.false()).select_from(tbl)
sql = self.database.compile_sqla_query(
qry,
catalog=self.catalog,

View File

@@ -1986,7 +1986,9 @@ class BaseEngineSpec: # pylint: disable=too-many-public-methods
fields = cls._get_fields(cols)
full_table_name = cls.quote_table(table, dialect)
qry = select(fields).select_from(text(full_table_name))
qry = select(*fields if isinstance(fields, list) else fields).select_from(
text(full_table_name)
)
qry = qry.limit(limit)
if latest_partition:

View File

@@ -368,7 +368,7 @@ class SupersetShillelaghAdapter(Adapter):
"""
Build SQLAlchemy query object.
"""
query = select([self._table])
query = select(self._table)
for column_name, filter_ in bounds.items():
column = self._table.c[column_name]
@@ -452,7 +452,7 @@ class SupersetShillelaghAdapter(Adapter):
if self._rowid:
return result.inserted_primary_key[0]
query = select([func.count()]).select_from(self._table)
query = select(func.count()).select_from(self._table)
return connection.execute(query).scalar()
@check_dml

View File

@@ -49,9 +49,9 @@ def upgrade():
clusters = sa.Table("clusters", metadata, autoload=True)
statement = datasources.update().values(
cluster_id=sa.select([clusters.c.id])
cluster_id=sa.select(clusters.c.id)
.where(datasources.c.cluster_name == clusters.c.cluster_name)
.as_scalar()
.scalar_subquery()
)
bind.execute(statement)
@@ -91,9 +91,9 @@ def downgrade():
clusters = sa.Table("clusters", metadata, autoload=True)
statement = datasources.update().values(
cluster_name=sa.select([clusters.c.cluster_name])
cluster_name=sa.select(clusters.c.cluster_name)
.where(datasources.c.cluster_id == clusters.c.id)
.as_scalar()
.scalar_subquery()
)
bind.execute(statement)

View File

@@ -49,7 +49,7 @@ def upgrade():
)
rlsf = sa.Table("row_level_security_filters", metadata, autoload=True)
filter_ids = sa.select([rlsf.c.id, rlsf.c.table_id])
filter_ids = sa.select(rlsf.c.id, rlsf.c.table_id)
for row in bind.execute(filter_ids):
move_table_id = rls_filter_tables.insert().values(
@@ -85,7 +85,7 @@ def downgrade():
rls_filter_tables = sa.Table("rls_filter_tables", metadata, autoload=True)
rls_filter_roles = sa.Table("rls_filter_roles", metadata, autoload=True)
filter_tables = sa.select([rls_filter_tables.c.rls_filter_id]).group_by(
filter_tables = sa.select(rls_filter_tables.c.rls_filter_id).group_by(
rls_filter_tables.c.rls_filter_id
)
@@ -95,7 +95,7 @@ def downgrade():
filter_params = dict(bind.execute(filter_query).fetchone())
origin_id = filter_params.pop("id", None)
table_ids = bind.execute(
sa.select([rls_filter_tables.c.table_id]).where(
sa.select(rls_filter_tables.c.table_id).where(
rls_filter_tables.c.rls_filter_id == row["rls_filter_id"]
)
).fetchall()

View File

@@ -336,23 +336,21 @@ def copy_tables(session: Session) -> None:
insert_from_select(
NewTable,
select(
[
# Tables need different uuid than datasets, since they are different
# entities. When INSERT FROM SELECT, we must provide a value for `uuid`,
# otherwise it'd use the default generated on Python side, which
# will cause duplicate values. They will be replaced by `assign_uuids` later. # noqa: E501
SqlaTable.uuid,
SqlaTable.id.label("sqlatable_id"),
SqlaTable.created_on,
SqlaTable.changed_on,
SqlaTable.created_by_fk,
SqlaTable.changed_by_fk,
SqlaTable.table_name.label("name"),
SqlaTable.schema,
SqlaTable.database_id,
SqlaTable.is_managed_externally,
SqlaTable.external_url,
]
# Tables need different uuid than datasets, since they are different
# entities. When INSERT FROM SELECT, we must provide a value for `uuid`,
# otherwise it'd use the default generated on Python side, which
# will cause duplicate values. They will be replaced by `assign_uuids` later. # noqa: E501
SqlaTable.uuid,
SqlaTable.id.label("sqlatable_id"),
SqlaTable.created_on,
SqlaTable.changed_on,
SqlaTable.created_by_fk,
SqlaTable.changed_by_fk,
SqlaTable.table_name.label("name"),
SqlaTable.schema,
SqlaTable.database_id,
SqlaTable.is_managed_externally,
SqlaTable.external_url,
)
# use an inner join to filter out only tables with valid database ids
.select_from(sa.join(SqlaTable, Database, SqlaTable.database_id == Database.id))
@@ -369,20 +367,18 @@ def copy_datasets(session: Session) -> None:
insert_from_select(
NewDataset,
select(
[
SqlaTable.uuid,
SqlaTable.created_on,
SqlaTable.changed_on,
SqlaTable.created_by_fk,
SqlaTable.changed_by_fk,
SqlaTable.database_id,
SqlaTable.table_name.label("name"),
func.coalesce(SqlaTable.sql, SqlaTable.table_name).label("expression"),
is_physical_table.label("is_physical"),
SqlaTable.is_managed_externally,
SqlaTable.external_url,
SqlaTable.extra.label("extra_json"),
]
SqlaTable.uuid,
SqlaTable.created_on,
SqlaTable.changed_on,
SqlaTable.created_by_fk,
SqlaTable.changed_by_fk,
SqlaTable.database_id,
SqlaTable.table_name.label("name"),
func.coalesce(SqlaTable.sql, SqlaTable.table_name).label("expression"),
is_physical_table.label("is_physical"),
SqlaTable.is_managed_externally,
SqlaTable.external_url,
SqlaTable.extra.label("extra_json"),
),
)
@@ -390,7 +386,7 @@ def copy_datasets(session: Session) -> None:
insert_from_select(
dataset_user_association_table,
select(
[NewDataset.id.label("dataset_id"), sqlatable_user_table.c.user_id]
NewDataset.id.label("dataset_id"), sqlatable_user_table.c.user_id
).select_from(
sqlatable_user_table.join(
SqlaTable, SqlaTable.id == sqlatable_user_table.c.table_id
@@ -402,10 +398,8 @@ def copy_datasets(session: Session) -> None:
insert_from_select(
dataset_table_association_table,
select(
[
NewDataset.id.label("dataset_id"),
NewTable.id.label("table_id"),
]
NewDataset.id.label("dataset_id"),
NewTable.id.label("table_id"),
).select_from(
sa.join(SqlaTable, NewTable, NewTable.sqlatable_id == SqlaTable.id).join(
NewDataset, NewDataset.uuid == SqlaTable.uuid
@@ -423,25 +417,23 @@ def copy_columns(session: Session) -> None:
insert_from_select(
NewColumn,
select(
[
TableColumn.uuid,
TableColumn.created_on,
TableColumn.changed_on,
TableColumn.created_by_fk,
TableColumn.changed_by_fk,
TableColumn.groupby.label("is_dimensional"),
TableColumn.filterable.label("is_filterable"),
TableColumn.column_name.label("name"),
TableColumn.description,
func.coalesce(TableColumn.expression, TableColumn.column_name).label(
"expression"
),
sa.literal(False).label("is_aggregation"),
is_physical_column.label("is_physical"),
func.coalesce(TableColumn.is_dttm, False).label("is_temporal"),
func.coalesce(TableColumn.type, UNKNOWN_TYPE).label("type"),
TableColumn.extra.label("extra_json"),
]
TableColumn.uuid,
TableColumn.created_on,
TableColumn.changed_on,
TableColumn.created_by_fk,
TableColumn.changed_by_fk,
TableColumn.groupby.label("is_dimensional"),
TableColumn.filterable.label("is_filterable"),
TableColumn.column_name.label("name"),
TableColumn.description,
func.coalesce(TableColumn.expression, TableColumn.column_name).label(
"expression"
),
sa.literal(False).label("is_aggregation"),
is_physical_column.label("is_physical"),
func.coalesce(TableColumn.is_dttm, False).label("is_temporal"),
func.coalesce(TableColumn.type, UNKNOWN_TYPE).label("type"),
TableColumn.extra.label("extra_json"),
).select_from(active_table_columns),
)
@@ -452,10 +444,8 @@ def copy_columns(session: Session) -> None:
insert_from_select(
dataset_column_association_table,
select(
[
NewDataset.id.label("dataset_id"),
NewColumn.id.label("column_id"),
],
NewDataset.id.label("dataset_id"),
NewColumn.id.label("column_id"),
).select_from(
joined_columns_table.join(NewDataset, NewDataset.uuid == SqlaTable.uuid)
),
@@ -472,33 +462,31 @@ def copy_metrics(session: Session) -> None:
insert_from_select(
NewColumn,
select(
[
SqlMetric.uuid,
SqlMetric.created_on,
SqlMetric.changed_on,
SqlMetric.created_by_fk,
SqlMetric.changed_by_fk,
SqlMetric.metric_name.label("name"),
SqlMetric.expression,
SqlMetric.description,
sa.literal(UNKNOWN_TYPE).label("type"),
(
func.coalesce(
sa.func.lower(SqlMetric.metric_type).in_(
ADDITIVE_METRIC_TYPES_LOWER
),
sa.literal(False),
).label("is_additive")
),
sa.literal(True).label("is_aggregation"),
# metrics are by default not filterable
sa.literal(False).label("is_filterable"),
sa.literal(False).label("is_dimensional"),
sa.literal(False).label("is_physical"),
sa.literal(False).label("is_temporal"),
SqlMetric.extra.label("extra_json"),
SqlMetric.warning_text,
]
SqlMetric.uuid,
SqlMetric.created_on,
SqlMetric.changed_on,
SqlMetric.created_by_fk,
SqlMetric.changed_by_fk,
SqlMetric.metric_name.label("name"),
SqlMetric.expression,
SqlMetric.description,
sa.literal(UNKNOWN_TYPE).label("type"),
(
func.coalesce(
sa.func.lower(SqlMetric.metric_type).in_(
ADDITIVE_METRIC_TYPES_LOWER
),
sa.literal(False),
).label("is_additive")
),
sa.literal(True).label("is_aggregation"),
# metrics are by default not filterable
sa.literal(False).label("is_filterable"),
sa.literal(False).label("is_dimensional"),
sa.literal(False).label("is_physical"),
sa.literal(False).label("is_temporal"),
SqlMetric.extra.label("extra_json"),
SqlMetric.warning_text,
).select_from(active_metrics),
)
@@ -506,10 +494,8 @@ def copy_metrics(session: Session) -> None:
insert_from_select(
dataset_column_association_table,
select(
[
NewDataset.id.label("dataset_id"),
NewColumn.id.label("column_id"),
],
NewDataset.id.label("dataset_id"),
NewColumn.id.label("column_id"),
).select_from(
active_metrics.join(NewDataset, NewDataset.uuid == SqlaTable.uuid).join(
NewColumn, NewColumn.uuid == SqlMetric.uuid
@@ -568,15 +554,13 @@ def postprocess_datasets(session: Session) -> None: # noqa: C901
sqlalchemy_uri,
) in session.execute(
select(
[
NewDataset.database_id,
NewDataset.id.label("dataset_id"),
NewDataset.expression,
SqlaTable.extra,
NewDataset.is_physical,
SqlaTable.schema,
Database.sqlalchemy_uri,
]
NewDataset.database_id,
NewDataset.id.label("dataset_id"),
NewDataset.expression,
SqlaTable.extra,
NewDataset.is_physical,
SqlaTable.schema,
Database.sqlalchemy_uri,
)
.select_from(joined_tables)
.offset(offset)
@@ -725,29 +709,27 @@ def postprocess_columns(session: Session) -> None: # noqa: C901
query = (
select(
# sorted alphabetically
[
NewColumn.id.label("column_id"),
TableColumn.column_name,
NewColumn.changed_by_fk,
NewColumn.changed_on,
NewColumn.created_on,
NewColumn.description,
SqlMetric.d3format,
NewDataset.external_url,
NewColumn.extra_json,
NewColumn.is_dimensional,
NewColumn.is_filterable,
NewDataset.is_managed_externally,
NewColumn.is_physical,
SqlMetric.metric_type,
TableColumn.python_date_format,
Database.sqlalchemy_uri,
dataset_table_association_table.c.table_id,
func.coalesce(
TableColumn.verbose_name, SqlMetric.verbose_name
).label("verbose_name"),
NewColumn.warning_text,
]
NewColumn.id.label("column_id"),
TableColumn.column_name,
NewColumn.changed_by_fk,
NewColumn.changed_on,
NewColumn.created_on,
NewColumn.description,
SqlMetric.d3format,
NewDataset.external_url,
NewColumn.extra_json,
NewColumn.is_dimensional,
NewColumn.is_filterable,
NewDataset.is_managed_externally,
NewColumn.is_physical,
SqlMetric.metric_type,
TableColumn.python_date_format,
Database.sqlalchemy_uri,
dataset_table_association_table.c.table_id,
func.coalesce(TableColumn.verbose_name, SqlMetric.verbose_name).label(
"verbose_name"
),
NewColumn.warning_text,
)
.select_from(get_joined_tables(offset, limit))
.where(
@@ -872,7 +854,7 @@ def postprocess_columns(session: Session) -> None: # noqa: C901
print(" Assign table column relations...")
insert_from_select(
table_column_association_table,
select([NewColumn.table_id, NewColumn.id.label("column_id")])
select(NewColumn.table_id, NewColumn.id.label("column_id"))
.select_from(NewColumn)
.where(and_(NewColumn.is_physical, NewColumn.table_id.isnot(None))),
)

View File

@@ -59,12 +59,10 @@ def upgrade():
# Delete duplicates if any
min_id_subquery = (
select(
[
func.min(tagged_object_table.c.id).label("min_id"),
tagged_object_table.c.tag_id,
tagged_object_table.c.object_id,
tagged_object_table.c.object_type,
]
func.min(tagged_object_table.c.id).label("min_id"),
tagged_object_table.c.tag_id,
tagged_object_table.c.object_id,
tagged_object_table.c.object_type,
)
.group_by(
tagged_object_table.c.tag_id,
@@ -75,7 +73,7 @@ def upgrade():
)
delete_query = tagged_object_table.delete().where(
tagged_object_table.c.id.notin_(select([min_id_subquery.c.min_id]))
tagged_object_table.c.id.notin_(select(min_id_subquery.c.min_id))
)
bind.execute(delete_query)

View File

@@ -117,6 +117,7 @@ from superset.utils.core import (
GenericDataType,
get_base_axis_labels,
get_column_name,
get_column_names,
get_metric_names,
get_non_base_axis_columns,
get_user_id,
@@ -1564,6 +1565,61 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
"""
return self.query(qry)
def _python_date_format(self, column: str | None) -> str | None:
"""Return the column's configured ``python_date_format`` (e.g. ``epoch_s``
or a strftime pattern), or ``None`` if the column declares no format.
Reads from either a column object or a dict, matching ``_is_dttm``."""
if not hasattr(self, "get_column") or not (col := self.get_column(column)):
return None
fmt = (
col.get("python_date_format")
if isinstance(col, dict)
else getattr(col, "python_date_format", None)
)
return str(fmt) if fmt else None
def _collect_dttm_labels(
self, query_object: QueryObject
) -> tuple[tuple[str, str | None], ...]:
"""``(label, python_date_format)`` for the columns whose values should be
normalized to datetimes: base-axis / granularity columns (aggregated
charts), plus raw/unaggregated temporal columns that declare a
``python_date_format``. The raw columns are gated on the declared format
rather than ``is_dttm`` alone so a plain integer column is not misread as
nanosecond timestamps. The format is resolved here with a single column
lookup per label so callers need not look it up again."""
def _resolve(label: str | None) -> tuple[bool, str | None]:
"""``(is_dttm, python_date_format)`` from one ``get_column`` lookup."""
if not hasattr(self, "get_column") or not (col := self.get_column(label)):
return False, None
if isinstance(col, dict):
fmt = col.get("python_date_format")
return bool(col.get("is_dttm")), str(fmt) if fmt else None
fmt = getattr(col, "python_date_format", None)
return bool(col.is_dttm), str(fmt) if fmt else None
labels: list[tuple[str, str | None]] = []
seen: set[str] = set()
for label in [
*get_base_axis_labels(query_object.columns),
query_object.granularity,
]:
if not label or label in seen:
continue
is_dttm, fmt = _resolve(label)
if is_dttm:
labels.append((label, fmt))
seen.add(label)
for label in get_column_names(query_object.columns):
if label in seen:
continue
is_dttm, fmt = _resolve(label)
if is_dttm and fmt:
labels.append((label, fmt))
seen.add(label)
return tuple(labels)
def normalize_df(self, df: pd.DataFrame, query_object: QueryObject) -> pd.DataFrame:
"""
Normalize the dataframe by converting datetime columns and ensuring
@@ -1573,46 +1629,22 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
:param query_object: The query object with metadata about columns
:return: Normalized dataframe
"""
def _get_timestamp_format(column: str | None) -> str | None:
if not hasattr(self, "get_column"):
return None
column_obj = self.get_column(column)
if (
column_obj
and hasattr(column_obj, "python_date_format")
and (formatter := column_obj.python_date_format)
):
return str(formatter)
return None
# Collect datetime columns
labels = tuple(
label
for label in [
*get_base_axis_labels(query_object.columns),
query_object.granularity,
]
if hasattr(self, "get_column")
and (col := self.get_column(label))
and (col.get("is_dttm") if isinstance(col, dict) else col.is_dttm)
)
labels = self._collect_dttm_labels(query_object)
dttm_cols = [
DateColumn(
timestamp_format=_get_timestamp_format(label),
timestamp_format=fmt,
offset=self.offset,
time_shift=query_object.time_shift,
col_label=label,
)
for label in labels
if label
for label, fmt in labels
]
if DTTM_ALIAS in df:
dttm_cols.append(
DateColumn.get_legacy_time_column(
timestamp_format=_get_timestamp_format(query_object.granularity),
timestamp_format=self._python_date_format(query_object.granularity),
offset=self.offset,
time_shift=query_object.time_shift,
)
@@ -2806,7 +2838,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
# automatically add a random alias to the projection because of the
# call to DISTINCT; others will uppercase the column names. This
# gives us a deterministic column name in the dataframe.
[target_col.get_sqla_col(template_processor=tp).label("column_values")]
target_col.get_sqla_col(template_processor=tp).label("column_values")
)
.select_from(tbl)
.distinct()
@@ -2900,15 +2932,15 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
) -> Select:
"""Build validation query based on expression type. Raises on error."""
if expression_type == SqlExpressionType.COLUMN:
return sa.select([sa.literal_column(expression).label("test_col")])
return sa.select(sa.literal_column(expression).label("test_col"))
elif expression_type == SqlExpressionType.METRIC:
return sa.select([sa.literal_column(expression).label("test_metric")])
return sa.select(sa.literal_column(expression).label("test_metric"))
elif expression_type == SqlExpressionType.WHERE:
return sa.select([sa.literal(1)]).where(sa.text(expression))
return sa.select(sa.literal(1)).where(sa.text(expression))
elif expression_type == SqlExpressionType.HAVING:
dummy_col = sa.literal("A").label("dummy")
return (
sa.select([dummy_col])
sa.select(dummy_col)
.group_by(sa.text("dummy"))
.having(sa.text(expression))
)
@@ -3345,7 +3377,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
if not db_engine_spec.allows_hidden_orderby_agg:
select_exprs = remove_duplicates(select_exprs + orderby_exprs)
qry = sa.select(select_exprs)
qry = sa.select(*select_exprs)
if groupby_all_columns:
qry = qry.group_by(*groupby_all_columns.values())
@@ -3685,7 +3717,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
inner_select_exprs.append(inner)
inner_select_exprs += [inner_main_metric_expr]
subq = sa.select(inner_select_exprs).select_from(tbl)
subq = sa.select(*inner_select_exprs).select_from(tbl)
inner_time_filter = []
if dttm_col and not db_engine_spec.time_groupby_inline:
@@ -3750,7 +3782,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
)
# Reconstruct query with modified expressions
qry = sa.select(select_exprs)
qry = sa.select(*select_exprs)
if groupby_all_columns:
qry = qry.group_by(*groupby_all_columns.values())
@@ -3824,7 +3856,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
)
# Reconstruct query with modified expressions
qry = sa.select(select_exprs)
qry = sa.select(*select_exprs)
if groupby_all_columns:
qry = qry.group_by(*groupby_all_columns.values())
@@ -3851,7 +3883,7 @@ class ExploreMixin: # pylint: disable=too-many-public-methods
)
label = "rowcount"
col = self.make_sqla_column_compatible(literal_column("COUNT(*)"), label)
qry = sa.select([col]).select_from(qry.alias("rowcount_qry"))
qry = sa.select(col).select_from(qry.alias("rowcount_qry"))
labels_expected = [label]
filter_columns = [flt.get("col") for flt in filter] if filter else []

View File

@@ -788,7 +788,7 @@ def pessimistic_connection_handling(some_engine: Engine) -> None:
# run a SELECT 1. use a core select() so that
# the SELECT of a scalar value without a table is
# appropriately formatted for the backend
connection.scalar(select([1]))
connection.scalar(select(1))
except exc.DBAPIError as err:
# catch SQLAlchemy's DBAPIError, which is a wrapper
# for the DBAPI's exception. It includes a .connection_invalidated
@@ -800,7 +800,7 @@ def pessimistic_connection_handling(some_engine: Engine) -> None:
# itself and establish a new connection. The disconnect detection
# here also causes the whole connection pool to be invalidated
# so that all stale connections are discarded.
connection.scalar(select([1]))
connection.scalar(select(1))
else:
raise
finally:

View File

@@ -80,7 +80,7 @@ def test_get_fields() -> None:
]
fields = BigQueryEngineSpec._get_fields(columns)
query = select(fields)
query = select(*fields)
assert str(query.compile(dialect=BigQueryDialect())) == (
"SELECT `limit` AS `limit`, `name` AS `name`, "
"`project`.`name` AS `project__name`"

View File

@@ -85,7 +85,7 @@ def test_get_fields() -> None:
]
fields = DatastoreEngineSpec._get_fields(columns)
query = select(fields)
query = select(*fields)
assert str(query.compile(dialect=CloudDatastoreDialect())) == (
'SELECT "limit" AS "limit", name AS name, "project.name" AS project__name'
)

View File

@@ -81,7 +81,7 @@ def test_where_clause_n_prefix() -> None:
tbl = table("tbl")
sel = (
select([str_col, unicode_col])
select(str_col, unicode_col)
.select_from(tbl)
.where(str_col == "abc")
.where(unicode_col == "abc")

View File

@@ -22,7 +22,7 @@ from __future__ import annotations
import copy
from contextlib import contextmanager
from typing import cast, TYPE_CHECKING
from unittest.mock import patch
from unittest.mock import MagicMock, patch
import pytest
from pytest_mock import MockerFixture
@@ -2616,6 +2616,239 @@ def test_adhoc_column_to_sqla_skips_probe_when_not_forced(
assert generic_type is None
def _normalize_df_datasource(column: object) -> MagicMock:
"""Bind ``ExploreMixin.normalize_df`` to a minimal datasource exposing a
single temporal ``column`` via ``get_column``."""
from superset.models.helpers import ExploreMixin
datasource = MagicMock()
datasource.offset = 0
datasource.enforce_numerical_metrics = False
datasource.columns = [column]
datasource.get_column = lambda name: {"ts": column}.get(name)
for method in ("_python_date_format", "_collect_dttm_labels", "normalize_df"):
setattr(datasource, method, getattr(ExploreMixin, method).__get__(datasource))
return datasource
def _raw_query_object() -> MagicMock:
"""A query object for a raw/unaggregated query: ``columns`` holds the plain
physical column name (no base-axis adhoc wrapper, no granularity)."""
query_object = MagicMock()
query_object.columns = ["ts"]
query_object.granularity = None
query_object.time_shift = None
return query_object
def test_normalize_df_applies_python_date_format_to_unaggregated_columns() -> None:
"""A temporal column selected in a raw/unaggregated query is a plain column
name rather than a base-axis adhoc column, so it is excluded from
``get_base_axis_labels``. It must still receive its ``python_date_format``,
so an ``epoch_s`` column is converted to datetimes the same way it is for
aggregated charts."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
ts_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format="epoch_s",
datetime_format=None,
)
datasource = _normalize_df_datasource(ts_col)
# 2020-01-01, 2021-01-01, 2022-01-01 as epoch seconds
df = pd.DataFrame({"ts": [1577836800, 1609459200, 1640995200]})
result = datasource.normalize_df(df, _raw_query_object())
assert is_datetime64_any_dtype(result["ts"])
assert result["ts"][0].strftime("%Y-%m-%d") == "2020-01-01"
assert result["ts"][2].strftime("%Y-%m-%d") == "2022-01-01"
def test_normalize_df_applies_epoch_ms_to_unaggregated_columns() -> None:
"""``epoch_ms`` is a separate conversion branch from ``epoch_s``; an
unaggregated column declaring it must also be converted to datetimes."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
ts_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format="epoch_ms",
datetime_format=None,
)
datasource = _normalize_df_datasource(ts_col)
# 2020-01-01, 2021-01-01, 2022-01-01 as epoch milliseconds
df = pd.DataFrame({"ts": [1577836800000, 1609459200000, 1640995200000]})
result = datasource.normalize_df(df, _raw_query_object())
assert is_datetime64_any_dtype(result["ts"])
assert result["ts"][0].strftime("%Y-%m-%d") == "2020-01-01"
assert result["ts"][2].strftime("%Y-%m-%d") == "2022-01-01"
def test_normalize_df_handles_dict_shaped_columns() -> None:
"""Some datasources expose columns as dicts rather than objects. The raw
temporal-column lookup must read is_dttm and python_date_format from either
shape, the same way the base-axis path already does."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
ts_col = {"column_name": "ts", "is_dttm": True, "python_date_format": "epoch_s"}
datasource = _normalize_df_datasource(ts_col)
df = pd.DataFrame({"ts": [1577836800, 1609459200]})
result = datasource.normalize_df(df, _raw_query_object())
assert is_datetime64_any_dtype(result["ts"])
assert result["ts"][0].strftime("%Y-%m-%d") == "2020-01-01"
def test_normalize_df_leaves_unconfigured_integer_dttm_columns_untouched() -> None:
"""A column flagged temporal but with no ``python_date_format`` must not be
coerced: a plain integer column would otherwise be reinterpreted as
nanoseconds since the epoch (1577836800 -> 1970-01-01 00:00:01.5...). Only
columns that declare a format are normalized."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
int_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format=None,
datetime_format=None,
)
datasource = _normalize_df_datasource(int_col)
df = pd.DataFrame({"ts": [1577836800, 1609459200, 1640995200]})
result = datasource.normalize_df(df, _raw_query_object())
assert not is_datetime64_any_dtype(result["ts"])
assert result["ts"].tolist() == [1577836800, 1609459200, 1640995200]
def test_normalize_df_without_get_column_is_a_noop() -> None:
"""Not every datasource exposes ``get_column``; for those, the temporal
lookup must short-circuit instead of raising, leaving the data untouched."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
from superset.models.helpers import ExploreMixin
class _NoGetColumnDatasource:
"""A datasource that does not implement ``get_column``."""
offset = 0
enforce_numerical_metrics = False
columns: list[object] = []
datasource = _NoGetColumnDatasource()
for method in ("_python_date_format", "_collect_dttm_labels", "normalize_df"):
setattr(datasource, method, getattr(ExploreMixin, method).__get__(datasource))
df = pd.DataFrame({"ts": [1577836800, 1609459200]})
result = datasource.normalize_df(df, _raw_query_object()) # type: ignore[attr-defined]
assert not is_datetime64_any_dtype(result["ts"])
assert result["ts"].tolist() == [1577836800, 1609459200]
def test_normalize_df_normalizes_base_axis_temporal_columns() -> None:
"""The aggregated path: a base-axis temporal column is normalized, and the
raw-column pass does not re-add it (it is already collected)."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
ts_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format="epoch_s",
datetime_format=None,
)
datasource = _normalize_df_datasource(ts_col)
query_object = MagicMock()
query_object.columns = [
{"label": "ts", "sqlExpression": "ts", "columnType": "BASE_AXIS"}
]
query_object.granularity = None
query_object.time_shift = None
df = pd.DataFrame({"ts": [1577836800, 1609459200]})
result = datasource.normalize_df(df, query_object)
assert is_datetime64_any_dtype(result["ts"])
assert result["ts"][0].strftime("%Y-%m-%d") == "2020-01-01"
def test_normalize_df_dedups_column_in_granularity_and_columns() -> None:
"""When the same temporal column is both the ``granularity`` and a selected
column, it is collected once (via the base path) and the raw pass does not
re-add it, so it is normalized a single time."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
ts_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format="epoch_s",
datetime_format=None,
)
datasource = _normalize_df_datasource(ts_col)
query_object = MagicMock()
query_object.columns = ["ts"]
query_object.granularity = "ts"
query_object.time_shift = None
assert datasource._collect_dttm_labels(query_object) == (("ts", "epoch_s"),)
df = pd.DataFrame({"ts": [1577836800, 1609459200]})
result = datasource.normalize_df(df, query_object)
assert is_datetime64_any_dtype(result["ts"])
assert result["ts"][0].strftime("%Y-%m-%d") == "2020-01-01"
def test_normalize_df_normalizes_legacy_time_column() -> None:
"""The legacy ``__timestamp`` column is normalized using the granularity
column's python_date_format."""
import pandas as pd
from pandas.api.types import is_datetime64_any_dtype
from superset.utils.core import DTTM_ALIAS
ts_col = MagicMock(
column_name="ts",
is_dttm=True,
python_date_format="epoch_s",
datetime_format=None,
)
datasource = _normalize_df_datasource(ts_col)
query_object = MagicMock()
query_object.columns = []
query_object.granularity = "ts"
query_object.time_shift = None
df = pd.DataFrame({DTTM_ALIAS: [1577836800, 1609459200]})
result = datasource.normalize_df(df, query_object)
assert is_datetime64_any_dtype(result[DTTM_ALIAS])
assert result[DTTM_ALIAS][0].strftime("%Y-%m-%d") == "2020-01-01"
def test_adhoc_column_to_sqla_returns_type_from_column_metadata(
database: Database,
) -> None: