Compare commits

...
Author SHA1 Message Date
Enzo Martellucci 9dd0c3be2a Merge branch 'master' into enxdev/feat/dashboard-excel-export-sync-fallback 2026-09-09 18:13:00 +02:00
Enzo Martellucci 70b52908b9 fix(dashboard): honor default Excel export row limit 2026-09-09 18:05:40 +02:00
Enzo Martellucci 44869b6c96 docs(dashboard): simplify Excel export guidance 2026-09-09 17:54:28 +02:00
Enzo Martellucci d137fd1c06 fix(dashboard): address Excel export review findings 2026-09-09 17:09:57 +02:00
Enzo Martellucci ccb5fb279f Merge branch 'master' into enxdev/feat/dashboard-excel-export-sync-fallback 2026-09-09 16:44:55 +02:00
Enzo MartellucciandClaude Opus 5 3108f41167 fix(dashboard): keep image exports queued, list skips in the workbook
Follow-up to the inline Excel export, addressing three gaps in it:

An image export draws every chart through the headless webdriver, one browser
session at a time. No row budget bounds that, so serving it inline was exactly
the gateway timeout the budget exists to prevent; without export storage it is
now refused with a message saying image exports run in the background.

The skipped-charts list moves into the workbook, on an "Export Summary" sheet
whenever any chart is skipped rather than only when every chart is. The list
then travels with the file however it is delivered, which is the only way an
export served as a download can report it -- and matches the sheet #43805 adds.

Query contexts are resolved once. The row budget already had to resolve them to
size the export, and the builder resolved them again to run it: with
EXCEL_EXPORT_QUERY_CONTEXT_BUILDER pointed at a service, that is both a doubled
cost and a real hazard, since one set of queries could be measured and a
different set run. The budget now returns an InlineExportPlan carrying what it
resolved, and the builder runs those contexts as-is.

The export action also reports progress: while a request is in flight both
Excel actions are disabled and the one clicked reads "Preparing export…",
restored when the file downloads, the queued message arrives, or it fails. The
server's lock prevents duplicate work but told the user nothing.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 15:04:00 +02:00
Enzo MartellucciandClaude Opus 5 f64b45f2c4 feat(dashboard): stream the Excel export inline when no storage is configured
"Export Data to Excel" was only usable where EXCEL_EXPORT_S3_BUCKET was set:
everywhere else the menu item rendered, the endpoint answered 501, and the
click produced nothing but an error toast.

The endpoint now picks its execution path from that same config rather than
refusing. With a bucket it queues the export and answers 202 with a job id,
unchanged. Without one it builds the workbook during the request and returns
it as the response, so the browser downloads the file directly -- no worker,
bucket or email required.

An export served this way has to finish inside one request, so its size is
settled before any query runs: the new EXCEL_EXPORT_SYNC_MAX_ROWS (100,000)
caps the combined row_limit of every query the export would run, and an export
over that total -- or one where any query has no finite limit -- is refused with
a 400 naming the bucket as the fix, rather than being started and left to hit
the request timeout.

Both paths build the same workbook from the same code: the builder moves out of
the Celery task into superset.dashboards.excel_export.workbook, leaving the task
to own only queueing, upload, email and its own error handling. Both acquire the
same per-user+dashboard lock; the inline path releases it, and deletes its temp
file, however the request ends.

Guest, embedded, anonymous and no-email sessions stay blocked on both paths --
guest support is handled separately in #43805 -- and are covered by a regression
test using a guest token that does grant access to the dashboard.

The storage check is isolated in a single helper so it can be repointed at
EXPORT_STORAGE when #43805 lands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-09 12:06:05 +02:00
16 changed files with 1651 additions and 551 deletions
+26 -20
View File
@@ -538,31 +538,37 @@ Note that a retried query returns partial data with no truncation indicator
(e.g. a filter dropdown may list only a subset of values on tables above the
row cap).
### Dashboard "Export Data to Excel" requires a Celery worker and S3 bucket
### Dashboard Excel exports support direct downloads
A new dashboard action exports every chart's data to a single multi-sheet
`.xlsx` asynchronously. It is disabled by default and turns on only when
`EXCEL_EXPORT_S3_BUCKET` is set (the endpoint returns `501` otherwise). It also
requires a running Celery worker and a configured SMTP transport, since the task
emails the requesting user a pre-signed download link. New config keys:
`EXCEL_EXPORT_S3_BUCKET`, `EXCEL_EXPORT_S3_KEY_PREFIX`,
`.xlsx`. Without an export bucket, Superset builds the workbook during the
request and returns it to the browser. When `EXCEL_EXPORT_S3_BUCKET` is set, a
Celery worker builds and uploads the workbook, then emails the user a pre-signed
download link. This queued path also needs a worker and SMTP transport.
Direct downloads are limited by `EXCEL_EXPORT_SYNC_MAX_ROWS` (default
`100_000`), based on the combined `row_limit` of the planned queries. Superset
uses `ROW_LIMIT` when a query omits its limit and returns `400` before querying
if the total exceeds the limit. Image exports also return `400` without an export
bucket because they require background webdriver rendering.
`POST /api/v1/dashboard/<id>/export_xlsx/` returns either `202` with a queued job
id or `200` with the workbook. It no longer returns `501` when no bucket is set.
New config keys: `EXCEL_EXPORT_S3_BUCKET`, `EXCEL_EXPORT_S3_KEY_PREFIX`,
`EXCEL_EXPORT_LINK_TTL_SECONDS`, `EXCEL_EXPORT_S3_CLIENT_KWARGS`,
`EXCEL_EXPORT_TABLE_VIZ_TYPES`, and `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
`EXCEL_EXPORT_SYNC_MAX_ROWS`, `EXCEL_EXPORT_TABLE_VIZ_TYPES`, and
`EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
The feature depends on `boto3`, which is **not** installed by default; install it
with `pip install apache-superset[excel-export]`.
The queued path depends on `boto3`, which is **not** installed by default; install
it with `pip install apache-superset[excel-export]`. The direct-download path
does not use it.
Charts store their `query_context` only once they have been (re-)saved in
Explore, so older charts may have none. For a fixed, conservative set of viz
types (`table`, `big_number_total`, `big_number`, `pie`) the export rebuilds a
query context from the chart's saved form data so those charts still export.
The rebuild is a single-query mapping and does **not** reproduce plugin
post-processing (pivot, rolling, forecast) or multi-query charts, so any chart of
another type without a saved query context is skipped and listed in the email for
the user to re-save. To cover those types, set `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`
to a callable that receives the chart's form data and returns a query-context
payload (or `None` to fall back to the built-in rebuild) — for example one backed
by a service that runs the chart's real frontend `buildQuery`.
For `table`, `big_number_total`, `big_number`, and `pie` charts without a saved
`query_context`, Superset rebuilds a single query from saved form data. Charts
that need post-processing or multiple queries are skipped and listed on the
workbook's "Export Summary" sheet. Use `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` to
support more chart types.
A second mode, **Export Images to Excel**, embeds non-table charts as rendered
images (which viz types stay tabular is controlled by
@@ -9,22 +9,31 @@ version: 1
Superset can export every chart on a dashboard to a single Excel workbook, with
each chart's underlying data rendered as its own worksheet. The export reflects
the dashboard's currently applied filters and runs asynchronously: when it
finishes, the requesting user receives an email with a time-limited download
link.
the dashboard's currently applied filters.
How the finished workbook reaches you depends on whether the deployment has
export storage configured:
- **With an export bucket**, a background worker builds the workbook and emails
the user a time-limited download link.
- **Without an export bucket**, Superset builds the workbook during the request
and returns it to the browser. This path only supports data exports within
`EXCEL_EXPORT_SYNC_MAX_ROWS` (see [Prerequisites](#prerequisites)).
## Using the export
From a dashboard, open the **... (actions) → Download** submenu and choose
**Export Data to Excel**. The action appears for users who have the dashboard
`can_export` permission. You'll see a confirmation that the export is being
prepared; the workbook arrives by email when it's ready.
`can_export` permission. Where the export is queued you'll see a confirmation
that it is being prepared, then receive the workbook by email. Direct exports
download when ready.
A second option, **Export Images to Excel**, embeds each non-table chart as a
rendered image (tables stay tabular) instead of exporting raw data. Because it
renders charts through the headless webdriver, this option only appears when the
webdriver screenshot feature flags are enabled (see the prerequisites below);
which viz types stay tabular is controlled by `EXCEL_EXPORT_TABLE_VIZ_TYPES`.
Image exports always run in the background and require an export bucket.
Notes on the generated workbook:
@@ -36,31 +45,37 @@ Notes on the generated workbook:
re-saved in Explore) still exports when it is a `table`, `big_number`,
`big_number_total` or `pie`, by rebuilding the query from the chart's saved
form data. Charts of other types — and charts relying on post-processing the
rebuild can't reproduce — are skipped and listed in the email; open the chart
in Explore and re-save it to include it next time, or configure
`EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
rebuild can't reproduce — are skipped; open the chart in Explore and re-save it
to include it next time, or configure `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
The workbook's **Export Summary** sheet lists skipped charts. Queued exports
also list them in the email.
- Row counts per sheet are capped the same way as the chart-level CSV/Excel
export (`ROW_LIMIT`, bounded by `SQL_MAX_ROW`), and never exceed Excel's
per-sheet maximum.
## Prerequisites
This feature is **disabled by default**. It requires:
Dashboard data exports need no configuration. By default, Superset builds the
workbook during the request and returns it to the browser. The combined
`row_limit` of its queries must not exceed `EXCEL_EXPORT_SYNC_MAX_ROWS` (100,000
by default). Queries without a `row_limit` use `ROW_LIMIT`. Keep this setting
within your web server's request timeout.
For larger data exports and all image exports, configure the background path:
1. **The `boto3` dependency.** It is not installed by default; install it with
`pip install apache-superset[excel-export]`. Without it, exports fail and the
user receives a failure email.
2. **An S3 bucket.** Set `EXCEL_EXPORT_S3_BUCKET`. Until it is set, the export
endpoint returns `501` and the menu action surfaces a "not configured"
message.
3. **A running Celery worker.** The export runs as a Celery task. If no worker
is running, the request is accepted but nothing is produced.
2. **An S3 bucket.** Set `EXCEL_EXPORT_S3_BUCKET` to queue exports and deliver
them by email.
3. **A running Celery worker.** The queued export runs as a Celery task. If no
worker is running, the request is accepted but nothing is produced.
4. **A configured SMTP transport.** The download link is delivered by email
using the same settings as alerts & reports (`SMTP_*`,
`EMAIL_REPORTS_SUBJECT_PREFIX`).
**Export Images to Excel** additionally requires a working headless webdriver
the same infrastructure scheduled reports and thumbnails use (`WEBDRIVER_*`,
**Export Images to Excel** also requires the headless webdriver used by scheduled
reports and thumbnails (`WEBDRIVER_*`,
plus the `ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS` and
`ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT` feature flags). The menu option
is hidden when those flags are off; if the webdriver is unreachable, image
@@ -72,14 +87,15 @@ will not register.
## Configuration keys
| Key | Default | Description |
| ------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `EXCEL_EXPORT_S3_BUCKET` | `None` | Destination bucket. Required; `501` if unset. |
| `EXCEL_EXPORT_S3_KEY_PREFIX` | `"dashboard-exports/"` | Key prefix: `{prefix}{dashboard_id}/{job_id}.xlsx`. |
| `EXCEL_EXPORT_LINK_TTL_SECONDS` | `86400` | Lifetime of the pre-signed download URL (24h). |
| `EXCEL_EXPORT_S3_CLIENT_KWARGS` | `{}` | Extra kwargs for `boto3.client("s3", ...)` — e.g. `region_name`, or `endpoint_url` for MinIO/LocalStack. |
| `EXCEL_EXPORT_TABLE_VIZ_TYPES` | `None` | Viz types kept tabular in **Export Images to Excel** mode; every other type is embedded as an image. `None` uses the built-in default (`table`, `pivot_table`, `pivot_table_v2`). |
| `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` | `None` | Optional `Callable[[form_data_dict], dict \| None]` to build a query context for a chart missing a saved one, tried before the built-in form-data rebuild. Point it at a service that runs the chart's real frontend `buildQuery` to faithfully export viz types the built-in rebuild can't handle. Must return `None` when it can't build faithfully, so the export falls back. |
| Key | Default | Description |
| ------------------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `EXCEL_EXPORT_S3_BUCKET` | `None` | Destination bucket. When unset, eligible data exports download directly. |
| `EXCEL_EXPORT_SYNC_MAX_ROWS` | `100000` | Maximum combined query `row_limit` for a direct download. Ignored when a bucket is configured. |
| `EXCEL_EXPORT_S3_KEY_PREFIX` | `"dashboard-exports/"` | Key prefix: `{prefix}{dashboard_id}/{job_id}.xlsx`. |
| `EXCEL_EXPORT_LINK_TTL_SECONDS` | `86400` | Lifetime of the pre-signed download URL (24h). |
| `EXCEL_EXPORT_S3_CLIENT_KWARGS` | `{}` | Extra arguments for `boto3.client("s3", ...)`, such as `region_name` or an `endpoint_url` for MinIO or LocalStack. |
| `EXCEL_EXPORT_TABLE_VIZ_TYPES` | `None` | Viz types kept tabular in image mode. `None` uses `table`, `pivot_table`, and `pivot_table_v2`. |
| `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` | `None` | Optional callable that builds a query context from `Slice.form_data`. Return `None` to use the built-in rebuild. |
Credentials and region resolve through the standard boto3 chain (environment
variables, shared config, or instance role) unless overridden via
@@ -88,8 +104,9 @@ variables, shared config, or instance role) unless overridden via
## Security considerations
- The emailed link is a **pre-signed S3 URL**: anyone who holds it can download
the workbook until it expires. Keep the bucket **private**, enable
encryption, and consider a lifecycle rule to delete objects after a few days.
the workbook until it expires. Direct downloads are not stored or linked.
Keep the bucket **private**, enable encryption, and consider a lifecycle rule
to delete objects after a few days.
Lower `EXCEL_EXPORT_LINK_TTL_SECONDS` if 24 hours is too long for your data.
- The export runs with the requesting user's permissions; each chart's query is
access-checked, so users only ever receive data they are entitled to.
@@ -97,9 +114,9 @@ variables, shared config, or instance role) unless overridden via
## Limitations
- **Embedded dashboards / guest tokens are not supported** in this version,
because guest users have no email address to deliver the link to. Logged-in
users viewing an embedded dashboard can still use the export.
including direct downloads. Logged-in users viewing an embedded dashboard can
still use the export.
- The default **Export Data to Excel** mode exports data only (no visual
styling). Use **Export Images to Excel** to embed rendered chart images, which
requires the webdriver infrastructure described in the prerequisites.
requires an export bucket and the webdriver setup described above.
- Scheduled/automated exports are not part of this feature.
+12 -4
View File
@@ -19521,7 +19521,7 @@
},
"/api/v1/dashboard/{pk}/export_xlsx/": {
"post": {
"description": "Enqueues an async task that writes each chart's data to its own worksheet, uploads the .xlsx to S3, and emails the requesting user a pre-signed download link. Returns immediately with a job id.",
"description": "Writes each chart to a worksheet. With export storage, the work is queued and the user receives a download link by email. Without it, eligible workbooks are returned in the response.",
"parameters": [
{
"description": "The dashboard id",
@@ -19543,6 +19543,17 @@
}
},
"responses": {
"200": {
"content": {
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": {
"schema": {
"format": "binary",
"type": "string"
}
}
},
"description": "The exported workbook, built during this request"
},
"202": {
"content": {
"application/json": {
@@ -19567,9 +19578,6 @@
},
"500": {
"$ref": "#/components/responses/500"
},
"501": {
"description": "Excel export is not configured on this server"
}
},
"security": [
@@ -141,11 +141,41 @@ test('Excel export items are hidden when userCanExport is false', () => {
expect(screen.getByText('Export YAML')).toBeInTheDocument();
});
test('Export Data to Excel posts mode "data" and shows a pending toast', async () => {
/** A queued export: 202 with a job id, delivered later by email. */
const mockQueuedResponse = (
body: Record<string, unknown> = { job_id: 'abc' },
) =>
mockSupersetClient.post.mockResolvedValue({
json: { job_id: 'abc' },
status: 202,
json: jest.fn().mockResolvedValue(body),
} as never);
/** An inline export: the workbook itself, as the response to the request. */
const mockWorkbookResponse = (
filename = 'World_Health_1.xlsx',
): { blob: jest.Mock } => {
const blob = jest.fn().mockResolvedValue(new Blob(['xlsx'])) as jest.Mock;
mockSupersetClient.post.mockResolvedValue({
status: 200,
blob,
headers: new Headers({
'Content-Disposition': `attachment; filename=${filename}`,
}),
} as never);
return { blob };
};
/** jsdom implements neither, and the download path needs both. */
const stubObjectUrls = (): { createObjectURL: jest.Mock } => {
const createObjectURL = jest.fn(() => 'blob:http://localhost/fake');
window.URL.createObjectURL = createObjectURL;
window.URL.revokeObjectURL = jest.fn();
return { createObjectURL };
};
test('Export Data to Excel posts mode "data" and shows a pending toast', async () => {
mockQueuedResponse();
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
@@ -153,7 +183,9 @@ test('Export Data to Excel posts mode "data" and shows a pending toast', async (
await waitFor(() => {
expect(mockSupersetClient.post).toHaveBeenCalledWith({
endpoint: '/api/v1/dashboard/123/export_xlsx/',
fetchRetryOptions: { retries: 0 },
jsonPayload: { active_data_mask: {}, mode: 'data' },
parseMethod: 'raw',
});
expect(mockAddSuccessToast).toHaveBeenCalledWith(
"Your export is being prepared. You'll receive an email when it's ready.",
@@ -163,9 +195,7 @@ test('Export Data to Excel posts mode "data" and shows a pending toast', async (
test('Export Images to Excel posts mode "images" and shows a pending toast', async () => {
enableWebDriverScreenshot();
mockSupersetClient.post.mockResolvedValue({
json: { job_id: 'abc' },
} as never);
mockQueuedResponse();
render(<MenuWrapper />, { useRedux: true });
@@ -174,7 +204,9 @@ test('Export Images to Excel posts mode "images" and shows a pending toast', asy
await waitFor(() => {
expect(mockSupersetClient.post).toHaveBeenCalledWith({
endpoint: '/api/v1/dashboard/123/export_xlsx/',
fetchRetryOptions: { retries: 0 },
jsonPayload: { active_data_mask: {}, mode: 'images' },
parseMethod: 'raw',
});
expect(mockAddSuccessToast).toHaveBeenCalledWith(
"Your export is being prepared. You'll receive an email when it's ready.",
@@ -182,13 +214,158 @@ test('Export Images to Excel posts mode "images" and shows a pending toast', asy
});
});
test('Export Data to Excel downloads the workbook when it arrives inline', async () => {
// Without storage, the response contains the workbook.
const { blob } = mockWorkbookResponse();
const { createObjectURL } = stubObjectUrls();
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(blob).toHaveBeenCalled();
expect(createObjectURL).toHaveBeenCalled();
expect(mockAddSuccessToast).toHaveBeenCalledWith(
'Dashboard data exported to Excel',
);
});
// Direct downloads do not show the queued-export message.
expect(mockAddSuccessToast).not.toHaveBeenCalledWith(
"Your export is being prepared. You'll receive an email when it's ready.",
);
});
test('Export Data to Excel names the downloaded file from the response', async () => {
mockWorkbookResponse('Sales_Overview_7.xlsx');
stubObjectUrls();
// Capture download names without navigating in jsdom.
const downloaded: string[] = [];
const click = jest
.spyOn(HTMLAnchorElement.prototype, 'click')
.mockImplementation(function recordDownload(this: HTMLAnchorElement) {
downloaded.push(this.download);
});
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => expect(downloaded).toEqual(['Sales_Overview_7.xlsx']));
click.mockRestore();
});
/** Mock a request that the test can settle later. */
const mockPendingResponse = (): { settle: (response: unknown) => void } => {
let settle: (response: unknown) => void = () => {};
mockSupersetClient.post.mockReturnValue(
new Promise(resolve => {
settle = resolve;
}) as never,
);
return { settle: response => settle(response) };
};
const menuItemFor = (label: string) =>
screen.getByText(label).closest('[role="menuitem"]');
test('Export Data to Excel reports progress while the export is running', async () => {
// Keep the menu responsive while the request runs.
const { settle } = mockPendingResponse();
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
});
expect(screen.queryByText('Export Data to Excel')).not.toBeInTheDocument();
expect(menuItemFor('Preparing export…')).toHaveAttribute(
'aria-disabled',
'true',
);
settle({ status: 202, json: jest.fn().mockResolvedValue({ job_id: 'abc' }) });
// Re-enable the action when the request finishes.
await waitFor(() => {
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
});
expect(mockAddSuccessToast).toHaveBeenCalledWith(
"Your export is being prepared. You'll receive an email when it's ready.",
);
});
test('Export Data to Excel is offered again once the download starts', async () => {
const { settle } = mockPendingResponse();
stubObjectUrls();
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
});
settle({
status: 200,
blob: jest.fn().mockResolvedValue(new Blob(['xlsx'])),
headers: new Headers({
'Content-Disposition': 'attachment; filename=dash.xlsx',
}),
});
await waitFor(() => {
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
});
expect(mockAddSuccessToast).toHaveBeenCalledWith(
'Dashboard data exported to Excel',
);
});
test('Export Data to Excel is offered again after a failure', async () => {
const { settle } = mockPendingResponse();
mockGetClientErrorObject.mockResolvedValue({ status: 500 });
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
});
// Re-enable the action after an error.
settle(Promise.reject(new Error('boom')));
await waitFor(() => {
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
});
expect(mockAddDangerToast).toHaveBeenCalledWith(
'Sorry, something went wrong. Try again later.',
);
});
test('Export Images to Excel is blocked while a data export is running', async () => {
// The server allows one export per dashboard and user.
enableWebDriverScreenshot();
mockPendingResponse();
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(menuItemFor('Export Images to Excel')).toHaveAttribute(
'aria-disabled',
'true',
);
});
await userEvent.click(screen.getByText('Export Images to Excel'));
expect(mockSupersetClient.post).toHaveBeenCalledTimes(1);
});
test('Export Data to Excel shows an "already in progress" toast when throttled', async () => {
// The throttle response is 202 with a message but no job_id.
mockSupersetClient.post.mockResolvedValue({
json: {
message: 'An Excel export for this dashboard is already in progress.',
},
} as never);
mockQueuedResponse({
message: 'An Excel export for this dashboard is already in progress.',
});
render(<MenuWrapper />, { useRedux: true });
@@ -201,18 +378,19 @@ test('Export Data to Excel shows an "already in progress" toast when throttled',
});
});
test('Export Data to Excel shows a config error toast on 501', async () => {
mockSupersetClient.post.mockRejectedValue(new Error('not configured'));
mockGetClientErrorObject.mockResolvedValue({ status: 501 });
test('Export Data to Excel surfaces the reason an export was refused', async () => {
// Show the server's actionable refusal.
const message =
'This dashboard requests too many rows to export in a single request.';
mockSupersetClient.post.mockRejectedValue(new Error('too big'));
mockGetClientErrorObject.mockResolvedValue({ status: 400, message });
render(<MenuWrapper />, { useRedux: true });
await userEvent.click(screen.getByText('Export Data to Excel'));
await waitFor(() => {
expect(mockAddDangerToast).toHaveBeenCalledWith(
'Excel export is not configured on this server.',
);
expect(mockAddDangerToast).toHaveBeenCalledWith(message);
});
});
@@ -16,7 +16,7 @@
* specific language governing permissions and limitations
* under the License.
*/
import { SyntheticEvent } from 'react';
import { SyntheticEvent, useState } from 'react';
import { useSelector } from 'react-redux';
import { logging } from '@apache-superset/core/utils';
import { t } from '@apache-superset/core/translation';
@@ -27,13 +27,15 @@ import {
SupersetClient,
} from '@superset-ui/core';
import { MenuItem } from '@superset-ui/core/components/Menu';
import { parse as parseContentDisposition } from 'content-disposition';
import { useDownloadScreenshot } from 'src/dashboard/hooks/useDownloadScreenshot';
import { NATIVE_FILTER_PREFIX } from 'src/dashboard/components/nativeFilters/FiltersConfigModal/utils';
import { MenuKeys, RootState } from 'src/dashboard/types';
import downloadAsPdf from 'src/utils/downloadAsPdf';
import downloadAsImage from 'src/utils/downloadAsImage';
import handleResourceExport from 'src/utils/export';
import handleResourceExport, {
downloadBlob,
getFilenameFromResponse,
} from 'src/utils/export';
import {
LOG_ACTIONS_DASHBOARD_DOWNLOAD_AS_PDF,
LOG_ACTIONS_DASHBOARD_DOWNLOAD_AS_IMAGE,
@@ -72,6 +74,10 @@ export const useDownloadMenuItems = (
const { addDangerToast, addSuccessToast } = useToasts();
const dataMask = useSelector((state: RootState) => state.dataMask);
// Disable both Excel actions while either export is running.
const [exportingXlsx, setExportingXlsx] = useState<'data' | 'images' | null>(
null,
);
const SCREENSHOT_NODE_SELECTOR = '.dashboard';
const buildActiveDataMask = (): Record<string, { extraFormData: object }> =>
@@ -130,35 +136,14 @@ export const useDownloadMenuItems = (
parseMethod: 'raw',
});
// Parse filename from Content-Disposition header
const disposition = response.headers.get('Content-Disposition');
let fileName = `dashboard_${dashboardId}_example.zip`;
if (disposition) {
try {
const parsed = parseContentDisposition(disposition);
if (parsed?.parameters?.filename) {
fileName = parsed.parameters.filename;
}
} catch (error) {
logging.warn('Failed to parse Content-Disposition header:', error);
}
}
// Convert response to blob and trigger download
const blob = await response.blob();
const url = window.URL.createObjectURL(blob);
try {
const a = document.createElement('a');
a.href = url;
a.download = fileName;
a.style.display = 'none';
document.body.appendChild(a);
a.click();
document.body.removeChild(a);
} finally {
window.URL.revokeObjectURL(url);
}
downloadBlob(
blob,
getFilenameFromResponse(
response,
`dashboard_${dashboardId}_example.zip`,
),
);
addSuccessToast(t('Dashboard exported as example successfully'));
} catch (error) {
@@ -168,14 +153,31 @@ export const useDownloadMenuItems = (
};
const onExportXlsx = async (mode: 'data' | 'images') => {
setExportingXlsx(mode);
try {
const { json } = await SupersetClient.post({
const response = await SupersetClient.post({
endpoint: `/api/v1/dashboard/${dashboardId}/export_xlsx/`,
jsonPayload: { active_data_mask: buildActiveDataMask(), mode },
// Parse the queued response or workbook after checking its status.
parseMethod: 'raw',
// A retry may hit the first request's lock and lose its file response.
fetchRetryOptions: { retries: 0 },
});
// The throttle response (an export is already running) returns 202 with a
// message but no job_id; only a freshly enqueued job carries a job_id.
if ((json as { job_id?: string })?.job_id) {
// A 202 is queued; any successful non-202 response is the workbook.
if (response.status !== 202) {
const blob = await response.blob();
downloadBlob(
blob,
getFilenameFromResponse(response, `dashboard_${dashboardId}.xlsx`),
);
addSuccessToast(t('Dashboard data exported to Excel'));
return;
}
// Only a newly queued export has a job id.
const json = (await response.json()) as { job_id?: string };
if (json?.job_id) {
addSuccessToast(
t(
"Your export is being prepared. You'll receive an email when it's ready.",
@@ -187,16 +189,20 @@ export const useDownloadMenuItems = (
);
}
} catch (error) {
// status comes from the response (Partial<SupersetClientResponse>), which
// the union type does not expose uniformly; read it via a narrow cast.
const { status } = (await getClientErrorObject(error)) as {
// The client error union does not expose response fields uniformly.
const { status, message } = (await getClientErrorObject(error)) as {
status?: number;
message?: string;
};
if (status === 501) {
addDangerToast(t('Excel export is not configured on this server.'));
// Show actionable client errors; keep server errors generic.
if (message && status && status >= 400 && status < 500) {
addDangerToast(message);
} else {
addDangerToast(t('Sorry, something went wrong. Try again later.'));
}
} finally {
// Re-enable the actions after success or failure.
setExportingXlsx(null);
}
};
@@ -244,23 +250,25 @@ export const useDownloadMenuItems = (
},
];
const xlsxExportLabel = (mode: 'data' | 'images', text: string) =>
exportingXlsx === mode ? t('Preparing export…') : text;
const exportMenuItems: MenuItem[] = [
...(userCanExport
? [
{
key: 'export-xlsx',
label: t('Export Data to Excel'),
label: xlsxExportLabel('data', t('Export Data to Excel')),
disabled: exportingXlsx !== null,
onClick: () => onExportXlsx('data'),
},
// Image export renders charts through the headless webdriver, so only
// offer it where that infrastructure is available (same signal as the
// PDF/PNG image downloads above); otherwise non-table charts would
// silently come back empty.
// Image exports require the same webdriver flags as PDF and PNG.
...(isWebDriverScreenshotEnabled
? [
{
key: 'export-xlsx-images',
label: t('Export Images to Excel'),
label: xlsxExportLabel('images', t('Export Images to Excel')),
disabled: exportingXlsx !== null,
onClick: () => onExportXlsx('images'),
},
]
+11 -15
View File
@@ -1539,10 +1539,10 @@ CSV_STREAMING_ROW_THRESHOLD = 100000
EXCEL_EXPORT: dict[str, Any] = {}
# ---------------------------------------------------
# Dashboard "Export Data to Excel" (async, S3-backed)
# Dashboard "Export Data to Excel"
# ---------------------------------------------------
# Destination S3 bucket for generated dashboard .xlsx exports. The feature is
# disabled until this is set: the export endpoint returns 501 when it is None.
# When set, dashboard .xlsx exports run in the background and arrive by email.
# Otherwise, eligible data exports are returned directly to the browser.
EXCEL_EXPORT_S3_BUCKET: str | None = None
# Key prefix for export objects: {prefix}{dashboard_id}/{job_id}.xlsx
EXCEL_EXPORT_S3_KEY_PREFIX = "dashboard-exports/"
@@ -1559,18 +1559,14 @@ EXCEL_EXPORT_S3_CLIENT_KWARGS: dict[str, Any] = {}
# a rendered image. Set to None to fall back to the built-in default.
EXCEL_EXPORT_TABLE_VIZ_TYPES: set[str] | None = None
# Optional hook to build a query context for a chart that has no saved
# ``query_context``, called before the built-in form-data rebuild. Receives the
# chart's form data (its ``params`` with ``viz_type`` and the
# ``datasource="{id}__{type}"`` string injected — i.e. ``Slice.form_data``) and
# returns a query-context payload dict (the shape ``ChartDataQueryContextSchema``
# loads) or ``None``. A deployment can point this at a service that runs the
# chart's real frontend ``buildQuery`` (faithful post-processing / multi-query)
# for viz types the built-in rebuild can't handle. Must return ``None`` — not a
# partial/stub context — whenever it cannot build the chart faithfully, so the
# export falls through to the built-in rebuild. The export deep-copies whatever
# it returns before applying dashboard filters, so a builder is free to memoize
# or share its payloads. Defaults to ``None`` (built-in behavior only).
# Maximum combined query ``row_limit`` for a direct download. Queries without a
# limit use ``ROW_LIMIT``. Keep this within the request timeout.
EXCEL_EXPORT_SYNC_MAX_ROWS = 100_000
# Optional query-context builder for charts without a saved ``query_context``.
# It receives ``Slice.form_data`` and returns a payload accepted by
# ``ChartDataQueryContextSchema``, or ``None`` to use the built-in rebuild.
# Superset copies returned payloads before applying dashboard filters.
EXCEL_EXPORT_QUERY_CONTEXT_BUILDER: (
Callable[[dict[str, Any]], dict[str, Any] | None] | None
) = None
+141 -27
View File
@@ -17,6 +17,8 @@
# pylint: disable=too-many-lines
import functools
import logging
import os
import tempfile
import uuid
from datetime import datetime
from io import BytesIO
@@ -92,6 +94,17 @@ from superset.commands.importers.v1.utils import get_contents_from_bundle
from superset.commands.purge import PurgeArchivedCommand, SoftDeleteBinding
from superset.constants import MODEL_API_RW_METHOD_PERMISSION_MAP, RouteMethod
from superset.daos.dashboard import DashboardDAO, EmbeddedDashboardDAO
from superset.dashboards.excel_export.storage import is_export_storage_configured
from superset.dashboards.excel_export.sync_budget import (
InlineExportPlan,
plan_inline_export,
)
from superset.dashboards.excel_export.workbook import (
build_workbook,
EXPORT_MODE_DATA,
EXPORT_MODE_IMAGES,
ResolvedQueryContexts,
)
from superset.dashboards.filter_scope import derive_json_metadata
from superset.dashboards.filters import (
DashboardAccessFilter,
@@ -180,6 +193,7 @@ from superset.versioning.api_helpers import (
)
from superset.versioning.etag import set_version_etag
from superset.versioning.schemas import VersionListItemSchema
from superset.views.base import generate_download_headers, XlsxResponse
from superset.views.base_api import (
BaseSupersetModelRestApi,
RelatedFieldFilter,
@@ -1726,15 +1740,15 @@ class DashboardRestApi(
action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.export_xlsx",
log_to_statsd=False,
)
def export_xlsx(self, pk: int) -> WerkzeugResponse:
"""Export all of a dashboard's chart data to an Excel workbook (async).
def export_xlsx(self, pk: int) -> WerkzeugResponse: # noqa: C901
"""Export all of a dashboard's chart data to an Excel workbook.
---
post:
summary: Export dashboard chart data to Excel
description: >-
Enqueues an async task that writes each chart's data to its own
worksheet, uploads the .xlsx to S3, and emails the requesting user a
pre-signed download link. Returns immediately with a job id.
Writes each chart to a worksheet. With export storage, the work is
queued and the user receives a download link by email. Without it,
eligible workbooks are returned in the response.
parameters:
- in: path
schema:
@@ -1747,6 +1761,13 @@ class DashboardRestApi(
schema:
$ref: '#/components/schemas/DashboardExportXlsxPostSchema'
responses:
200:
description: The exported workbook, built during this request
content:
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet:
schema:
type: string
format: binary
202:
description: Export task accepted
content:
@@ -1763,13 +1784,10 @@ class DashboardRestApi(
$ref: '#/components/responses/404'
500:
$ref: '#/components/responses/500'
501:
description: Excel export is not configured on this server
"""
if not current_app.config["EXCEL_EXPORT_S3_BUCKET"]:
return self.response(
501, message="Excel export is not configured on this server."
)
# Keep the request guards and two delivery paths together.
# Resolve the path once so its checks and delivery cannot diverge.
queued = is_export_storage_configured()
try:
# Tolerate an empty/non-JSON body (e.g. a POST with no Content-Type);
# request.json would otherwise raise 415.
@@ -1779,10 +1797,7 @@ class DashboardRestApi(
except ValidationError as error:
return self.response_400(message=error.messages)
# Image export drives the headless webdriver, so it is only available
# when the same screenshot flags the UI checks are enabled. The decorator
# form (``@validate_feature_flags``) can't be used here because it would
# also block ``mode="data"``; mirror its 404 behavior inline instead.
# Image exports require the screenshot feature flags; data exports do not.
if payload.get("mode") == "images" and not (
is_feature_enabled("ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS")
and is_feature_enabled("ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT")
@@ -1797,8 +1812,7 @@ class DashboardRestApi(
except SupersetSecurityException:
return self.response_403()
# Email delivery is the only result channel, so an account with an email
# address is required; embedded guest users are excluded in this version.
# Both delivery paths require a non-guest account with an email address.
if isinstance(g.user, GuestUser) or not getattr(g.user, "email", None):
return self.response_400(
message="Excel export requires an account with an email address."
@@ -1806,11 +1820,21 @@ class DashboardRestApi(
if not dashboard.slices:
return self.response_400(message="Dashboard has no charts to export.")
# Throttle: one concurrent export per user+dashboard. Acquire a shared,
# atomic distributed lock (Redis when configured, the metadata DB
# otherwise) so the guard works across the web server and workers and is
# not a no-op under the default cache. The task releases it when it
# settles; the TTL is the backstop if that release is ever lost.
active_data_mask = payload.get("active_data_mask", {})
mode = payload.get("mode", "data")
if not queued and mode == EXPORT_MODE_IMAGES:
# Webdriver rendering is too slow and unbounded for a web request.
return self.response_400(
message=(
"Exporting images to Excel runs in the background. "
"Configure EXCEL_EXPORT_S3_BUCKET to use it, or export "
"the dashboard's data instead."
)
)
# Allow one export per user and dashboard across web and worker processes.
# The TTL releases the lock if normal cleanup fails.
lock_params = export_lock_params(g.user.id, dashboard.id)
try:
AcquireDistributedLock(
@@ -1825,25 +1849,115 @@ class DashboardRestApi(
)
job_id = str(uuid.uuid4())
if queued:
return self._export_xlsx_queued(
dashboard, active_data_mask, mode, job_id, lock_params
)
# Plan after locking because query-context resolution can be expensive.
# Release here unless the inline exporter takes over cleanup.
lock_delegated = False
try:
plan: InlineExportPlan = plan_inline_export(dashboard)
if not plan.fits_row_budget:
return self.response_400(
message=(
"This dashboard requests too many rows to export in a "
"single request. Configure EXCEL_EXPORT_S3_BUCKET to "
"export it in the background, or lower the row limits of "
"its charts."
)
)
lock_delegated = True
return self._export_xlsx_inline(
dashboard,
active_data_mask,
job_id,
lock_params,
plan.query_contexts,
)
finally:
if not lock_delegated:
try:
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
except Exception: # pylint: disable=broad-except
# The TTL is the fallback if release fails.
logger.exception(
"Failed to release in-flight export lock for dashboard %s",
dashboard.id,
)
def _export_xlsx_queued( # pylint: disable=too-many-arguments
self,
dashboard: Dashboard,
active_data_mask: dict[str, Any],
mode: str,
job_id: str,
lock_params: dict[str, int],
) -> WerkzeugResponse:
"""Queue an export for upload and email delivery."""
try:
export_dashboard_excel.apply_async(
kwargs={
"dashboard_id": dashboard.id,
"user_id": g.user.id,
"active_data_mask": payload.get("active_data_mask", {}),
"active_data_mask": active_data_mask,
"job_id": job_id,
"mode": payload.get("mode", "data"),
"mode": mode,
},
task_id=job_id,
)
except Exception:
# If enqueuing fails (e.g. broker down) the task will never run to
# release the lock, so free it now rather than block exports until
# the TTL expires.
# No task will release the lock if enqueueing fails.
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
raise
return self.response(202, job_id=job_id)
@staticmethod
def _export_xlsx_inline( # pylint: disable=too-many-arguments
dashboard: Dashboard,
active_data_mask: dict[str, Any],
job_id: str,
lock_params: dict[str, int],
query_contexts: ResolvedQueryContexts,
) -> WerkzeugResponse:
"""Build a planned data export and return it in the response."""
tmp_path: str | None = None
try:
file_descriptor, tmp_path = tempfile.mkstemp(
suffix=".xlsx", prefix=f"dash-export-{job_id}-"
)
os.close(file_descriptor)
build_workbook(
tmp_path,
dashboard,
active_data_mask,
job_id,
EXPORT_MODE_DATA,
g.user,
query_contexts=query_contexts,
)
with open(tmp_path, "rb") as workbook:
content = workbook.read()
finally:
# Always release the lock and remove the temporary workbook.
try:
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
except Exception: # pylint: disable=broad-except
# The TTL is the fallback if release fails.
logger.exception(
"Failed to release in-flight export lock for dashboard %s",
dashboard.id,
)
if tmp_path and os.path.exists(tmp_path):
os.remove(tmp_path)
filename = get_filename(dashboard.dashboard_title, dashboard.id, skip_id=False)
return XlsxResponse(
content, headers=generate_download_headers("xlsx", filename)
)
def _validate_permalink_for_dashboard(
self, permalink_key: str, dashboard: Dashboard
) -> WerkzeugResponse | None:
@@ -0,0 +1,26 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Check whether dashboard Excel export storage is configured."""
from __future__ import annotations
from flask import current_app
def is_export_storage_configured() -> bool:
"""Return whether exports can be uploaded and shared by link."""
return bool(current_app.config["EXCEL_EXPORT_S3_BUCKET"])
@@ -0,0 +1,85 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Plan and size dashboard Excel exports served in the HTTP response."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Any
from flask import current_app
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
from superset.dashboards.excel_export.workbook import (
resolve_query_context,
ResolvedQueryContexts,
)
@dataclass(frozen=True)
class InlineExportPlan:
"""Queries planned for a direct download and their row budget."""
#: Resolved query contexts by chart id. ``None`` marks a skipped chart.
query_contexts: ResolvedQueryContexts
#: Combined row limit, or ``None`` when any query has no finite limit.
requested_rows: int | None
#: Configured limit for direct downloads.
max_rows: int
@property
def fits_row_budget(self) -> bool:
"""Return whether the export can run during the request."""
return self.requested_rows is not None and self.requested_rows <= self.max_rows
def _finite_row_limit(query: Any) -> int | None:
"""Return the query's row limit, using ``ROW_LIMIT`` when omitted."""
if not isinstance(query, dict):
return None
row_limit = query.get("row_limit") or current_app.config["ROW_LIMIT"]
if isinstance(row_limit, bool) or not isinstance(row_limit, int):
return None
return row_limit if row_limit > 0 else None
def _row_total(query_contexts: ResolvedQueryContexts) -> int | None:
"""Rows every resolved query may return, or ``None`` if any is unbounded."""
total = 0
for query_context in query_contexts.values():
if query_context is None:
# Skipped charts do not add to the row budget.
continue
for query in query_context["queries"]:
row_limit = _finite_row_limit(query)
if row_limit is None:
return None
total += row_limit
return total
def plan_inline_export(dashboard: Any) -> InlineExportPlan:
"""Resolve a dashboard's queries and calculate its direct-download size."""
query_contexts: ResolvedQueryContexts = {
chart.id: resolve_query_context(chart)
for chart in get_charts_in_layout_order(dashboard)
}
return InlineExportPlan(
query_contexts=query_contexts,
requested_rows=_row_total(query_contexts),
max_rows=current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"],
)
@@ -0,0 +1,302 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Build dashboard Excel workbooks for direct and queued exports."""
from __future__ import annotations
import copy
import logging
from typing import Any
from celery.exceptions import SoftTimeLimitExceeded
from flask import current_app, g
from superset.charts.data.dashboard_filter_context import (
apply_dashboard_filter_context,
get_dashboard_filter_context,
)
from superset.charts.schemas import ChartDataQueryContextSchema
from superset.commands.chart.data.get_data_command import ChartDataCommand
from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
from superset.common.form_data_query_context import (
build_query_context_from_form_data,
is_raw_query_mode,
)
from superset.dashboards.excel_export import email
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
from superset.dashboards.excel_export.screenshot import render_chart_image
from superset.utils import json
from superset.utils.excel_streaming import StreamingXlsxWriter
logger = logging.getLogger(__name__)
# Data mode writes chart results; image mode renders non-table charts.
EXPORT_MODE_DATA = "data"
EXPORT_MODE_IMAGES = "images"
# Viz types that remain tabular in image mode.
TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"}
# Viz types that can be rebuilt as a single query without post-processing.
REBUILD_VIZ_TYPES = {"table", "big_number_total", "big_number", "pie"}
#: Resolved query contexts by chart id. ``None`` marks a skipped chart.
ResolvedQueryContexts = dict[int, dict[str, Any] | None]
class ChartSkippedError(Exception):
"""Raised when a chart should be listed as skipped."""
def chart_label(chart: Any) -> str:
"""Return the chart label used in the export summary."""
return f"{chart.id} - {chart.slice_name or ''}".strip()
def _usable_query_context(value: Any) -> dict[str, Any] | None:
"""Return a query context with a non-empty ``queries`` list."""
if not isinstance(value, dict) or not isinstance(value.get("queries"), list):
return None
return value if value["queries"] else None
def _saved_query_context(raw: Any) -> dict[str, Any] | None:
"""Parse a saved query context, returning ``None`` if it is unusable."""
if not raw:
return None
try:
parsed = json.loads(raw)
except (TypeError, ValueError):
return None
return _usable_query_context(parsed)
# Form-data features the single-query rebuild cannot reproduce.
_UNSUPPORTED_PROCESSING_KEYS = ("time_compare", "rolling_type", "resample_rule")
def _needs_unsupported_processing(form_data: dict[str, Any]) -> bool:
"""Return whether the chart needs unsupported post-processing."""
# ``percent_metrics`` are produced by contribution post-processing.
if form_data.get("percent_metrics"):
return True
# Aggregate table totals require a second query.
if form_data.get("show_totals") and not is_raw_query_mode(form_data):
return True
for key in _UNSUPPORTED_PROCESSING_KEYS:
value = form_data.get(key)
# ``rolling_type`` may contain the string ``"None"`` when unset.
if value and value != "None":
return True
return form_data.get("aggregation") == "raw"
def resolve_query_context(chart: Any) -> dict[str, Any] | None:
"""Resolve a saved, custom-built, or built-in query context for a chart."""
if saved := _saved_query_context(chart.query_context):
return saved
# Builder failures fall back to the built-in query-context rebuild.
if builder := current_app.config.get("EXCEL_EXPORT_QUERY_CONTEXT_BUILDER"):
try:
built = builder(chart.form_data)
except SoftTimeLimitExceeded:
# A task timeout must stop the whole export.
raise
except Exception: # pylint: disable=broad-except
logger.warning(
"EXCEL_EXPORT_QUERY_CONTEXT_BUILDER failed for chart %s; "
"falling back to the built-in rebuild",
chart.id,
exc_info=True,
)
built = None
if (from_builder := _usable_query_context(built)) is not None:
# Filters mutate nested queries, so do not modify a shared payload.
return copy.deepcopy(from_builder)
# Only the built-in rebuild is limited to simple chart types.
if chart.viz_type not in REBUILD_VIZ_TYPES or chart.datasource_id is None:
return None
try:
form_data = json.loads(chart.params) if chart.params else {}
except (TypeError, ValueError):
return None
if not isinstance(form_data, dict) or not form_data:
return None
if _needs_unsupported_processing(form_data):
return None
return build_query_context_from_form_data(
form_data,
{"id": chart.datasource_id, "type": chart.datasource_type or "table"},
chart.viz_type,
)
def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]:
return [record.get(col) for col in colnames]
def _table_viz_types() -> set[str]:
"""Viz types kept tabular in image mode (config override or built-in default)."""
return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES
def renders_as_image(chart: Any, mode: str) -> bool:
"""Return whether the chart is exported as an image."""
return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types()
def _write_chart_image_sheet(
writer: StreamingXlsxWriter,
chart: Any,
dashboard_id: int,
active_data_mask: dict[str, Any],
user: Any,
) -> None:
"""Render a chart into a worksheet.
:raises ChartSkippedError: if rendering fails
"""
image = render_chart_image(chart, dashboard_id, active_data_mask, user)
if image is None:
raise ChartSkippedError
writer.add_image_sheet(chart_label(chart), image)
def _write_chart_sheets(
writer: StreamingXlsxWriter,
chart: Any,
json_body: dict[str, Any],
dashboard_id: int,
active_data_mask: dict[str, Any],
) -> None:
"""Run a chart's queries and write each result to a worksheet."""
# Preserve the caller's top-level query-context values.
json_body = dict(json_body)
# Export full JSON results, regardless of saved values.
json_body["result_format"] = ChartDataResultFormat.JSON
json_body["result_type"] = ChartDataResultType.FULL
json_body.pop("force", None)
filter_context = get_dashboard_filter_context(
dashboard_id=dashboard_id,
chart_id=chart.id,
active_data_mask=active_data_mask,
)
if filter_context.extra_form_data:
apply_dashboard_filter_context(json_body, filter_context.extra_form_data)
# Jinja macros read the query context from ``g.form_data``.
g.form_data = json_body
query_context = ChartDataQueryContextSchema().load(json_body)
command = ChartDataCommand(query_context)
command.validate()
result = command.run()
for index, query in enumerate(result["queries"]):
colnames = query.get("colnames") or []
data = query.get("data") or []
if index == 0:
name = f"{chart.id} - {chart.slice_name or ''}"
else:
name = f"{chart.id}.{index} - {chart.slice_name or ''}"
writer.add_sheet(
name,
colnames,
(_record_to_row(record, colnames) for record in data),
)
def build_workbook( # pylint: disable=too-many-arguments
path: str,
dashboard: Any,
active_data_mask: dict[str, Any],
job_id: str,
mode: str,
user: Any,
query_contexts: ResolvedQueryContexts | None = None,
) -> dict[str, list[str]]:
"""Build the workbook on disk.
Return skipped charts grouped by ``email.ERROR_*`` reason.
:param path: Destination path for the ``.xlsx`` file
:param dashboard: The dashboard whose charts to export
:param active_data_mask: Live dashboard filter state keyed by native filter id
:param job_id: Correlation id used in log lines
:param mode: ``"data"`` or ``"images"``
:param user: The requesting user (used to render images)
:param query_contexts: Pre-resolved contexts. Missing charts are resolved here.
"""
errored: dict[str, list[str]] = {}
resolved = query_contexts or {}
writer = StreamingXlsxWriter(path)
try:
for chart in get_charts_in_layout_order(dashboard):
label = chart_label(chart)
try:
if renders_as_image(chart, mode):
# Image charts do not need a query context.
_write_chart_image_sheet(
writer, chart, dashboard.id, active_data_mask, user
)
else:
# Reuse a planned context or resolve one here.
json_body = (
resolved[chart.id]
if chart.id in resolved
else resolve_query_context(chart)
)
if json_body is None:
errored.setdefault(email.ERROR_NO_QUERY_CONTEXT, []).append(
label
)
continue
_write_chart_sheets(
writer, chart, json_body, dashboard.id, active_data_mask
)
except SoftTimeLimitExceeded:
# Let the task handler report the timeout and clean up.
raise
except ChartSkippedError:
logger.warning(
"Skipping chart %s in dashboard export %s (could not render)",
chart.id,
job_id,
)
errored.setdefault(email.ERROR_GENERAL, []).append(label)
except Exception: # pylint: disable=broad-except
logger.exception(
"Skipping chart %s in dashboard export %s", chart.id, job_id
)
errored.setdefault(email.ERROR_GENERAL, []).append(label)
# Include skipped charts in the workbook for both delivery paths.
if writer.sheet_count == 0 or errored:
flat = [label for labels in errored.values() for label in labels]
header = (
"No chart data could be exported."
if writer.sheet_count == 0
else "Charts that could not be exported:"
)
writer.add_summary_sheet("Export Summary", [header, *flat])
finally:
writer.close()
return errored
+8 -357
View File
@@ -14,22 +14,10 @@
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""
Celery task that exports every chart on a dashboard to a single multi-sheet
``.xlsx`` file, uploads it to S3, and emails the requesting user a pre-signed
download link.
In ``"data"`` mode the task re-runs each chart's saved query context under the
requesting user, applies the live dashboard filter state, and streams the results
row-by-row into a constant-memory workbook so large dashboards never load all
data at once. In ``"images"`` mode non-table charts are instead rendered to
images (through the same headless path as scheduled reports, reflecting the live
filters) and embedded, while table-like charts stay tabular.
"""
"""Build dashboard Excel exports in Celery and deliver them by email."""
from __future__ import annotations
import copy
import logging
import os
import tempfile
@@ -37,58 +25,23 @@ from datetime import datetime, timedelta, timezone
from typing import Any
from celery.exceptions import SoftTimeLimitExceeded
from flask import current_app, g
from flask import current_app
from superset import db, security_manager
from superset.charts.data.dashboard_filter_context import (
apply_dashboard_filter_context,
get_dashboard_filter_context,
)
from superset.charts.schemas import ChartDataQueryContextSchema
from superset.commands.chart.data.get_data_command import ChartDataCommand
from superset.commands.distributed_lock.release import ReleaseDistributedLock
from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
from superset.common.form_data_query_context import (
build_query_context_from_form_data,
is_raw_query_mode,
)
from superset.dashboards.excel_export import email
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
from superset.dashboards.excel_export.screenshot import render_chart_image
from superset.dashboards.excel_export.workbook import build_workbook, EXPORT_MODE_DATA
from superset.extensions import celery_app
from superset.utils import json, s3
from superset.utils import s3
from superset.utils.core import override_user
from superset.utils.excel_streaming import StreamingXlsxWriter
logger = logging.getLogger(__name__)
# Export modes: "data" streams every chart's tabular result (the default,
# unchanged behavior); "images" embeds non-table charts as rendered images and
# keeps only table-like charts tabular.
EXPORT_MODE_DATA = "data"
EXPORT_MODE_IMAGES = "images"
# Viz types kept as tabular data in image mode; everything else is rendered as an
# image. Operators can override the set via ``EXCEL_EXPORT_TABLE_VIZ_TYPES``.
TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"}
# Viz types whose missing query context may be rebuilt from saved form data.
# Conservative: only charts whose data maps faithfully to a single plain query
# (no post-processing, no multi-query fan-out). Every other viz type without a
# saved query context is skipped and listed for the user to re-save in Explore.
REBUILD_VIZ_TYPES = {"table", "big_number_total", "big_number", "pie"}
EXPORT_SOFT_TIME_LIMIT = 600
EXPORT_HARD_TIME_LIMIT = 660
# Namespace + TTL for the per-user+dashboard in-flight lock the API acquires
# before enqueue and this task releases when it settles. The lock uses the
# shared, atomic DistributedLock backend (Redis when configured, the metadata
# DB otherwise) so it actually synchronizes across the web server and workers —
# unlike a plain cache, which is a no-op under the default ``NullCache``.
# The TTL outlives the hard time limit so a worker killed at that limit (which
# skips the ``finally`` release) cannot hold the lock forever; the release in
# ``finally`` is the fast path that frees it as soon as the task settles.
# The API acquires this cross-process lock before either export path runs. The
# task releases it, while the TTL covers hard worker shutdowns.
EXPORT_LOCK_NAMESPACE = "excel_export"
EXPORT_LOCK_TTL_SECONDS = EXPORT_HARD_TIME_LIMIT + 60
@@ -98,308 +51,6 @@ def export_lock_params(user_id: int, dashboard_id: int) -> dict[str, int]:
return {"user_id": user_id, "dashboard_id": dashboard_id}
class _ChartSkippedError(Exception):
"""Signals a chart that could not be exported and should be listed as skipped."""
def _chart_label(chart: Any) -> str:
"""Human-readable label for a chart in the skipped-charts list."""
return f"{chart.id} - {chart.slice_name or ''}".strip()
def _usable_query_context(value: Any) -> dict[str, Any] | None:
"""
``value`` when it is a usable query-context payload, else ``None``.
A payload is usable only if it is a dict with a non-empty ``queries`` list; a
blank, query-less, mistyped, or non-object value (e.g. ``{}``,
``{"queries": []}``, ``{"queries": "oops"}``, ``None``) is treated the same as
a missing context. Shared by the saved-context path and the builder hook so
both apply the same validity rule and so a malformed builder return falls
through to the built-in rebuild instead of failing later in the general
error bucket.
"""
if not isinstance(value, dict) or not isinstance(value.get("queries"), list):
return None
return value if value["queries"] else None
def _saved_query_context(raw: Any) -> dict[str, Any] | None:
"""
The chart's saved query context parsed to a dict, or ``None`` when it is
missing or unusable.
Returns ``None`` for a blank value, a string that does not parse as JSON, and
any value that is not a dict with a non-empty ``queries`` list.
"""
if not raw:
return None
try:
parsed = json.loads(raw)
except (TypeError, ValueError):
return None
return _usable_query_context(parsed)
# Form-data keys whose behavior needs plugin post-processing or extra queries
# (contribution/time comparison, rolling window, resampling, raw big-number
# aggregation) that the single-query rebuild cannot reproduce. A chart using any
# of these is skipped rather than exported with values that differ from the chart.
_UNSUPPORTED_PROCESSING_KEYS = ("time_compare", "rolling_type", "resample_rule")
def _needs_unsupported_processing(form_data: dict[str, Any]) -> bool:
"""Whether the form data relies on processing the rebuild can't reproduce."""
# ``percent_metrics`` are "% of total" columns produced by contribution
# post-processing the rebuild can't apply; skip so the export doesn't silently
# omit columns the user sees.
if form_data.get("percent_metrics"):
return True
# ``show_totals`` adds a totals row via a *second* query
# (``plugin-chart-table/src/buildQuery.ts``, gated on aggregate mode); the
# single-query rebuild would silently drop that row. The mode check mirrors
# the frontend so a raw-mode table carrying a stale value still exports.
if form_data.get("show_totals") and not is_raw_query_mode(form_data):
return True
for key in _UNSUPPORTED_PROCESSING_KEYS:
value = form_data.get(key)
# ``rolling_type`` is often the literal string ``"None"`` when unset.
if value and value != "None":
return True
return form_data.get("aggregation") == "raw"
def _resolve_query_context(chart: Any) -> dict[str, Any] | None:
"""
The query-context payload to run for a chart's data export, or ``None`` when
none can be obtained.
Resolution order:
1. the chart's saved ``query_context``;
2. an optional ``EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`` hook, letting a deployment
supply a faithful context (e.g. from a service running the chart's real
frontend ``buildQuery``) for viz types the built-in rebuild can't handle;
3. the built-in form-data rebuild, restricted to viz types whose data maps
faithfully to a single plain query (``REBUILD_VIZ_TYPES``) without
post-processing or extra queries.
Returns ``None`` when none apply, so the caller lists the chart for re-saving
rather than exporting inaccurate data.
"""
if saved := _saved_query_context(chart.query_context):
return saved
# The hook receives the chart's form data and must return ``None`` — not a
# partial/stub context — whenever it can't build the chart faithfully, so we
# fall through to the built-in rebuild (which handles the allowlisted viz types
# well). A hook failure falls through too, preserving "builder problem →
# rebuild, don't fail the export" — the one exception being a task-level
# timeout, which has to abort the whole export rather than this chart.
if builder := current_app.config.get("EXCEL_EXPORT_QUERY_CONTEXT_BUILDER"):
try:
built = builder(chart.form_data)
except SoftTimeLimitExceeded:
# A soft timeout is a task-level signal, not a builder failure: let it
# propagate to _build_workbook so the export aborts cleanly instead of
# continuing on to rebuild this chart and start the next one.
raise
except Exception: # pylint: disable=broad-except
logger.warning(
"EXCEL_EXPORT_QUERY_CONTEXT_BUILDER failed for chart %s; "
"falling back to the built-in rebuild",
chart.id,
exc_info=True,
)
built = None
if (from_builder := _usable_query_context(built)) is not None:
# Copy: the payload's ``queries`` are mutated in place downstream (by
# ``apply_dashboard_filter_context``), and a builder is free to
# memoize or otherwise share its return value — which would then
# accumulate filters across charts and across exports.
return copy.deepcopy(from_builder)
# The allowlist and ``_needs_unsupported_processing`` bound only the built-in
# rebuild below; the builder hook above is intentionally not gated by them (a
# faithful builder includes the post-processing the built-in rebuild lacks).
if chart.viz_type not in REBUILD_VIZ_TYPES or chart.datasource_id is None:
return None
try:
form_data = json.loads(chart.params) if chart.params else {}
except (TypeError, ValueError):
return None
if not isinstance(form_data, dict) or not form_data:
return None
if _needs_unsupported_processing(form_data):
return None
return build_query_context_from_form_data(
form_data,
{"id": chart.datasource_id, "type": chart.datasource_type or "table"},
chart.viz_type,
)
def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]:
return [record.get(col) for col in colnames]
def _table_viz_types() -> set[str]:
"""Viz types kept tabular in image mode (config override or built-in default)."""
return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES
def _renders_as_image(chart: Any, mode: str) -> bool:
"""Whether this chart is embedded as an image rather than streamed as data."""
return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types()
def _write_chart_image_sheet(
writer: StreamingXlsxWriter,
chart: Any,
dashboard_id: int,
active_data_mask: dict[str, Any],
user: Any,
) -> None:
"""
Render a single chart to an image and embed it as its own sheet.
:raises _ChartSkippedError: if the chart could not be rendered
"""
image = render_chart_image(chart, dashboard_id, active_data_mask, user)
if image is None:
raise _ChartSkippedError
writer.add_image_sheet(_chart_label(chart), image)
def _write_chart_sheets(
writer: StreamingXlsxWriter,
chart: Any,
json_body: dict[str, Any],
dashboard_id: int,
active_data_mask: dict[str, Any],
) -> None:
"""
Run a single chart's query and stream its result(s) into the workbook.
``json_body`` is the resolved query-context payload (the chart's saved
context or one synthesized from its form data). Charts may yield more than
one query (e.g. mixed-series charts); each becomes its own sheet. Raises if
the chart cannot be exported, so the caller can skip it and note it in the
email.
"""
# Shallow-copy before setting our own top-level keys so the caller's payload
# keeps its original result_format/result_type. (The nested ``queries`` are
# mutated in place by apply_dashboard_filter_context below, which is safe
# because every payload ``_resolve_query_context`` returns is this chart's
# alone: freshly parsed, freshly built, or deep-copied from the builder hook.)
json_body = dict(json_body)
# Override any stale saved values: we always want full JSON results.
json_body["result_format"] = ChartDataResultFormat.JSON
json_body["result_type"] = ChartDataResultType.FULL
json_body.pop("force", None)
filter_context = get_dashboard_filter_context(
dashboard_id=dashboard_id,
chart_id=chart.id,
active_data_mask=active_data_mask,
)
if filter_context.extra_form_data:
apply_dashboard_filter_context(json_body, filter_context.extra_form_data)
# Jinja macros resolve form data from g.form_data; expose the saved context.
g.form_data = json_body
query_context = ChartDataQueryContextSchema().load(json_body)
command = ChartDataCommand(query_context)
command.validate()
result = command.run()
for index, query in enumerate(result["queries"]):
colnames = query.get("colnames") or []
data = query.get("data") or []
if index == 0:
name = f"{chart.id} - {chart.slice_name or ''}"
else:
name = f"{chart.id}.{index} - {chart.slice_name or ''}"
writer.add_sheet(
name,
colnames,
(_record_to_row(record, colnames) for record in data),
)
def _build_workbook(
path: str,
dashboard: Any,
active_data_mask: dict[str, Any],
job_id: str,
mode: str,
user: Any,
) -> dict[str, list[str]]:
"""Build the workbook on disk.
Return the charts that could not be exported, grouped by the reason they
were omitted (see the ``email.ERROR_*`` reason keys), so the notification
can explain each group separately.
"""
errored: dict[str, list[str]] = {}
writer = StreamingXlsxWriter(path)
try:
for chart in get_charts_in_layout_order(dashboard):
label = _chart_label(chart)
try:
if _renders_as_image(chart, mode):
# Image charts render from their saved params via the
# webdriver and don't need a query context.
_write_chart_image_sheet(
writer, chart, dashboard.id, active_data_mask, user
)
else:
# Data charts need a query context: use the saved one, or
# rebuild it from form data for eligible viz types. Skip
# cleanly when none is available rather than failing.
json_body = _resolve_query_context(chart)
if json_body is None:
errored.setdefault(email.ERROR_NO_QUERY_CONTEXT, []).append(
label
)
continue
_write_chart_sheets(
writer, chart, json_body, dashboard.id, active_data_mask
)
except SoftTimeLimitExceeded:
# A soft timeout is a task-level signal, not a per-chart failure:
# let it propagate so the outer handler emails a failure and runs
# cleanup, rather than continuing until the hard limit kills the
# worker (which would skip cleanup, leak temp files, and hold the
# in-flight lock until its TTL). ``except Exception`` below would
# otherwise swallow it, since it subclasses ``Exception``.
raise
except _ChartSkippedError:
logger.warning(
"Skipping chart %s in dashboard export %s (could not render)",
chart.id,
job_id,
)
errored.setdefault(email.ERROR_GENERAL, []).append(label)
except Exception: # pylint: disable=broad-except
logger.exception(
"Skipping chart %s in dashboard export %s", chart.id, job_id
)
errored.setdefault(email.ERROR_GENERAL, []).append(label)
if writer.sheet_count == 0:
flat = [label for labels in errored.values() for label in labels]
writer.add_summary_sheet(
"Export Summary",
["No chart data could be exported.", *flat],
)
finally:
writer.close()
return errored
def _send_failure_email(
user: Any, dashboard_title: str, requested_at: datetime
) -> None:
@@ -462,7 +113,7 @@ def export_dashboard_excel(
)
os.close(file_descriptor)
errored = _build_workbook(
errored = build_workbook(
tmp_path, dashboard, active_data_mask, job_id, mode, user
)
@@ -510,7 +161,7 @@ def export_dashboard_excel(
export_lock_params(user_id, dashboard_id),
).run()
except Exception: # pylint: disable=broad-except
# Best-effort: the lock's TTL is the backstop if this fails.
# The TTL is the fallback if release fails.
logger.exception(
"Failed to release in-flight export lock for user %s dashboard %s",
user_id,
+341 -4
View File
@@ -28,11 +28,15 @@ import pytest
import rison
import yaml
from flask import current_app, g
from freezegun import freeze_time
from sqlalchemy import and_
from superset import db, security_manager # noqa: F401
from superset.commands.dashboard.permalink.create import CreateDashboardPermalinkCommand
from superset.daos.dashboard import EmbeddedDashboardDAO
from superset.dashboards.excel_export.sync_budget import InlineExportPlan
from superset.exceptions import LockAlreadyHeldException
from superset.security.guest_token import GuestTokenResourceType, GuestUser
from superset.models.dashboard import Dashboard
from superset.models.core import FavStar, FavStarClassName
from superset.reports.models import ReportSchedule, ReportScheduleType
@@ -3558,14 +3562,16 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
response = json.loads(rv.data.decode("utf-8"))
assert response["count"] > 0
def test_export_xlsx_501_when_bucket_unset(self):
"""Dashboard API: export_xlsx returns 501 when the S3 bucket is unset."""
def test_export_xlsx_400_for_empty_dashboard_without_storage(self):
"""Dashboard API: with no storage configured the request is still validated
before an export runs, so a dashboard with no charts is rejected rather
than streaming an empty workbook."""
admin = self.get_user("admin")
dashboard = self.insert_dashboard("xlsx-501", None, [admin.id])
dashboard = self.insert_dashboard("xlsx-sync-empty", None, [admin.id])
self.login(ADMIN_USERNAME)
try:
rv = self.client.post(f"api/v1/dashboard/{dashboard.id}/export_xlsx/")
assert rv.status_code == 501
assert rv.status_code == 400
finally:
db.session.delete(dashboard)
db.session.commit()
@@ -3687,6 +3693,10 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": "exports"})
@with_feature_flags(
ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS=False,
ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT=False,
)
@patch("superset.dashboards.api.export_dashboard_excel")
def test_export_xlsx_images_404_when_screenshot_flags_off(self, mock_task):
"""Dashboard API: ``mode=images`` is rejected with 404 when the webdriver
@@ -3725,6 +3735,333 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
_, kwargs = mock_task.apply_async.call_args
assert kwargs["kwargs"]["mode"] == "images"
# Direct download without export storage
@staticmethod
def _write_stub_workbook(path, *args, **kwargs):
"""Stand in for the shared workbook builder, writing a real .xlsx."""
from superset.utils.excel_streaming import StreamingXlsxWriter
writer = StreamingXlsxWriter(path)
writer.add_sheet("10 - Chart", ["a"], [[1]])
writer.close()
return {}
@staticmethod
def _export_temp_files():
"""Temp files the export path creates, so a leak can be detected."""
import glob
import os
import tempfile
return glob.glob(os.path.join(tempfile.gettempdir(), "dash-export-*"))
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.export_dashboard_excel")
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_200_streams_workbook_without_storage(
self, mock_build, mock_task
):
"""Dashboard API: with no storage configured the workbook is built inline
and returned as the response, instead of the request dead-ending."""
mock_build.side_effect = self._write_stub_workbook
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 200
assert rv.mimetype == (
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
)
assert "attachment" in rv.headers["Content-Disposition"]
assert ".xlsx" in rv.headers["Content-Disposition"]
# XLSX files are ZIP archives.
assert rv.data.startswith(b"PK")
assert is_zipfile(BytesIO(rv.data))
# Direct downloads do not queue a task.
mock_task.apply_async.assert_not_called()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_builds_with_the_same_inputs_as_the_task(self, mock_build):
"""Dashboard API: the synchronous path hands the shared builder the same
dashboard, filter state and mode the Celery task would, so both paths
produce the same workbook."""
mock_build.side_effect = self._write_stub_workbook
data_mask = {"NATIVE_FILTER-abc": {"extraFormData": {"time_range": "No"}}}
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": data_mask},
)
assert rv.status_code == 200
args, _ = mock_build.call_args
path, built_dashboard, active_data_mask, _job_id, mode, user = args
assert path.endswith(".xlsx")
assert built_dashboard.id == dashboard.id
assert active_data_mask == data_mask
assert mode == "data"
assert user.username == ADMIN_USERNAME
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.ReleaseDistributedLock")
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.build_workbook")
@patch("superset.dashboards.api.plan_inline_export")
def test_export_xlsx_sync_refused_when_over_the_row_budget(
self, mock_plan, mock_build, mock_acquire, mock_release
):
"""Dashboard API: an export too large to serve inline is refused up front
with a message naming the fix, rather than being started and timing out."""
mock_plan.return_value = InlineExportPlan(
query_contexts={}, requested_rows=250_000, max_rows=100_000
)
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 400
message = rv.data.decode("utf-8")
assert "EXCEL_EXPORT_S3_BUCKET" in message
# A budget refusal releases the lock without querying charts.
mock_plan.assert_called_once()
mock_build.assert_not_called()
mock_acquire.return_value.run.assert_called_once()
mock_release.return_value.run.assert_called_once()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.ReleaseDistributedLock")
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.plan_inline_export")
def test_export_xlsx_sync_releases_the_lock_when_planning_fails(
self, mock_plan, mock_acquire, mock_release
):
"""Dashboard API: a context-builder failure while planning must not keep
the user locked out until the lock's TTL expires."""
mock_plan.side_effect = RuntimeError("builder failed")
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 500
mock_acquire.return_value.run.assert_called_once()
mock_release.return_value.run.assert_called_once()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.build_workbook")
@patch("superset.dashboards.api.plan_inline_export")
def test_export_xlsx_sync_runs_the_contexts_the_budget_measured(
self, mock_plan, mock_build
):
"""Dashboard API: the export runs the query contexts the row budget was
measured against. Resolving them a second time would risk vouching for one
set of queries and running another, since a deployment's context builder
need not be deterministic."""
measured = {10: {"queries": [{"row_limit": 5}]}, 20: None}
mock_plan.return_value = InlineExportPlan(
query_contexts=measured, requested_rows=5, max_rows=100_000
)
mock_build.side_effect = self._write_stub_workbook
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 200
assert mock_build.call_args.kwargs["query_contexts"] is measured
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@with_feature_flags(
ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS=True,
ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT=True,
)
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_images_refused_without_storage(self, mock_build, mock_acquire):
"""Dashboard API: image export renders every chart through the headless
webdriver, which no row budget bounds and no request should wait on, so it
is refused rather than served inline -- even with the webdriver enabled."""
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}, "mode": "images"},
)
assert rv.status_code == 400
assert "EXCEL_EXPORT_S3_BUCKET" in rv.data.decode("utf-8")
mock_build.assert_not_called()
mock_acquire.return_value.run.assert_not_called()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.ReleaseDistributedLock")
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_releases_the_lock_on_success(
self, mock_build, mock_acquire, mock_release
):
"""Dashboard API: the in-flight lock the synchronous path takes is released
once the response is ready, so the next export is not locked out."""
mock_build.side_effect = self._write_stub_workbook
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 200
mock_acquire.return_value.run.assert_called_once()
mock_release.return_value.run.assert_called_once()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.ReleaseDistributedLock")
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_releases_the_lock_when_building_fails(
self, mock_build, mock_acquire, mock_release
):
"""Dashboard API: a failure while building must not leave the user locked
out of their own dashboard until the lock's TTL expires."""
mock_build.side_effect = RuntimeError("boom")
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 500
mock_acquire.return_value.run.assert_called_once()
mock_release.return_value.run.assert_called_once()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_deletes_the_temp_file_on_success(self, mock_build):
"""Dashboard API: the workbook is built through a temp file, which must not
outlive the response."""
mock_build.side_effect = self._write_stub_workbook
before = self._export_temp_files()
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 200
assert self._export_temp_files() == before
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_deletes_the_temp_file_when_building_fails(
self, mock_build
):
"""Dashboard API: a half-written workbook is cleaned up too, so a failing
export does not fill the web server's disk."""
mock_build.side_effect = RuntimeError("boom")
before = self._export_temp_files()
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 500
assert self._export_temp_files() == before
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@patch("superset.dashboards.api.AcquireDistributedLock")
@patch("superset.dashboards.api.build_workbook")
@patch("superset.dashboards.api.plan_inline_export")
def test_export_xlsx_sync_rejected_when_export_already_in_progress(
self, mock_plan, mock_build, mock_acquire
):
"""Dashboard API: the synchronous path honors the same per-user+dashboard
lock as the queued one, so one user cannot run two exports at once."""
mock_acquire.return_value.run.side_effect = LockAlreadyHeldException("held")
self.login(ADMIN_USERNAME)
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
rv = self.client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
)
assert rv.status_code == 202
assert "already in progress" in rv.data.decode("utf-8")
mock_plan.assert_not_called()
mock_build.assert_not_called()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
@with_feature_flags(EMBEDDED_SUPERSET=True)
@patch("superset.dashboards.api.build_workbook")
def test_export_xlsx_sync_still_blocks_guest_sessions(self, mock_build):
"""Dashboard API: the synchronous path does not become a way for an
embedded guest session to export a dashboard. Guest support is deliberately
out of scope here, so a guest holding a token that *does* grant access to
this dashboard is still refused, before any workbook is built."""
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
embedded = EmbeddedDashboardDAO.upsert(dashboard, ["superset.example"])
db.session.commit()
token = security_manager.create_guest_access_token(
{"username": "xlsx_guest"},
[{"type": GuestTokenResourceType.DASHBOARD, "id": str(embedded.uuid)}],
[],
)
with self.client as client:
rv = client.post(
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
json={"active_data_mask": {}},
headers={
current_app.config["GUEST_TOKEN_HEADER_NAME"]: token.decode("utf-8")
if isinstance(token, bytes)
else token
},
)
assert isinstance(g.user, GuestUser)
assert rv.status_code == 400
assert "email address" in rv.data.decode("utf-8")
mock_build.assert_not_called()
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
def test_embedded_dashboards(self):
self.login(ADMIN_USERNAME)
@@ -0,0 +1,39 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations
import pytest
from superset.app import SupersetApp
from superset.dashboards.excel_export.storage import is_export_storage_configured
@pytest.mark.parametrize(
("bucket", "configured"),
[
("exports-bucket", True),
(None, False),
("", False),
],
)
def test_storage_is_configured_only_with_a_bucket(
app: SupersetApp, bucket: str | None, configured: bool
) -> None:
from flask import current_app
current_app.config["EXCEL_EXPORT_S3_BUCKET"] = bucket
assert is_export_storage_configured() is configured
@@ -0,0 +1,173 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
from __future__ import annotations
from collections.abc import Iterator
from typing import Any
from unittest import mock
import pytest
from flask import current_app
from superset.dashboards.excel_export.sync_budget import plan_inline_export
from superset.utils import json
MODULE = "superset.dashboards.excel_export.sync_budget"
def _chart(chart_id: int, *queries: dict[str, Any]) -> mock.MagicMock:
"""A chart whose saved query context holds ``queries``."""
chart = mock.MagicMock()
chart.id = chart_id
chart.slice_name = f"Chart {chart_id}"
chart.viz_type = "table"
chart.query_context = json.dumps({"queries": list(queries)})
return chart
def _unexportable_chart(chart_id: int) -> mock.MagicMock:
"""A chart the export has to skip: no context, and no rebuilding it."""
chart = _chart(chart_id)
chart.query_context = None
chart.viz_type = "mixed_timeseries" # outside the rebuild allowlist
return chart
@pytest.fixture
def charts() -> Iterator[mock.MagicMock]:
"""Patch the layout walk so tests supply the dashboard's charts directly."""
with mock.patch(f"{MODULE}.get_charts_in_layout_order") as ordered:
yield ordered
@pytest.fixture(autouse=True)
def restore_config() -> Iterator[None]:
"""Undo config edits: the app fixture is shared by every test in the module."""
original_sync_max_rows = current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"]
original_row_limit = current_app.config["ROW_LIMIT"]
yield
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = original_sync_max_rows
current_app.config["ROW_LIMIT"] = original_row_limit
def test_plan_sums_the_row_limit_of_every_chart(charts: mock.MagicMock) -> None:
charts.return_value = [
_chart(10, {"row_limit": 1000}),
_chart(20, {"row_limit": 250}),
]
assert plan_inline_export(mock.MagicMock()).requested_rows == 1250
def test_plan_counts_every_query_of_a_multi_query_chart(
charts: mock.MagicMock,
) -> None:
# Count every query in a multi-query chart.
charts.return_value = [_chart(10, {"row_limit": 100}, {"row_limit": 400})]
assert plan_inline_export(mock.MagicMock()).requested_rows == 500
@pytest.mark.parametrize(
"query",
[
{"row_limit": "1000"}, # not an integer
{"row_limit": -5},
],
)
def test_plan_row_total_is_indeterminate_without_a_finite_row_limit(
charts: mock.MagicMock, query: dict[str, Any]
) -> None:
# Every query needs a finite limit.
charts.return_value = [_chart(10, {"row_limit": 100}), _chart(20, query)]
plan = plan_inline_export(mock.MagicMock())
assert plan.requested_rows is None
assert plan.fits_row_budget is False
@pytest.mark.parametrize("query", [{}, {"row_limit": 0}, {"row_limit": None}])
def test_plan_uses_default_when_row_limit_is_omitted(
charts: mock.MagicMock, query: dict[str, Any]
) -> None:
current_app.config["ROW_LIMIT"] = 250
charts.return_value = [_chart(10, {"row_limit": 100}), _chart(20, query)]
assert plan_inline_export(mock.MagicMock()).requested_rows == 350
def test_plan_ignores_charts_that_cannot_be_exported(charts: mock.MagicMock) -> None:
# Skipped charts add no rows.
charts.return_value = [_chart(10, {"row_limit": 100}), _unexportable_chart(20)]
assert plan_inline_export(mock.MagicMock()).requested_rows == 100
@pytest.mark.parametrize(
("row_limit", "fits"),
[
(99_999, True), # below the limit
(100_000, True), # exactly at the limit
(100_001, False), # above the limit
],
)
def test_plan_fits_totals_up_to_and_including_the_limit(
charts: mock.MagicMock, row_limit: int, fits: bool
) -> None:
charts.return_value = [_chart(10, {"row_limit": row_limit})]
assert plan_inline_export(mock.MagicMock()).fits_row_budget is fits
def test_plan_honors_the_configured_limit(charts: mock.MagicMock) -> None:
charts.return_value = [_chart(10, {"row_limit": 5_000})]
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = 1_000
assert plan_inline_export(mock.MagicMock()).fits_row_budget is False
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = 10_000
assert plan_inline_export(mock.MagicMock()).fits_row_budget is True
def test_plan_carries_the_resolved_context_of_every_chart(
charts: mock.MagicMock,
) -> None:
# Return the contexts used to calculate the budget.
exportable = _chart(10, {"row_limit": 100})
charts.return_value = [exportable, _unexportable_chart(20)]
plan = plan_inline_export(mock.MagicMock())
assert plan.query_contexts[10] == {"queries": [{"row_limit": 100}]}
# ``None`` marks a resolved chart that cannot be exported.
assert 20 in plan.query_contexts
assert plan.query_contexts[20] is None
def test_plan_resolves_each_chart_exactly_once(charts: mock.MagicMock) -> None:
# Resolve each chart once so planning and export use the same context.
first = _chart(10, {"row_limit": 1})
second = _chart(20, {"row_limit": 2})
charts.return_value = [first, second]
with mock.patch(f"{MODULE}.resolve_query_context") as resolve:
resolve.return_value = {"queries": [{"row_limit": 1}]}
plan_inline_export(mock.MagicMock())
assert resolve.call_args_list == [mock.call(first), mock.call(second)]
@@ -0,0 +1,125 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
"""Tests for passing already-resolved query contexts into the workbook builder.
The builder's own behavior (sheet naming, skipped charts, filter application) is
covered through the Celery task in
``tests/unit_tests/tasks/test_export_dashboard_excel.py``.
"""
from __future__ import annotations
import os
import tempfile
from collections.abc import Iterator
from typing import Any
from unittest import mock
import pytest
from superset.dashboards.excel_export.workbook import build_workbook
from superset.utils import json
MODULE = "superset.dashboards.excel_export.workbook"
def _chart(chart_id: int, name: str) -> mock.MagicMock:
chart = mock.MagicMock()
chart.id = chart_id
chart.slice_name = name
chart.viz_type = "table"
chart.query_context = json.dumps({"queries": [{"row_limit": 100}]})
return chart
@pytest.fixture
def mocks() -> Iterator[dict[str, Any]]:
"""Patch the builder's collaborators; keep the real xlsx writer."""
with mock.patch.multiple(
MODULE,
get_charts_in_layout_order=mock.DEFAULT,
get_dashboard_filter_context=mock.DEFAULT,
ChartDataQueryContextSchema=mock.DEFAULT,
ChartDataCommand=mock.DEFAULT,
resolve_query_context=mock.DEFAULT,
) as patched:
patched["get_dashboard_filter_context"].return_value.extra_form_data = {}
patched["ChartDataCommand"].return_value.run.return_value = {
"queries": [{"colnames": ["a"], "data": [{"a": 1}]}]
}
yield patched
@pytest.fixture
def workbook_path() -> Iterator[str]:
file_descriptor, path = tempfile.mkstemp(suffix=".xlsx")
os.close(file_descriptor)
yield path
if os.path.exists(path):
os.remove(path)
def _build(path: str, **kwargs: Any) -> Any:
dashboard = mock.MagicMock()
dashboard.id = 1
return build_workbook(
path, dashboard, {}, "job-1", "data", mock.MagicMock(), **kwargs
)
def test_provided_query_context_is_used_without_resolving_again(
mocks: dict[str, Any], workbook_path: str
) -> None:
# Use the context measured by the row budget.
chart = _chart(10, "First")
mocks["get_charts_in_layout_order"].return_value = [chart]
provided = {"queries": [{"row_limit": 7, "metrics": ["count"]}]}
_build(workbook_path, query_contexts={10: provided})
mocks["resolve_query_context"].assert_not_called()
loaded = mocks["ChartDataQueryContextSchema"].return_value.load.call_args.args[0]
assert loaded["queries"] == provided["queries"]
def test_a_chart_resolved_to_none_is_skipped_without_resolving_again(
mocks: dict[str, Any], workbook_path: str
) -> None:
# ``None`` marks a chart already resolved as unexportable.
chart = _chart(20, "Skipped")
mocks["get_charts_in_layout_order"].return_value = [chart]
errored = _build(workbook_path, query_contexts={20: None})
mocks["resolve_query_context"].assert_not_called()
mocks["ChartDataCommand"].return_value.run.assert_not_called()
assert [label for labels in errored.values() for label in labels] == [
"20 - Skipped"
]
def test_a_chart_missing_from_the_map_is_resolved_by_the_builder(
mocks: dict[str, Any], workbook_path: str
) -> None:
# Resolve charts missing from a partial context map.
chart = _chart(30, "Unmapped")
mocks["get_charts_in_layout_order"].return_value = [chart]
mocks["resolve_query_context"].return_value = {"queries": [{"row_limit": 5}]}
_build(workbook_path, query_contexts={})
mocks["resolve_query_context"].assert_called_once_with(chart)
@@ -30,6 +30,8 @@ from celery.exceptions import SoftTimeLimitExceeded
from superset.utils import json
MODULE = "superset.tasks.export_dashboard_excel"
# Workbook building is shared by queued and direct exports.
WORKBOOK_MODULE = "superset.dashboards.excel_export.workbook"
# A minimal valid 1x1 transparent PNG for image-mode tests.
@@ -71,21 +73,25 @@ def mocks() -> Iterator[dict[str, Any]]:
# would make calls like security_manager.get_user_by_id() return coroutines.
patched = {
name: stack.enter_context(
mock.patch(f"{MODULE}.{name}", new=mock.MagicMock())
mock.patch(f"{module}.{name}", new=mock.MagicMock())
)
for name in (
"security_manager",
"db",
"get_charts_in_layout_order",
"get_dashboard_filter_context",
"ChartDataQueryContextSchema",
"ChartDataCommand",
"render_chart_image",
"s3",
"email",
"ReleaseDistributedLock",
for module, name in (
(MODULE, "security_manager"),
(MODULE, "db"),
(MODULE, "s3"),
(MODULE, "ReleaseDistributedLock"),
(WORKBOOK_MODULE, "get_charts_in_layout_order"),
(WORKBOOK_MODULE, "get_dashboard_filter_context"),
(WORKBOOK_MODULE, "ChartDataQueryContextSchema"),
(WORKBOOK_MODULE, "ChartDataCommand"),
(WORKBOOK_MODULE, "render_chart_image"),
)
}
# Both modules must use the same mocked error keys.
shared_email = mock.MagicMock()
for module in (MODULE, WORKBOOK_MODULE):
stack.enter_context(mock.patch(f"{module}.email", new=shared_email))
patched["email"] = shared_email
user = mock.MagicMock()
user.email = "user@example.com"
patched["security_manager"].get_user_by_id.return_value = user
@@ -287,8 +293,6 @@ def _rebuildable_chart(
@contextmanager
def _builder_hook(builder: Any) -> Iterator[None]:
"""Patch current_app so EXCEL_EXPORT_QUERY_CONTEXT_BUILDER resolves to builder."""
from superset.tasks import export_dashboard_excel as module
fake_app = mock.MagicMock()
fake_app.config.get.side_effect = lambda key, default=None: (
builder if key == "EXCEL_EXPORT_QUERY_CONTEXT_BUILDER" else default
@@ -300,14 +304,17 @@ def _builder_hook(builder: Any) -> Iterator[None]:
"EXCEL_EXPORT_S3_KEY_PREFIX": "dashboard-exports/",
"EXCEL_EXPORT_LINK_TTL_SECONDS": 3600,
}.__getitem__
with mock.patch.object(module, "current_app", fake_app):
# Both the task and workbook read configuration.
with ExitStack() as stack:
for module in (MODULE, WORKBOOK_MODULE):
stack.enter_context(mock.patch(f"{module}.current_app", fake_app))
yield
def test_builder_hook_context_is_used_for_any_viz_type() -> None:
# A configured builder can supply a context for a viz type outside the
# built-in allowlist (pivot_table_v2), and is called with the chart's form data.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
ctx = {
"datasource": {"id": 5, "type": "table"},
@@ -317,7 +324,7 @@ def test_builder_hook_context_is_used_for_any_viz_type() -> None:
chart = _rebuildable_chart(viz_type="pivot_table_v2")
with _builder_hook(builder):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
assert result == ctx
builder.assert_called_once_with(chart.form_data)
@@ -326,13 +333,13 @@ def test_builder_hook_context_is_used_for_any_viz_type() -> None:
def test_builder_hook_none_falls_through_to_builtin_rebuild() -> None:
# When the builder returns None (can't build faithfully) the export falls
# through to the built-in rebuild, so an allowlisted table is unaffected.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
builder = mock.MagicMock(return_value=None)
chart = _rebuildable_chart(viz_type="table")
with _builder_hook(builder):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
builder.assert_called_once_with(chart.form_data)
assert result is not None
@@ -354,13 +361,13 @@ def test_builder_hook_none_falls_through_to_builtin_rebuild() -> None:
def test_builder_hook_malformed_result_falls_through(built: Any) -> None:
# A stub / empty / malformed builder result is treated as "not built" and
# falls through to the built-in rebuild rather than shipping an empty context.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
builder = mock.MagicMock(return_value=built)
chart = _rebuildable_chart(viz_type="table")
with _builder_hook(builder):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
builder.assert_called_once_with(chart.form_data)
assert result is not None
@@ -370,13 +377,13 @@ def test_builder_hook_malformed_result_falls_through(built: Any) -> None:
def test_builder_hook_exception_falls_through() -> None:
# A raising builder (e.g. sidecar down) must not fail the chart; the export
# falls through to the built-in rebuild and no exception escapes.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
builder = mock.MagicMock(side_effect=RuntimeError("sidecar down"))
chart = _rebuildable_chart(viz_type="table")
with _builder_hook(builder):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
builder.assert_called_once_with(chart.form_data)
assert result is not None
@@ -387,13 +394,13 @@ def test_builder_hook_soft_time_limit_propagates() -> None:
# A soft timeout raised while the builder is in flight is a task-level signal,
# not a builder failure: it must escape _resolve_query_context so the export
# aborts cleanly, rather than being swallowed by the broad fall-through guard.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
builder = mock.MagicMock(side_effect=SoftTimeLimitExceeded())
chart = _rebuildable_chart(viz_type="table")
with _builder_hook(builder), pytest.raises(SoftTimeLimitExceeded):
module._resolve_query_context(chart)
module.resolve_query_context(chart)
builder.assert_called_once_with(chart.form_data)
@@ -401,11 +408,11 @@ def test_builder_hook_soft_time_limit_propagates() -> None:
def test_no_builder_hook_leaves_builtin_behavior_unchanged() -> None:
# With no builder configured, an allowlisted chart is rebuilt and an
# ineligible one is skipped — identical to the pre-hook behavior.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
with _builder_hook(None):
table = module._resolve_query_context(_rebuildable_chart(viz_type="table"))
ineligible = module._resolve_query_context(
table = module.resolve_query_context(_rebuildable_chart(viz_type="table"))
ineligible = module.resolve_query_context(
_rebuildable_chart(viz_type="mixed_timeseries")
)
@@ -416,14 +423,14 @@ def test_no_builder_hook_leaves_builtin_behavior_unchanged() -> None:
def test_saved_context_short_circuits_builder_hook() -> None:
# A saved query context wins over the builder hook, which is never called.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
builder = mock.MagicMock(return_value={"queries": [{"from": "hook"}]})
chart = _rebuildable_chart(viz_type="table")
chart.query_context = json.dumps({"queries": [{"from": "saved"}]})
with _builder_hook(builder):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
assert result == {"queries": [{"from": "saved"}]}
builder.assert_not_called()
@@ -595,7 +602,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
# gates the totals query on queryMode === Aggregate), and the control isn't
# reset when hidden. A raw-mode table carrying a stale value must still
# rebuild rather than be needlessly skipped.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
chart = _rebuildable_chart(
viz_type="table",
@@ -603,7 +610,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
)
with _builder_hook(None):
result = module._resolve_query_context(chart)
result = module.resolve_query_context(chart)
assert result is not None
assert result["queries"][0]["columns"] == ["a"]
@@ -612,7 +619,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
def test_rebuild_viz_types_is_the_conservative_default() -> None:
# The rebuild allow-list is a fixed fallback (no config override): only viz
# types whose data maps faithfully to a single plain query.
from superset.tasks import export_dashboard_excel as module
from superset.dashboards.excel_export import workbook as module
assert module.REBUILD_VIZ_TYPES == {
"table",
@@ -699,6 +706,34 @@ def test_all_charts_skipped_writes_summary(mocks: dict[str, Any]) -> None:
mocks["email"].build_success_email.assert_called_once()
def test_partial_failure_appends_summary_sheet(mocks: dict[str, Any]) -> None:
"""When some charts export and others are skipped, the workbook itself lists
the skipped charts: a download served as the response to the request has no
email to list them in."""
mocks["get_charts_in_layout_order"].return_value = [
_chart(10, "Good"),
_chart(20, "Bad", has_context=False, viz_type="sunburst"),
]
mocks["ChartDataCommand"].return_value.run.return_value = {
"queries": [{"colnames": ["a"], "data": [{"a": 1}]}]
}
uploaded: dict[str, Any] = {}
def _capture(path: str, bucket: str, key: str) -> None:
uploaded["sheets"] = _read_sheets(path)
mocks["s3"].upload_file_to_s3.side_effect = _capture
_run()
assert "Export Summary" in uploaded["sheets"]
flat = [str(cell) for row in uploaded["sheets"]["Export Summary"] for cell in row]
assert any("20 - Bad" in cell for cell in flat)
# Keep successful sheets alongside the summary.
assert "10 - Good" in uploaded["sheets"]
def test_upload_failure_sends_failure_email_and_cleans_up(
mocks: dict[str, Any],
) -> None: