mirror of
https://github.com/apache/superset.git
synced 2026-09-09 16:54:29 +00:00
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
9dd0c3be2a | ||
|
|
70b52908b9 | ||
|
|
44869b6c96 | ||
|
|
d137fd1c06 | ||
|
|
ccb5fb279f | ||
|
|
3108f41167 | ||
|
|
f64b45f2c4 |
+26
-20
@@ -538,31 +538,37 @@ Note that a retried query returns partial data with no truncation indicator
|
||||
(e.g. a filter dropdown may list only a subset of values on tables above the
|
||||
row cap).
|
||||
|
||||
### Dashboard "Export Data to Excel" requires a Celery worker and S3 bucket
|
||||
### Dashboard Excel exports support direct downloads
|
||||
|
||||
A new dashboard action exports every chart's data to a single multi-sheet
|
||||
`.xlsx` asynchronously. It is disabled by default and turns on only when
|
||||
`EXCEL_EXPORT_S3_BUCKET` is set (the endpoint returns `501` otherwise). It also
|
||||
requires a running Celery worker and a configured SMTP transport, since the task
|
||||
emails the requesting user a pre-signed download link. New config keys:
|
||||
`EXCEL_EXPORT_S3_BUCKET`, `EXCEL_EXPORT_S3_KEY_PREFIX`,
|
||||
`.xlsx`. Without an export bucket, Superset builds the workbook during the
|
||||
request and returns it to the browser. When `EXCEL_EXPORT_S3_BUCKET` is set, a
|
||||
Celery worker builds and uploads the workbook, then emails the user a pre-signed
|
||||
download link. This queued path also needs a worker and SMTP transport.
|
||||
|
||||
Direct downloads are limited by `EXCEL_EXPORT_SYNC_MAX_ROWS` (default
|
||||
`100_000`), based on the combined `row_limit` of the planned queries. Superset
|
||||
uses `ROW_LIMIT` when a query omits its limit and returns `400` before querying
|
||||
if the total exceeds the limit. Image exports also return `400` without an export
|
||||
bucket because they require background webdriver rendering.
|
||||
|
||||
`POST /api/v1/dashboard/<id>/export_xlsx/` returns either `202` with a queued job
|
||||
id or `200` with the workbook. It no longer returns `501` when no bucket is set.
|
||||
|
||||
New config keys: `EXCEL_EXPORT_S3_BUCKET`, `EXCEL_EXPORT_S3_KEY_PREFIX`,
|
||||
`EXCEL_EXPORT_LINK_TTL_SECONDS`, `EXCEL_EXPORT_S3_CLIENT_KWARGS`,
|
||||
`EXCEL_EXPORT_TABLE_VIZ_TYPES`, and `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
|
||||
`EXCEL_EXPORT_SYNC_MAX_ROWS`, `EXCEL_EXPORT_TABLE_VIZ_TYPES`, and
|
||||
`EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
|
||||
|
||||
The feature depends on `boto3`, which is **not** installed by default; install it
|
||||
with `pip install apache-superset[excel-export]`.
|
||||
The queued path depends on `boto3`, which is **not** installed by default; install
|
||||
it with `pip install apache-superset[excel-export]`. The direct-download path
|
||||
does not use it.
|
||||
|
||||
Charts store their `query_context` only once they have been (re-)saved in
|
||||
Explore, so older charts may have none. For a fixed, conservative set of viz
|
||||
types (`table`, `big_number_total`, `big_number`, `pie`) the export rebuilds a
|
||||
query context from the chart's saved form data so those charts still export.
|
||||
The rebuild is a single-query mapping and does **not** reproduce plugin
|
||||
post-processing (pivot, rolling, forecast) or multi-query charts, so any chart of
|
||||
another type without a saved query context is skipped and listed in the email for
|
||||
the user to re-save. To cover those types, set `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`
|
||||
to a callable that receives the chart's form data and returns a query-context
|
||||
payload (or `None` to fall back to the built-in rebuild) — for example one backed
|
||||
by a service that runs the chart's real frontend `buildQuery`.
|
||||
For `table`, `big_number_total`, `big_number`, and `pie` charts without a saved
|
||||
`query_context`, Superset rebuilds a single query from saved form data. Charts
|
||||
that need post-processing or multiple queries are skipped and listed on the
|
||||
workbook's "Export Summary" sheet. Use `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` to
|
||||
support more chart types.
|
||||
|
||||
A second mode, **Export Images to Excel**, embeds non-table charts as rendered
|
||||
images (which viz types stay tabular is controlled by
|
||||
|
||||
@@ -9,22 +9,31 @@ version: 1
|
||||
|
||||
Superset can export every chart on a dashboard to a single Excel workbook, with
|
||||
each chart's underlying data rendered as its own worksheet. The export reflects
|
||||
the dashboard's currently applied filters and runs asynchronously: when it
|
||||
finishes, the requesting user receives an email with a time-limited download
|
||||
link.
|
||||
the dashboard's currently applied filters.
|
||||
|
||||
How the finished workbook reaches you depends on whether the deployment has
|
||||
export storage configured:
|
||||
|
||||
- **With an export bucket**, a background worker builds the workbook and emails
|
||||
the user a time-limited download link.
|
||||
- **Without an export bucket**, Superset builds the workbook during the request
|
||||
and returns it to the browser. This path only supports data exports within
|
||||
`EXCEL_EXPORT_SYNC_MAX_ROWS` (see [Prerequisites](#prerequisites)).
|
||||
|
||||
## Using the export
|
||||
|
||||
From a dashboard, open the **... (actions) → Download** submenu and choose
|
||||
**Export Data to Excel**. The action appears for users who have the dashboard
|
||||
`can_export` permission. You'll see a confirmation that the export is being
|
||||
prepared; the workbook arrives by email when it's ready.
|
||||
`can_export` permission. Where the export is queued you'll see a confirmation
|
||||
that it is being prepared, then receive the workbook by email. Direct exports
|
||||
download when ready.
|
||||
|
||||
A second option, **Export Images to Excel**, embeds each non-table chart as a
|
||||
rendered image (tables stay tabular) instead of exporting raw data. Because it
|
||||
renders charts through the headless webdriver, this option only appears when the
|
||||
webdriver screenshot feature flags are enabled (see the prerequisites below);
|
||||
which viz types stay tabular is controlled by `EXCEL_EXPORT_TABLE_VIZ_TYPES`.
|
||||
Image exports always run in the background and require an export bucket.
|
||||
|
||||
Notes on the generated workbook:
|
||||
|
||||
@@ -36,31 +45,37 @@ Notes on the generated workbook:
|
||||
re-saved in Explore) still exports when it is a `table`, `big_number`,
|
||||
`big_number_total` or `pie`, by rebuilding the query from the chart's saved
|
||||
form data. Charts of other types — and charts relying on post-processing the
|
||||
rebuild can't reproduce — are skipped and listed in the email; open the chart
|
||||
in Explore and re-save it to include it next time, or configure
|
||||
`EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
|
||||
rebuild can't reproduce — are skipped; open the chart in Explore and re-save it
|
||||
to include it next time, or configure `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`.
|
||||
The workbook's **Export Summary** sheet lists skipped charts. Queued exports
|
||||
also list them in the email.
|
||||
- Row counts per sheet are capped the same way as the chart-level CSV/Excel
|
||||
export (`ROW_LIMIT`, bounded by `SQL_MAX_ROW`), and never exceed Excel's
|
||||
per-sheet maximum.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
This feature is **disabled by default**. It requires:
|
||||
Dashboard data exports need no configuration. By default, Superset builds the
|
||||
workbook during the request and returns it to the browser. The combined
|
||||
`row_limit` of its queries must not exceed `EXCEL_EXPORT_SYNC_MAX_ROWS` (100,000
|
||||
by default). Queries without a `row_limit` use `ROW_LIMIT`. Keep this setting
|
||||
within your web server's request timeout.
|
||||
|
||||
For larger data exports and all image exports, configure the background path:
|
||||
|
||||
1. **The `boto3` dependency.** It is not installed by default; install it with
|
||||
`pip install apache-superset[excel-export]`. Without it, exports fail and the
|
||||
user receives a failure email.
|
||||
2. **An S3 bucket.** Set `EXCEL_EXPORT_S3_BUCKET`. Until it is set, the export
|
||||
endpoint returns `501` and the menu action surfaces a "not configured"
|
||||
message.
|
||||
3. **A running Celery worker.** The export runs as a Celery task. If no worker
|
||||
is running, the request is accepted but nothing is produced.
|
||||
2. **An S3 bucket.** Set `EXCEL_EXPORT_S3_BUCKET` to queue exports and deliver
|
||||
them by email.
|
||||
3. **A running Celery worker.** The queued export runs as a Celery task. If no
|
||||
worker is running, the request is accepted but nothing is produced.
|
||||
4. **A configured SMTP transport.** The download link is delivered by email
|
||||
using the same settings as alerts & reports (`SMTP_*`,
|
||||
`EMAIL_REPORTS_SUBJECT_PREFIX`).
|
||||
|
||||
**Export Images to Excel** additionally requires a working headless webdriver —
|
||||
the same infrastructure scheduled reports and thumbnails use (`WEBDRIVER_*`,
|
||||
**Export Images to Excel** also requires the headless webdriver used by scheduled
|
||||
reports and thumbnails (`WEBDRIVER_*`,
|
||||
plus the `ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS` and
|
||||
`ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT` feature flags). The menu option
|
||||
is hidden when those flags are off; if the webdriver is unreachable, image
|
||||
@@ -72,14 +87,15 @@ will not register.
|
||||
|
||||
## Configuration keys
|
||||
|
||||
| Key | Default | Description |
|
||||
| ------------------------------- | ---------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `EXCEL_EXPORT_S3_BUCKET` | `None` | Destination bucket. Required; `501` if unset. |
|
||||
| `EXCEL_EXPORT_S3_KEY_PREFIX` | `"dashboard-exports/"` | Key prefix: `{prefix}{dashboard_id}/{job_id}.xlsx`. |
|
||||
| `EXCEL_EXPORT_LINK_TTL_SECONDS` | `86400` | Lifetime of the pre-signed download URL (24h). |
|
||||
| `EXCEL_EXPORT_S3_CLIENT_KWARGS` | `{}` | Extra kwargs for `boto3.client("s3", ...)` — e.g. `region_name`, or `endpoint_url` for MinIO/LocalStack. |
|
||||
| `EXCEL_EXPORT_TABLE_VIZ_TYPES` | `None` | Viz types kept tabular in **Export Images to Excel** mode; every other type is embedded as an image. `None` uses the built-in default (`table`, `pivot_table`, `pivot_table_v2`). |
|
||||
| `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` | `None` | Optional `Callable[[form_data_dict], dict \| None]` to build a query context for a chart missing a saved one, tried before the built-in form-data rebuild. Point it at a service that runs the chart's real frontend `buildQuery` to faithfully export viz types the built-in rebuild can't handle. Must return `None` when it can't build faithfully, so the export falls back. |
|
||||
| Key | Default | Description |
|
||||
| ------------------------------------ | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| `EXCEL_EXPORT_S3_BUCKET` | `None` | Destination bucket. When unset, eligible data exports download directly. |
|
||||
| `EXCEL_EXPORT_SYNC_MAX_ROWS` | `100000` | Maximum combined query `row_limit` for a direct download. Ignored when a bucket is configured. |
|
||||
| `EXCEL_EXPORT_S3_KEY_PREFIX` | `"dashboard-exports/"` | Key prefix: `{prefix}{dashboard_id}/{job_id}.xlsx`. |
|
||||
| `EXCEL_EXPORT_LINK_TTL_SECONDS` | `86400` | Lifetime of the pre-signed download URL (24h). |
|
||||
| `EXCEL_EXPORT_S3_CLIENT_KWARGS` | `{}` | Extra arguments for `boto3.client("s3", ...)`, such as `region_name` or an `endpoint_url` for MinIO or LocalStack. |
|
||||
| `EXCEL_EXPORT_TABLE_VIZ_TYPES` | `None` | Viz types kept tabular in image mode. `None` uses `table`, `pivot_table`, and `pivot_table_v2`. |
|
||||
| `EXCEL_EXPORT_QUERY_CONTEXT_BUILDER` | `None` | Optional callable that builds a query context from `Slice.form_data`. Return `None` to use the built-in rebuild. |
|
||||
|
||||
Credentials and region resolve through the standard boto3 chain (environment
|
||||
variables, shared config, or instance role) unless overridden via
|
||||
@@ -88,8 +104,9 @@ variables, shared config, or instance role) unless overridden via
|
||||
## Security considerations
|
||||
|
||||
- The emailed link is a **pre-signed S3 URL**: anyone who holds it can download
|
||||
the workbook until it expires. Keep the bucket **private**, enable
|
||||
encryption, and consider a lifecycle rule to delete objects after a few days.
|
||||
the workbook until it expires. Direct downloads are not stored or linked.
|
||||
Keep the bucket **private**, enable encryption, and consider a lifecycle rule
|
||||
to delete objects after a few days.
|
||||
Lower `EXCEL_EXPORT_LINK_TTL_SECONDS` if 24 hours is too long for your data.
|
||||
- The export runs with the requesting user's permissions; each chart's query is
|
||||
access-checked, so users only ever receive data they are entitled to.
|
||||
@@ -97,9 +114,9 @@ variables, shared config, or instance role) unless overridden via
|
||||
## Limitations
|
||||
|
||||
- **Embedded dashboards / guest tokens are not supported** in this version,
|
||||
because guest users have no email address to deliver the link to. Logged-in
|
||||
users viewing an embedded dashboard can still use the export.
|
||||
including direct downloads. Logged-in users viewing an embedded dashboard can
|
||||
still use the export.
|
||||
- The default **Export Data to Excel** mode exports data only (no visual
|
||||
styling). Use **Export Images to Excel** to embed rendered chart images, which
|
||||
requires the webdriver infrastructure described in the prerequisites.
|
||||
requires an export bucket and the webdriver setup described above.
|
||||
- Scheduled/automated exports are not part of this feature.
|
||||
|
||||
Vendored
+12
-4
@@ -19521,7 +19521,7 @@
|
||||
},
|
||||
"/api/v1/dashboard/{pk}/export_xlsx/": {
|
||||
"post": {
|
||||
"description": "Enqueues an async task that writes each chart's data to its own worksheet, uploads the .xlsx to S3, and emails the requesting user a pre-signed download link. Returns immediately with a job id.",
|
||||
"description": "Writes each chart to a worksheet. With export storage, the work is queued and the user receives a download link by email. Without it, eligible workbooks are returned in the response.",
|
||||
"parameters": [
|
||||
{
|
||||
"description": "The dashboard id",
|
||||
@@ -19543,6 +19543,17 @@
|
||||
}
|
||||
},
|
||||
"responses": {
|
||||
"200": {
|
||||
"content": {
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet": {
|
||||
"schema": {
|
||||
"format": "binary",
|
||||
"type": "string"
|
||||
}
|
||||
}
|
||||
},
|
||||
"description": "The exported workbook, built during this request"
|
||||
},
|
||||
"202": {
|
||||
"content": {
|
||||
"application/json": {
|
||||
@@ -19567,9 +19578,6 @@
|
||||
},
|
||||
"500": {
|
||||
"$ref": "#/components/responses/500"
|
||||
},
|
||||
"501": {
|
||||
"description": "Excel export is not configured on this server"
|
||||
}
|
||||
},
|
||||
"security": [
|
||||
|
||||
+194
-16
@@ -141,11 +141,41 @@ test('Excel export items are hidden when userCanExport is false', () => {
|
||||
expect(screen.getByText('Export YAML')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
test('Export Data to Excel posts mode "data" and shows a pending toast', async () => {
|
||||
/** A queued export: 202 with a job id, delivered later by email. */
|
||||
const mockQueuedResponse = (
|
||||
body: Record<string, unknown> = { job_id: 'abc' },
|
||||
) =>
|
||||
mockSupersetClient.post.mockResolvedValue({
|
||||
json: { job_id: 'abc' },
|
||||
status: 202,
|
||||
json: jest.fn().mockResolvedValue(body),
|
||||
} as never);
|
||||
|
||||
/** An inline export: the workbook itself, as the response to the request. */
|
||||
const mockWorkbookResponse = (
|
||||
filename = 'World_Health_1.xlsx',
|
||||
): { blob: jest.Mock } => {
|
||||
const blob = jest.fn().mockResolvedValue(new Blob(['xlsx'])) as jest.Mock;
|
||||
mockSupersetClient.post.mockResolvedValue({
|
||||
status: 200,
|
||||
blob,
|
||||
headers: new Headers({
|
||||
'Content-Disposition': `attachment; filename=${filename}`,
|
||||
}),
|
||||
} as never);
|
||||
return { blob };
|
||||
};
|
||||
|
||||
/** jsdom implements neither, and the download path needs both. */
|
||||
const stubObjectUrls = (): { createObjectURL: jest.Mock } => {
|
||||
const createObjectURL = jest.fn(() => 'blob:http://localhost/fake');
|
||||
window.URL.createObjectURL = createObjectURL;
|
||||
window.URL.revokeObjectURL = jest.fn();
|
||||
return { createObjectURL };
|
||||
};
|
||||
|
||||
test('Export Data to Excel posts mode "data" and shows a pending toast', async () => {
|
||||
mockQueuedResponse();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
@@ -153,7 +183,9 @@ test('Export Data to Excel posts mode "data" and shows a pending toast', async (
|
||||
await waitFor(() => {
|
||||
expect(mockSupersetClient.post).toHaveBeenCalledWith({
|
||||
endpoint: '/api/v1/dashboard/123/export_xlsx/',
|
||||
fetchRetryOptions: { retries: 0 },
|
||||
jsonPayload: { active_data_mask: {}, mode: 'data' },
|
||||
parseMethod: 'raw',
|
||||
});
|
||||
expect(mockAddSuccessToast).toHaveBeenCalledWith(
|
||||
"Your export is being prepared. You'll receive an email when it's ready.",
|
||||
@@ -163,9 +195,7 @@ test('Export Data to Excel posts mode "data" and shows a pending toast', async (
|
||||
|
||||
test('Export Images to Excel posts mode "images" and shows a pending toast', async () => {
|
||||
enableWebDriverScreenshot();
|
||||
mockSupersetClient.post.mockResolvedValue({
|
||||
json: { job_id: 'abc' },
|
||||
} as never);
|
||||
mockQueuedResponse();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
@@ -174,7 +204,9 @@ test('Export Images to Excel posts mode "images" and shows a pending toast', asy
|
||||
await waitFor(() => {
|
||||
expect(mockSupersetClient.post).toHaveBeenCalledWith({
|
||||
endpoint: '/api/v1/dashboard/123/export_xlsx/',
|
||||
fetchRetryOptions: { retries: 0 },
|
||||
jsonPayload: { active_data_mask: {}, mode: 'images' },
|
||||
parseMethod: 'raw',
|
||||
});
|
||||
expect(mockAddSuccessToast).toHaveBeenCalledWith(
|
||||
"Your export is being prepared. You'll receive an email when it's ready.",
|
||||
@@ -182,13 +214,158 @@ test('Export Images to Excel posts mode "images" and shows a pending toast', asy
|
||||
});
|
||||
});
|
||||
|
||||
test('Export Data to Excel downloads the workbook when it arrives inline', async () => {
|
||||
// Without storage, the response contains the workbook.
|
||||
const { blob } = mockWorkbookResponse();
|
||||
const { createObjectURL } = stubObjectUrls();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(blob).toHaveBeenCalled();
|
||||
expect(createObjectURL).toHaveBeenCalled();
|
||||
expect(mockAddSuccessToast).toHaveBeenCalledWith(
|
||||
'Dashboard data exported to Excel',
|
||||
);
|
||||
});
|
||||
// Direct downloads do not show the queued-export message.
|
||||
expect(mockAddSuccessToast).not.toHaveBeenCalledWith(
|
||||
"Your export is being prepared. You'll receive an email when it's ready.",
|
||||
);
|
||||
});
|
||||
|
||||
test('Export Data to Excel names the downloaded file from the response', async () => {
|
||||
mockWorkbookResponse('Sales_Overview_7.xlsx');
|
||||
stubObjectUrls();
|
||||
// Capture download names without navigating in jsdom.
|
||||
const downloaded: string[] = [];
|
||||
const click = jest
|
||||
.spyOn(HTMLAnchorElement.prototype, 'click')
|
||||
.mockImplementation(function recordDownload(this: HTMLAnchorElement) {
|
||||
downloaded.push(this.download);
|
||||
});
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
|
||||
await waitFor(() => expect(downloaded).toEqual(['Sales_Overview_7.xlsx']));
|
||||
click.mockRestore();
|
||||
});
|
||||
|
||||
/** Mock a request that the test can settle later. */
|
||||
const mockPendingResponse = (): { settle: (response: unknown) => void } => {
|
||||
let settle: (response: unknown) => void = () => {};
|
||||
mockSupersetClient.post.mockReturnValue(
|
||||
new Promise(resolve => {
|
||||
settle = resolve;
|
||||
}) as never,
|
||||
);
|
||||
return { settle: response => settle(response) };
|
||||
};
|
||||
|
||||
const menuItemFor = (label: string) =>
|
||||
screen.getByText(label).closest('[role="menuitem"]');
|
||||
|
||||
test('Export Data to Excel reports progress while the export is running', async () => {
|
||||
// Keep the menu responsive while the request runs.
|
||||
const { settle } = mockPendingResponse();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
|
||||
});
|
||||
expect(screen.queryByText('Export Data to Excel')).not.toBeInTheDocument();
|
||||
expect(menuItemFor('Preparing export…')).toHaveAttribute(
|
||||
'aria-disabled',
|
||||
'true',
|
||||
);
|
||||
|
||||
settle({ status: 202, json: jest.fn().mockResolvedValue({ job_id: 'abc' }) });
|
||||
|
||||
// Re-enable the action when the request finishes.
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
|
||||
});
|
||||
expect(mockAddSuccessToast).toHaveBeenCalledWith(
|
||||
"Your export is being prepared. You'll receive an email when it's ready.",
|
||||
);
|
||||
});
|
||||
|
||||
test('Export Data to Excel is offered again once the download starts', async () => {
|
||||
const { settle } = mockPendingResponse();
|
||||
stubObjectUrls();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
settle({
|
||||
status: 200,
|
||||
blob: jest.fn().mockResolvedValue(new Blob(['xlsx'])),
|
||||
headers: new Headers({
|
||||
'Content-Disposition': 'attachment; filename=dash.xlsx',
|
||||
}),
|
||||
});
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
|
||||
});
|
||||
expect(mockAddSuccessToast).toHaveBeenCalledWith(
|
||||
'Dashboard data exported to Excel',
|
||||
);
|
||||
});
|
||||
|
||||
test('Export Data to Excel is offered again after a failure', async () => {
|
||||
const { settle } = mockPendingResponse();
|
||||
mockGetClientErrorObject.mockResolvedValue({ status: 500 });
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Preparing export…')).toBeInTheDocument();
|
||||
});
|
||||
|
||||
// Re-enable the action after an error.
|
||||
settle(Promise.reject(new Error('boom')));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(screen.getByText('Export Data to Excel')).toBeInTheDocument();
|
||||
});
|
||||
expect(mockAddDangerToast).toHaveBeenCalledWith(
|
||||
'Sorry, something went wrong. Try again later.',
|
||||
);
|
||||
});
|
||||
|
||||
test('Export Images to Excel is blocked while a data export is running', async () => {
|
||||
// The server allows one export per dashboard and user.
|
||||
enableWebDriverScreenshot();
|
||||
mockPendingResponse();
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(menuItemFor('Export Images to Excel')).toHaveAttribute(
|
||||
'aria-disabled',
|
||||
'true',
|
||||
);
|
||||
});
|
||||
await userEvent.click(screen.getByText('Export Images to Excel'));
|
||||
expect(mockSupersetClient.post).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
test('Export Data to Excel shows an "already in progress" toast when throttled', async () => {
|
||||
// The throttle response is 202 with a message but no job_id.
|
||||
mockSupersetClient.post.mockResolvedValue({
|
||||
json: {
|
||||
message: 'An Excel export for this dashboard is already in progress.',
|
||||
},
|
||||
} as never);
|
||||
mockQueuedResponse({
|
||||
message: 'An Excel export for this dashboard is already in progress.',
|
||||
});
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
@@ -201,18 +378,19 @@ test('Export Data to Excel shows an "already in progress" toast when throttled',
|
||||
});
|
||||
});
|
||||
|
||||
test('Export Data to Excel shows a config error toast on 501', async () => {
|
||||
mockSupersetClient.post.mockRejectedValue(new Error('not configured'));
|
||||
mockGetClientErrorObject.mockResolvedValue({ status: 501 });
|
||||
test('Export Data to Excel surfaces the reason an export was refused', async () => {
|
||||
// Show the server's actionable refusal.
|
||||
const message =
|
||||
'This dashboard requests too many rows to export in a single request.';
|
||||
mockSupersetClient.post.mockRejectedValue(new Error('too big'));
|
||||
mockGetClientErrorObject.mockResolvedValue({ status: 400, message });
|
||||
|
||||
render(<MenuWrapper />, { useRedux: true });
|
||||
|
||||
await userEvent.click(screen.getByText('Export Data to Excel'));
|
||||
|
||||
await waitFor(() => {
|
||||
expect(mockAddDangerToast).toHaveBeenCalledWith(
|
||||
'Excel export is not configured on this server.',
|
||||
);
|
||||
expect(mockAddDangerToast).toHaveBeenCalledWith(message);
|
||||
});
|
||||
});
|
||||
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
import { SyntheticEvent } from 'react';
|
||||
import { SyntheticEvent, useState } from 'react';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { logging } from '@apache-superset/core/utils';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
@@ -27,13 +27,15 @@ import {
|
||||
SupersetClient,
|
||||
} from '@superset-ui/core';
|
||||
import { MenuItem } from '@superset-ui/core/components/Menu';
|
||||
import { parse as parseContentDisposition } from 'content-disposition';
|
||||
import { useDownloadScreenshot } from 'src/dashboard/hooks/useDownloadScreenshot';
|
||||
import { NATIVE_FILTER_PREFIX } from 'src/dashboard/components/nativeFilters/FiltersConfigModal/utils';
|
||||
import { MenuKeys, RootState } from 'src/dashboard/types';
|
||||
import downloadAsPdf from 'src/utils/downloadAsPdf';
|
||||
import downloadAsImage from 'src/utils/downloadAsImage';
|
||||
import handleResourceExport from 'src/utils/export';
|
||||
import handleResourceExport, {
|
||||
downloadBlob,
|
||||
getFilenameFromResponse,
|
||||
} from 'src/utils/export';
|
||||
import {
|
||||
LOG_ACTIONS_DASHBOARD_DOWNLOAD_AS_PDF,
|
||||
LOG_ACTIONS_DASHBOARD_DOWNLOAD_AS_IMAGE,
|
||||
@@ -72,6 +74,10 @@ export const useDownloadMenuItems = (
|
||||
|
||||
const { addDangerToast, addSuccessToast } = useToasts();
|
||||
const dataMask = useSelector((state: RootState) => state.dataMask);
|
||||
// Disable both Excel actions while either export is running.
|
||||
const [exportingXlsx, setExportingXlsx] = useState<'data' | 'images' | null>(
|
||||
null,
|
||||
);
|
||||
const SCREENSHOT_NODE_SELECTOR = '.dashboard';
|
||||
|
||||
const buildActiveDataMask = (): Record<string, { extraFormData: object }> =>
|
||||
@@ -130,35 +136,14 @@ export const useDownloadMenuItems = (
|
||||
parseMethod: 'raw',
|
||||
});
|
||||
|
||||
// Parse filename from Content-Disposition header
|
||||
const disposition = response.headers.get('Content-Disposition');
|
||||
let fileName = `dashboard_${dashboardId}_example.zip`;
|
||||
|
||||
if (disposition) {
|
||||
try {
|
||||
const parsed = parseContentDisposition(disposition);
|
||||
if (parsed?.parameters?.filename) {
|
||||
fileName = parsed.parameters.filename;
|
||||
}
|
||||
} catch (error) {
|
||||
logging.warn('Failed to parse Content-Disposition header:', error);
|
||||
}
|
||||
}
|
||||
|
||||
// Convert response to blob and trigger download
|
||||
const blob = await response.blob();
|
||||
const url = window.URL.createObjectURL(blob);
|
||||
try {
|
||||
const a = document.createElement('a');
|
||||
a.href = url;
|
||||
a.download = fileName;
|
||||
a.style.display = 'none';
|
||||
document.body.appendChild(a);
|
||||
a.click();
|
||||
document.body.removeChild(a);
|
||||
} finally {
|
||||
window.URL.revokeObjectURL(url);
|
||||
}
|
||||
downloadBlob(
|
||||
blob,
|
||||
getFilenameFromResponse(
|
||||
response,
|
||||
`dashboard_${dashboardId}_example.zip`,
|
||||
),
|
||||
);
|
||||
|
||||
addSuccessToast(t('Dashboard exported as example successfully'));
|
||||
} catch (error) {
|
||||
@@ -168,14 +153,31 @@ export const useDownloadMenuItems = (
|
||||
};
|
||||
|
||||
const onExportXlsx = async (mode: 'data' | 'images') => {
|
||||
setExportingXlsx(mode);
|
||||
try {
|
||||
const { json } = await SupersetClient.post({
|
||||
const response = await SupersetClient.post({
|
||||
endpoint: `/api/v1/dashboard/${dashboardId}/export_xlsx/`,
|
||||
jsonPayload: { active_data_mask: buildActiveDataMask(), mode },
|
||||
// Parse the queued response or workbook after checking its status.
|
||||
parseMethod: 'raw',
|
||||
// A retry may hit the first request's lock and lose its file response.
|
||||
fetchRetryOptions: { retries: 0 },
|
||||
});
|
||||
// The throttle response (an export is already running) returns 202 with a
|
||||
// message but no job_id; only a freshly enqueued job carries a job_id.
|
||||
if ((json as { job_id?: string })?.job_id) {
|
||||
|
||||
// A 202 is queued; any successful non-202 response is the workbook.
|
||||
if (response.status !== 202) {
|
||||
const blob = await response.blob();
|
||||
downloadBlob(
|
||||
blob,
|
||||
getFilenameFromResponse(response, `dashboard_${dashboardId}.xlsx`),
|
||||
);
|
||||
addSuccessToast(t('Dashboard data exported to Excel'));
|
||||
return;
|
||||
}
|
||||
|
||||
// Only a newly queued export has a job id.
|
||||
const json = (await response.json()) as { job_id?: string };
|
||||
if (json?.job_id) {
|
||||
addSuccessToast(
|
||||
t(
|
||||
"Your export is being prepared. You'll receive an email when it's ready.",
|
||||
@@ -187,16 +189,20 @@ export const useDownloadMenuItems = (
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
// status comes from the response (Partial<SupersetClientResponse>), which
|
||||
// the union type does not expose uniformly; read it via a narrow cast.
|
||||
const { status } = (await getClientErrorObject(error)) as {
|
||||
// The client error union does not expose response fields uniformly.
|
||||
const { status, message } = (await getClientErrorObject(error)) as {
|
||||
status?: number;
|
||||
message?: string;
|
||||
};
|
||||
if (status === 501) {
|
||||
addDangerToast(t('Excel export is not configured on this server.'));
|
||||
// Show actionable client errors; keep server errors generic.
|
||||
if (message && status && status >= 400 && status < 500) {
|
||||
addDangerToast(message);
|
||||
} else {
|
||||
addDangerToast(t('Sorry, something went wrong. Try again later.'));
|
||||
}
|
||||
} finally {
|
||||
// Re-enable the actions after success or failure.
|
||||
setExportingXlsx(null);
|
||||
}
|
||||
};
|
||||
|
||||
@@ -244,23 +250,25 @@ export const useDownloadMenuItems = (
|
||||
},
|
||||
];
|
||||
|
||||
const xlsxExportLabel = (mode: 'data' | 'images', text: string) =>
|
||||
exportingXlsx === mode ? t('Preparing export…') : text;
|
||||
|
||||
const exportMenuItems: MenuItem[] = [
|
||||
...(userCanExport
|
||||
? [
|
||||
{
|
||||
key: 'export-xlsx',
|
||||
label: t('Export Data to Excel'),
|
||||
label: xlsxExportLabel('data', t('Export Data to Excel')),
|
||||
disabled: exportingXlsx !== null,
|
||||
onClick: () => onExportXlsx('data'),
|
||||
},
|
||||
// Image export renders charts through the headless webdriver, so only
|
||||
// offer it where that infrastructure is available (same signal as the
|
||||
// PDF/PNG image downloads above); otherwise non-table charts would
|
||||
// silently come back empty.
|
||||
// Image exports require the same webdriver flags as PDF and PNG.
|
||||
...(isWebDriverScreenshotEnabled
|
||||
? [
|
||||
{
|
||||
key: 'export-xlsx-images',
|
||||
label: t('Export Images to Excel'),
|
||||
label: xlsxExportLabel('images', t('Export Images to Excel')),
|
||||
disabled: exportingXlsx !== null,
|
||||
onClick: () => onExportXlsx('images'),
|
||||
},
|
||||
]
|
||||
|
||||
+11
-15
@@ -1539,10 +1539,10 @@ CSV_STREAMING_ROW_THRESHOLD = 100000
|
||||
EXCEL_EXPORT: dict[str, Any] = {}
|
||||
|
||||
# ---------------------------------------------------
|
||||
# Dashboard "Export Data to Excel" (async, S3-backed)
|
||||
# Dashboard "Export Data to Excel"
|
||||
# ---------------------------------------------------
|
||||
# Destination S3 bucket for generated dashboard .xlsx exports. The feature is
|
||||
# disabled until this is set: the export endpoint returns 501 when it is None.
|
||||
# When set, dashboard .xlsx exports run in the background and arrive by email.
|
||||
# Otherwise, eligible data exports are returned directly to the browser.
|
||||
EXCEL_EXPORT_S3_BUCKET: str | None = None
|
||||
# Key prefix for export objects: {prefix}{dashboard_id}/{job_id}.xlsx
|
||||
EXCEL_EXPORT_S3_KEY_PREFIX = "dashboard-exports/"
|
||||
@@ -1559,18 +1559,14 @@ EXCEL_EXPORT_S3_CLIENT_KWARGS: dict[str, Any] = {}
|
||||
# a rendered image. Set to None to fall back to the built-in default.
|
||||
EXCEL_EXPORT_TABLE_VIZ_TYPES: set[str] | None = None
|
||||
|
||||
# Optional hook to build a query context for a chart that has no saved
|
||||
# ``query_context``, called before the built-in form-data rebuild. Receives the
|
||||
# chart's form data (its ``params`` with ``viz_type`` and the
|
||||
# ``datasource="{id}__{type}"`` string injected — i.e. ``Slice.form_data``) and
|
||||
# returns a query-context payload dict (the shape ``ChartDataQueryContextSchema``
|
||||
# loads) or ``None``. A deployment can point this at a service that runs the
|
||||
# chart's real frontend ``buildQuery`` (faithful post-processing / multi-query)
|
||||
# for viz types the built-in rebuild can't handle. Must return ``None`` — not a
|
||||
# partial/stub context — whenever it cannot build the chart faithfully, so the
|
||||
# export falls through to the built-in rebuild. The export deep-copies whatever
|
||||
# it returns before applying dashboard filters, so a builder is free to memoize
|
||||
# or share its payloads. Defaults to ``None`` (built-in behavior only).
|
||||
# Maximum combined query ``row_limit`` for a direct download. Queries without a
|
||||
# limit use ``ROW_LIMIT``. Keep this within the request timeout.
|
||||
EXCEL_EXPORT_SYNC_MAX_ROWS = 100_000
|
||||
|
||||
# Optional query-context builder for charts without a saved ``query_context``.
|
||||
# It receives ``Slice.form_data`` and returns a payload accepted by
|
||||
# ``ChartDataQueryContextSchema``, or ``None`` to use the built-in rebuild.
|
||||
# Superset copies returned payloads before applying dashboard filters.
|
||||
EXCEL_EXPORT_QUERY_CONTEXT_BUILDER: (
|
||||
Callable[[dict[str, Any]], dict[str, Any] | None] | None
|
||||
) = None
|
||||
|
||||
+141
-27
@@ -17,6 +17,8 @@
|
||||
# pylint: disable=too-many-lines
|
||||
import functools
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
import uuid
|
||||
from datetime import datetime
|
||||
from io import BytesIO
|
||||
@@ -92,6 +94,17 @@ from superset.commands.importers.v1.utils import get_contents_from_bundle
|
||||
from superset.commands.purge import PurgeArchivedCommand, SoftDeleteBinding
|
||||
from superset.constants import MODEL_API_RW_METHOD_PERMISSION_MAP, RouteMethod
|
||||
from superset.daos.dashboard import DashboardDAO, EmbeddedDashboardDAO
|
||||
from superset.dashboards.excel_export.storage import is_export_storage_configured
|
||||
from superset.dashboards.excel_export.sync_budget import (
|
||||
InlineExportPlan,
|
||||
plan_inline_export,
|
||||
)
|
||||
from superset.dashboards.excel_export.workbook import (
|
||||
build_workbook,
|
||||
EXPORT_MODE_DATA,
|
||||
EXPORT_MODE_IMAGES,
|
||||
ResolvedQueryContexts,
|
||||
)
|
||||
from superset.dashboards.filter_scope import derive_json_metadata
|
||||
from superset.dashboards.filters import (
|
||||
DashboardAccessFilter,
|
||||
@@ -180,6 +193,7 @@ from superset.versioning.api_helpers import (
|
||||
)
|
||||
from superset.versioning.etag import set_version_etag
|
||||
from superset.versioning.schemas import VersionListItemSchema
|
||||
from superset.views.base import generate_download_headers, XlsxResponse
|
||||
from superset.views.base_api import (
|
||||
BaseSupersetModelRestApi,
|
||||
RelatedFieldFilter,
|
||||
@@ -1726,15 +1740,15 @@ class DashboardRestApi(
|
||||
action=lambda self, *args, **kwargs: f"{self.__class__.__name__}.export_xlsx",
|
||||
log_to_statsd=False,
|
||||
)
|
||||
def export_xlsx(self, pk: int) -> WerkzeugResponse:
|
||||
"""Export all of a dashboard's chart data to an Excel workbook (async).
|
||||
def export_xlsx(self, pk: int) -> WerkzeugResponse: # noqa: C901
|
||||
"""Export all of a dashboard's chart data to an Excel workbook.
|
||||
---
|
||||
post:
|
||||
summary: Export dashboard chart data to Excel
|
||||
description: >-
|
||||
Enqueues an async task that writes each chart's data to its own
|
||||
worksheet, uploads the .xlsx to S3, and emails the requesting user a
|
||||
pre-signed download link. Returns immediately with a job id.
|
||||
Writes each chart to a worksheet. With export storage, the work is
|
||||
queued and the user receives a download link by email. Without it,
|
||||
eligible workbooks are returned in the response.
|
||||
parameters:
|
||||
- in: path
|
||||
schema:
|
||||
@@ -1747,6 +1761,13 @@ class DashboardRestApi(
|
||||
schema:
|
||||
$ref: '#/components/schemas/DashboardExportXlsxPostSchema'
|
||||
responses:
|
||||
200:
|
||||
description: The exported workbook, built during this request
|
||||
content:
|
||||
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet:
|
||||
schema:
|
||||
type: string
|
||||
format: binary
|
||||
202:
|
||||
description: Export task accepted
|
||||
content:
|
||||
@@ -1763,13 +1784,10 @@ class DashboardRestApi(
|
||||
$ref: '#/components/responses/404'
|
||||
500:
|
||||
$ref: '#/components/responses/500'
|
||||
501:
|
||||
description: Excel export is not configured on this server
|
||||
"""
|
||||
if not current_app.config["EXCEL_EXPORT_S3_BUCKET"]:
|
||||
return self.response(
|
||||
501, message="Excel export is not configured on this server."
|
||||
)
|
||||
# Keep the request guards and two delivery paths together.
|
||||
# Resolve the path once so its checks and delivery cannot diverge.
|
||||
queued = is_export_storage_configured()
|
||||
try:
|
||||
# Tolerate an empty/non-JSON body (e.g. a POST with no Content-Type);
|
||||
# request.json would otherwise raise 415.
|
||||
@@ -1779,10 +1797,7 @@ class DashboardRestApi(
|
||||
except ValidationError as error:
|
||||
return self.response_400(message=error.messages)
|
||||
|
||||
# Image export drives the headless webdriver, so it is only available
|
||||
# when the same screenshot flags the UI checks are enabled. The decorator
|
||||
# form (``@validate_feature_flags``) can't be used here because it would
|
||||
# also block ``mode="data"``; mirror its 404 behavior inline instead.
|
||||
# Image exports require the screenshot feature flags; data exports do not.
|
||||
if payload.get("mode") == "images" and not (
|
||||
is_feature_enabled("ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS")
|
||||
and is_feature_enabled("ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT")
|
||||
@@ -1797,8 +1812,7 @@ class DashboardRestApi(
|
||||
except SupersetSecurityException:
|
||||
return self.response_403()
|
||||
|
||||
# Email delivery is the only result channel, so an account with an email
|
||||
# address is required; embedded guest users are excluded in this version.
|
||||
# Both delivery paths require a non-guest account with an email address.
|
||||
if isinstance(g.user, GuestUser) or not getattr(g.user, "email", None):
|
||||
return self.response_400(
|
||||
message="Excel export requires an account with an email address."
|
||||
@@ -1806,11 +1820,21 @@ class DashboardRestApi(
|
||||
if not dashboard.slices:
|
||||
return self.response_400(message="Dashboard has no charts to export.")
|
||||
|
||||
# Throttle: one concurrent export per user+dashboard. Acquire a shared,
|
||||
# atomic distributed lock (Redis when configured, the metadata DB
|
||||
# otherwise) so the guard works across the web server and workers and is
|
||||
# not a no-op under the default cache. The task releases it when it
|
||||
# settles; the TTL is the backstop if that release is ever lost.
|
||||
active_data_mask = payload.get("active_data_mask", {})
|
||||
mode = payload.get("mode", "data")
|
||||
|
||||
if not queued and mode == EXPORT_MODE_IMAGES:
|
||||
# Webdriver rendering is too slow and unbounded for a web request.
|
||||
return self.response_400(
|
||||
message=(
|
||||
"Exporting images to Excel runs in the background. "
|
||||
"Configure EXCEL_EXPORT_S3_BUCKET to use it, or export "
|
||||
"the dashboard's data instead."
|
||||
)
|
||||
)
|
||||
|
||||
# Allow one export per user and dashboard across web and worker processes.
|
||||
# The TTL releases the lock if normal cleanup fails.
|
||||
lock_params = export_lock_params(g.user.id, dashboard.id)
|
||||
try:
|
||||
AcquireDistributedLock(
|
||||
@@ -1825,25 +1849,115 @@ class DashboardRestApi(
|
||||
)
|
||||
|
||||
job_id = str(uuid.uuid4())
|
||||
if queued:
|
||||
return self._export_xlsx_queued(
|
||||
dashboard, active_data_mask, mode, job_id, lock_params
|
||||
)
|
||||
|
||||
# Plan after locking because query-context resolution can be expensive.
|
||||
# Release here unless the inline exporter takes over cleanup.
|
||||
lock_delegated = False
|
||||
try:
|
||||
plan: InlineExportPlan = plan_inline_export(dashboard)
|
||||
if not plan.fits_row_budget:
|
||||
return self.response_400(
|
||||
message=(
|
||||
"This dashboard requests too many rows to export in a "
|
||||
"single request. Configure EXCEL_EXPORT_S3_BUCKET to "
|
||||
"export it in the background, or lower the row limits of "
|
||||
"its charts."
|
||||
)
|
||||
)
|
||||
lock_delegated = True
|
||||
return self._export_xlsx_inline(
|
||||
dashboard,
|
||||
active_data_mask,
|
||||
job_id,
|
||||
lock_params,
|
||||
plan.query_contexts,
|
||||
)
|
||||
finally:
|
||||
if not lock_delegated:
|
||||
try:
|
||||
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
|
||||
except Exception: # pylint: disable=broad-except
|
||||
# The TTL is the fallback if release fails.
|
||||
logger.exception(
|
||||
"Failed to release in-flight export lock for dashboard %s",
|
||||
dashboard.id,
|
||||
)
|
||||
|
||||
def _export_xlsx_queued( # pylint: disable=too-many-arguments
|
||||
self,
|
||||
dashboard: Dashboard,
|
||||
active_data_mask: dict[str, Any],
|
||||
mode: str,
|
||||
job_id: str,
|
||||
lock_params: dict[str, int],
|
||||
) -> WerkzeugResponse:
|
||||
"""Queue an export for upload and email delivery."""
|
||||
try:
|
||||
export_dashboard_excel.apply_async(
|
||||
kwargs={
|
||||
"dashboard_id": dashboard.id,
|
||||
"user_id": g.user.id,
|
||||
"active_data_mask": payload.get("active_data_mask", {}),
|
||||
"active_data_mask": active_data_mask,
|
||||
"job_id": job_id,
|
||||
"mode": payload.get("mode", "data"),
|
||||
"mode": mode,
|
||||
},
|
||||
task_id=job_id,
|
||||
)
|
||||
except Exception:
|
||||
# If enqueuing fails (e.g. broker down) the task will never run to
|
||||
# release the lock, so free it now rather than block exports until
|
||||
# the TTL expires.
|
||||
# No task will release the lock if enqueueing fails.
|
||||
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
|
||||
raise
|
||||
return self.response(202, job_id=job_id)
|
||||
|
||||
@staticmethod
|
||||
def _export_xlsx_inline( # pylint: disable=too-many-arguments
|
||||
dashboard: Dashboard,
|
||||
active_data_mask: dict[str, Any],
|
||||
job_id: str,
|
||||
lock_params: dict[str, int],
|
||||
query_contexts: ResolvedQueryContexts,
|
||||
) -> WerkzeugResponse:
|
||||
"""Build a planned data export and return it in the response."""
|
||||
tmp_path: str | None = None
|
||||
try:
|
||||
file_descriptor, tmp_path = tempfile.mkstemp(
|
||||
suffix=".xlsx", prefix=f"dash-export-{job_id}-"
|
||||
)
|
||||
os.close(file_descriptor)
|
||||
|
||||
build_workbook(
|
||||
tmp_path,
|
||||
dashboard,
|
||||
active_data_mask,
|
||||
job_id,
|
||||
EXPORT_MODE_DATA,
|
||||
g.user,
|
||||
query_contexts=query_contexts,
|
||||
)
|
||||
with open(tmp_path, "rb") as workbook:
|
||||
content = workbook.read()
|
||||
finally:
|
||||
# Always release the lock and remove the temporary workbook.
|
||||
try:
|
||||
ReleaseDistributedLock(EXPORT_LOCK_NAMESPACE, lock_params).run()
|
||||
except Exception: # pylint: disable=broad-except
|
||||
# The TTL is the fallback if release fails.
|
||||
logger.exception(
|
||||
"Failed to release in-flight export lock for dashboard %s",
|
||||
dashboard.id,
|
||||
)
|
||||
if tmp_path and os.path.exists(tmp_path):
|
||||
os.remove(tmp_path)
|
||||
|
||||
filename = get_filename(dashboard.dashboard_title, dashboard.id, skip_id=False)
|
||||
return XlsxResponse(
|
||||
content, headers=generate_download_headers("xlsx", filename)
|
||||
)
|
||||
|
||||
def _validate_permalink_for_dashboard(
|
||||
self, permalink_key: str, dashboard: Dashboard
|
||||
) -> WerkzeugResponse | None:
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""Check whether dashboard Excel export storage is configured."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from flask import current_app
|
||||
|
||||
|
||||
def is_export_storage_configured() -> bool:
|
||||
"""Return whether exports can be uploaded and shared by link."""
|
||||
return bool(current_app.config["EXCEL_EXPORT_S3_BUCKET"])
|
||||
@@ -0,0 +1,85 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""Plan and size dashboard Excel exports served in the HTTP response."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass
|
||||
from typing import Any
|
||||
|
||||
from flask import current_app
|
||||
|
||||
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
|
||||
from superset.dashboards.excel_export.workbook import (
|
||||
resolve_query_context,
|
||||
ResolvedQueryContexts,
|
||||
)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class InlineExportPlan:
|
||||
"""Queries planned for a direct download and their row budget."""
|
||||
|
||||
#: Resolved query contexts by chart id. ``None`` marks a skipped chart.
|
||||
query_contexts: ResolvedQueryContexts
|
||||
#: Combined row limit, or ``None`` when any query has no finite limit.
|
||||
requested_rows: int | None
|
||||
#: Configured limit for direct downloads.
|
||||
max_rows: int
|
||||
|
||||
@property
|
||||
def fits_row_budget(self) -> bool:
|
||||
"""Return whether the export can run during the request."""
|
||||
return self.requested_rows is not None and self.requested_rows <= self.max_rows
|
||||
|
||||
|
||||
def _finite_row_limit(query: Any) -> int | None:
|
||||
"""Return the query's row limit, using ``ROW_LIMIT`` when omitted."""
|
||||
if not isinstance(query, dict):
|
||||
return None
|
||||
row_limit = query.get("row_limit") or current_app.config["ROW_LIMIT"]
|
||||
if isinstance(row_limit, bool) or not isinstance(row_limit, int):
|
||||
return None
|
||||
return row_limit if row_limit > 0 else None
|
||||
|
||||
|
||||
def _row_total(query_contexts: ResolvedQueryContexts) -> int | None:
|
||||
"""Rows every resolved query may return, or ``None`` if any is unbounded."""
|
||||
total = 0
|
||||
for query_context in query_contexts.values():
|
||||
if query_context is None:
|
||||
# Skipped charts do not add to the row budget.
|
||||
continue
|
||||
for query in query_context["queries"]:
|
||||
row_limit = _finite_row_limit(query)
|
||||
if row_limit is None:
|
||||
return None
|
||||
total += row_limit
|
||||
return total
|
||||
|
||||
|
||||
def plan_inline_export(dashboard: Any) -> InlineExportPlan:
|
||||
"""Resolve a dashboard's queries and calculate its direct-download size."""
|
||||
query_contexts: ResolvedQueryContexts = {
|
||||
chart.id: resolve_query_context(chart)
|
||||
for chart in get_charts_in_layout_order(dashboard)
|
||||
}
|
||||
return InlineExportPlan(
|
||||
query_contexts=query_contexts,
|
||||
requested_rows=_row_total(query_contexts),
|
||||
max_rows=current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"],
|
||||
)
|
||||
@@ -0,0 +1,302 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""Build dashboard Excel workbooks for direct and queued exports."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
from typing import Any
|
||||
|
||||
from celery.exceptions import SoftTimeLimitExceeded
|
||||
from flask import current_app, g
|
||||
|
||||
from superset.charts.data.dashboard_filter_context import (
|
||||
apply_dashboard_filter_context,
|
||||
get_dashboard_filter_context,
|
||||
)
|
||||
from superset.charts.schemas import ChartDataQueryContextSchema
|
||||
from superset.commands.chart.data.get_data_command import ChartDataCommand
|
||||
from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
|
||||
from superset.common.form_data_query_context import (
|
||||
build_query_context_from_form_data,
|
||||
is_raw_query_mode,
|
||||
)
|
||||
from superset.dashboards.excel_export import email
|
||||
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
|
||||
from superset.dashboards.excel_export.screenshot import render_chart_image
|
||||
from superset.utils import json
|
||||
from superset.utils.excel_streaming import StreamingXlsxWriter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Data mode writes chart results; image mode renders non-table charts.
|
||||
EXPORT_MODE_DATA = "data"
|
||||
EXPORT_MODE_IMAGES = "images"
|
||||
|
||||
# Viz types that remain tabular in image mode.
|
||||
TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"}
|
||||
|
||||
# Viz types that can be rebuilt as a single query without post-processing.
|
||||
REBUILD_VIZ_TYPES = {"table", "big_number_total", "big_number", "pie"}
|
||||
|
||||
#: Resolved query contexts by chart id. ``None`` marks a skipped chart.
|
||||
ResolvedQueryContexts = dict[int, dict[str, Any] | None]
|
||||
|
||||
|
||||
class ChartSkippedError(Exception):
|
||||
"""Raised when a chart should be listed as skipped."""
|
||||
|
||||
|
||||
def chart_label(chart: Any) -> str:
|
||||
"""Return the chart label used in the export summary."""
|
||||
return f"{chart.id} - {chart.slice_name or ''}".strip()
|
||||
|
||||
|
||||
def _usable_query_context(value: Any) -> dict[str, Any] | None:
|
||||
"""Return a query context with a non-empty ``queries`` list."""
|
||||
if not isinstance(value, dict) or not isinstance(value.get("queries"), list):
|
||||
return None
|
||||
return value if value["queries"] else None
|
||||
|
||||
|
||||
def _saved_query_context(raw: Any) -> dict[str, Any] | None:
|
||||
"""Parse a saved query context, returning ``None`` if it is unusable."""
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return _usable_query_context(parsed)
|
||||
|
||||
|
||||
# Form-data features the single-query rebuild cannot reproduce.
|
||||
_UNSUPPORTED_PROCESSING_KEYS = ("time_compare", "rolling_type", "resample_rule")
|
||||
|
||||
|
||||
def _needs_unsupported_processing(form_data: dict[str, Any]) -> bool:
|
||||
"""Return whether the chart needs unsupported post-processing."""
|
||||
# ``percent_metrics`` are produced by contribution post-processing.
|
||||
if form_data.get("percent_metrics"):
|
||||
return True
|
||||
# Aggregate table totals require a second query.
|
||||
if form_data.get("show_totals") and not is_raw_query_mode(form_data):
|
||||
return True
|
||||
for key in _UNSUPPORTED_PROCESSING_KEYS:
|
||||
value = form_data.get(key)
|
||||
# ``rolling_type`` may contain the string ``"None"`` when unset.
|
||||
if value and value != "None":
|
||||
return True
|
||||
return form_data.get("aggregation") == "raw"
|
||||
|
||||
|
||||
def resolve_query_context(chart: Any) -> dict[str, Any] | None:
|
||||
"""Resolve a saved, custom-built, or built-in query context for a chart."""
|
||||
if saved := _saved_query_context(chart.query_context):
|
||||
return saved
|
||||
|
||||
# Builder failures fall back to the built-in query-context rebuild.
|
||||
if builder := current_app.config.get("EXCEL_EXPORT_QUERY_CONTEXT_BUILDER"):
|
||||
try:
|
||||
built = builder(chart.form_data)
|
||||
except SoftTimeLimitExceeded:
|
||||
# A task timeout must stop the whole export.
|
||||
raise
|
||||
except Exception: # pylint: disable=broad-except
|
||||
logger.warning(
|
||||
"EXCEL_EXPORT_QUERY_CONTEXT_BUILDER failed for chart %s; "
|
||||
"falling back to the built-in rebuild",
|
||||
chart.id,
|
||||
exc_info=True,
|
||||
)
|
||||
built = None
|
||||
if (from_builder := _usable_query_context(built)) is not None:
|
||||
# Filters mutate nested queries, so do not modify a shared payload.
|
||||
return copy.deepcopy(from_builder)
|
||||
|
||||
# Only the built-in rebuild is limited to simple chart types.
|
||||
if chart.viz_type not in REBUILD_VIZ_TYPES or chart.datasource_id is None:
|
||||
return None
|
||||
try:
|
||||
form_data = json.loads(chart.params) if chart.params else {}
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not isinstance(form_data, dict) or not form_data:
|
||||
return None
|
||||
if _needs_unsupported_processing(form_data):
|
||||
return None
|
||||
|
||||
return build_query_context_from_form_data(
|
||||
form_data,
|
||||
{"id": chart.datasource_id, "type": chart.datasource_type or "table"},
|
||||
chart.viz_type,
|
||||
)
|
||||
|
||||
|
||||
def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]:
|
||||
return [record.get(col) for col in colnames]
|
||||
|
||||
|
||||
def _table_viz_types() -> set[str]:
|
||||
"""Viz types kept tabular in image mode (config override or built-in default)."""
|
||||
return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES
|
||||
|
||||
|
||||
def renders_as_image(chart: Any, mode: str) -> bool:
|
||||
"""Return whether the chart is exported as an image."""
|
||||
return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types()
|
||||
|
||||
|
||||
def _write_chart_image_sheet(
|
||||
writer: StreamingXlsxWriter,
|
||||
chart: Any,
|
||||
dashboard_id: int,
|
||||
active_data_mask: dict[str, Any],
|
||||
user: Any,
|
||||
) -> None:
|
||||
"""Render a chart into a worksheet.
|
||||
|
||||
:raises ChartSkippedError: if rendering fails
|
||||
"""
|
||||
image = render_chart_image(chart, dashboard_id, active_data_mask, user)
|
||||
if image is None:
|
||||
raise ChartSkippedError
|
||||
writer.add_image_sheet(chart_label(chart), image)
|
||||
|
||||
|
||||
def _write_chart_sheets(
|
||||
writer: StreamingXlsxWriter,
|
||||
chart: Any,
|
||||
json_body: dict[str, Any],
|
||||
dashboard_id: int,
|
||||
active_data_mask: dict[str, Any],
|
||||
) -> None:
|
||||
"""Run a chart's queries and write each result to a worksheet."""
|
||||
# Preserve the caller's top-level query-context values.
|
||||
json_body = dict(json_body)
|
||||
# Export full JSON results, regardless of saved values.
|
||||
json_body["result_format"] = ChartDataResultFormat.JSON
|
||||
json_body["result_type"] = ChartDataResultType.FULL
|
||||
json_body.pop("force", None)
|
||||
|
||||
filter_context = get_dashboard_filter_context(
|
||||
dashboard_id=dashboard_id,
|
||||
chart_id=chart.id,
|
||||
active_data_mask=active_data_mask,
|
||||
)
|
||||
if filter_context.extra_form_data:
|
||||
apply_dashboard_filter_context(json_body, filter_context.extra_form_data)
|
||||
|
||||
# Jinja macros read the query context from ``g.form_data``.
|
||||
g.form_data = json_body
|
||||
|
||||
query_context = ChartDataQueryContextSchema().load(json_body)
|
||||
command = ChartDataCommand(query_context)
|
||||
command.validate()
|
||||
result = command.run()
|
||||
|
||||
for index, query in enumerate(result["queries"]):
|
||||
colnames = query.get("colnames") or []
|
||||
data = query.get("data") or []
|
||||
if index == 0:
|
||||
name = f"{chart.id} - {chart.slice_name or ''}"
|
||||
else:
|
||||
name = f"{chart.id}.{index} - {chart.slice_name or ''}"
|
||||
writer.add_sheet(
|
||||
name,
|
||||
colnames,
|
||||
(_record_to_row(record, colnames) for record in data),
|
||||
)
|
||||
|
||||
|
||||
def build_workbook( # pylint: disable=too-many-arguments
|
||||
path: str,
|
||||
dashboard: Any,
|
||||
active_data_mask: dict[str, Any],
|
||||
job_id: str,
|
||||
mode: str,
|
||||
user: Any,
|
||||
query_contexts: ResolvedQueryContexts | None = None,
|
||||
) -> dict[str, list[str]]:
|
||||
"""Build the workbook on disk.
|
||||
|
||||
Return skipped charts grouped by ``email.ERROR_*`` reason.
|
||||
|
||||
:param path: Destination path for the ``.xlsx`` file
|
||||
:param dashboard: The dashboard whose charts to export
|
||||
:param active_data_mask: Live dashboard filter state keyed by native filter id
|
||||
:param job_id: Correlation id used in log lines
|
||||
:param mode: ``"data"`` or ``"images"``
|
||||
:param user: The requesting user (used to render images)
|
||||
:param query_contexts: Pre-resolved contexts. Missing charts are resolved here.
|
||||
"""
|
||||
errored: dict[str, list[str]] = {}
|
||||
resolved = query_contexts or {}
|
||||
writer = StreamingXlsxWriter(path)
|
||||
try:
|
||||
for chart in get_charts_in_layout_order(dashboard):
|
||||
label = chart_label(chart)
|
||||
try:
|
||||
if renders_as_image(chart, mode):
|
||||
# Image charts do not need a query context.
|
||||
_write_chart_image_sheet(
|
||||
writer, chart, dashboard.id, active_data_mask, user
|
||||
)
|
||||
else:
|
||||
# Reuse a planned context or resolve one here.
|
||||
json_body = (
|
||||
resolved[chart.id]
|
||||
if chart.id in resolved
|
||||
else resolve_query_context(chart)
|
||||
)
|
||||
if json_body is None:
|
||||
errored.setdefault(email.ERROR_NO_QUERY_CONTEXT, []).append(
|
||||
label
|
||||
)
|
||||
continue
|
||||
_write_chart_sheets(
|
||||
writer, chart, json_body, dashboard.id, active_data_mask
|
||||
)
|
||||
except SoftTimeLimitExceeded:
|
||||
# Let the task handler report the timeout and clean up.
|
||||
raise
|
||||
except ChartSkippedError:
|
||||
logger.warning(
|
||||
"Skipping chart %s in dashboard export %s (could not render)",
|
||||
chart.id,
|
||||
job_id,
|
||||
)
|
||||
errored.setdefault(email.ERROR_GENERAL, []).append(label)
|
||||
except Exception: # pylint: disable=broad-except
|
||||
logger.exception(
|
||||
"Skipping chart %s in dashboard export %s", chart.id, job_id
|
||||
)
|
||||
errored.setdefault(email.ERROR_GENERAL, []).append(label)
|
||||
|
||||
# Include skipped charts in the workbook for both delivery paths.
|
||||
if writer.sheet_count == 0 or errored:
|
||||
flat = [label for labels in errored.values() for label in labels]
|
||||
header = (
|
||||
"No chart data could be exported."
|
||||
if writer.sheet_count == 0
|
||||
else "Charts that could not be exported:"
|
||||
)
|
||||
writer.add_summary_sheet("Export Summary", [header, *flat])
|
||||
finally:
|
||||
writer.close()
|
||||
return errored
|
||||
@@ -14,22 +14,10 @@
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""
|
||||
Celery task that exports every chart on a dashboard to a single multi-sheet
|
||||
``.xlsx`` file, uploads it to S3, and emails the requesting user a pre-signed
|
||||
download link.
|
||||
|
||||
In ``"data"`` mode the task re-runs each chart's saved query context under the
|
||||
requesting user, applies the live dashboard filter state, and streams the results
|
||||
row-by-row into a constant-memory workbook so large dashboards never load all
|
||||
data at once. In ``"images"`` mode non-table charts are instead rendered to
|
||||
images (through the same headless path as scheduled reports, reflecting the live
|
||||
filters) and embedded, while table-like charts stay tabular.
|
||||
"""
|
||||
"""Build dashboard Excel exports in Celery and deliver them by email."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import copy
|
||||
import logging
|
||||
import os
|
||||
import tempfile
|
||||
@@ -37,58 +25,23 @@ from datetime import datetime, timedelta, timezone
|
||||
from typing import Any
|
||||
|
||||
from celery.exceptions import SoftTimeLimitExceeded
|
||||
from flask import current_app, g
|
||||
from flask import current_app
|
||||
|
||||
from superset import db, security_manager
|
||||
from superset.charts.data.dashboard_filter_context import (
|
||||
apply_dashboard_filter_context,
|
||||
get_dashboard_filter_context,
|
||||
)
|
||||
from superset.charts.schemas import ChartDataQueryContextSchema
|
||||
from superset.commands.chart.data.get_data_command import ChartDataCommand
|
||||
from superset.commands.distributed_lock.release import ReleaseDistributedLock
|
||||
from superset.common.chart_data import ChartDataResultFormat, ChartDataResultType
|
||||
from superset.common.form_data_query_context import (
|
||||
build_query_context_from_form_data,
|
||||
is_raw_query_mode,
|
||||
)
|
||||
from superset.dashboards.excel_export import email
|
||||
from superset.dashboards.excel_export.layout import get_charts_in_layout_order
|
||||
from superset.dashboards.excel_export.screenshot import render_chart_image
|
||||
from superset.dashboards.excel_export.workbook import build_workbook, EXPORT_MODE_DATA
|
||||
from superset.extensions import celery_app
|
||||
from superset.utils import json, s3
|
||||
from superset.utils import s3
|
||||
from superset.utils.core import override_user
|
||||
from superset.utils.excel_streaming import StreamingXlsxWriter
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
# Export modes: "data" streams every chart's tabular result (the default,
|
||||
# unchanged behavior); "images" embeds non-table charts as rendered images and
|
||||
# keeps only table-like charts tabular.
|
||||
EXPORT_MODE_DATA = "data"
|
||||
EXPORT_MODE_IMAGES = "images"
|
||||
|
||||
# Viz types kept as tabular data in image mode; everything else is rendered as an
|
||||
# image. Operators can override the set via ``EXCEL_EXPORT_TABLE_VIZ_TYPES``.
|
||||
TABLE_VIZ_TYPES = {"table", "pivot_table_v2", "pivot_table"}
|
||||
|
||||
# Viz types whose missing query context may be rebuilt from saved form data.
|
||||
# Conservative: only charts whose data maps faithfully to a single plain query
|
||||
# (no post-processing, no multi-query fan-out). Every other viz type without a
|
||||
# saved query context is skipped and listed for the user to re-save in Explore.
|
||||
REBUILD_VIZ_TYPES = {"table", "big_number_total", "big_number", "pie"}
|
||||
|
||||
EXPORT_SOFT_TIME_LIMIT = 600
|
||||
EXPORT_HARD_TIME_LIMIT = 660
|
||||
|
||||
# Namespace + TTL for the per-user+dashboard in-flight lock the API acquires
|
||||
# before enqueue and this task releases when it settles. The lock uses the
|
||||
# shared, atomic DistributedLock backend (Redis when configured, the metadata
|
||||
# DB otherwise) so it actually synchronizes across the web server and workers —
|
||||
# unlike a plain cache, which is a no-op under the default ``NullCache``.
|
||||
# The TTL outlives the hard time limit so a worker killed at that limit (which
|
||||
# skips the ``finally`` release) cannot hold the lock forever; the release in
|
||||
# ``finally`` is the fast path that frees it as soon as the task settles.
|
||||
# The API acquires this cross-process lock before either export path runs. The
|
||||
# task releases it, while the TTL covers hard worker shutdowns.
|
||||
EXPORT_LOCK_NAMESPACE = "excel_export"
|
||||
EXPORT_LOCK_TTL_SECONDS = EXPORT_HARD_TIME_LIMIT + 60
|
||||
|
||||
@@ -98,308 +51,6 @@ def export_lock_params(user_id: int, dashboard_id: int) -> dict[str, int]:
|
||||
return {"user_id": user_id, "dashboard_id": dashboard_id}
|
||||
|
||||
|
||||
class _ChartSkippedError(Exception):
|
||||
"""Signals a chart that could not be exported and should be listed as skipped."""
|
||||
|
||||
|
||||
def _chart_label(chart: Any) -> str:
|
||||
"""Human-readable label for a chart in the skipped-charts list."""
|
||||
return f"{chart.id} - {chart.slice_name or ''}".strip()
|
||||
|
||||
|
||||
def _usable_query_context(value: Any) -> dict[str, Any] | None:
|
||||
"""
|
||||
``value`` when it is a usable query-context payload, else ``None``.
|
||||
|
||||
A payload is usable only if it is a dict with a non-empty ``queries`` list; a
|
||||
blank, query-less, mistyped, or non-object value (e.g. ``{}``,
|
||||
``{"queries": []}``, ``{"queries": "oops"}``, ``None``) is treated the same as
|
||||
a missing context. Shared by the saved-context path and the builder hook so
|
||||
both apply the same validity rule — and so a malformed builder return falls
|
||||
through to the built-in rebuild instead of failing later in the general
|
||||
error bucket.
|
||||
"""
|
||||
if not isinstance(value, dict) or not isinstance(value.get("queries"), list):
|
||||
return None
|
||||
return value if value["queries"] else None
|
||||
|
||||
|
||||
def _saved_query_context(raw: Any) -> dict[str, Any] | None:
|
||||
"""
|
||||
The chart's saved query context parsed to a dict, or ``None`` when it is
|
||||
missing or unusable.
|
||||
|
||||
Returns ``None`` for a blank value, a string that does not parse as JSON, and
|
||||
any value that is not a dict with a non-empty ``queries`` list.
|
||||
"""
|
||||
if not raw:
|
||||
return None
|
||||
try:
|
||||
parsed = json.loads(raw)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
return _usable_query_context(parsed)
|
||||
|
||||
|
||||
# Form-data keys whose behavior needs plugin post-processing or extra queries
|
||||
# (contribution/time comparison, rolling window, resampling, raw big-number
|
||||
# aggregation) that the single-query rebuild cannot reproduce. A chart using any
|
||||
# of these is skipped rather than exported with values that differ from the chart.
|
||||
_UNSUPPORTED_PROCESSING_KEYS = ("time_compare", "rolling_type", "resample_rule")
|
||||
|
||||
|
||||
def _needs_unsupported_processing(form_data: dict[str, Any]) -> bool:
|
||||
"""Whether the form data relies on processing the rebuild can't reproduce."""
|
||||
# ``percent_metrics`` are "% of total" columns produced by contribution
|
||||
# post-processing the rebuild can't apply; skip so the export doesn't silently
|
||||
# omit columns the user sees.
|
||||
if form_data.get("percent_metrics"):
|
||||
return True
|
||||
# ``show_totals`` adds a totals row via a *second* query
|
||||
# (``plugin-chart-table/src/buildQuery.ts``, gated on aggregate mode); the
|
||||
# single-query rebuild would silently drop that row. The mode check mirrors
|
||||
# the frontend so a raw-mode table carrying a stale value still exports.
|
||||
if form_data.get("show_totals") and not is_raw_query_mode(form_data):
|
||||
return True
|
||||
for key in _UNSUPPORTED_PROCESSING_KEYS:
|
||||
value = form_data.get(key)
|
||||
# ``rolling_type`` is often the literal string ``"None"`` when unset.
|
||||
if value and value != "None":
|
||||
return True
|
||||
return form_data.get("aggregation") == "raw"
|
||||
|
||||
|
||||
def _resolve_query_context(chart: Any) -> dict[str, Any] | None:
|
||||
"""
|
||||
The query-context payload to run for a chart's data export, or ``None`` when
|
||||
none can be obtained.
|
||||
|
||||
Resolution order:
|
||||
|
||||
1. the chart's saved ``query_context``;
|
||||
2. an optional ``EXCEL_EXPORT_QUERY_CONTEXT_BUILDER`` hook, letting a deployment
|
||||
supply a faithful context (e.g. from a service running the chart's real
|
||||
frontend ``buildQuery``) for viz types the built-in rebuild can't handle;
|
||||
3. the built-in form-data rebuild, restricted to viz types whose data maps
|
||||
faithfully to a single plain query (``REBUILD_VIZ_TYPES``) without
|
||||
post-processing or extra queries.
|
||||
|
||||
Returns ``None`` when none apply, so the caller lists the chart for re-saving
|
||||
rather than exporting inaccurate data.
|
||||
"""
|
||||
if saved := _saved_query_context(chart.query_context):
|
||||
return saved
|
||||
|
||||
# The hook receives the chart's form data and must return ``None`` — not a
|
||||
# partial/stub context — whenever it can't build the chart faithfully, so we
|
||||
# fall through to the built-in rebuild (which handles the allowlisted viz types
|
||||
# well). A hook failure falls through too, preserving "builder problem →
|
||||
# rebuild, don't fail the export" — the one exception being a task-level
|
||||
# timeout, which has to abort the whole export rather than this chart.
|
||||
if builder := current_app.config.get("EXCEL_EXPORT_QUERY_CONTEXT_BUILDER"):
|
||||
try:
|
||||
built = builder(chart.form_data)
|
||||
except SoftTimeLimitExceeded:
|
||||
# A soft timeout is a task-level signal, not a builder failure: let it
|
||||
# propagate to _build_workbook so the export aborts cleanly instead of
|
||||
# continuing on to rebuild this chart and start the next one.
|
||||
raise
|
||||
except Exception: # pylint: disable=broad-except
|
||||
logger.warning(
|
||||
"EXCEL_EXPORT_QUERY_CONTEXT_BUILDER failed for chart %s; "
|
||||
"falling back to the built-in rebuild",
|
||||
chart.id,
|
||||
exc_info=True,
|
||||
)
|
||||
built = None
|
||||
if (from_builder := _usable_query_context(built)) is not None:
|
||||
# Copy: the payload's ``queries`` are mutated in place downstream (by
|
||||
# ``apply_dashboard_filter_context``), and a builder is free to
|
||||
# memoize or otherwise share its return value — which would then
|
||||
# accumulate filters across charts and across exports.
|
||||
return copy.deepcopy(from_builder)
|
||||
|
||||
# The allowlist and ``_needs_unsupported_processing`` bound only the built-in
|
||||
# rebuild below; the builder hook above is intentionally not gated by them (a
|
||||
# faithful builder includes the post-processing the built-in rebuild lacks).
|
||||
if chart.viz_type not in REBUILD_VIZ_TYPES or chart.datasource_id is None:
|
||||
return None
|
||||
try:
|
||||
form_data = json.loads(chart.params) if chart.params else {}
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
if not isinstance(form_data, dict) or not form_data:
|
||||
return None
|
||||
if _needs_unsupported_processing(form_data):
|
||||
return None
|
||||
|
||||
return build_query_context_from_form_data(
|
||||
form_data,
|
||||
{"id": chart.datasource_id, "type": chart.datasource_type or "table"},
|
||||
chart.viz_type,
|
||||
)
|
||||
|
||||
|
||||
def _record_to_row(record: dict[str, Any], colnames: list[str]) -> list[Any]:
|
||||
return [record.get(col) for col in colnames]
|
||||
|
||||
|
||||
def _table_viz_types() -> set[str]:
|
||||
"""Viz types kept tabular in image mode (config override or built-in default)."""
|
||||
return current_app.config.get("EXCEL_EXPORT_TABLE_VIZ_TYPES") or TABLE_VIZ_TYPES
|
||||
|
||||
|
||||
def _renders_as_image(chart: Any, mode: str) -> bool:
|
||||
"""Whether this chart is embedded as an image rather than streamed as data."""
|
||||
return mode == EXPORT_MODE_IMAGES and chart.viz_type not in _table_viz_types()
|
||||
|
||||
|
||||
def _write_chart_image_sheet(
|
||||
writer: StreamingXlsxWriter,
|
||||
chart: Any,
|
||||
dashboard_id: int,
|
||||
active_data_mask: dict[str, Any],
|
||||
user: Any,
|
||||
) -> None:
|
||||
"""
|
||||
Render a single chart to an image and embed it as its own sheet.
|
||||
|
||||
:raises _ChartSkippedError: if the chart could not be rendered
|
||||
"""
|
||||
image = render_chart_image(chart, dashboard_id, active_data_mask, user)
|
||||
if image is None:
|
||||
raise _ChartSkippedError
|
||||
writer.add_image_sheet(_chart_label(chart), image)
|
||||
|
||||
|
||||
def _write_chart_sheets(
|
||||
writer: StreamingXlsxWriter,
|
||||
chart: Any,
|
||||
json_body: dict[str, Any],
|
||||
dashboard_id: int,
|
||||
active_data_mask: dict[str, Any],
|
||||
) -> None:
|
||||
"""
|
||||
Run a single chart's query and stream its result(s) into the workbook.
|
||||
|
||||
``json_body`` is the resolved query-context payload (the chart's saved
|
||||
context or one synthesized from its form data). Charts may yield more than
|
||||
one query (e.g. mixed-series charts); each becomes its own sheet. Raises if
|
||||
the chart cannot be exported, so the caller can skip it and note it in the
|
||||
email.
|
||||
"""
|
||||
# Shallow-copy before setting our own top-level keys so the caller's payload
|
||||
# keeps its original result_format/result_type. (The nested ``queries`` are
|
||||
# mutated in place by apply_dashboard_filter_context below, which is safe
|
||||
# because every payload ``_resolve_query_context`` returns is this chart's
|
||||
# alone: freshly parsed, freshly built, or deep-copied from the builder hook.)
|
||||
json_body = dict(json_body)
|
||||
# Override any stale saved values: we always want full JSON results.
|
||||
json_body["result_format"] = ChartDataResultFormat.JSON
|
||||
json_body["result_type"] = ChartDataResultType.FULL
|
||||
json_body.pop("force", None)
|
||||
|
||||
filter_context = get_dashboard_filter_context(
|
||||
dashboard_id=dashboard_id,
|
||||
chart_id=chart.id,
|
||||
active_data_mask=active_data_mask,
|
||||
)
|
||||
if filter_context.extra_form_data:
|
||||
apply_dashboard_filter_context(json_body, filter_context.extra_form_data)
|
||||
|
||||
# Jinja macros resolve form data from g.form_data; expose the saved context.
|
||||
g.form_data = json_body
|
||||
|
||||
query_context = ChartDataQueryContextSchema().load(json_body)
|
||||
command = ChartDataCommand(query_context)
|
||||
command.validate()
|
||||
result = command.run()
|
||||
|
||||
for index, query in enumerate(result["queries"]):
|
||||
colnames = query.get("colnames") or []
|
||||
data = query.get("data") or []
|
||||
if index == 0:
|
||||
name = f"{chart.id} - {chart.slice_name or ''}"
|
||||
else:
|
||||
name = f"{chart.id}.{index} - {chart.slice_name or ''}"
|
||||
writer.add_sheet(
|
||||
name,
|
||||
colnames,
|
||||
(_record_to_row(record, colnames) for record in data),
|
||||
)
|
||||
|
||||
|
||||
def _build_workbook(
|
||||
path: str,
|
||||
dashboard: Any,
|
||||
active_data_mask: dict[str, Any],
|
||||
job_id: str,
|
||||
mode: str,
|
||||
user: Any,
|
||||
) -> dict[str, list[str]]:
|
||||
"""Build the workbook on disk.
|
||||
|
||||
Return the charts that could not be exported, grouped by the reason they
|
||||
were omitted (see the ``email.ERROR_*`` reason keys), so the notification
|
||||
can explain each group separately.
|
||||
"""
|
||||
errored: dict[str, list[str]] = {}
|
||||
writer = StreamingXlsxWriter(path)
|
||||
try:
|
||||
for chart in get_charts_in_layout_order(dashboard):
|
||||
label = _chart_label(chart)
|
||||
try:
|
||||
if _renders_as_image(chart, mode):
|
||||
# Image charts render from their saved params via the
|
||||
# webdriver and don't need a query context.
|
||||
_write_chart_image_sheet(
|
||||
writer, chart, dashboard.id, active_data_mask, user
|
||||
)
|
||||
else:
|
||||
# Data charts need a query context: use the saved one, or
|
||||
# rebuild it from form data for eligible viz types. Skip
|
||||
# cleanly when none is available rather than failing.
|
||||
json_body = _resolve_query_context(chart)
|
||||
if json_body is None:
|
||||
errored.setdefault(email.ERROR_NO_QUERY_CONTEXT, []).append(
|
||||
label
|
||||
)
|
||||
continue
|
||||
_write_chart_sheets(
|
||||
writer, chart, json_body, dashboard.id, active_data_mask
|
||||
)
|
||||
except SoftTimeLimitExceeded:
|
||||
# A soft timeout is a task-level signal, not a per-chart failure:
|
||||
# let it propagate so the outer handler emails a failure and runs
|
||||
# cleanup, rather than continuing until the hard limit kills the
|
||||
# worker (which would skip cleanup, leak temp files, and hold the
|
||||
# in-flight lock until its TTL). ``except Exception`` below would
|
||||
# otherwise swallow it, since it subclasses ``Exception``.
|
||||
raise
|
||||
except _ChartSkippedError:
|
||||
logger.warning(
|
||||
"Skipping chart %s in dashboard export %s (could not render)",
|
||||
chart.id,
|
||||
job_id,
|
||||
)
|
||||
errored.setdefault(email.ERROR_GENERAL, []).append(label)
|
||||
except Exception: # pylint: disable=broad-except
|
||||
logger.exception(
|
||||
"Skipping chart %s in dashboard export %s", chart.id, job_id
|
||||
)
|
||||
errored.setdefault(email.ERROR_GENERAL, []).append(label)
|
||||
|
||||
if writer.sheet_count == 0:
|
||||
flat = [label for labels in errored.values() for label in labels]
|
||||
writer.add_summary_sheet(
|
||||
"Export Summary",
|
||||
["No chart data could be exported.", *flat],
|
||||
)
|
||||
finally:
|
||||
writer.close()
|
||||
return errored
|
||||
|
||||
|
||||
def _send_failure_email(
|
||||
user: Any, dashboard_title: str, requested_at: datetime
|
||||
) -> None:
|
||||
@@ -462,7 +113,7 @@ def export_dashboard_excel(
|
||||
)
|
||||
os.close(file_descriptor)
|
||||
|
||||
errored = _build_workbook(
|
||||
errored = build_workbook(
|
||||
tmp_path, dashboard, active_data_mask, job_id, mode, user
|
||||
)
|
||||
|
||||
@@ -510,7 +161,7 @@ def export_dashboard_excel(
|
||||
export_lock_params(user_id, dashboard_id),
|
||||
).run()
|
||||
except Exception: # pylint: disable=broad-except
|
||||
# Best-effort: the lock's TTL is the backstop if this fails.
|
||||
# The TTL is the fallback if release fails.
|
||||
logger.exception(
|
||||
"Failed to release in-flight export lock for user %s dashboard %s",
|
||||
user_id,
|
||||
|
||||
@@ -28,11 +28,15 @@ import pytest
|
||||
import rison
|
||||
import yaml
|
||||
|
||||
from flask import current_app, g
|
||||
from freezegun import freeze_time
|
||||
from sqlalchemy import and_
|
||||
from superset import db, security_manager # noqa: F401
|
||||
from superset.commands.dashboard.permalink.create import CreateDashboardPermalinkCommand
|
||||
from superset.daos.dashboard import EmbeddedDashboardDAO
|
||||
from superset.dashboards.excel_export.sync_budget import InlineExportPlan
|
||||
from superset.exceptions import LockAlreadyHeldException
|
||||
from superset.security.guest_token import GuestTokenResourceType, GuestUser
|
||||
from superset.models.dashboard import Dashboard
|
||||
from superset.models.core import FavStar, FavStarClassName
|
||||
from superset.reports.models import ReportSchedule, ReportScheduleType
|
||||
@@ -3558,14 +3562,16 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
|
||||
response = json.loads(rv.data.decode("utf-8"))
|
||||
assert response["count"] > 0
|
||||
|
||||
def test_export_xlsx_501_when_bucket_unset(self):
|
||||
"""Dashboard API: export_xlsx returns 501 when the S3 bucket is unset."""
|
||||
def test_export_xlsx_400_for_empty_dashboard_without_storage(self):
|
||||
"""Dashboard API: with no storage configured the request is still validated
|
||||
before an export runs, so a dashboard with no charts is rejected rather
|
||||
than streaming an empty workbook."""
|
||||
admin = self.get_user("admin")
|
||||
dashboard = self.insert_dashboard("xlsx-501", None, [admin.id])
|
||||
dashboard = self.insert_dashboard("xlsx-sync-empty", None, [admin.id])
|
||||
self.login(ADMIN_USERNAME)
|
||||
try:
|
||||
rv = self.client.post(f"api/v1/dashboard/{dashboard.id}/export_xlsx/")
|
||||
assert rv.status_code == 501
|
||||
assert rv.status_code == 400
|
||||
finally:
|
||||
db.session.delete(dashboard)
|
||||
db.session.commit()
|
||||
@@ -3687,6 +3693,10 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": "exports"})
|
||||
@with_feature_flags(
|
||||
ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS=False,
|
||||
ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT=False,
|
||||
)
|
||||
@patch("superset.dashboards.api.export_dashboard_excel")
|
||||
def test_export_xlsx_images_404_when_screenshot_flags_off(self, mock_task):
|
||||
"""Dashboard API: ``mode=images`` is rejected with 404 when the webdriver
|
||||
@@ -3725,6 +3735,333 @@ class TestDashboardApi(ApiEditorsTestCaseMixin, InsertChartMixin, SupersetTestCa
|
||||
_, kwargs = mock_task.apply_async.call_args
|
||||
assert kwargs["kwargs"]["mode"] == "images"
|
||||
|
||||
# Direct download without export storage
|
||||
|
||||
@staticmethod
|
||||
def _write_stub_workbook(path, *args, **kwargs):
|
||||
"""Stand in for the shared workbook builder, writing a real .xlsx."""
|
||||
from superset.utils.excel_streaming import StreamingXlsxWriter
|
||||
|
||||
writer = StreamingXlsxWriter(path)
|
||||
writer.add_sheet("10 - Chart", ["a"], [[1]])
|
||||
writer.close()
|
||||
return {}
|
||||
|
||||
@staticmethod
|
||||
def _export_temp_files():
|
||||
"""Temp files the export path creates, so a leak can be detected."""
|
||||
import glob
|
||||
import os
|
||||
import tempfile
|
||||
|
||||
return glob.glob(os.path.join(tempfile.gettempdir(), "dash-export-*"))
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.export_dashboard_excel")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_200_streams_workbook_without_storage(
|
||||
self, mock_build, mock_task
|
||||
):
|
||||
"""Dashboard API: with no storage configured the workbook is built inline
|
||||
and returned as the response, instead of the request dead-ending."""
|
||||
mock_build.side_effect = self._write_stub_workbook
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 200
|
||||
assert rv.mimetype == (
|
||||
"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet"
|
||||
)
|
||||
assert "attachment" in rv.headers["Content-Disposition"]
|
||||
assert ".xlsx" in rv.headers["Content-Disposition"]
|
||||
# XLSX files are ZIP archives.
|
||||
assert rv.data.startswith(b"PK")
|
||||
assert is_zipfile(BytesIO(rv.data))
|
||||
# Direct downloads do not queue a task.
|
||||
mock_task.apply_async.assert_not_called()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_builds_with_the_same_inputs_as_the_task(self, mock_build):
|
||||
"""Dashboard API: the synchronous path hands the shared builder the same
|
||||
dashboard, filter state and mode the Celery task would, so both paths
|
||||
produce the same workbook."""
|
||||
mock_build.side_effect = self._write_stub_workbook
|
||||
data_mask = {"NATIVE_FILTER-abc": {"extraFormData": {"time_range": "No"}}}
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": data_mask},
|
||||
)
|
||||
|
||||
assert rv.status_code == 200
|
||||
args, _ = mock_build.call_args
|
||||
path, built_dashboard, active_data_mask, _job_id, mode, user = args
|
||||
assert path.endswith(".xlsx")
|
||||
assert built_dashboard.id == dashboard.id
|
||||
assert active_data_mask == data_mask
|
||||
assert mode == "data"
|
||||
assert user.username == ADMIN_USERNAME
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.ReleaseDistributedLock")
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
@patch("superset.dashboards.api.plan_inline_export")
|
||||
def test_export_xlsx_sync_refused_when_over_the_row_budget(
|
||||
self, mock_plan, mock_build, mock_acquire, mock_release
|
||||
):
|
||||
"""Dashboard API: an export too large to serve inline is refused up front
|
||||
with a message naming the fix, rather than being started and timing out."""
|
||||
mock_plan.return_value = InlineExportPlan(
|
||||
query_contexts={}, requested_rows=250_000, max_rows=100_000
|
||||
)
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 400
|
||||
message = rv.data.decode("utf-8")
|
||||
assert "EXCEL_EXPORT_S3_BUCKET" in message
|
||||
# A budget refusal releases the lock without querying charts.
|
||||
mock_plan.assert_called_once()
|
||||
mock_build.assert_not_called()
|
||||
mock_acquire.return_value.run.assert_called_once()
|
||||
mock_release.return_value.run.assert_called_once()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.ReleaseDistributedLock")
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.plan_inline_export")
|
||||
def test_export_xlsx_sync_releases_the_lock_when_planning_fails(
|
||||
self, mock_plan, mock_acquire, mock_release
|
||||
):
|
||||
"""Dashboard API: a context-builder failure while planning must not keep
|
||||
the user locked out until the lock's TTL expires."""
|
||||
mock_plan.side_effect = RuntimeError("builder failed")
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 500
|
||||
mock_acquire.return_value.run.assert_called_once()
|
||||
mock_release.return_value.run.assert_called_once()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
@patch("superset.dashboards.api.plan_inline_export")
|
||||
def test_export_xlsx_sync_runs_the_contexts_the_budget_measured(
|
||||
self, mock_plan, mock_build
|
||||
):
|
||||
"""Dashboard API: the export runs the query contexts the row budget was
|
||||
measured against. Resolving them a second time would risk vouching for one
|
||||
set of queries and running another, since a deployment's context builder
|
||||
need not be deterministic."""
|
||||
measured = {10: {"queries": [{"row_limit": 5}]}, 20: None}
|
||||
mock_plan.return_value = InlineExportPlan(
|
||||
query_contexts=measured, requested_rows=5, max_rows=100_000
|
||||
)
|
||||
mock_build.side_effect = self._write_stub_workbook
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 200
|
||||
assert mock_build.call_args.kwargs["query_contexts"] is measured
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@with_feature_flags(
|
||||
ENABLE_DASHBOARD_SCREENSHOT_ENDPOINTS=True,
|
||||
ENABLE_DASHBOARD_DOWNLOAD_WEBDRIVER_SCREENSHOT=True,
|
||||
)
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_images_refused_without_storage(self, mock_build, mock_acquire):
|
||||
"""Dashboard API: image export renders every chart through the headless
|
||||
webdriver, which no row budget bounds and no request should wait on, so it
|
||||
is refused rather than served inline -- even with the webdriver enabled."""
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}, "mode": "images"},
|
||||
)
|
||||
|
||||
assert rv.status_code == 400
|
||||
assert "EXCEL_EXPORT_S3_BUCKET" in rv.data.decode("utf-8")
|
||||
mock_build.assert_not_called()
|
||||
mock_acquire.return_value.run.assert_not_called()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.ReleaseDistributedLock")
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_releases_the_lock_on_success(
|
||||
self, mock_build, mock_acquire, mock_release
|
||||
):
|
||||
"""Dashboard API: the in-flight lock the synchronous path takes is released
|
||||
once the response is ready, so the next export is not locked out."""
|
||||
mock_build.side_effect = self._write_stub_workbook
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 200
|
||||
mock_acquire.return_value.run.assert_called_once()
|
||||
mock_release.return_value.run.assert_called_once()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.ReleaseDistributedLock")
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_releases_the_lock_when_building_fails(
|
||||
self, mock_build, mock_acquire, mock_release
|
||||
):
|
||||
"""Dashboard API: a failure while building must not leave the user locked
|
||||
out of their own dashboard until the lock's TTL expires."""
|
||||
mock_build.side_effect = RuntimeError("boom")
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 500
|
||||
mock_acquire.return_value.run.assert_called_once()
|
||||
mock_release.return_value.run.assert_called_once()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_deletes_the_temp_file_on_success(self, mock_build):
|
||||
"""Dashboard API: the workbook is built through a temp file, which must not
|
||||
outlive the response."""
|
||||
mock_build.side_effect = self._write_stub_workbook
|
||||
before = self._export_temp_files()
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 200
|
||||
assert self._export_temp_files() == before
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_deletes_the_temp_file_when_building_fails(
|
||||
self, mock_build
|
||||
):
|
||||
"""Dashboard API: a half-written workbook is cleaned up too, so a failing
|
||||
export does not fill the web server's disk."""
|
||||
mock_build.side_effect = RuntimeError("boom")
|
||||
before = self._export_temp_files()
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 500
|
||||
assert self._export_temp_files() == before
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@patch("superset.dashboards.api.AcquireDistributedLock")
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
@patch("superset.dashboards.api.plan_inline_export")
|
||||
def test_export_xlsx_sync_rejected_when_export_already_in_progress(
|
||||
self, mock_plan, mock_build, mock_acquire
|
||||
):
|
||||
"""Dashboard API: the synchronous path honors the same per-user+dashboard
|
||||
lock as the queued one, so one user cannot run two exports at once."""
|
||||
mock_acquire.return_value.run.side_effect = LockAlreadyHeldException("held")
|
||||
self.login(ADMIN_USERNAME)
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
|
||||
rv = self.client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
)
|
||||
|
||||
assert rv.status_code == 202
|
||||
assert "already in progress" in rv.data.decode("utf-8")
|
||||
mock_plan.assert_not_called()
|
||||
mock_build.assert_not_called()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
@with_config({"EXCEL_EXPORT_S3_BUCKET": None})
|
||||
@with_feature_flags(EMBEDDED_SUPERSET=True)
|
||||
@patch("superset.dashboards.api.build_workbook")
|
||||
def test_export_xlsx_sync_still_blocks_guest_sessions(self, mock_build):
|
||||
"""Dashboard API: the synchronous path does not become a way for an
|
||||
embedded guest session to export a dashboard. Guest support is deliberately
|
||||
out of scope here, so a guest holding a token that *does* grant access to
|
||||
this dashboard is still refused, before any workbook is built."""
|
||||
dashboard = db.session.query(Dashboard).filter_by(slug="world_health").first()
|
||||
embedded = EmbeddedDashboardDAO.upsert(dashboard, ["superset.example"])
|
||||
db.session.commit()
|
||||
token = security_manager.create_guest_access_token(
|
||||
{"username": "xlsx_guest"},
|
||||
[{"type": GuestTokenResourceType.DASHBOARD, "id": str(embedded.uuid)}],
|
||||
[],
|
||||
)
|
||||
|
||||
with self.client as client:
|
||||
rv = client.post(
|
||||
f"api/v1/dashboard/{dashboard.id}/export_xlsx/",
|
||||
json={"active_data_mask": {}},
|
||||
headers={
|
||||
current_app.config["GUEST_TOKEN_HEADER_NAME"]: token.decode("utf-8")
|
||||
if isinstance(token, bytes)
|
||||
else token
|
||||
},
|
||||
)
|
||||
assert isinstance(g.user, GuestUser)
|
||||
|
||||
assert rv.status_code == 400
|
||||
assert "email address" in rv.data.decode("utf-8")
|
||||
mock_build.assert_not_called()
|
||||
|
||||
@pytest.mark.usefixtures("load_world_bank_dashboard_with_slices")
|
||||
def test_embedded_dashboards(self):
|
||||
self.login(ADMIN_USERNAME)
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
from __future__ import annotations
|
||||
|
||||
import pytest
|
||||
|
||||
from superset.app import SupersetApp
|
||||
from superset.dashboards.excel_export.storage import is_export_storage_configured
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("bucket", "configured"),
|
||||
[
|
||||
("exports-bucket", True),
|
||||
(None, False),
|
||||
("", False),
|
||||
],
|
||||
)
|
||||
def test_storage_is_configured_only_with_a_bucket(
|
||||
app: SupersetApp, bucket: str | None, configured: bool
|
||||
) -> None:
|
||||
from flask import current_app
|
||||
|
||||
current_app.config["EXCEL_EXPORT_S3_BUCKET"] = bucket
|
||||
assert is_export_storage_configured() is configured
|
||||
@@ -0,0 +1,173 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
from __future__ import annotations
|
||||
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
from flask import current_app
|
||||
|
||||
from superset.dashboards.excel_export.sync_budget import plan_inline_export
|
||||
from superset.utils import json
|
||||
|
||||
MODULE = "superset.dashboards.excel_export.sync_budget"
|
||||
|
||||
|
||||
def _chart(chart_id: int, *queries: dict[str, Any]) -> mock.MagicMock:
|
||||
"""A chart whose saved query context holds ``queries``."""
|
||||
chart = mock.MagicMock()
|
||||
chart.id = chart_id
|
||||
chart.slice_name = f"Chart {chart_id}"
|
||||
chart.viz_type = "table"
|
||||
chart.query_context = json.dumps({"queries": list(queries)})
|
||||
return chart
|
||||
|
||||
|
||||
def _unexportable_chart(chart_id: int) -> mock.MagicMock:
|
||||
"""A chart the export has to skip: no context, and no rebuilding it."""
|
||||
chart = _chart(chart_id)
|
||||
chart.query_context = None
|
||||
chart.viz_type = "mixed_timeseries" # outside the rebuild allowlist
|
||||
return chart
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def charts() -> Iterator[mock.MagicMock]:
|
||||
"""Patch the layout walk so tests supply the dashboard's charts directly."""
|
||||
with mock.patch(f"{MODULE}.get_charts_in_layout_order") as ordered:
|
||||
yield ordered
|
||||
|
||||
|
||||
@pytest.fixture(autouse=True)
|
||||
def restore_config() -> Iterator[None]:
|
||||
"""Undo config edits: the app fixture is shared by every test in the module."""
|
||||
original_sync_max_rows = current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"]
|
||||
original_row_limit = current_app.config["ROW_LIMIT"]
|
||||
yield
|
||||
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = original_sync_max_rows
|
||||
current_app.config["ROW_LIMIT"] = original_row_limit
|
||||
|
||||
|
||||
def test_plan_sums_the_row_limit_of_every_chart(charts: mock.MagicMock) -> None:
|
||||
charts.return_value = [
|
||||
_chart(10, {"row_limit": 1000}),
|
||||
_chart(20, {"row_limit": 250}),
|
||||
]
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).requested_rows == 1250
|
||||
|
||||
|
||||
def test_plan_counts_every_query_of_a_multi_query_chart(
|
||||
charts: mock.MagicMock,
|
||||
) -> None:
|
||||
# Count every query in a multi-query chart.
|
||||
charts.return_value = [_chart(10, {"row_limit": 100}, {"row_limit": 400})]
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).requested_rows == 500
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
"query",
|
||||
[
|
||||
{"row_limit": "1000"}, # not an integer
|
||||
{"row_limit": -5},
|
||||
],
|
||||
)
|
||||
def test_plan_row_total_is_indeterminate_without_a_finite_row_limit(
|
||||
charts: mock.MagicMock, query: dict[str, Any]
|
||||
) -> None:
|
||||
# Every query needs a finite limit.
|
||||
charts.return_value = [_chart(10, {"row_limit": 100}), _chart(20, query)]
|
||||
|
||||
plan = plan_inline_export(mock.MagicMock())
|
||||
|
||||
assert plan.requested_rows is None
|
||||
assert plan.fits_row_budget is False
|
||||
|
||||
|
||||
@pytest.mark.parametrize("query", [{}, {"row_limit": 0}, {"row_limit": None}])
|
||||
def test_plan_uses_default_when_row_limit_is_omitted(
|
||||
charts: mock.MagicMock, query: dict[str, Any]
|
||||
) -> None:
|
||||
current_app.config["ROW_LIMIT"] = 250
|
||||
charts.return_value = [_chart(10, {"row_limit": 100}), _chart(20, query)]
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).requested_rows == 350
|
||||
|
||||
|
||||
def test_plan_ignores_charts_that_cannot_be_exported(charts: mock.MagicMock) -> None:
|
||||
# Skipped charts add no rows.
|
||||
charts.return_value = [_chart(10, {"row_limit": 100}), _unexportable_chart(20)]
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).requested_rows == 100
|
||||
|
||||
|
||||
@pytest.mark.parametrize(
|
||||
("row_limit", "fits"),
|
||||
[
|
||||
(99_999, True), # below the limit
|
||||
(100_000, True), # exactly at the limit
|
||||
(100_001, False), # above the limit
|
||||
],
|
||||
)
|
||||
def test_plan_fits_totals_up_to_and_including_the_limit(
|
||||
charts: mock.MagicMock, row_limit: int, fits: bool
|
||||
) -> None:
|
||||
charts.return_value = [_chart(10, {"row_limit": row_limit})]
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).fits_row_budget is fits
|
||||
|
||||
|
||||
def test_plan_honors_the_configured_limit(charts: mock.MagicMock) -> None:
|
||||
charts.return_value = [_chart(10, {"row_limit": 5_000})]
|
||||
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = 1_000
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).fits_row_budget is False
|
||||
|
||||
current_app.config["EXCEL_EXPORT_SYNC_MAX_ROWS"] = 10_000
|
||||
|
||||
assert plan_inline_export(mock.MagicMock()).fits_row_budget is True
|
||||
|
||||
|
||||
def test_plan_carries_the_resolved_context_of_every_chart(
|
||||
charts: mock.MagicMock,
|
||||
) -> None:
|
||||
# Return the contexts used to calculate the budget.
|
||||
exportable = _chart(10, {"row_limit": 100})
|
||||
charts.return_value = [exportable, _unexportable_chart(20)]
|
||||
|
||||
plan = plan_inline_export(mock.MagicMock())
|
||||
|
||||
assert plan.query_contexts[10] == {"queries": [{"row_limit": 100}]}
|
||||
# ``None`` marks a resolved chart that cannot be exported.
|
||||
assert 20 in plan.query_contexts
|
||||
assert plan.query_contexts[20] is None
|
||||
|
||||
|
||||
def test_plan_resolves_each_chart_exactly_once(charts: mock.MagicMock) -> None:
|
||||
# Resolve each chart once so planning and export use the same context.
|
||||
first = _chart(10, {"row_limit": 1})
|
||||
second = _chart(20, {"row_limit": 2})
|
||||
charts.return_value = [first, second]
|
||||
|
||||
with mock.patch(f"{MODULE}.resolve_query_context") as resolve:
|
||||
resolve.return_value = {"queries": [{"row_limit": 1}]}
|
||||
plan_inline_export(mock.MagicMock())
|
||||
|
||||
assert resolve.call_args_list == [mock.call(first), mock.call(second)]
|
||||
@@ -0,0 +1,125 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
"""Tests for passing already-resolved query contexts into the workbook builder.
|
||||
|
||||
The builder's own behavior (sheet naming, skipped charts, filter application) is
|
||||
covered through the Celery task in
|
||||
``tests/unit_tests/tasks/test_export_dashboard_excel.py``.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import tempfile
|
||||
from collections.abc import Iterator
|
||||
from typing import Any
|
||||
from unittest import mock
|
||||
|
||||
import pytest
|
||||
|
||||
from superset.dashboards.excel_export.workbook import build_workbook
|
||||
from superset.utils import json
|
||||
|
||||
MODULE = "superset.dashboards.excel_export.workbook"
|
||||
|
||||
|
||||
def _chart(chart_id: int, name: str) -> mock.MagicMock:
|
||||
chart = mock.MagicMock()
|
||||
chart.id = chart_id
|
||||
chart.slice_name = name
|
||||
chart.viz_type = "table"
|
||||
chart.query_context = json.dumps({"queries": [{"row_limit": 100}]})
|
||||
return chart
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def mocks() -> Iterator[dict[str, Any]]:
|
||||
"""Patch the builder's collaborators; keep the real xlsx writer."""
|
||||
with mock.patch.multiple(
|
||||
MODULE,
|
||||
get_charts_in_layout_order=mock.DEFAULT,
|
||||
get_dashboard_filter_context=mock.DEFAULT,
|
||||
ChartDataQueryContextSchema=mock.DEFAULT,
|
||||
ChartDataCommand=mock.DEFAULT,
|
||||
resolve_query_context=mock.DEFAULT,
|
||||
) as patched:
|
||||
patched["get_dashboard_filter_context"].return_value.extra_form_data = {}
|
||||
patched["ChartDataCommand"].return_value.run.return_value = {
|
||||
"queries": [{"colnames": ["a"], "data": [{"a": 1}]}]
|
||||
}
|
||||
yield patched
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def workbook_path() -> Iterator[str]:
|
||||
file_descriptor, path = tempfile.mkstemp(suffix=".xlsx")
|
||||
os.close(file_descriptor)
|
||||
yield path
|
||||
if os.path.exists(path):
|
||||
os.remove(path)
|
||||
|
||||
|
||||
def _build(path: str, **kwargs: Any) -> Any:
|
||||
dashboard = mock.MagicMock()
|
||||
dashboard.id = 1
|
||||
return build_workbook(
|
||||
path, dashboard, {}, "job-1", "data", mock.MagicMock(), **kwargs
|
||||
)
|
||||
|
||||
|
||||
def test_provided_query_context_is_used_without_resolving_again(
|
||||
mocks: dict[str, Any], workbook_path: str
|
||||
) -> None:
|
||||
# Use the context measured by the row budget.
|
||||
chart = _chart(10, "First")
|
||||
mocks["get_charts_in_layout_order"].return_value = [chart]
|
||||
provided = {"queries": [{"row_limit": 7, "metrics": ["count"]}]}
|
||||
|
||||
_build(workbook_path, query_contexts={10: provided})
|
||||
|
||||
mocks["resolve_query_context"].assert_not_called()
|
||||
loaded = mocks["ChartDataQueryContextSchema"].return_value.load.call_args.args[0]
|
||||
assert loaded["queries"] == provided["queries"]
|
||||
|
||||
|
||||
def test_a_chart_resolved_to_none_is_skipped_without_resolving_again(
|
||||
mocks: dict[str, Any], workbook_path: str
|
||||
) -> None:
|
||||
# ``None`` marks a chart already resolved as unexportable.
|
||||
chart = _chart(20, "Skipped")
|
||||
mocks["get_charts_in_layout_order"].return_value = [chart]
|
||||
|
||||
errored = _build(workbook_path, query_contexts={20: None})
|
||||
|
||||
mocks["resolve_query_context"].assert_not_called()
|
||||
mocks["ChartDataCommand"].return_value.run.assert_not_called()
|
||||
assert [label for labels in errored.values() for label in labels] == [
|
||||
"20 - Skipped"
|
||||
]
|
||||
|
||||
|
||||
def test_a_chart_missing_from_the_map_is_resolved_by_the_builder(
|
||||
mocks: dict[str, Any], workbook_path: str
|
||||
) -> None:
|
||||
# Resolve charts missing from a partial context map.
|
||||
chart = _chart(30, "Unmapped")
|
||||
mocks["get_charts_in_layout_order"].return_value = [chart]
|
||||
mocks["resolve_query_context"].return_value = {"queries": [{"row_limit": 5}]}
|
||||
|
||||
_build(workbook_path, query_contexts={})
|
||||
|
||||
mocks["resolve_query_context"].assert_called_once_with(chart)
|
||||
@@ -30,6 +30,8 @@ from celery.exceptions import SoftTimeLimitExceeded
|
||||
from superset.utils import json
|
||||
|
||||
MODULE = "superset.tasks.export_dashboard_excel"
|
||||
# Workbook building is shared by queued and direct exports.
|
||||
WORKBOOK_MODULE = "superset.dashboards.excel_export.workbook"
|
||||
|
||||
|
||||
# A minimal valid 1x1 transparent PNG for image-mode tests.
|
||||
@@ -71,21 +73,25 @@ def mocks() -> Iterator[dict[str, Any]]:
|
||||
# would make calls like security_manager.get_user_by_id() return coroutines.
|
||||
patched = {
|
||||
name: stack.enter_context(
|
||||
mock.patch(f"{MODULE}.{name}", new=mock.MagicMock())
|
||||
mock.patch(f"{module}.{name}", new=mock.MagicMock())
|
||||
)
|
||||
for name in (
|
||||
"security_manager",
|
||||
"db",
|
||||
"get_charts_in_layout_order",
|
||||
"get_dashboard_filter_context",
|
||||
"ChartDataQueryContextSchema",
|
||||
"ChartDataCommand",
|
||||
"render_chart_image",
|
||||
"s3",
|
||||
"email",
|
||||
"ReleaseDistributedLock",
|
||||
for module, name in (
|
||||
(MODULE, "security_manager"),
|
||||
(MODULE, "db"),
|
||||
(MODULE, "s3"),
|
||||
(MODULE, "ReleaseDistributedLock"),
|
||||
(WORKBOOK_MODULE, "get_charts_in_layout_order"),
|
||||
(WORKBOOK_MODULE, "get_dashboard_filter_context"),
|
||||
(WORKBOOK_MODULE, "ChartDataQueryContextSchema"),
|
||||
(WORKBOOK_MODULE, "ChartDataCommand"),
|
||||
(WORKBOOK_MODULE, "render_chart_image"),
|
||||
)
|
||||
}
|
||||
# Both modules must use the same mocked error keys.
|
||||
shared_email = mock.MagicMock()
|
||||
for module in (MODULE, WORKBOOK_MODULE):
|
||||
stack.enter_context(mock.patch(f"{module}.email", new=shared_email))
|
||||
patched["email"] = shared_email
|
||||
user = mock.MagicMock()
|
||||
user.email = "user@example.com"
|
||||
patched["security_manager"].get_user_by_id.return_value = user
|
||||
@@ -287,8 +293,6 @@ def _rebuildable_chart(
|
||||
@contextmanager
|
||||
def _builder_hook(builder: Any) -> Iterator[None]:
|
||||
"""Patch current_app so EXCEL_EXPORT_QUERY_CONTEXT_BUILDER resolves to builder."""
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
|
||||
fake_app = mock.MagicMock()
|
||||
fake_app.config.get.side_effect = lambda key, default=None: (
|
||||
builder if key == "EXCEL_EXPORT_QUERY_CONTEXT_BUILDER" else default
|
||||
@@ -300,14 +304,17 @@ def _builder_hook(builder: Any) -> Iterator[None]:
|
||||
"EXCEL_EXPORT_S3_KEY_PREFIX": "dashboard-exports/",
|
||||
"EXCEL_EXPORT_LINK_TTL_SECONDS": 3600,
|
||||
}.__getitem__
|
||||
with mock.patch.object(module, "current_app", fake_app):
|
||||
# Both the task and workbook read configuration.
|
||||
with ExitStack() as stack:
|
||||
for module in (MODULE, WORKBOOK_MODULE):
|
||||
stack.enter_context(mock.patch(f"{module}.current_app", fake_app))
|
||||
yield
|
||||
|
||||
|
||||
def test_builder_hook_context_is_used_for_any_viz_type() -> None:
|
||||
# A configured builder can supply a context for a viz type outside the
|
||||
# built-in allowlist (pivot_table_v2), and is called with the chart's form data.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
ctx = {
|
||||
"datasource": {"id": 5, "type": "table"},
|
||||
@@ -317,7 +324,7 @@ def test_builder_hook_context_is_used_for_any_viz_type() -> None:
|
||||
chart = _rebuildable_chart(viz_type="pivot_table_v2")
|
||||
|
||||
with _builder_hook(builder):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
assert result == ctx
|
||||
builder.assert_called_once_with(chart.form_data)
|
||||
@@ -326,13 +333,13 @@ def test_builder_hook_context_is_used_for_any_viz_type() -> None:
|
||||
def test_builder_hook_none_falls_through_to_builtin_rebuild() -> None:
|
||||
# When the builder returns None (can't build faithfully) the export falls
|
||||
# through to the built-in rebuild, so an allowlisted table is unaffected.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
builder = mock.MagicMock(return_value=None)
|
||||
chart = _rebuildable_chart(viz_type="table")
|
||||
|
||||
with _builder_hook(builder):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
builder.assert_called_once_with(chart.form_data)
|
||||
assert result is not None
|
||||
@@ -354,13 +361,13 @@ def test_builder_hook_none_falls_through_to_builtin_rebuild() -> None:
|
||||
def test_builder_hook_malformed_result_falls_through(built: Any) -> None:
|
||||
# A stub / empty / malformed builder result is treated as "not built" and
|
||||
# falls through to the built-in rebuild rather than shipping an empty context.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
builder = mock.MagicMock(return_value=built)
|
||||
chart = _rebuildable_chart(viz_type="table")
|
||||
|
||||
with _builder_hook(builder):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
builder.assert_called_once_with(chart.form_data)
|
||||
assert result is not None
|
||||
@@ -370,13 +377,13 @@ def test_builder_hook_malformed_result_falls_through(built: Any) -> None:
|
||||
def test_builder_hook_exception_falls_through() -> None:
|
||||
# A raising builder (e.g. sidecar down) must not fail the chart; the export
|
||||
# falls through to the built-in rebuild and no exception escapes.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
builder = mock.MagicMock(side_effect=RuntimeError("sidecar down"))
|
||||
chart = _rebuildable_chart(viz_type="table")
|
||||
|
||||
with _builder_hook(builder):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
builder.assert_called_once_with(chart.form_data)
|
||||
assert result is not None
|
||||
@@ -387,13 +394,13 @@ def test_builder_hook_soft_time_limit_propagates() -> None:
|
||||
# A soft timeout raised while the builder is in flight is a task-level signal,
|
||||
# not a builder failure: it must escape _resolve_query_context so the export
|
||||
# aborts cleanly, rather than being swallowed by the broad fall-through guard.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
builder = mock.MagicMock(side_effect=SoftTimeLimitExceeded())
|
||||
chart = _rebuildable_chart(viz_type="table")
|
||||
|
||||
with _builder_hook(builder), pytest.raises(SoftTimeLimitExceeded):
|
||||
module._resolve_query_context(chart)
|
||||
module.resolve_query_context(chart)
|
||||
|
||||
builder.assert_called_once_with(chart.form_data)
|
||||
|
||||
@@ -401,11 +408,11 @@ def test_builder_hook_soft_time_limit_propagates() -> None:
|
||||
def test_no_builder_hook_leaves_builtin_behavior_unchanged() -> None:
|
||||
# With no builder configured, an allowlisted chart is rebuilt and an
|
||||
# ineligible one is skipped — identical to the pre-hook behavior.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
with _builder_hook(None):
|
||||
table = module._resolve_query_context(_rebuildable_chart(viz_type="table"))
|
||||
ineligible = module._resolve_query_context(
|
||||
table = module.resolve_query_context(_rebuildable_chart(viz_type="table"))
|
||||
ineligible = module.resolve_query_context(
|
||||
_rebuildable_chart(viz_type="mixed_timeseries")
|
||||
)
|
||||
|
||||
@@ -416,14 +423,14 @@ def test_no_builder_hook_leaves_builtin_behavior_unchanged() -> None:
|
||||
|
||||
def test_saved_context_short_circuits_builder_hook() -> None:
|
||||
# A saved query context wins over the builder hook, which is never called.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
builder = mock.MagicMock(return_value={"queries": [{"from": "hook"}]})
|
||||
chart = _rebuildable_chart(viz_type="table")
|
||||
chart.query_context = json.dumps({"queries": [{"from": "saved"}]})
|
||||
|
||||
with _builder_hook(builder):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
assert result == {"queries": [{"from": "saved"}]}
|
||||
builder.assert_not_called()
|
||||
@@ -595,7 +602,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
|
||||
# gates the totals query on queryMode === Aggregate), and the control isn't
|
||||
# reset when hidden. A raw-mode table carrying a stale value must still
|
||||
# rebuild rather than be needlessly skipped.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
chart = _rebuildable_chart(
|
||||
viz_type="table",
|
||||
@@ -603,7 +610,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
|
||||
)
|
||||
|
||||
with _builder_hook(None):
|
||||
result = module._resolve_query_context(chart)
|
||||
result = module.resolve_query_context(chart)
|
||||
|
||||
assert result is not None
|
||||
assert result["queries"][0]["columns"] == ["a"]
|
||||
@@ -612,7 +619,7 @@ def test_raw_mode_table_ignores_stale_show_totals() -> None:
|
||||
def test_rebuild_viz_types_is_the_conservative_default() -> None:
|
||||
# The rebuild allow-list is a fixed fallback (no config override): only viz
|
||||
# types whose data maps faithfully to a single plain query.
|
||||
from superset.tasks import export_dashboard_excel as module
|
||||
from superset.dashboards.excel_export import workbook as module
|
||||
|
||||
assert module.REBUILD_VIZ_TYPES == {
|
||||
"table",
|
||||
@@ -699,6 +706,34 @@ def test_all_charts_skipped_writes_summary(mocks: dict[str, Any]) -> None:
|
||||
mocks["email"].build_success_email.assert_called_once()
|
||||
|
||||
|
||||
def test_partial_failure_appends_summary_sheet(mocks: dict[str, Any]) -> None:
|
||||
"""When some charts export and others are skipped, the workbook itself lists
|
||||
the skipped charts: a download served as the response to the request has no
|
||||
email to list them in."""
|
||||
mocks["get_charts_in_layout_order"].return_value = [
|
||||
_chart(10, "Good"),
|
||||
_chart(20, "Bad", has_context=False, viz_type="sunburst"),
|
||||
]
|
||||
mocks["ChartDataCommand"].return_value.run.return_value = {
|
||||
"queries": [{"colnames": ["a"], "data": [{"a": 1}]}]
|
||||
}
|
||||
|
||||
uploaded: dict[str, Any] = {}
|
||||
|
||||
def _capture(path: str, bucket: str, key: str) -> None:
|
||||
uploaded["sheets"] = _read_sheets(path)
|
||||
|
||||
mocks["s3"].upload_file_to_s3.side_effect = _capture
|
||||
|
||||
_run()
|
||||
|
||||
assert "Export Summary" in uploaded["sheets"]
|
||||
flat = [str(cell) for row in uploaded["sheets"]["Export Summary"] for cell in row]
|
||||
assert any("20 - Bad" in cell for cell in flat)
|
||||
# Keep successful sheets alongside the summary.
|
||||
assert "10 - Good" in uploaded["sheets"]
|
||||
|
||||
|
||||
def test_upload_failure_sends_failure_email_and_cleans_up(
|
||||
mocks: dict[str, Any],
|
||||
) -> None:
|
||||
|
||||
Reference in New Issue
Block a user