mirror of
https://github.com/apache/superset.git
synced 2026-07-20 21:55:46 +00:00
Compare commits
6 Commits
oauth-duri
...
research-s
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7c2135a3ad | ||
|
|
0f220e976e | ||
|
|
a75f9b67b2 | ||
|
|
3f0858e35d | ||
|
|
68c145adc3 | ||
|
|
4a9aecda4a |
540
RESEARCH.md
Normal file
540
RESEARCH.md
Normal file
@@ -0,0 +1,540 @@
|
||||
# Semantic Layer MCP — Research
|
||||
|
||||
**Story**: SC-98803 — Semantic Layer & MCP
|
||||
**Branch**: `research-semantic-layer-mcp`
|
||||
**Date**: 2026-05-20
|
||||
|
||||
---
|
||||
|
||||
## Story Summary
|
||||
|
||||
Story #98803 is titled "Semantic Layer & MCP" with no written description or acceptance criteria. Two
|
||||
comments clarify intent:
|
||||
|
||||
1. "Probably worth meeting with Beto to start discussing what Semantic Layer work means for MCP"
|
||||
2. "not putting any more stories in mcp v1 and rather triaging future work in the Preset MCP V2 epic"
|
||||
|
||||
This is a **research spike** to map out what "Semantic Layer MCP" would mean, what already exists, what
|
||||
gaps remain, and what would go into MCP v2. No implementation is expected from this story.
|
||||
|
||||
---
|
||||
|
||||
## What Superset's Semantic Layer Is
|
||||
|
||||
Superset has **two distinct semantic layer concepts** that must be treated separately:
|
||||
|
||||
### 1. Built-in Dataset Semantic Layer (SqlaTable)
|
||||
|
||||
Every Superset dataset (`SqlaTable`) carries a lightweight semantic layer composed of:
|
||||
|
||||
| Concept | Model | Storage |
|
||||
|---|---|---|
|
||||
| **Saved metrics** | `SqlMetric` | `sql_metrics` table; `expression` column holds SQL (e.g. `COUNT(*)`, `SUM(revenue)`) |
|
||||
| **Calculated columns** | `TableColumn.expression` | `table_columns` table; non-null `expression` = virtual column |
|
||||
| **Dimensions** | `TableColumn` with `groupby=True` | Physical or calculated columns marked as groupable |
|
||||
| **Filters** | `TableColumn` with `filterable=True` | Drives filter dropdowns |
|
||||
| **Time grains** | `SqlaTable.main_dttm_col` | Primary datetime column for temporal slicing |
|
||||
| **RLS** | `RowLevelSecurityFilter` | Per-user WHERE clause predicates |
|
||||
|
||||
At query time, `TableColumn.get_sqla_col()` either wraps the physical column (`column(name)`) or inlines
|
||||
the SQL expression as `literal_column(expression)`. Jinja2 templates are processed before injection.
|
||||
Saved metrics follow the same pattern via `SqlMetric.get_sqla_col()`.
|
||||
|
||||
Physical vs. virtual datasets:
|
||||
- **Physical** — `sql=NULL`; metadata discovered from DB schema
|
||||
- **Virtual** — `sql=<SELECT ...>`; metadata discovered by running the SQL; `is_sqllab_view=True`
|
||||
|
||||
### 2. External Semantic Layer (SemanticLayer / SemanticView)
|
||||
|
||||
A newer plugin-based system for connecting to external semantic catalogs:
|
||||
|
||||
- `SemanticLayer` model (`superset/semantic_layers/models.py`) — a connection record (type, encrypted config)
|
||||
- `SemanticView` model — a queryable view within a layer (maps to a dbt model, a Cube cube, etc.)
|
||||
- `superset/semantic_layers/registry.py` — plugin registry mapping `type → implementation class`
|
||||
- `superset/semantic_layers/mapper.py` — translates Superset's `QueryObject` → `SemanticQuery` for
|
||||
dispatch to the implementation; handles metrics, dimensions, filters, time grains, and time comparison
|
||||
- REST API: `GET/POST /api/v1/semantic_layer/`, `GET/POST /api/v1/semantic_layer/{id}/semantic_view/`
|
||||
- Query path uses `datasource_type="semantic_view"` (distinct from `"table"` for SqlaTable)
|
||||
|
||||
The registry is currently empty in OSS (`registry: dict[str, type[SemanticLayer]] = {}`); implementations
|
||||
are contributed via the extension/plugin mechanism referenced by the TODO in `models.py`.
|
||||
|
||||
---
|
||||
|
||||
## Existing MCP Tool Surface
|
||||
|
||||
The MCP service (`superset/mcp_service/`) already covers Superset's built-in semantic layer well:
|
||||
|
||||
| Tool | Tag | What it does |
|
||||
|---|---|---|
|
||||
| `list_datasets` | discovery | List/search datasets by name, schema, database |
|
||||
| `get_dataset_info` | discovery | Full dataset metadata: **columns** (dimensions) + **metrics** (saved metrics) |
|
||||
| `query_dataset` | data | Query using metric names + column names + filters + time range — **no SQL needed** |
|
||||
| `create_virtual_dataset` | mutate | Save a SQL SELECT as a named virtual dataset |
|
||||
| `execute_sql` | data | Raw SQL against any DB (bypasses semantic layer entirely) |
|
||||
|
||||
`query_dataset` is the core semantic layer MCP tool. It:
|
||||
1. Resolves dataset by ID or UUID
|
||||
2. Validates requested metric/column names against the dataset's saved metrics and columns (with typo
|
||||
suggestions via `difflib.get_close_matches`)
|
||||
3. Builds a `QueryContext` with `datasource_type="table"` and dispatches via `ChartDataCommand`
|
||||
4. Returns tabular rows, column metadata, cache status, and applied filters
|
||||
|
||||
**What is NOT covered:**
|
||||
- No MCP tools for `SemanticLayer` or `SemanticView` models (external semantic layers)
|
||||
- No cross-dataset metric/dimension discovery (no `list_metrics`, `list_dimensions` tools)
|
||||
- No tool for querying external semantic layers via the `datasource_type="semantic_view"` path
|
||||
|
||||
---
|
||||
|
||||
## Proposed MCP Tools
|
||||
|
||||
The following analysis groups proposed tools by priority tier.
|
||||
|
||||
### Tier 1 — External Semantic Layer Discovery (highest value, unblocked)
|
||||
|
||||
These tools mirror what `list_datasets` / `get_dataset_info` do for SqlaTable but for the external
|
||||
semantic layer model.
|
||||
|
||||
---
|
||||
|
||||
**`list_semantic_layers`**
|
||||
- **Description**: List available external semantic layer connections (dbt, Cube, etc.)
|
||||
- **Inputs**: `page`, `page_size`, `search` (optional name filter)
|
||||
- **Outputs**: `[{id, uuid, name, type, description, view_count}]`, pagination
|
||||
- **Pattern**: `ModelListCore(SemanticLayerDAO, ...)`
|
||||
- **Notes**: Requires `SemanticLayer` FAB permission; respects `semantic_layer_access` PVM.
|
||||
|
||||
---
|
||||
|
||||
**`get_semantic_layer_info`**
|
||||
- **Description**: Get details of a semantic layer including its registered views
|
||||
- **Inputs**: `identifier` (UUID)
|
||||
- **Outputs**: `{uuid, name, type, description, views: [{id, uuid, name, description}]}`
|
||||
- **Pattern**: `ModelGetInfoCore(SemanticLayerDAO, ...)`
|
||||
- **Notes**: Views list drives the LLM to choose the right view before querying.
|
||||
|
||||
---
|
||||
|
||||
**`list_semantic_views`**
|
||||
- **Description**: List views within a specific semantic layer
|
||||
- **Inputs**: `semantic_layer_uuid`, `page`, `page_size`, `search`
|
||||
- **Outputs**: `[{id, uuid, name, description, metric_count, dimension_count}]`
|
||||
- **Pattern**: DAO query on `SemanticView` filtered by `semantic_layer_uuid`
|
||||
- **Notes**: Separate from `get_semantic_layer_info` to support pagination on large layers.
|
||||
|
||||
---
|
||||
|
||||
### Tier 2 — External Semantic Layer Query
|
||||
|
||||
---
|
||||
|
||||
**`get_semantic_view_schema`**
|
||||
- **Description**: Get available metrics and dimensions for a semantic view
|
||||
- **Inputs**: `semantic_view_id` (int or UUID)
|
||||
- **Outputs**: `{metrics: [{name, type, description, ...}], dimensions: [{name, type, is_dttm, ...}]}`
|
||||
- **Notes**: Calls the layer's `implementation` to fetch live schema from the external system. This is
|
||||
distinct from `get_dataset_info` because the metadata comes from the external catalog, not Superset's DB.
|
||||
The `MetricMetadata` and `ColumnMetadata` dataclasses in `semantic_layers/models.py` define the shape.
|
||||
|
||||
---
|
||||
|
||||
**`query_semantic_view`**
|
||||
- **Description**: Query an external semantic layer view by metric and dimension names
|
||||
- **Inputs**: `semantic_view_id`, `metrics: list[str]`, `dimensions: list[str]`, `filters`, `time_range`,
|
||||
`time_grain`, `row_limit`
|
||||
- **Outputs**: Same shape as `query_dataset` (tabular rows + column metadata + performance)
|
||||
- **Notes**: Uses the `datasource_type="semantic_view"` path through `QueryContextFactory`. The
|
||||
`mapper.py` `get_results()` function already handles the translation. This is a peer to `query_dataset`,
|
||||
not a replacement.
|
||||
- **Key design question**: How to handle authentication to the external system (dbt Cloud token, Cube API
|
||||
key, etc.) — stored in encrypted `SemanticLayer.configuration`, fetched at query time.
|
||||
|
||||
---
|
||||
|
||||
### Tier 3 — Cross-Dataset Discovery (nice-to-have, lower priority)
|
||||
|
||||
These are useful for LLM agents that need to find metrics without knowing which dataset to start from.
|
||||
|
||||
---
|
||||
|
||||
**`list_metrics`**
|
||||
- **Description**: Search saved metrics across all accessible datasets
|
||||
- **Inputs**: `search` (metric name / description), `dataset_id` (optional), `page`, `page_size`
|
||||
- **Outputs**: `[{metric_name, expression, dataset_id, dataset_name, description, d3format}]`
|
||||
- **Notes**: Direct DAO query on `SqlMetric` with RBAC filtering. High value for LLMs that start with
|
||||
"find me a revenue metric" rather than "query dataset 42."
|
||||
|
||||
---
|
||||
|
||||
**`list_dimensions`**
|
||||
- **Description**: Search groupable columns across accessible datasets
|
||||
- **Inputs**: `search`, `dataset_id` (optional), `is_dttm` (optional), `page`, `page_size`
|
||||
- **Outputs**: `[{column_name, type, is_dttm, dataset_id, dataset_name, description}]`
|
||||
- **Notes**: Same pattern as `list_metrics` but on `TableColumn` with `groupby=True`.
|
||||
Complementary to `list_metrics` for fully metric-driven workflows.
|
||||
|
||||
---
|
||||
|
||||
### Summary Table
|
||||
|
||||
| Tool | Tier | Effort | Depends on |
|
||||
|---|---|---|---|
|
||||
| `list_semantic_layers` | 1 | S | `SemanticLayerDAO` (exists) |
|
||||
| `get_semantic_layer_info` | 1 | S | `SemanticLayerDAO` (exists) |
|
||||
| `list_semantic_views` | 1 | S | `SemanticLayerDAO` (exists) |
|
||||
| `get_semantic_view_schema` | 2 | M | External layer implementation |
|
||||
| `query_semantic_view` | 2 | M | `mapper.py` (exists), `QueryContextFactory` |
|
||||
| `list_metrics` | 3 | S | `SqlMetric` DAO query |
|
||||
| `list_dimensions` | 3 | S | `TableColumn` DAO query |
|
||||
|
||||
Effort scale: S = 1–2 days, M = 3–5 days, L = 1–2 weeks, XL = 2+ weeks.
|
||||
|
||||
---
|
||||
|
||||
## Landscape Survey
|
||||
|
||||
### How Other Tools Do It
|
||||
|
||||
**Looker** (Google)
|
||||
- Looker has a full Semantic Layer API: `/api/4.0/lookml_models/{name}/explores/{name}/fields` returns
|
||||
dimensions, measures, and filters from LookML
|
||||
- LookML is the semantic definition language; AI/LLM access is via Looker's Explore REST API
|
||||
- Looker Studio integration reads the semantic layer directly
|
||||
|
||||
**Cube.dev**
|
||||
- Cube ships an official MCP server (`@cubejs-backend/mcp`) with:
|
||||
- `list_cubes` — returns all cubes with measure/dimension/segment definitions
|
||||
- `query_cube` — runs a semantic query by cube+measures+dimensions+filters
|
||||
- `get_cube_meta` — live schema for a specific cube
|
||||
- This is the closest prior art to what Superset's Tier 1+2 tools would look like
|
||||
|
||||
**dbt Semantic Layer (MetricFlow)**
|
||||
- dbt Cloud exposes metrics via JDBC/GraphQL Semantic Layer API
|
||||
- MetricFlow's conceptual model: `Metric`, `Dimension`, `Entity`, `TimeDimension`
|
||||
- LLM integrations (e.g., dbt Copilot) use `list_metrics`, `list_dimensions`, and a `query_metrics`
|
||||
endpoint that returns results as Arrow IPC
|
||||
|
||||
**MetricFlow**
|
||||
- Query API: `list_metrics()`, `list_dimensions()`, `query_metrics(metrics=[], groupBy=[])`
|
||||
- Self-describing schema: each metric knows its required dimensions and time grains
|
||||
|
||||
**Key takeaway**: The industry pattern is consistent — a 3-tool surface: `list_metrics`, `list_dimensions`,
|
||||
`query_metrics`. Superset's `get_dataset_info` + `query_dataset` already implement this pattern for the
|
||||
built-in semantic layer. The gap is the external semantic layer equivalents and cross-dataset discovery.
|
||||
|
||||
### Prior Art in the Superset Community
|
||||
|
||||
- The `semantic_layers/` module (models, mapper, registry, API) was built ahead of any specific
|
||||
implementation — it is the extension point for dbt, Cube, Snowflake Cortex, etc.
|
||||
- No OSS implementations are registered; implementations come via the Preset plugin mechanism
|
||||
- No Preset-specific semantic layer MCP tools exist in `superset-shell/preset/mcp/` today
|
||||
|
||||
---
|
||||
|
||||
## Implementation Approach
|
||||
|
||||
### Phase 1: External Semantic Layer Discovery (Tier 1)
|
||||
|
||||
The `SemanticLayerDAO` already exists and the REST API is live. Adding `list_semantic_layers`,
|
||||
`get_semantic_layer_info`, and `list_semantic_views` follows the exact same pattern as the existing
|
||||
dataset tools. Use `ModelListCore` and `ModelGetInfoCore` from `mcp_core.py`. This is straightforward
|
||||
work (3 × S = ~3–4 days total) and unblocked.
|
||||
|
||||
Create `superset/mcp_service/semantic_layer/` module with:
|
||||
- `schemas.py` — Pydantic schemas for SemanticLayer and SemanticView
|
||||
- `tool/list_semantic_layers.py`
|
||||
- `tool/get_semantic_layer_info.py`
|
||||
- `tool/list_semantic_views.py`
|
||||
- `tool/__init__.py`
|
||||
|
||||
Register all three in `app.py`.
|
||||
|
||||
### Phase 2: External Semantic Layer Query (Tier 2)
|
||||
|
||||
`get_semantic_view_schema` requires calling the external implementation's schema method, which may do
|
||||
a network call. This needs timeout handling and graceful error messaging.
|
||||
|
||||
`query_semantic_view` is the most complex: it must go through the `mapper.py` + `QueryContextFactory`
|
||||
path with `datasource_type="semantic_view"`. The query execution path exists but is untested from MCP.
|
||||
Recommend writing this alongside integration tests against a mock external layer.
|
||||
|
||||
### Phase 3: Cross-Dataset Discovery (Tier 3)
|
||||
|
||||
`list_metrics` and `list_dimensions` are pure DAO queries on `SqlMetric` and `TableColumn`. They need
|
||||
RBAC filtering (only return metrics/columns from datasets the user can access) which adds complexity.
|
||||
The simplest safe approach is to call `DatasetDAO.find_all()` to get accessible datasets and then filter
|
||||
metrics/columns in Python — this avoids writing a custom RBAC-aware JOIN.
|
||||
|
||||
### Key Design Decisions
|
||||
|
||||
1. **Unified `query` tool vs. split tools**: Should `query_dataset` be extended to detect
|
||||
`semantic_view_id` inputs, or should `query_semantic_view` be a separate tool? Recommendation:
|
||||
**separate tools** — the query paths are genuinely different (different datasource types, different
|
||||
schema shapes) and separate tools are clearer for LLMs.
|
||||
|
||||
2. **Schema exposure**: Should `get_semantic_view_schema` return the full external schema or only the
|
||||
subset Superset knows about? Recommendation: **live from external system** — this is the value prop.
|
||||
|
||||
3. **`list_metrics` RBAC**: Should it join dataset permissions in SQL or filter in Python? Recommendation:
|
||||
**Python filtering via DatasetDAO** until proven to be a performance problem.
|
||||
|
||||
4. **Prompt updates**: `DEFAULT_INSTRUCTIONS` in `app.py` should be updated to mention the semantic layer
|
||||
tools and when to prefer them over `execute_sql`.
|
||||
|
||||
---
|
||||
|
||||
## Questions for Beto — with Answers (2026-06-04)
|
||||
|
||||
1. **What does MCP v2 mean architecturally?** The story comment says these tools belong in "MCP v2
|
||||
epic." Are there breaking changes (new tool naming, new auth model, new transport) that affect how
|
||||
these tools should be designed?
|
||||
|
||||
> **💬 Beto:** No. It's just the second release of the MCP service — not a breaking change. Build
|
||||
> these tools with the same conventions as v1.
|
||||
|
||||
2. **Which external semantic layer implementations exist at Preset?** The `registry` is empty in OSS.
|
||||
What types are registered in `superset-shell`? Which implementations should `query_semantic_view`
|
||||
be tested against first (dbt, Snowflake Cortex, Cube)?
|
||||
|
||||
> **💬 Beto:** Our first target is Snowflake, followed by MetricFlow, Cube, DJ. But it shouldn't
|
||||
> matter — the interface is identical for all of them.
|
||||
|
||||
3. **Is `SemanticLayer.implementation` wiring complete?** There's a `TODO` comment in
|
||||
`semantic_layers/models.py` referencing an `extension_manager` that isn't fully wired yet (currently
|
||||
falls back directly to `registry[self.type]`). Is the plugin mechanism stable enough to build MCP
|
||||
tools on top of?
|
||||
|
||||
> **💬 Beto:** It is. The TODO in the discovery mechanism is because we've been discussing changing
|
||||
> how installed extensions are discovered in the future, but it won't affect any APIs — it's purely
|
||||
> internal. Safe to build on top of today.
|
||||
|
||||
4. **RLS for external semantic layers**: SqlaTable RLS is handled by `get_sqla_row_level_filters()`.
|
||||
External layers have their own permission models (dbt Cloud access, Cube tenant isolation). How should
|
||||
MCP enforce per-user data access when querying external layers?
|
||||
|
||||
> **💬 Beto:** We don't currently support RLS in semantic layers/views, but will eventually. Don't
|
||||
> worry about this now — when we add support it will be transparent to the MCP service.
|
||||
|
||||
5. **Should `list_metrics` / `list_dimensions` span built-in + external layers?** A unified
|
||||
`list_metrics` spanning both SqlaTable saved metrics and external semantic layer metrics could be
|
||||
very powerful but complex. Or should they be separate surfaces?
|
||||
|
||||
> **💬 Beto:** Yes — unified. Most users will have only a single semantic layer (the whole point is
|
||||
> a single source of truth). Because of this, we should never force the user to choose a layer before
|
||||
> listing metrics — most of the time there will be only one, and we should never prompt to choose from
|
||||
> a single-element list. Even when there are multiple, start with global search.
|
||||
|
||||
6. **What is the intended LLM workflow for external semantic layers?** For built-in datasets the
|
||||
workflow is: `list_datasets` → `get_dataset_info` → `query_dataset`. What's the analogous flow
|
||||
for external layers?
|
||||
|
||||
> **💬 Beto:** The flow is `list_metrics` → metrics come with related dimensions already — then
|
||||
> `get_table(metrics, dimensions, filters)`. Also worth exposing the `get_compatible_*()` methods
|
||||
> from `superset-core/src/superset_core/semantic_layers/view.py`. When listing metrics, compatible
|
||||
> dimensions should be part of the metric payload itself.
|
||||
|
||||
---
|
||||
|
||||
## Effort Estimate (original)
|
||||
|
||||
| Scope | Effort | Notes |
|
||||
|---|---|---|
|
||||
| `list_semantic_layers` | S (1–2 days) | Follows exact ModelListCore pattern |
|
||||
| `get_semantic_layer_info` | S (1–2 days) | Follows exact ModelGetInfoCore pattern |
|
||||
| `list_semantic_views` | S (1 day) | Subset of get_semantic_layer_info |
|
||||
| `get_semantic_view_schema` | M (3 days) | Network call to external system + error handling |
|
||||
| `query_semantic_view` | M (4–5 days) | New query path + integration tests |
|
||||
| `list_metrics` | S (2 days) | DAO query + RBAC filter |
|
||||
| `list_dimensions` | S (1–2 days) | Same pattern as list_metrics |
|
||||
| Prompt/instructions updates | S (1 day) | Update DEFAULT_INSTRUCTIONS + tool docs |
|
||||
| **Total Tier 1** | **~1 week** | |
|
||||
| **Total Tier 1+2** | **~2.5 weeks** | Assuming external implementation is available |
|
||||
| **Total Tier 1+2+3** | **~3.5 weeks** | |
|
||||
|
||||
---
|
||||
|
||||
## Final Architecture Proposal
|
||||
|
||||
*Synthesized from the research above and Beto's 2026-06-04 answers.*
|
||||
|
||||
### Core Insight: Flat Metric-First Workflow
|
||||
|
||||
Beto's answer to Q6 collapses the original three-tier hierarchy into a much flatter design. The intended
|
||||
LLM workflow is:
|
||||
|
||||
```
|
||||
list_metrics → get_table(metrics, dimensions, filters)
|
||||
```
|
||||
|
||||
Not the hierarchical: `list_semantic_layers → list_semantic_views → get_semantic_view_schema → query_semantic_view`.
|
||||
|
||||
This means `list_metrics` is the **primary entry point** — not an administrative helper — and it must
|
||||
work globally across both built-in datasets and external semantic layers without the user choosing a
|
||||
layer first. Compatible dimensions travel with each metric in the response payload.
|
||||
|
||||
The abstract interface in `superset-core/src/superset_core/semantic_layers/view.py` is the contract
|
||||
we're wrapping. Every proposed MCP tool maps to a method there:
|
||||
|
||||
| MCP Tool | `view.py` method | Notes |
|
||||
|---|---|---|
|
||||
| `list_metrics` | `get_metrics()` | + `SqlMetric` for built-in |
|
||||
| `get_compatible_dimensions` | `get_compatible_dimensions(metrics, dims)` | Refinement during selection |
|
||||
| `get_compatible_metrics` | `get_compatible_metrics(metrics, dims)` | Refinement during selection |
|
||||
| `get_table` | `get_table(query)` | Unified query; replaces `query_dataset` + `query_semantic_view` |
|
||||
| `get_dimension_values` | `get_values(dimension, filters)` | Distinct values for filter UI |
|
||||
| `get_row_count` | `get_row_count(query)` | Preview row count before full fetch |
|
||||
|
||||
### Revised Tool Design
|
||||
|
||||
#### `list_metrics` ★ Primary tool
|
||||
|
||||
Search all accessible metrics globally — spanning both built-in (`SqlMetric`) and external semantic
|
||||
layer metrics (via `SemanticView.get_metrics()`). No layer selection required.
|
||||
|
||||
- **Inputs:** `search` (optional text filter), `page`, `page_size`
|
||||
- **Outputs:**
|
||||
```json
|
||||
[{
|
||||
"name": "total_revenue",
|
||||
"description": "...",
|
||||
"source": {"type": "dataset" | "semantic_view", "id": "<uuid>", "name": "Sales"},
|
||||
"compatible_dimensions": [
|
||||
{"name": "region", "type": "VARCHAR", "is_dttm": false},
|
||||
{"name": "order_date", "type": "TIMESTAMP", "is_dttm": true}
|
||||
]
|
||||
}]
|
||||
```
|
||||
- **RBAC:** Built-in metrics: filter by accessible datasets via `DatasetDAO`. External: filter by
|
||||
accessible `SemanticLayer` connections.
|
||||
- **Effort:** M (3–4 days) — unified across both sources is more complex than a single DAO query.
|
||||
|
||||
---
|
||||
|
||||
#### `get_table` ★ Unified query tool
|
||||
|
||||
Execute a semantic query against any metric source. Routes to the built-in path or the external
|
||||
`SemanticView.get_table()` path based on where the metrics come from.
|
||||
|
||||
- **Inputs:** `metrics: list[str]`, `dimensions: list[str]`, `filters`, `time_range`, `time_grain`,
|
||||
`row_limit`
|
||||
- **Outputs:** Tabular rows + column metadata + performance info (same shape as existing `query_dataset`)
|
||||
- **Routing logic:**
|
||||
- All metrics from `SqlMetric` → existing `ChartDataCommand` path (`datasource_type="table"`)
|
||||
- Metrics from `SemanticView` → `mapper.py` + `datasource_type="semantic_view"` path
|
||||
- Mixed → error (metrics must come from a single source)
|
||||
- **Notes:** This is the peer/replacement for `query_dataset` in semantic-layer-first workflows.
|
||||
Existing `query_dataset` is unchanged — it remains the direct-dataset tool.
|
||||
- **Effort:** M (4–5 days) — routing logic + integration tests against both paths.
|
||||
|
||||
---
|
||||
|
||||
#### `get_compatible_dimensions`
|
||||
|
||||
Refine dimension selection given already-chosen metrics and dimensions. Surfaces
|
||||
`view.get_compatible_dimensions()`.
|
||||
|
||||
- **Inputs:** `metric_names: list[str]`, `selected_dimensions: list[str]` (already picked)
|
||||
- **Outputs:** `[{name, type, is_dttm, description}]` — dimensions compatible with the current selection
|
||||
- **Use case:** LLM is helping user build a query step-by-step; user has picked 2 metrics and now asks
|
||||
"what dimensions can I group by?"
|
||||
- **Effort:** S (1–2 days)
|
||||
|
||||
---
|
||||
|
||||
#### `get_compatible_metrics`
|
||||
|
||||
Mirror of `get_compatible_dimensions` — refine metric selection given selected dimensions.
|
||||
|
||||
- **Inputs:** `selected_metrics: list[str]`, `dimension_names: list[str]`
|
||||
- **Outputs:** `[{name, description, source}]` — metrics compatible with the selected dimensions
|
||||
- **Effort:** S (1 day)
|
||||
|
||||
---
|
||||
|
||||
#### `get_dimension_values`
|
||||
|
||||
Fetch distinct values for a dimension (for filter UI or LLM-driven filtering). Surfaces
|
||||
`view.get_values()`.
|
||||
|
||||
- **Inputs:** `dimension_name: str`, `source_id` (dataset UUID or semantic_view UUID), `filters`
|
||||
(optional pre-filters)
|
||||
- **Outputs:** `[{value, label}]`
|
||||
- **Effort:** S (1–2 days)
|
||||
|
||||
---
|
||||
|
||||
#### `get_row_count`
|
||||
|
||||
Count rows a query would return before executing it — useful for previews and safety checks. Surfaces
|
||||
`view.get_row_count()`.
|
||||
|
||||
- **Inputs:** `metrics: list[str]`, `dimensions: list[str]`, `filters`, `time_range`
|
||||
- **Outputs:** `{row_count: int}`
|
||||
- **Effort:** S (1 day)
|
||||
|
||||
---
|
||||
|
||||
#### Admin Discovery Tools (lower priority)
|
||||
|
||||
These are useful for admin/setup workflows but are NOT part of the primary LLM query flow:
|
||||
|
||||
| Tool | Effort | Purpose |
|
||||
|---|---|---|
|
||||
| `list_semantic_layers` | S (1–2 days) | List configured external connections |
|
||||
| `get_semantic_layer_info` | S (1–2 days) | Details of a connection + its views |
|
||||
| `list_semantic_views` | S (1 day) | Paginated views within a layer |
|
||||
|
||||
---
|
||||
|
||||
### What Changed from the Original Proposal
|
||||
|
||||
| Original | Revised | Reason |
|
||||
|---|---|---|
|
||||
| Tier 3 (`list_metrics`) = lowest priority | `list_metrics` = **highest priority** | Beto: it's the primary entry point |
|
||||
| Separate `query_dataset` + `query_semantic_view` | Unified `get_table` | Beto: single query tool for the semantic workflow |
|
||||
| `get_semantic_view_schema` for schema discovery | `list_metrics` includes compatible dims inline | Beto: schema travels with metrics |
|
||||
| Layer selection required before listing metrics | Global search, no layer selection | Beto: single source of truth, never prompt to choose |
|
||||
| `get_compatible_*()` not in scope | Now explicit MCP tools | Beto: expose these directly |
|
||||
|
||||
### Implementation Phases
|
||||
|
||||
**Phase 1 — Core semantic workflow (~2 weeks, mostly unblocked)**
|
||||
|
||||
All built-in layer support; external layer support added in Phase 2.
|
||||
|
||||
1. `list_metrics` (built-in SqlMetric + stub for external)
|
||||
2. `get_table` (built-in path first; `datasource_type="table"`)
|
||||
3. `get_compatible_dimensions` / `get_compatible_metrics`
|
||||
4. Update `DEFAULT_INSTRUCTIONS` in `app.py`
|
||||
|
||||
**Phase 2 — External layer support (~1.5 weeks, needs Snowflake plugin)**
|
||||
|
||||
1. Extend `list_metrics` to query external `SemanticView.get_metrics()`
|
||||
2. Extend `get_table` to route to `datasource_type="semantic_view"` path
|
||||
3. `get_dimension_values` (via `view.get_values()`)
|
||||
4. `get_row_count` (via `view.get_row_count()`)
|
||||
|
||||
**Phase 3 — Admin tools (~0.5 weeks, independent)**
|
||||
|
||||
1. `list_semantic_layers` / `get_semantic_layer_info` / `list_semantic_views`
|
||||
|
||||
### Revised Effort Summary
|
||||
|
||||
| Tool | Phase | Effort | Blocked? |
|
||||
|---|---|---|---|
|
||||
| `list_metrics` (built-in) | 1 | M (3 days) | No |
|
||||
| `get_table` (built-in path) | 1 | M (3–4 days) | No |
|
||||
| `get_compatible_dimensions` | 1 | S (1–2 days) | No |
|
||||
| `get_compatible_metrics` | 1 | S (1 day) | No |
|
||||
| `DEFAULT_INSTRUCTIONS` update | 1 | S (1 day) | No |
|
||||
| `list_metrics` (external) | 2 | M (2 days) | Yes — needs Snowflake plugin |
|
||||
| `get_table` (external path) | 2 | M (2–3 days) | Yes — needs Snowflake plugin |
|
||||
| `get_dimension_values` | 2 | S (1–2 days) | Yes — needs Snowflake plugin |
|
||||
| `get_row_count` | 2 | S (1 day) | Yes — needs Snowflake plugin |
|
||||
| `list_semantic_layers` | 3 | S (1–2 days) | No |
|
||||
| `get_semantic_layer_info` | 3 | S (1–2 days) | No |
|
||||
| `list_semantic_views` | 3 | S (1 day) | No |
|
||||
| **Total Phase 1** | | **~2 weeks** | |
|
||||
| **Total Phase 1+2** | | **~3.5 weeks** | |
|
||||
| **Total Phase 1+2+3** | | **~4 weeks** | |
|
||||
@@ -92,6 +92,26 @@ class Dimension:
|
||||
grain: Grain | None = None
|
||||
|
||||
|
||||
class AggregationType(str, enum.Enum):
|
||||
"""
|
||||
Aggregation function applied by a metric.
|
||||
|
||||
Additivity (across an arbitrary set of grouping dimensions):
|
||||
* ``SUM``, ``COUNT``: fully additive — sub-group sums roll up via ``sum``.
|
||||
* ``MIN``, ``MAX``: roll up via ``min`` / ``max`` of sub-group values.
|
||||
* ``AVG``, ``COUNT_DISTINCT``, ``OTHER``: not safely roll-uppable from
|
||||
sub-aggregates without auxiliary data.
|
||||
"""
|
||||
|
||||
SUM = "SUM"
|
||||
COUNT = "COUNT"
|
||||
MIN = "MIN"
|
||||
MAX = "MAX"
|
||||
AVG = "AVG"
|
||||
COUNT_DISTINCT = "COUNT_DISTINCT"
|
||||
OTHER = "OTHER"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Metric:
|
||||
id: str
|
||||
@@ -100,6 +120,7 @@ class Metric:
|
||||
|
||||
definition: str
|
||||
description: str | None = None
|
||||
aggregation: AggregationType | None = None
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
|
||||
@@ -95,8 +95,11 @@ class FakeMessageChannel {
|
||||
const port2 = new FakeMessagePort();
|
||||
port1.otherPort = port2;
|
||||
port2.otherPort = port1;
|
||||
this.port1 = port1;
|
||||
this.port2 = port2;
|
||||
// FakeMessagePort only implements the subset of MessagePort that
|
||||
// Switchboard exercises; cast at the boundary so the fake satisfies
|
||||
// the consumer signature without weakening the production type.
|
||||
this.port1 = port1 as unknown as MessagePort;
|
||||
this.port2 = port2 as unknown as MessagePort;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -88,7 +88,7 @@ function isError(message: Message): message is ErrorMessage {
|
||||
* Calling methods on the switchboard causes messages to be sent through the channel.
|
||||
*/
|
||||
export class Switchboard {
|
||||
port: MessagePort;
|
||||
port!: MessagePort;
|
||||
|
||||
name = '';
|
||||
|
||||
@@ -97,9 +97,9 @@ export class Switchboard {
|
||||
// used to make unique ids
|
||||
incrementor = 1;
|
||||
|
||||
debugMode: boolean;
|
||||
debugMode = false;
|
||||
|
||||
private isInitialised: boolean;
|
||||
private isInitialised = false;
|
||||
|
||||
constructor(params?: Params) {
|
||||
if (!params) {
|
||||
|
||||
@@ -1712,7 +1712,7 @@ export function createDatasource(
|
||||
|
||||
export function createCtasDatasource(
|
||||
vizOptions: Record<string, unknown>,
|
||||
): SqlLabThunkAction<Promise<{ id: number }>> {
|
||||
): SqlLabThunkAction<Promise<{ table_id: number }>> {
|
||||
return (dispatch: AppDispatch) => {
|
||||
dispatch(createDatasourceStarted());
|
||||
return SupersetClient.post({
|
||||
@@ -1720,9 +1720,14 @@ export function createCtasDatasource(
|
||||
jsonPayload: vizOptions,
|
||||
})
|
||||
.then(({ json }) => {
|
||||
dispatch(createDatasourceSuccess(json.result));
|
||||
const result = json.result as { table_id: number };
|
||||
// The endpoint's `result.table_id` IS the dataset id; normalize so
|
||||
// createDatasourceSuccess's `${data.id}__table` resolves correctly.
|
||||
// Without this, the CTAS Explore button silently produced
|
||||
// `"undefined__table"` because `result.id` doesn't exist.
|
||||
dispatch(createDatasourceSuccess({ id: result.table_id }));
|
||||
|
||||
return json.result;
|
||||
return result;
|
||||
})
|
||||
.catch(() => {
|
||||
const errorMsg = t('An error occurred while creating the data source');
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
|
||||
import { useRef, useEffect, FC, useMemo } from 'react';
|
||||
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { logging } from '@apache-superset/core/utils';
|
||||
import {
|
||||
SqlLabRootState,
|
||||
@@ -86,7 +87,7 @@ const EditorAutoSync: FC = () => {
|
||||
const editorTabLastUpdatedAt = useSelector<SqlLabRootState, number>(
|
||||
state => state.sqlLab.editorTabLastUpdatedAt,
|
||||
);
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const lastSavedTimestampRef = useRef<number>(editorTabLastUpdatedAt);
|
||||
|
||||
const currentQueryEditorId = useSelector<SqlLabRootState, string>(
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useState, useEffect, useRef, useCallback, useMemo } from 'react';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { usePrevious } from '@superset-ui/core';
|
||||
import { css, useTheme } from '@apache-superset/core/theme';
|
||||
import { Global } from '@emotion/react';
|
||||
@@ -136,7 +137,7 @@ const EditorWrapper = ({
|
||||
height,
|
||||
hotkeys,
|
||||
}: EditorWrapperProps) => {
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const queryEditor = useQueryEditor(queryEditorId, [
|
||||
'id',
|
||||
'dbId',
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useEffect, useMemo, useRef } from 'react';
|
||||
import { useDispatch, useStore } from 'react-redux';
|
||||
import { useStore } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import { getExtensionsRegistry } from '@superset-ui/core';
|
||||
|
||||
@@ -68,7 +69,7 @@ export function useKeywords(
|
||||
catalog,
|
||||
schema,
|
||||
});
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const hasFetchedKeywords = useRef(false);
|
||||
// skipFetch is used to prevent re-evaluating memoized keywords
|
||||
// due to updated api results by skip flag
|
||||
|
||||
@@ -16,9 +16,10 @@
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
import { useSelector, useDispatch } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import { JsonObject, VizType } from '@superset-ui/core';
|
||||
import { VizType } from '@superset-ui/core';
|
||||
import {
|
||||
createCtasDatasource,
|
||||
addInfoToast,
|
||||
@@ -45,7 +46,7 @@ const ExploreCtasResultsButton = ({
|
||||
const errorMessage = useSelector(
|
||||
(state: SqlLabRootState) => state.sqlLab.errorMessage,
|
||||
);
|
||||
const dispatch = useDispatch<(dispatch: any) => Promise<JsonObject>>();
|
||||
const dispatch = useAppDispatch();
|
||||
|
||||
const buildVizOptions = {
|
||||
table_name: table,
|
||||
@@ -56,7 +57,7 @@ const ExploreCtasResultsButton = ({
|
||||
|
||||
const visualize = () => {
|
||||
dispatch(createCtasDatasource(buildVizOptions))
|
||||
.then((data: { table_id: number }) => {
|
||||
.then(data => {
|
||||
const formData = {
|
||||
datasource: `${data.table_id}__table`,
|
||||
metrics: ['count'],
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useEffect, useState } from 'react';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import URI from 'urijs';
|
||||
import { pick } from 'lodash';
|
||||
import { useComponentDidUpdate } from '@superset-ui/core';
|
||||
@@ -49,7 +50,7 @@ const PopEditorTab: React.FC<{ children?: React.ReactNode }> = ({
|
||||
({ sqlLab: { tabHistory } }) => tabHistory.slice(-1)[0],
|
||||
);
|
||||
const [updatedUrl, setUpdatedUrl] = useState<string>(SQL_LAB_URL);
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
useComponentDidUpdate(() => {
|
||||
setQueryEditorId(assigned => assigned ?? activeQueryEditorId);
|
||||
if (activeQueryEditorId) {
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useRef } from 'react';
|
||||
import { useSelector, useDispatch } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { isObject } from 'lodash';
|
||||
import rison from 'rison';
|
||||
import {
|
||||
@@ -82,7 +83,7 @@ function QueryAutoRefresh({
|
||||
.map(({ id }) => id),
|
||||
),
|
||||
);
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
|
||||
const checkForRefresh = () => {
|
||||
const shouldRequestChecking = shouldCheckForQueries(queries);
|
||||
|
||||
@@ -16,7 +16,7 @@
|
||||
* specific language governing permissions and limitations
|
||||
* under the License.
|
||||
*/
|
||||
import { useDispatch } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import { Dropdown, Button } from '@superset-ui/core/components';
|
||||
import { Menu } from '@superset-ui/core/components/Menu';
|
||||
@@ -75,7 +75,7 @@ const QueryLimitSelect = ({
|
||||
maxRow,
|
||||
defaultQueryLimit,
|
||||
}: QueryLimitSelectProps) => {
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
|
||||
const queryEditor = useQueryEditor(queryEditorId, ['id', 'queryLimit']);
|
||||
const queryLimit = queryEditor.queryLimit || defaultQueryLimit;
|
||||
|
||||
@@ -30,7 +30,8 @@ import ProgressBar from '@superset-ui/core/components/ProgressBar';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import { QueryResponse, QueryState } from '@superset-ui/core';
|
||||
import { useTheme } from '@apache-superset/core/theme';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
|
||||
import {
|
||||
queryEditorSetSql,
|
||||
@@ -92,7 +93,7 @@ const QueryTable = ({
|
||||
latestQueryId,
|
||||
}: QueryTableProps) => {
|
||||
const theme = useTheme();
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const [selectedQuery, setSelectedQuery] = useState<QueryResponse | null>(
|
||||
null,
|
||||
);
|
||||
|
||||
@@ -27,7 +27,8 @@ import {
|
||||
} from 'react';
|
||||
|
||||
import AutoSizer from 'react-virtualized-auto-sizer';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { useHistory } from 'react-router-dom';
|
||||
import { pick } from 'lodash';
|
||||
import {
|
||||
@@ -231,7 +232,7 @@ const ResultSet = ({
|
||||
canCopyClipboardSqlLab: canCopyClipboard,
|
||||
} = usePermissions();
|
||||
const history = useHistory();
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const logAction = useLogAction({ queryId, sqlEditorId: query.sqlEditorId });
|
||||
const { showConfirm, ConfirmModal } = useConfirmModal();
|
||||
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { createRef, useCallback, useMemo } from 'react';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { nanoid } from 'nanoid';
|
||||
import Tabs from '@superset-ui/core/components/Tabs';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
@@ -105,7 +106,7 @@ const SouthPane = ({
|
||||
const { id, tabViewId } = useQueryEditor(queryEditorId, ['tabViewId']);
|
||||
const editorId = tabViewId ?? id;
|
||||
const theme = useTheme();
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const viewItems = views.getViews(ViewLocations.sqllab.panels) || [];
|
||||
const { offline, tables } = useSelector(
|
||||
({ sqlLab: { offline, tables } }: SqlLabRootState) => ({
|
||||
|
||||
@@ -30,7 +30,8 @@ import {
|
||||
|
||||
import type { editors } from '@apache-superset/core';
|
||||
import useEffectEvent from 'src/hooks/useEffectEvent';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import AutoSizer from 'react-virtualized-auto-sizer';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import {
|
||||
@@ -237,7 +238,7 @@ const SqlEditor: FC<Props> = ({
|
||||
scheduleQueryWarning,
|
||||
}) => {
|
||||
const theme = useTheme();
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
|
||||
const { database, latestQuery, currentQueryEditorId, hasSqlStatement } =
|
||||
useSelector<
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useCallback, useState } from 'react';
|
||||
import { useDispatch } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
|
||||
import { resetState } from 'src/SqlLab/actions/sqlLab';
|
||||
import {
|
||||
@@ -69,7 +69,7 @@ const SqlEditorLeftBar = ({ queryEditorId }: SqlEditorLeftBarProps) => {
|
||||
const { db, catalog, schema, onDbChange, onCatalogChange, onSchemaChange } =
|
||||
dbSelectorProps;
|
||||
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const shouldShowReset = window.location.search === '?reset=1';
|
||||
|
||||
// Modal state for Database/Catalog/Schema selector
|
||||
|
||||
@@ -19,7 +19,8 @@
|
||||
import { useMemo, FC } from 'react';
|
||||
|
||||
import { bindActionCreators } from 'redux';
|
||||
import { useSelector, useDispatch, shallowEqual } from 'react-redux';
|
||||
import { useSelector, shallowEqual } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { MenuDotsDropdown } from '@superset-ui/core/components';
|
||||
import { Menu, MenuItemType } from '@superset-ui/core/components/Menu';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
@@ -90,7 +91,7 @@ const SqlEditorTabHeader: FC<Props> = ({ queryEditor }) => {
|
||||
);
|
||||
const StatusIcon = queryState ? STATE_ICONS[queryState] : STATE_ICONS.running;
|
||||
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const actions = useMemo(
|
||||
() =>
|
||||
bindActionCreators(
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useEffect, useCallback, useMemo, useState } from 'react';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
|
||||
import { SqlLabRootState } from 'src/SqlLab/types';
|
||||
import {
|
||||
@@ -41,7 +42,7 @@ export default function useDatabaseSelector(queryEditorId: string) {
|
||||
SqlLabRootState,
|
||||
SqlLabRootState['sqlLab']['databases']
|
||||
>(({ sqlLab }) => sqlLab.databases);
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const queryEditor = useQueryEditor(queryEditorId, [
|
||||
'dbId',
|
||||
'catalog',
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useState, useRef, useEffect } from 'react';
|
||||
import { useDispatch, useSelector } from 'react-redux';
|
||||
import { useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import type { QueryEditor, SqlLabRootState, Table } from 'src/SqlLab/types';
|
||||
import {
|
||||
ButtonGroup,
|
||||
@@ -75,7 +76,7 @@ const Fade = styled.div`
|
||||
const TableElement = ({ table, ...props }: TableElementProps) => {
|
||||
const { dbId, catalog, schema, name, expanded, id } = table;
|
||||
const theme = useTheme();
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const {
|
||||
currentData: tableMetadata,
|
||||
isSuccess: isMetadataSuccess,
|
||||
|
||||
@@ -25,7 +25,8 @@ import {
|
||||
type ChangeEvent,
|
||||
useMemo,
|
||||
} from 'react';
|
||||
import { useSelector, useDispatch, shallowEqual } from 'react-redux';
|
||||
import { useSelector, shallowEqual } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { styled, css, useTheme } from '@apache-superset/core/theme';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import AutoSizer from 'react-virtualized-auto-sizer';
|
||||
@@ -163,7 +164,7 @@ const savePinnedSchemasToStorage = (
|
||||
};
|
||||
|
||||
const TableExploreTree: React.FC<Props> = ({ queryEditorId }) => {
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const theme = useTheme();
|
||||
const treeRef = useRef<TreeApi<TreeNodeData>>(null);
|
||||
const tables = useSelector(
|
||||
|
||||
@@ -17,7 +17,7 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { useMemo, useReducer, useCallback } from 'react';
|
||||
import { useDispatch } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import {
|
||||
Table,
|
||||
@@ -130,7 +130,7 @@ const useTreeData = ({
|
||||
catalog,
|
||||
pinnedTables,
|
||||
}: UseTreeDataParams): UseTreeDataResult => {
|
||||
const reduxDispatch = useDispatch();
|
||||
const reduxDispatch = useAppDispatch();
|
||||
// Schema data from API
|
||||
const {
|
||||
currentData: schemaData,
|
||||
|
||||
@@ -17,7 +17,8 @@
|
||||
* under the License.
|
||||
*/
|
||||
import { type FC, useCallback, useMemo, useRef, useState } from 'react';
|
||||
import { shallowEqual, useDispatch, useSelector } from 'react-redux';
|
||||
import { shallowEqual, useSelector } from 'react-redux';
|
||||
import { useAppDispatch } from 'src/views/store';
|
||||
import { nanoid } from 'nanoid';
|
||||
import { t } from '@apache-superset/core/translation';
|
||||
import { ClientErrorObject, getExtensionsRegistry } from '@superset-ui/core';
|
||||
@@ -110,7 +111,7 @@ const renderWell = (partitions: TableMetaData['partitions']) => {
|
||||
};
|
||||
|
||||
const TablePreview: FC<Props> = ({ dbId, catalog, schema, tableName }) => {
|
||||
const dispatch = useDispatch();
|
||||
const dispatch = useAppDispatch();
|
||||
const theme = useTheme();
|
||||
const [databaseName, backend, disableDataPreview] = useSelector<
|
||||
SqlLabRootState,
|
||||
|
||||
@@ -22,12 +22,13 @@ import {
|
||||
createListenerMiddleware,
|
||||
StoreEnhancer,
|
||||
} from '@reduxjs/toolkit';
|
||||
import type { AnyAction } from 'redux';
|
||||
import {
|
||||
useDispatch,
|
||||
useSelector,
|
||||
type TypedUseSelectorHook,
|
||||
} from 'react-redux';
|
||||
import thunk from 'redux-thunk';
|
||||
import thunk, { type ThunkDispatch } from 'redux-thunk';
|
||||
import { api } from 'src/hooks/apiResources/queryApi';
|
||||
import messageToastReducer from 'src/components/MessageToasts/reducers';
|
||||
import charts from 'src/components/Chart/chartReducer';
|
||||
@@ -188,6 +189,14 @@ export type RootState = ReturnType<typeof store.getState>;
|
||||
// thunks resolve correctly), and `useAppSelector` infers `RootState` without
|
||||
// callers having to annotate every selector. Required ahead of the
|
||||
// react-redux v8+ bump, which tightens dispatch typing — see #39927.
|
||||
export type AppDispatch = typeof store.dispatch;
|
||||
//
|
||||
// AppDispatch is declared as ThunkDispatch & store.dispatch rather than
|
||||
// `typeof store.dispatch` because Superset annotates getMiddleware as
|
||||
// ConfigureStoreOptions['middleware'], which erases the middleware tuple type
|
||||
// and leaves store.dispatch typed as Dispatch<AnyAction>. The intersection
|
||||
// restores thunk support without requiring a wider refactor of the middleware
|
||||
// setup.
|
||||
export type AppDispatch = ThunkDispatch<RootState, undefined, AnyAction> &
|
||||
typeof store.dispatch;
|
||||
export const useAppDispatch: () => AppDispatch = useDispatch;
|
||||
export const useAppSelector: TypedUseSelectorHook<RootState> = useSelector;
|
||||
|
||||
@@ -62,6 +62,21 @@ def build_uuid_to_id_map(position: dict[str, Any]) -> dict[str, int]:
|
||||
}
|
||||
|
||||
|
||||
def _remap_charts_in_scope(container: dict[str, Any], id_map: dict[int, int]) -> None:
|
||||
"""Remap source-env chart IDs in ``container["chartsInScope"]`` in place.
|
||||
|
||||
``chartsInScope`` is a denormalized cache of the charts a filter (native
|
||||
or cross-filter) currently applies to. Both surfaces share this contract,
|
||||
so they share this remap. Unresolvable IDs are dropped rather than
|
||||
passed through, matching the convention used for ``scope.excluded``.
|
||||
"""
|
||||
charts_in_scope = container.get("chartsInScope")
|
||||
if isinstance(charts_in_scope, list):
|
||||
container["chartsInScope"] = [
|
||||
id_map[old_id] for old_id in charts_in_scope if old_id in id_map
|
||||
]
|
||||
|
||||
|
||||
def update_id_refs( # pylint: disable=too-many-locals # noqa: C901
|
||||
config: dict[str, Any],
|
||||
chart_ids: dict[str, int],
|
||||
@@ -145,6 +160,8 @@ def update_id_refs( # pylint: disable=too-many-locals # noqa: C901
|
||||
id_map[old_id] for old_id in scope_excluded if old_id in id_map
|
||||
]
|
||||
|
||||
_remap_charts_in_scope(native_filter, id_map)
|
||||
|
||||
# fix display control dataset references
|
||||
for customization in (
|
||||
fixed.get("metadata", {}).get("chart_customization_config") or []
|
||||
@@ -170,7 +187,7 @@ def update_id_refs( # pylint: disable=too-many-locals # noqa: C901
|
||||
return fixed
|
||||
|
||||
|
||||
def update_cross_filter_scoping(
|
||||
def update_cross_filter_scoping( # noqa: C901
|
||||
config: dict[str, Any], id_map: dict[int, int]
|
||||
) -> dict[str, Any]:
|
||||
# fix cross filter references
|
||||
@@ -185,6 +202,9 @@ def update_cross_filter_scoping(
|
||||
id_map[old_id] for old_id in scope_excluded if old_id in id_map
|
||||
]
|
||||
|
||||
# Global cross-filter chartsInScope mirrors the native-filter case.
|
||||
_remap_charts_in_scope(cross_filter_global_config, id_map)
|
||||
|
||||
if "chart_configuration" in (metadata := fixed.get("metadata", {})):
|
||||
# Build remapped configuration in a single pass for clarity/readability.
|
||||
new_chart_configuration: dict[str, Any] = {}
|
||||
@@ -212,6 +232,11 @@ def update_cross_filter_scoping(
|
||||
if old_id in id_map
|
||||
]
|
||||
|
||||
# Cross-filter chartsInScope mirrors the native-filter case.
|
||||
cross_filters = chart_config.get("crossFilters")
|
||||
if isinstance(cross_filters, dict):
|
||||
_remap_charts_in_scope(cross_filters, id_map)
|
||||
|
||||
new_chart_configuration[str(new_id)] = chart_config
|
||||
|
||||
metadata["chart_configuration"] = new_chart_configuration
|
||||
|
||||
@@ -238,6 +238,136 @@ def test_update_native_filter_config_default_rootpath_preserved():
|
||||
assert scope["excluded"] == []
|
||||
|
||||
|
||||
def test_update_id_refs_remaps_charts_in_scope():
|
||||
"""
|
||||
Regression for #26338: ``chartsInScope`` on a native filter holds chart
|
||||
IDs and must be remapped from source-env IDs to destination-env IDs
|
||||
during import.
|
||||
|
||||
The export side already converts ``chartsInScope`` IDs to UUIDs (see
|
||||
``export_example.py:325``). The import side must symmetrically convert
|
||||
them back to the destination environment's chart IDs. Without that
|
||||
remap, the field carries stale source IDs into the imported dashboard
|
||||
and breaks ``filtersInScope`` / ``filtersOutScope`` computation —
|
||||
filters end up applied to the wrong charts (or none at all).
|
||||
"""
|
||||
from superset.commands.dashboard.importers.v1.utils import update_id_refs
|
||||
|
||||
config: dict[str, Any] = {
|
||||
"position": {
|
||||
"CHART1": {
|
||||
"id": "CHART1",
|
||||
"meta": {"chartId": 101, "uuid": "uuid1"},
|
||||
"type": "CHART",
|
||||
},
|
||||
"CHART2": {
|
||||
"id": "CHART2",
|
||||
"meta": {"chartId": 102, "uuid": "uuid2"},
|
||||
"type": "CHART",
|
||||
},
|
||||
},
|
||||
"metadata": {
|
||||
"native_filter_configuration": [
|
||||
{
|
||||
"id": "NATIVE_FILTER-region",
|
||||
"scope": {"rootPath": ["ROOT_ID"], "excluded": []},
|
||||
# chartsInScope contains source-env chart IDs.
|
||||
"chartsInScope": [101, 102, 103],
|
||||
}
|
||||
],
|
||||
},
|
||||
}
|
||||
chart_ids = {"uuid1": 1, "uuid2": 2}
|
||||
dataset_info: dict[str, dict[str, Any]] = {}
|
||||
|
||||
fixed = update_id_refs(config, chart_ids, dataset_info)
|
||||
filter_config = fixed["metadata"]["native_filter_configuration"][0]
|
||||
|
||||
# Resolved IDs are remapped; unknown IDs (103) are dropped rather than
|
||||
# left to silently bind to whatever chart owns that integer in the
|
||||
# destination environment.
|
||||
assert filter_config["chartsInScope"] == [1, 2]
|
||||
|
||||
|
||||
def test_update_id_refs_remaps_cross_filter_charts_in_scope():
|
||||
"""
|
||||
Companion to test_update_id_refs_remaps_charts_in_scope. Cross-filter
|
||||
config also stores ``chartsInScope`` (under ``crossFilters`` per chart)
|
||||
and must be remapped on import for the same reason.
|
||||
"""
|
||||
from superset.commands.dashboard.importers.v1.utils import update_id_refs
|
||||
|
||||
config: dict[str, Any] = {
|
||||
"position": {
|
||||
"CHART1": {
|
||||
"id": "CHART1",
|
||||
"meta": {"chartId": 101, "uuid": "uuid1"},
|
||||
"type": "CHART",
|
||||
},
|
||||
"CHART2": {
|
||||
"id": "CHART2",
|
||||
"meta": {"chartId": 102, "uuid": "uuid2"},
|
||||
"type": "CHART",
|
||||
},
|
||||
},
|
||||
"metadata": {
|
||||
"chart_configuration": {
|
||||
"101": {
|
||||
"id": 101,
|
||||
"crossFilters": {
|
||||
"scope": {"rootPath": ["ROOT_ID"], "excluded": []},
|
||||
"chartsInScope": [101, 102, 103],
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
chart_ids = {"uuid1": 1, "uuid2": 2}
|
||||
dataset_info: dict[str, dict[str, Any]] = {}
|
||||
|
||||
fixed = update_id_refs(config, chart_ids, dataset_info)
|
||||
cross_filters = fixed["metadata"]["chart_configuration"]["1"]["crossFilters"]
|
||||
|
||||
assert cross_filters["chartsInScope"] == [1, 2]
|
||||
|
||||
|
||||
def test_update_id_refs_remaps_global_chart_configuration_charts_in_scope():
|
||||
"""
|
||||
Per-chart and native-filter ``chartsInScope`` are remapped by their own
|
||||
branches; ``global_chart_configuration.chartsInScope`` lives next to
|
||||
``global_chart_configuration.scope.excluded`` and needs the same treatment
|
||||
so the global cross-filter scope cache doesn't keep stale source-env IDs.
|
||||
"""
|
||||
from superset.commands.dashboard.importers.v1.utils import update_id_refs
|
||||
|
||||
config: dict[str, Any] = {
|
||||
"position": {
|
||||
"CHART1": {
|
||||
"id": "CHART1",
|
||||
"meta": {"chartId": 101, "uuid": "uuid1"},
|
||||
"type": "CHART",
|
||||
},
|
||||
"CHART2": {
|
||||
"id": "CHART2",
|
||||
"meta": {"chartId": 102, "uuid": "uuid2"},
|
||||
"type": "CHART",
|
||||
},
|
||||
},
|
||||
"metadata": {
|
||||
"global_chart_configuration": {
|
||||
"scope": {"rootPath": ["ROOT_ID"], "excluded": []},
|
||||
"chartsInScope": [101, 102, 103],
|
||||
},
|
||||
},
|
||||
}
|
||||
chart_ids = {"uuid1": 1, "uuid2": 2}
|
||||
dataset_info: dict[str, dict[str, Any]] = {}
|
||||
|
||||
fixed = update_id_refs(config, chart_ids, dataset_info)
|
||||
|
||||
assert fixed["metadata"]["global_chart_configuration"]["chartsInScope"] == [1, 2]
|
||||
|
||||
|
||||
def test_update_id_refs_cross_filter_chart_configuration_key_and_excluded_mapping():
|
||||
from superset.commands.dashboard.importers.v1.utils import update_id_refs
|
||||
|
||||
|
||||
58
tests/unit_tests/semantic_layers/types_test.py
Normal file
58
tests/unit_tests/semantic_layers/types_test.py
Normal file
@@ -0,0 +1,58 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pyarrow as pa
|
||||
from superset_core.semantic_layers.types import AggregationType, Metric
|
||||
|
||||
|
||||
def test_metric_aggregation_defaults_to_none() -> None:
|
||||
metric = Metric(
|
||||
id="x",
|
||||
name="x",
|
||||
type=pa.float64(),
|
||||
definition="sum(x)",
|
||||
)
|
||||
assert metric.aggregation is None
|
||||
|
||||
|
||||
def test_metric_accepts_aggregation_type() -> None:
|
||||
metric = Metric(
|
||||
id="x",
|
||||
name="x",
|
||||
type=pa.float64(),
|
||||
definition="sum(x)",
|
||||
aggregation=AggregationType.SUM,
|
||||
)
|
||||
assert metric.aggregation is AggregationType.SUM
|
||||
|
||||
|
||||
def test_aggregation_type_is_string_enum() -> None:
|
||||
# Behaves as a string for equality and serialization, so it can be sent
|
||||
# over JSON without an explicit converter.
|
||||
assert AggregationType.SUM == "SUM"
|
||||
assert AggregationType.COUNT_DISTINCT.value == "COUNT_DISTINCT"
|
||||
assert {a.value for a in AggregationType} == {
|
||||
"SUM",
|
||||
"COUNT",
|
||||
"MIN",
|
||||
"MAX",
|
||||
"AVG",
|
||||
"COUNT_DISTINCT",
|
||||
"OTHER",
|
||||
}
|
||||
Reference in New Issue
Block a user