mirror of
https://github.com/apache/superset.git
synced 2026-04-07 18:35:15 +00:00
docs: bifurcate documentation into user and admin sections (#38196)
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
Evan Rusackas
239
docs/developer_docs/extensions/architecture.md
Normal file
239
docs/developer_docs/extensions/architecture.md
Normal file
@@ -0,0 +1,239 @@
|
||||
---
|
||||
title: Architecture
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Architecture
|
||||
|
||||
Apache Superset's extension system is designed to enable powerful customization while maintaining stability, security, and performance. This page explains the architectural principles, system design, and technical mechanisms that make the extension ecosystem possible.
|
||||
|
||||
## Architectural Principles
|
||||
|
||||
The extension architecture is built on six core principles that guide all technical decisions and ensure extensions can be developed safely and predictably:
|
||||
|
||||
### 1. Lean Core
|
||||
|
||||
Superset's core should remain minimal, with many features delegated to extensions. Built-in features use the same APIs and extension mechanisms available to external developers. This approach:
|
||||
- Reduces maintenance burden and complexity
|
||||
- Encourages modularity
|
||||
- Allows the community to innovate independently of the main codebase
|
||||
|
||||
### 2. Explicit Contribution Points
|
||||
|
||||
All extension points are clearly defined and documented. Extension authors know exactly where and how they can interact with the host system. Each extension declares its capabilities in a metadata file, enabling the host to:
|
||||
- Manage the extension lifecycle
|
||||
- Provide a consistent user experience
|
||||
- Validate extension compatibility
|
||||
|
||||
### 3. Versioned and Stable APIs
|
||||
|
||||
Public interfaces for extensions follow semantic versioning, allowing for:
|
||||
- Safe evolution of the platform
|
||||
- Backward compatibility
|
||||
- Clear upgrade paths for extension authors
|
||||
|
||||
### 4. Lazy Loading and Activation
|
||||
|
||||
Extensions are loaded and activated only when needed, which:
|
||||
- Minimizes performance overhead
|
||||
- Reduces resource consumption
|
||||
- Improves startup time
|
||||
|
||||
### 5. Composability and Reuse
|
||||
|
||||
The architecture encourages reusing extension points and patterns across different modules, promoting:
|
||||
- Consistency across extensions
|
||||
- Reduced duplication
|
||||
- Shared best practices
|
||||
|
||||
### 6. Community-Driven Evolution
|
||||
|
||||
The system evolves based on real-world feedback and contributions. New extension points and capabilities are added as needs emerge, ensuring the platform remains relevant and flexible.
|
||||
|
||||
## System Overview
|
||||
|
||||
The extension architecture is built around three main components that work together to create a flexible, maintainable ecosystem:
|
||||
|
||||
### Core Packages
|
||||
|
||||
Two core packages provide the foundation for extension development:
|
||||
|
||||
**Frontend: `@apache-superset/core`**
|
||||
|
||||
This package provides essential building blocks for frontend extensions and the host application:
|
||||
- Shared UI components
|
||||
- Utility functions
|
||||
- APIs and hooks
|
||||
- Type definitions
|
||||
|
||||
By centralizing these resources, both extensions and built-in features use the same APIs, ensuring consistency, type safety, and a seamless user experience. The package is versioned to support safe platform evolution while maintaining compatibility.
|
||||
|
||||
**Backend: `apache-superset-core`**
|
||||
|
||||
This package exposes key classes and APIs for backend extensions:
|
||||
- Database connectors
|
||||
- API extensions
|
||||
- Security manager customization
|
||||
- Core utilities and models
|
||||
|
||||
It includes dependencies on critical libraries like Flask-AppBuilder and SQLAlchemy, and follows semantic versioning for compatibility and stability.
|
||||
|
||||
### Developer Tools
|
||||
|
||||
**`apache-superset-extensions-cli`**
|
||||
|
||||
The CLI provides comprehensive commands for extension development:
|
||||
- Project scaffolding
|
||||
- Code generation
|
||||
- Building and bundling
|
||||
- Packaging for distribution
|
||||
|
||||
By standardizing these processes, the CLI ensures extensions are built consistently, remain compatible with evolving versions of Superset, and follow best practices.
|
||||
|
||||
### Host Application
|
||||
|
||||
The Superset host application serves as the runtime environment for extensions:
|
||||
|
||||
**Extension Management**
|
||||
- Exposes `/api/v1/extensions` endpoint for registration and management
|
||||
- Provides a dedicated UI for managing extensions
|
||||
- Stores extension metadata in the `extensions` database table
|
||||
|
||||
**Extension Storage**
|
||||
|
||||
The extensions table contains:
|
||||
- Extension name, version, and author
|
||||
- Contributed features and exposed modules
|
||||
- Metadata and configuration
|
||||
- Built frontend and/or backend code
|
||||
|
||||
### Architecture Diagram
|
||||
|
||||
The following diagram illustrates how these components work together:
|
||||
|
||||
<img width="955" height="586" alt="Extension System Architecture" src="https://github.com/user-attachments/assets/cc2a41df-55a4-48c8-b056-35f7a1e567c6" />
|
||||
|
||||
The diagram shows:
|
||||
1. **Extension projects** depend on core packages for development
|
||||
2. **Core packages** provide APIs and type definitions
|
||||
3. **The host application** implements the APIs and manages extensions
|
||||
4. **Extensions** integrate seamlessly with the host through well-defined interfaces
|
||||
|
||||
## Dynamic Module Loading
|
||||
|
||||
One of the most sophisticated aspects of the extension architecture is how frontend code is dynamically loaded at runtime using Webpack's Module Federation.
|
||||
|
||||
### Module Federation
|
||||
|
||||
The architecture leverages Webpack's Module Federation to enable dynamic loading of frontend assets. This allows extensions to be built independently from Superset.
|
||||
|
||||
### How It Works
|
||||
|
||||
**Extension Configuration**
|
||||
|
||||
Extensions configure Webpack to expose their entry points:
|
||||
|
||||
``` typescript
|
||||
new ModuleFederationPlugin({
|
||||
name: 'my_extension',
|
||||
filename: 'remoteEntry.[contenthash].js',
|
||||
exposes: {
|
||||
'./index': './src/index.tsx',
|
||||
},
|
||||
externalsType: 'window',
|
||||
externals: {
|
||||
'@apache-superset/core': 'superset',
|
||||
},
|
||||
shared: {
|
||||
react: { singleton: true },
|
||||
'react-dom': { singleton: true },
|
||||
'antd-v5': { singleton: true }
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
This configuration does several important things:
|
||||
|
||||
**`exposes`** - Declares which modules are available to the host application. The extension makes `./index` available as its entry point.
|
||||
|
||||
**`externals` and `externalsType`** - Tell Webpack that when the extension imports `@apache-superset/core`, it should use `window.superset` at runtime instead of bundling its own copy. This ensures extensions use the host's implementation of shared packages.
|
||||
|
||||
**`shared`** - Prevents duplication of common libraries like React and Ant Design. The `singleton: true` setting ensures only one instance of each library exists, avoiding version conflicts and reducing bundle size.
|
||||
|
||||
### Runtime Resolution
|
||||
|
||||
The following diagram illustrates the module loading process:
|
||||
|
||||
<img width="913" height="558" alt="Module Federation Flow" src="https://github.com/user-attachments/assets/e5e4d2ae-e8b5-4d17-a2a1-3667c65f25ca" />
|
||||
|
||||
Here's what happens at runtime:
|
||||
|
||||
1. **Extension Registration**: When an extension is registered, Superset stores its remote entry URL
|
||||
2. **Dynamic Loading**: When the extension is activated, the host fetches the remote entry file
|
||||
3. **Module Resolution**: The extension imports `@apache-superset/core`, which resolves to `window.superset`
|
||||
4. **Execution**: The extension code runs with access to the host's APIs and shared dependencies
|
||||
|
||||
### Host API Setup
|
||||
|
||||
On the Superset side, the APIs are mapped to `window.superset` during application bootstrap:
|
||||
|
||||
``` typescript
|
||||
import * as supersetCore from '@apache-superset/core';
|
||||
import {
|
||||
authentication,
|
||||
core,
|
||||
commands,
|
||||
extensions,
|
||||
sqlLab,
|
||||
} from 'src/extensions';
|
||||
|
||||
export default function setupExtensionsAPI() {
|
||||
window.superset = {
|
||||
...supersetCore,
|
||||
authentication,
|
||||
core,
|
||||
commands,
|
||||
extensions,
|
||||
sqlLab,
|
||||
};
|
||||
}
|
||||
```
|
||||
|
||||
This function runs before any extensions are loaded, ensuring the APIs are available when extensions import from `@apache-superset/core`.
|
||||
|
||||
### Benefits
|
||||
|
||||
This architecture provides several key benefits:
|
||||
|
||||
- **Independent development**: Extensions can be built separately from Superset's codebase
|
||||
- **Version isolation**: Each extension can be developed with its own release cycle
|
||||
- **Shared dependencies**: Common libraries are shared, reducing memory usage and bundle size
|
||||
- **Type safety**: TypeScript types flow from the core package to extensions
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you understand the architecture, explore:
|
||||
|
||||
- **[Dependencies](./dependencies)** - Managing dependencies and understanding API stability
|
||||
- **[Quick Start](./quick-start)** - Build your first extension
|
||||
- **[Contribution Types](./contribution-types)** - What kinds of extensions you can build
|
||||
- **[Development](./development)** - Project structure, APIs, and development workflow
|
||||
131
docs/developer_docs/extensions/components/alert.mdx
Normal file
131
docs/developer_docs/extensions/components/alert.mdx
Normal file
@@ -0,0 +1,131 @@
|
||||
---
|
||||
title: Alert
|
||||
sidebar_label: Alert
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
import { StoryWithControls } from '../../../src/components/StorybookWrapper';
|
||||
import { Alert } from '@apache-superset/core/ui';
|
||||
|
||||
# Alert
|
||||
|
||||
Alert component for displaying important messages to users. Wraps Ant Design Alert with sensible defaults and improved accessibility.
|
||||
|
||||
## Live Example
|
||||
|
||||
<StoryWithControls
|
||||
component={Alert}
|
||||
props={{
|
||||
closable: true,
|
||||
type: 'info',
|
||||
message: 'This is a sample alert message.',
|
||||
description: 'Sample description for additional context.',
|
||||
showIcon: true
|
||||
}}
|
||||
controls={[
|
||||
{
|
||||
name: 'type',
|
||||
label: 'Type',
|
||||
type: 'select',
|
||||
options: [
|
||||
'info',
|
||||
'error',
|
||||
'warning',
|
||||
'success'
|
||||
]
|
||||
},
|
||||
{
|
||||
name: 'closable',
|
||||
label: 'Closable',
|
||||
type: 'boolean'
|
||||
},
|
||||
{
|
||||
name: 'showIcon',
|
||||
label: 'Show Icon',
|
||||
type: 'boolean'
|
||||
},
|
||||
{
|
||||
name: 'message',
|
||||
label: 'Message',
|
||||
type: 'text'
|
||||
},
|
||||
{
|
||||
name: 'description',
|
||||
label: 'Description',
|
||||
type: 'text'
|
||||
}
|
||||
]}
|
||||
/>
|
||||
|
||||
## Try It
|
||||
|
||||
Edit the code below to experiment with the component:
|
||||
|
||||
```tsx live
|
||||
function Demo() {
|
||||
return (
|
||||
<Alert
|
||||
closable
|
||||
type="info"
|
||||
message="This is a sample alert message."
|
||||
description="Sample description for additional context."
|
||||
showIcon
|
||||
/>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Props
|
||||
|
||||
| Prop | Type | Default | Description |
|
||||
|------|------|---------|-------------|
|
||||
| `closable` | `boolean` | `true` | Whether the Alert can be closed with a close button. |
|
||||
| `type` | `string` | `"info"` | Type of the alert (e.g., info, error, warning, success). |
|
||||
| `message` | `string` | `"This is a sample alert message."` | Message |
|
||||
| `description` | `string` | `"Sample description for additional context."` | Description |
|
||||
| `showIcon` | `boolean` | `true` | Whether to display an icon in the Alert. |
|
||||
|
||||
## Usage in Extensions
|
||||
|
||||
This component is available in the `@apache-superset/core/ui` package, which is automatically available to Superset extensions.
|
||||
|
||||
```tsx
|
||||
import { Alert } from '@apache-superset/core/ui';
|
||||
|
||||
function MyExtension() {
|
||||
return (
|
||||
<Alert
|
||||
closable
|
||||
type="info"
|
||||
message="This is a sample alert message."
|
||||
/>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Source Links
|
||||
|
||||
- [Story file](https://github.com/apache/superset/blob/master/superset-frontend/packages/superset-core/src/ui/components/Alert/Alert.stories.tsx)
|
||||
- [Component source](https://github.com/apache/superset/blob/master/superset-frontend/packages/superset-core/src/ui/components/Alert/index.tsx)
|
||||
|
||||
---
|
||||
|
||||
*This page was auto-generated from the component's Storybook story.*
|
||||
93
docs/developer_docs/extensions/components/index.mdx
Normal file
93
docs/developer_docs/extensions/components/index.mdx
Normal file
@@ -0,0 +1,93 @@
|
||||
---
|
||||
title: Extension Components
|
||||
sidebar_label: Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Extension Components
|
||||
|
||||
These UI components are available to Superset extension developers through the `@apache-superset/core/ui` package. They provide a consistent look and feel with the rest of Superset and are designed to be used in extension panels, views, and other UI elements.
|
||||
|
||||
## Available Components
|
||||
|
||||
- [Alert](./alert)
|
||||
|
||||
## Usage
|
||||
|
||||
All components are exported from the `@apache-superset/core/ui` package:
|
||||
|
||||
```tsx
|
||||
import { Alert } from '@apache-superset/core/ui';
|
||||
|
||||
export function MyExtensionPanel() {
|
||||
return (
|
||||
<Alert type="info">
|
||||
Welcome to my extension!
|
||||
</Alert>
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
## Adding New Components
|
||||
|
||||
Components in `@apache-superset/core/ui` are automatically documented here. To add a new extension component:
|
||||
|
||||
1. Add the component to `superset-frontend/packages/superset-core/src/ui/components/`
|
||||
2. Export it from `superset-frontend/packages/superset-core/src/ui/components/index.ts`
|
||||
3. Create a Storybook story with an `Interactive` export:
|
||||
|
||||
```tsx
|
||||
export default {
|
||||
title: 'Extension Components/MyComponent',
|
||||
component: MyComponent,
|
||||
parameters: {
|
||||
docs: {
|
||||
description: {
|
||||
component: 'Description of the component...',
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
export const InteractiveMyComponent = (args) => <MyComponent {...args} />;
|
||||
|
||||
InteractiveMyComponent.args = {
|
||||
variant: 'primary',
|
||||
disabled: false,
|
||||
};
|
||||
|
||||
InteractiveMyComponent.argTypes = {
|
||||
variant: {
|
||||
control: { type: 'select' },
|
||||
options: ['primary', 'secondary'],
|
||||
},
|
||||
disabled: {
|
||||
control: { type: 'boolean' },
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
4. Run `yarn start` in `docs/` - the page generates automatically!
|
||||
|
||||
## Interactive Documentation
|
||||
|
||||
For interactive examples with controls, visit the [Storybook](/storybook/?path=/docs/extension-components--docs).
|
||||
155
docs/developer_docs/extensions/contribution-types.md
Normal file
155
docs/developer_docs/extensions/contribution-types.md
Normal file
@@ -0,0 +1,155 @@
|
||||
---
|
||||
title: Contribution Types
|
||||
sidebar_position: 5
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Contribution Types
|
||||
|
||||
To facilitate the development of extensions, we define a set of well-defined contribution types that extensions can implement. These contribution types serve as the building blocks for extensions, allowing them to interact with the host application and provide new functionality.
|
||||
|
||||
## Frontend
|
||||
|
||||
Frontend contribution types allow extensions to extend Superset's user interface with new views, commands, and menu items.
|
||||
|
||||
### Views
|
||||
|
||||
Extensions can add new views or panels to the host application, such as custom SQL Lab panels, dashboards, or other UI components. Each view is registered with a unique ID and can be activated or deactivated as needed. Contribution areas are uniquely identified (e.g., `sqllab.panels` for SQL Lab panels), enabling seamless integration into specific parts of the application.
|
||||
|
||||
```json
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"views": {
|
||||
"sqllab": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "my_extension.main",
|
||||
"name": "My Panel Name"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Commands
|
||||
|
||||
Extensions can define custom commands that can be executed within the host application, such as context-aware actions or menu options. Each command can specify properties like a unique command identifier, an icon, a title, and a description. These commands can be invoked by users through menus, keyboard shortcuts, or other UI elements, enabling extensions to add rich, interactive functionality to Superset.
|
||||
|
||||
```json
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"commands": [
|
||||
{
|
||||
"command": "my_extension.copy_query",
|
||||
"icon": "CopyOutlined",
|
||||
"title": "Copy Query",
|
||||
"description": "Copy the current query to clipboard"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Menus
|
||||
|
||||
Extensions can contribute new menu items or context menus to the host application, providing users with additional actions and options. Each menu item can specify properties such as the target view, the command to execute, its placement (primary, secondary, or context), and conditions for when it should be displayed. Menu contribution areas are uniquely identified (e.g., `sqllab.editor` for the SQL Lab editor), allowing extensions to seamlessly integrate their functionality into specific menus and workflows within Superset.
|
||||
|
||||
```json
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"menus": {
|
||||
"sqllab": {
|
||||
"editor": {
|
||||
"primary": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "my_extension.copy_query"
|
||||
}
|
||||
],
|
||||
"secondary": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "my_extension.prettify"
|
||||
}
|
||||
],
|
||||
"context": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "my_extension.clear"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Editors
|
||||
|
||||
Extensions can replace Superset's default text editors with custom implementations. This enables enhanced editing experiences using alternative editor frameworks like Monaco, CodeMirror, or custom solutions. When an extension registers an editor for a language, it replaces the default Ace editor in all locations that use that language (SQL Lab, Dashboard Properties, CSS editors, etc.).
|
||||
|
||||
```json
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"editors": [
|
||||
{
|
||||
"id": "my_extension.monaco_sql",
|
||||
"name": "Monaco SQL Editor",
|
||||
"languages": ["sql"],
|
||||
"description": "Monaco-based SQL editor with IntelliSense"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
See [Editors Extension Point](./extension-points/editors) for implementation details.
|
||||
|
||||
## Backend
|
||||
|
||||
Backend contribution types allow extensions to extend Superset's server-side capabilities with new API endpoints, MCP tools, and MCP prompts.
|
||||
|
||||
### REST API Endpoints
|
||||
|
||||
Extensions can register custom REST API endpoints under the `/api/v1/extensions/` namespace. This dedicated namespace prevents conflicts with built-in endpoints and provides a clear separation between core and extension functionality.
|
||||
|
||||
```json
|
||||
"backend": {
|
||||
"entryPoints": ["my_extension.entrypoint"],
|
||||
"files": ["backend/src/my_extension/**/*.py"]
|
||||
}
|
||||
```
|
||||
|
||||
The entry point module registers the API with Superset:
|
||||
|
||||
```python
|
||||
from superset_core.api.rest_api import add_extension_api
|
||||
from .api import MyExtensionAPI
|
||||
|
||||
add_extension_api(MyExtensionAPI)
|
||||
```
|
||||
|
||||
### MCP Tools and Prompts
|
||||
|
||||
Extensions can contribute Model Context Protocol (MCP) tools and prompts that AI agents can discover and use. See [MCP Integration](./mcp) for detailed documentation.
|
||||
166
docs/developer_docs/extensions/dependencies.md
Normal file
166
docs/developer_docs/extensions/dependencies.md
Normal file
@@ -0,0 +1,166 @@
|
||||
---
|
||||
title: Dependencies
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Dependencies
|
||||
|
||||
This guide explains how to manage dependencies in your Superset extensions, including the difference between public APIs and internal code, and best practices for maintaining stable extensions.
|
||||
|
||||
## Core Packages vs Internal Code
|
||||
|
||||
Extensions run in the same context as Superset during runtime. This means extension developers can technically import any module from the Superset codebase, not just the public APIs. Understanding the distinction between public and internal code is critical for building maintainable extensions.
|
||||
|
||||
### Public APIs (Stable)
|
||||
|
||||
The core packages follow [semantic versioning](https://semver.org/) and provide stable, documented APIs:
|
||||
|
||||
| Package | Language | Description |
|
||||
|---------|----------|-------------|
|
||||
| `@apache-superset/core` | JavaScript/TypeScript | Frontend APIs, UI components, hooks, and utilities |
|
||||
| `apache-superset-core` | Python | Backend APIs, models, DAOs, and utilities |
|
||||
|
||||
**Benefits of using core packages:**
|
||||
|
||||
- **Semantic versioning**: Breaking changes are communicated through version numbers
|
||||
- **Documentation**: APIs are documented with clear usage examples
|
||||
- **Stability commitment**: We strive to maintain backward compatibility
|
||||
- **Type safety**: Full TypeScript and Python type definitions
|
||||
|
||||
### Internal Code (Unstable)
|
||||
|
||||
Any code that is not exported through the core packages is considered internal. This includes:
|
||||
|
||||
- Direct imports from `superset-frontend/src/` modules
|
||||
- Direct imports from `superset/` Python modules (outside of `superset_core`)
|
||||
- Undocumented functions, classes, or utilities
|
||||
|
||||
:::warning Use at Your Own Risk
|
||||
Internal code can change at any time without notice. If you depend on internal modules, your extension may break when Superset is upgraded. There is no guarantee of backward compatibility for internal code.
|
||||
:::
|
||||
|
||||
**Example of internal vs public imports:**
|
||||
|
||||
```typescript
|
||||
// ✅ Public API - stable
|
||||
import { Button, sqlLab } from '@apache-superset/core';
|
||||
|
||||
// ❌ Internal code - may break without notice
|
||||
import { someInternalFunction } from 'src/explore/components/SomeComponent';
|
||||
```
|
||||
|
||||
```python
|
||||
# ✅ Public API - stable
|
||||
from superset_core.api.models import Database
|
||||
from superset_core.api.daos import DatabaseDAO
|
||||
|
||||
# ❌ Internal code - may break without notice
|
||||
from superset.views.core import SomeInternalClass
|
||||
```
|
||||
|
||||
## API Evolution
|
||||
|
||||
The core packages are still evolving. While we follow semantic versioning, the APIs may change as we add new extension points and refine existing ones based on community feedback.
|
||||
|
||||
**What this means for extension developers:**
|
||||
|
||||
- Check the release notes when upgrading Superset
|
||||
- Test your extensions against new Superset versions before deploying
|
||||
- Participate in discussions about API changes to influence the direction
|
||||
- In some cases, using internal dependencies may be acceptable while the public API is being developed for your use case
|
||||
|
||||
### When Internal Dependencies May Be Acceptable
|
||||
|
||||
While public APIs are always preferred, there are situations where using internal code may be reasonable:
|
||||
|
||||
1. **Missing functionality**: The public API doesn't yet expose what you need
|
||||
2. **Prototype/experimental extensions**: You're exploring capabilities before committing to a stable implementation
|
||||
3. **Bridge period**: You need functionality that's planned for the public API but not yet released
|
||||
|
||||
In these cases, document your internal dependencies clearly and plan to migrate to public APIs when they become available.
|
||||
|
||||
## Core Library Dependencies
|
||||
|
||||
An important architectural principle of the Superset extension system is that **we do not provide abstractions on top of core dependencies** like React (frontend) or SQLAlchemy (backend).
|
||||
|
||||
### Why We Don't Abstract Core Libraries
|
||||
|
||||
Abstracting libraries like React or SQLAlchemy would:
|
||||
|
||||
- Create maintenance overhead keeping abstractions in sync with upstream
|
||||
- Limit access to the full power of these libraries
|
||||
- Add unnecessary abstraction layers
|
||||
- Fragment the ecosystem with Superset-specific variants
|
||||
|
||||
### Depending on Core Libraries Directly
|
||||
|
||||
Extension developers should depend on and use core libraries directly:
|
||||
|
||||
**Frontend (examples):**
|
||||
- [React](https://react.dev/) - UI framework
|
||||
- [Ant Design](https://ant.design/) - UI component library (prefer Superset components from `@apache-superset/core/ui` when available to preserve visual consistency)
|
||||
- [Emotion](https://emotion.sh/) - CSS-in-JS styling
|
||||
- ...
|
||||
|
||||
**Backend (examples):**
|
||||
- [SQLAlchemy](https://www.sqlalchemy.org/) - Database toolkit
|
||||
- [Flask](https://flask.palletsprojects.com/) - Web framework
|
||||
- [Flask-AppBuilder](https://flask-appbuilder.readthedocs.io/) - Application framework
|
||||
- ...
|
||||
|
||||
:::info Version Compatibility
|
||||
When Superset upgrades its core dependencies (e.g., a new major version of Ant Design or SQLAlchemy), extension developers should upgrade their extensions accordingly. This ensures compatibility and access to the latest features and security fixes.
|
||||
:::
|
||||
|
||||
## API Versioning and Changelog
|
||||
|
||||
Once the extensions API reaches **v1**, we will maintain a dedicated `CHANGELOG.md` file to track all changes to the public APIs. This will include:
|
||||
|
||||
- New APIs and features
|
||||
- Deprecation notices
|
||||
- Breaking changes with migration guides
|
||||
- Bug fixes affecting API behavior
|
||||
|
||||
Until then, monitor the Superset release notes and test your extensions with each new release.
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Do
|
||||
|
||||
- **Prefer public APIs**: Always check if functionality exists in `@apache-superset/core` or `apache-superset-core` before using internal code
|
||||
- **Pin versions**: Specify compatible Superset versions in your extension metadata
|
||||
- **Test upgrades**: Verify your extension works with new Superset releases before deploying
|
||||
- **Report missing APIs**: If you need functionality not in the public API, open a GitHub issue to request it
|
||||
- **Use core libraries directly**: Leverage Ant Design, SQLAlchemy, and other core libraries directly
|
||||
|
||||
### Don't
|
||||
|
||||
- **Assume stability of internal code**: Internal modules can change or be removed in any release
|
||||
- **Depend on implementation details**: Even if something works, it may not be supported
|
||||
- **Skip upgrade testing**: Always test your extension against new Superset versions
|
||||
- **Expect abstractions**: Use core dependencies directly rather than expecting Superset-specific abstractions
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[Architecture](./architecture)** - Understand the extension system design
|
||||
- **[Development](./development)** - Learn about APIs and development workflow
|
||||
- **[Quick Start](./quick-start)** - Build your first extension
|
||||
49
docs/developer_docs/extensions/deployment.md
Normal file
49
docs/developer_docs/extensions/deployment.md
Normal file
@@ -0,0 +1,49 @@
|
||||
---
|
||||
title: Deployment
|
||||
sidebar_position: 7
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Deployment
|
||||
|
||||
Once an extension has been developed, the deployment process involves packaging and uploading it to the host application.
|
||||
|
||||
Packaging is handled by the `superset-extensions bundle` command, which:
|
||||
|
||||
1. Builds frontend assets using Webpack (with Module Federation configuration).
|
||||
2. Collects backend Python source files and all necessary resources.
|
||||
3. Generates a `manifest.json` with build-time metadata, including the contents of `extension.json` and references to built assets.
|
||||
4. Packages everything into a `.supx` file (a zip archive with a specific structure required by Superset).
|
||||
|
||||
To deploy an extension, place the `.supx` file in the extensions directory configured via `EXTENSIONS_PATH` in your `superset_config.py`:
|
||||
|
||||
``` python
|
||||
EXTENSIONS_PATH = "/path/to/extensions"
|
||||
```
|
||||
|
||||
During application startup, Superset automatically discovers and loads all `.supx` files from this directory:
|
||||
|
||||
1. Scans the configured directory for `.supx` files.
|
||||
2. Validates each file is a properly formatted zip archive.
|
||||
3. Extracts and validates the extension manifest and metadata.
|
||||
4. Loads the extension, making it available for use.
|
||||
|
||||
This file-based approach simplifies deployment in containerized environments and enables version control of extensions alongside infrastructure configuration.
|
||||
317
docs/developer_docs/extensions/development.md
Normal file
317
docs/developer_docs/extensions/development.md
Normal file
@@ -0,0 +1,317 @@
|
||||
---
|
||||
title: Development
|
||||
sidebar_position: 6
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Development
|
||||
|
||||
This guide covers everything you need to know about developing extensions for Superset, from project structure to development workflow.
|
||||
|
||||
## Project Structure
|
||||
|
||||
The [apache-superset-extensions-cli](https://github.com/apache/superset/tree/master/superset-extensions-cli) package provides a command-line interface (CLI) that streamlines the extension development workflow. It offers the following commands:
|
||||
|
||||
```
|
||||
superset-extensions init: Generates the initial folder structure and scaffolds a new extension project.
|
||||
|
||||
superset-extensions build: Builds extension assets.
|
||||
|
||||
superset-extensions bundle: Packages the extension into a .supx file.
|
||||
|
||||
superset-extensions dev: Automatically rebuilds the extension as files change.
|
||||
```
|
||||
|
||||
When creating a new extension with `superset-extensions init`, the CLI generates a standardized folder structure:
|
||||
|
||||
```
|
||||
dataset-references/
|
||||
├── extension.json
|
||||
├── frontend/
|
||||
│ ├── src/
|
||||
│ ├── webpack.config.js
|
||||
│ ├── tsconfig.json
|
||||
│ └── package.json
|
||||
├── backend/
|
||||
│ ├── src/
|
||||
│ │ └── superset_extensions/
|
||||
│ │ └── dataset_references/
|
||||
│ ├── tests/
|
||||
│ ├── pyproject.toml
|
||||
│ └── requirements.txt
|
||||
├── dist/
|
||||
│ ├── manifest.json
|
||||
│ ├── frontend
|
||||
│ │ └── dist/
|
||||
│ │ ├── remoteEntry.d7a9225d042e4ccb6354.js
|
||||
│ │ └── 900.038b20cdff6d49cfa8d9.js
|
||||
│ └── backend
|
||||
│ └── superset_extensions/
|
||||
│ └── dataset_references/
|
||||
│ ├── __init__.py
|
||||
│ ├── api.py
|
||||
│ └── entrypoint.py
|
||||
├── dataset-references-1.0.0.supx
|
||||
└── README.md
|
||||
```
|
||||
|
||||
**Note**: The extension ID (`dataset-references`) serves as the basis for all technical names:
|
||||
- Directory name: `dataset-references` (kebab-case)
|
||||
- Backend Python package: `dataset_references` (snake_case)
|
||||
- Frontend package name: `dataset-references` (kebab-case)
|
||||
- Module Federation name: `datasetReferences` (camelCase)
|
||||
|
||||
The `extension.json` file serves as the declared metadata for the extension, containing the extension's name, version, author, description, and a list of capabilities. This file is essential for the host application to understand how to load and manage the extension.
|
||||
|
||||
The `frontend` directory contains the source code for the frontend components of the extension, including React components, styles, and assets. The `webpack.config.js` file is used to configure Webpack for building the frontend code, while the `tsconfig.json` file defines the TypeScript configuration for the project. The `package.json` file specifies the dependencies and scripts for building and testing the frontend code.
|
||||
|
||||
The `backend` directory contains the source code for the backend components of the extension, including Python modules, tests, and configuration files. The `pyproject.toml` file is used to define the Python package and its dependencies, while the `requirements.txt` file lists the required Python packages for the extension. The `src` folder contains the functional backend source files, `tests` directory contains unit tests for the backend code, ensuring that the extension behaves as expected and meets the defined requirements.
|
||||
|
||||
The `dist` directory is built when running the `build` or `dev` command, and contains the files that will be included in the bundle. The `manifest.json` file contains critical metadata about the extension, including the majority of the contents of the `extension.json` file, but also other build-time information, like the name of the built Webpack Module Federation remote entry file. The files in the `dist` directory will be zipped into the final `.supx` file. Although this file is technically a zip archive, the `.supx` extension makes it clear that it is a Superset extension package and follows a specific file layout. This packaged file can be distributed and installed in Superset instances.
|
||||
|
||||
The `README.md` file provides documentation and instructions for using the extension, including how to install, configure, and use its functionality.
|
||||
|
||||
## Extension Metadata
|
||||
|
||||
The `extension.json` file contains all metadata necessary for the host application to understand and manage the extension:
|
||||
|
||||
```json
|
||||
{
|
||||
"id": "dataset-references",
|
||||
"name": "Dataset References",
|
||||
"version": "1.0.0",
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"views": {
|
||||
"sqllab": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "dataset-references.main",
|
||||
"name": "Dataset References"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"moduleFederation": {
|
||||
"exposes": ["./index"],
|
||||
"name": "datasetReferences"
|
||||
}
|
||||
},
|
||||
"backend": {
|
||||
"entryPoints": ["superset_extensions.dataset_references.entrypoint"],
|
||||
"files": ["backend/src/superset_extensions/dataset_references/**/*.py"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The `contributions` section declares how the extension extends Superset's functionality through views, commands, menus, and other contribution types. The `backend` section specifies entry points and files to include in the bundle.
|
||||
|
||||
## Interacting with the Host
|
||||
|
||||
Extensions interact with Superset through well-defined, versioned APIs provided by the `@apache-superset/core` (frontend) and `apache-superset-core` (backend) packages. These APIs are designed to be stable, discoverable, and consistent for both built-in and external extensions.
|
||||
|
||||
**Note**: The `superset_core.api` module provides abstract classes that are replaced with concrete implementations via dependency injection when Superset initializes. This allows extensions to use the same interfaces as the host application.
|
||||
|
||||
### Frontend APIs
|
||||
|
||||
The frontend extension APIs (via `@apache-superset/core`) are organized into logical namespaces such as `authentication`, `commands`, `extensions`, `sqlLab`, and others. Each namespace groups related functionality, making it easy for extension authors to discover and use the APIs relevant to their needs. For example, the `sqlLab` namespace provides events and methods specific to SQL Lab, allowing extensions to react to user actions and interact with the SQL Lab environment:
|
||||
|
||||
```typescript
|
||||
export const getCurrentTab: () => Tab | undefined;
|
||||
|
||||
export const getDatabases: () => Database[];
|
||||
|
||||
export const getTabs: () => Tab[];
|
||||
|
||||
export const onDidChangeActivePanel: Event<Panel>;
|
||||
|
||||
export const onDidChangeTabTitle: Event<string>;
|
||||
|
||||
export const onDidQueryRun: Event<QueryContext>;
|
||||
|
||||
export const onDidQueryStop: Event<QueryContext>;
|
||||
```
|
||||
|
||||
The following code demonstrates more examples of the existing frontend APIs:
|
||||
|
||||
```typescript
|
||||
import { core, commands, sqlLab, authentication, Button } from '@apache-superset/core';
|
||||
import MyPanel from './MyPanel';
|
||||
|
||||
export function activate(context) {
|
||||
// Register a new panel (view) in SQL Lab and use shared UI components in your extension's React code
|
||||
const panelDisposable = core.registerView('my_extension.panel', <MyPanel><Button/></MyPanel>);
|
||||
|
||||
// Register a custom command
|
||||
const commandDisposable = commands.registerCommand(
|
||||
'my_extension.copy_query',
|
||||
() => {
|
||||
// Command logic here
|
||||
},
|
||||
);
|
||||
|
||||
// Listen for query run events in SQL Lab
|
||||
const eventDisposable = sqlLab.onDidQueryRun(queryContext => {
|
||||
console.log('Query started on database:', queryContext.tab.databaseId);
|
||||
});
|
||||
|
||||
// Access a CSRF token for secure API requests
|
||||
authentication.getCSRFToken().then(token => {
|
||||
// Use token as needed
|
||||
});
|
||||
|
||||
// Add all disposables for automatic cleanup on deactivation
|
||||
context.subscriptions.push(panelDisposable, commandDisposable, eventDisposable);
|
||||
}
|
||||
```
|
||||
|
||||
### Backend APIs
|
||||
|
||||
Backend APIs (via `apache-superset-core`) follow a similar pattern, providing access to Superset's models, sessions, and query capabilities. Extensions can register REST API endpoints, access the metadata database, and interact with Superset's core functionality.
|
||||
|
||||
Extension endpoints are registered under a dedicated `/extensions` namespace to avoid conflicting with built-in endpoints and also because they don't share the same version constraints. By grouping all extension endpoints under `/extensions`, Superset establishes a clear boundary between core and extension functionality, making it easier to manage, document, and secure both types of APIs.
|
||||
|
||||
```python
|
||||
from superset_core.api.models import Database, get_session
|
||||
from superset_core.api.daos import DatabaseDAO
|
||||
from superset_core.api.rest_api import add_extension_api
|
||||
from .api import DatasetReferencesAPI
|
||||
|
||||
# Register a new extension REST API
|
||||
add_extension_api(DatasetReferencesAPI)
|
||||
|
||||
# Fetch Superset entities via the DAO to apply base filters that filter out entities
|
||||
# that the user doesn't have access to
|
||||
databases = DatabaseDAO.find_all()
|
||||
|
||||
# ..or apply simple filters on top of base filters
|
||||
databases = DatabaseDAO.filter_by(uuid=database.uuid)
|
||||
if not databases:
|
||||
raise Exception("Database not found")
|
||||
|
||||
return databases[0]
|
||||
|
||||
# Perform complex queries using SQLAlchemy Query, also filtering out
|
||||
# inaccessible entities
|
||||
session = get_session()
|
||||
databases_query = session.query(Database).filter(
|
||||
Database.database_name.ilike("%abc%")
|
||||
)
|
||||
return DatabaseDAO.query(databases_query)
|
||||
```
|
||||
|
||||
In the future, we plan to expand the backend APIs to support configuring security models, database engines, SQL Alchemy dialects, etc.
|
||||
|
||||
## Development Mode
|
||||
|
||||
Development mode accelerates extension development by letting developers see changes in Superset quickly, without the need for repeated packaging and uploading. To enable development mode, set the `LOCAL_EXTENSIONS` configuration in your `superset_config.py`:
|
||||
|
||||
```python
|
||||
LOCAL_EXTENSIONS = [
|
||||
"/path/to/your/extension1",
|
||||
"/path/to/your/extension2",
|
||||
]
|
||||
```
|
||||
|
||||
This instructs Superset to load and serve extensions directly from disk, so you can iterate quickly. Running `superset-extensions dev` watches for file changes and rebuilds assets automatically, while the Webpack development server (started separately with `npm run dev-server`) serves updated files as soon as they're modified. This enables immediate feedback for React components, styles, and other frontend code. Changes to backend files are also detected automatically and immediately synced, ensuring that both frontend and backend updates are reflected in your development environment.
|
||||
|
||||
Example output when running in development mode:
|
||||
|
||||
```
|
||||
superset-extensions dev
|
||||
|
||||
⚙️ Building frontend assets…
|
||||
✅ Frontend rebuilt
|
||||
✅ Backend files synced
|
||||
✅ Manifest updated
|
||||
👀 Watching for changes in: /dataset_references/frontend, /dataset_references/backend
|
||||
```
|
||||
|
||||
## Contributing Extension-Compatible Components
|
||||
|
||||
Components in `@apache-superset/core` are automatically documented in the Developer Docs. Simply add a component to the package and it will appear in the extension documentation.
|
||||
|
||||
### Requirements
|
||||
|
||||
1. **Location**: The component must be in `superset-frontend/packages/superset-core/src/ui/components/`
|
||||
2. **Exported**: The component must be exported from the package's `index.ts`
|
||||
3. **Story**: The component must have a Storybook story
|
||||
|
||||
### Creating a Story for Your Component
|
||||
|
||||
Create a story file with an `Interactive` export that defines args and argTypes:
|
||||
|
||||
```typescript
|
||||
// MyComponent.stories.tsx
|
||||
import { MyComponent } from '.';
|
||||
|
||||
export default {
|
||||
title: 'Extension Components/MyComponent',
|
||||
component: MyComponent,
|
||||
parameters: {
|
||||
docs: {
|
||||
description: {
|
||||
component: 'A brief description of what this component does.',
|
||||
},
|
||||
},
|
||||
},
|
||||
};
|
||||
|
||||
// Define an interactive story with args
|
||||
export const InteractiveMyComponent = (args) => <MyComponent {...args} />;
|
||||
|
||||
InteractiveMyComponent.args = {
|
||||
variant: 'primary',
|
||||
disabled: false,
|
||||
};
|
||||
|
||||
InteractiveMyComponent.argTypes = {
|
||||
variant: {
|
||||
control: { type: 'select' },
|
||||
options: ['primary', 'secondary', 'danger'],
|
||||
},
|
||||
disabled: {
|
||||
control: { type: 'boolean' },
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
### How Documentation is Generated
|
||||
|
||||
When the docs site is built (`yarn start` or `yarn build` in the `docs/` directory):
|
||||
|
||||
1. The `generate-extension-components` script scans all stories in `superset-core`
|
||||
2. For each story, it generates an MDX page with:
|
||||
- Component description
|
||||
- **Live interactive example** with controls extracted from `argTypes`
|
||||
- **Editable code playground** for experimentation
|
||||
- Props table from story `args`
|
||||
- Usage code snippet
|
||||
- Links to source files
|
||||
3. Pages appear automatically in **Developer Docs → Extensions → Components**
|
||||
|
||||
### Best Practices
|
||||
|
||||
- **Use descriptive titles**: The title path determines the component's location in docs (e.g., `Extension Components/Alert`)
|
||||
- **Define argTypes**: These become interactive controls in the documentation
|
||||
- **Provide default args**: These populate the initial state of the live example
|
||||
- **Write clear descriptions**: Help extension developers understand when to use each component
|
||||
241
docs/developer_docs/extensions/extension-points/editors.md
Normal file
241
docs/developer_docs/extensions/extension-points/editors.md
Normal file
@@ -0,0 +1,241 @@
|
||||
---
|
||||
title: Editors
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Editor Contributions
|
||||
|
||||
Extensions can replace Superset's default text editors with custom implementations. This allows you to provide enhanced editing experiences using alternative editor frameworks like Monaco, CodeMirror, or custom solutions.
|
||||
|
||||
## Overview
|
||||
|
||||
Superset uses text editors in various places throughout the application:
|
||||
|
||||
| Language | Locations |
|
||||
|----------|-----------|
|
||||
| `sql` | SQL Lab, Metric/Filter Popovers |
|
||||
| `json` | Dashboard Properties, Annotation Modal, Theme Modal |
|
||||
| `css` | Dashboard Properties, CSS Template Modal |
|
||||
| `markdown` | Dashboard Markdown component |
|
||||
| `yaml` | Template Params Editor |
|
||||
|
||||
By registering an editor provider for a language, your extension replaces the default Ace editor in **all** locations that use that language.
|
||||
|
||||
## Manifest Configuration
|
||||
|
||||
Declare editor contributions in your `extension.json` manifest:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "monaco-editor",
|
||||
"version": "1.0.0",
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"editors": [
|
||||
{
|
||||
"id": "monaco-editor.sql",
|
||||
"name": "Monaco SQL Editor",
|
||||
"languages": ["sql"],
|
||||
"description": "Monaco-based SQL editor with IntelliSense"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementing an Editor
|
||||
|
||||
Your editor component must implement the `EditorProps` interface and expose an `EditorHandle` via `forwardRef`. For the complete interface definitions, see `@apache-superset/core/api/editors.ts`.
|
||||
|
||||
### Key EditorProps
|
||||
|
||||
```typescript
|
||||
interface EditorProps {
|
||||
/** Controlled value */
|
||||
value: string;
|
||||
/** Content change handler */
|
||||
onChange: (value: string) => void;
|
||||
/** Language mode for syntax highlighting */
|
||||
language: EditorLanguage;
|
||||
/** Keyboard shortcuts to register */
|
||||
hotkeys?: EditorHotkey[];
|
||||
/** Callback when editor is ready with imperative handle */
|
||||
onReady?: (handle: EditorHandle) => void;
|
||||
/** Host-specific context (e.g., database info from SQL Lab) */
|
||||
metadata?: Record<string, unknown>;
|
||||
// ... additional props for styling, annotations, etc.
|
||||
}
|
||||
```
|
||||
|
||||
### Key EditorHandle Methods
|
||||
|
||||
```typescript
|
||||
interface EditorHandle {
|
||||
/** Focus the editor */
|
||||
focus(): void;
|
||||
/** Get the current editor content */
|
||||
getValue(): string;
|
||||
/** Get the current cursor position */
|
||||
getCursorPosition(): Position;
|
||||
/** Move the cursor to a specific position */
|
||||
moveCursorToPosition(position: Position): void;
|
||||
/** Set the selection range */
|
||||
setSelection(selection: Range): void;
|
||||
/** Scroll to a specific line */
|
||||
scrollToLine(line: number): void;
|
||||
// ... additional methods for text manipulation, annotations, etc.
|
||||
}
|
||||
```
|
||||
|
||||
## Example Implementation
|
||||
|
||||
Here's an example of a Monaco-based SQL editor implementing the key interfaces shown above:
|
||||
|
||||
### MonacoSQLEditor.tsx
|
||||
|
||||
```typescript
|
||||
import { forwardRef, useRef, useImperativeHandle, useEffect } from 'react';
|
||||
import * as monaco from 'monaco-editor';
|
||||
import type { editors } from '@apache-superset/core';
|
||||
|
||||
const MonacoSQLEditor = forwardRef<editors.EditorHandle, editors.EditorProps>(
|
||||
(props, ref) => {
|
||||
const { value, onChange, hotkeys, onReady } = props;
|
||||
const containerRef = useRef<HTMLDivElement>(null);
|
||||
const editorRef = useRef<monaco.editor.IStandaloneCodeEditor | null>(null);
|
||||
|
||||
// Implement EditorHandle interface
|
||||
const handle: editors.EditorHandle = {
|
||||
focus: () => editorRef.current?.focus(),
|
||||
getValue: () => editorRef.current?.getValue() ?? '',
|
||||
getCursorPosition: () => {
|
||||
const pos = editorRef.current?.getPosition();
|
||||
return { line: (pos?.lineNumber ?? 1) - 1, column: (pos?.column ?? 1) - 1 };
|
||||
},
|
||||
// ... implement remaining methods
|
||||
};
|
||||
|
||||
useImperativeHandle(ref, () => handle, []);
|
||||
|
||||
useEffect(() => {
|
||||
if (!containerRef.current) return;
|
||||
|
||||
const editor = monaco.editor.create(containerRef.current, { value, language: 'sql' });
|
||||
editorRef.current = editor;
|
||||
|
||||
editor.onDidChangeModelContent(() => onChange(editor.getValue()));
|
||||
|
||||
// Register hotkeys
|
||||
hotkeys?.forEach(hotkey => {
|
||||
editor.addAction({
|
||||
id: hotkey.name,
|
||||
label: hotkey.name,
|
||||
run: () => hotkey.exec(handle),
|
||||
});
|
||||
});
|
||||
|
||||
onReady?.(handle);
|
||||
return () => editor.dispose();
|
||||
}, []);
|
||||
|
||||
return <div ref={containerRef} style={{ height: '100%', width: '100%' }} />;
|
||||
},
|
||||
);
|
||||
|
||||
export default MonacoSQLEditor;
|
||||
```
|
||||
|
||||
### activate.ts
|
||||
|
||||
```typescript
|
||||
import { editors } from '@apache-superset/core';
|
||||
import MonacoSQLEditor from './MonacoSQLEditor';
|
||||
|
||||
export function activate(context) {
|
||||
// Register the Monaco editor for SQL using the contribution ID from extension.json
|
||||
const disposable = editors.registerEditorProvider(
|
||||
'monaco-sql-editor.sql',
|
||||
MonacoSQLEditor,
|
||||
);
|
||||
|
||||
context.subscriptions.push(disposable);
|
||||
}
|
||||
```
|
||||
|
||||
## Handling Hotkeys
|
||||
|
||||
Superset passes keyboard shortcuts via the `hotkeys` prop. Each hotkey includes an `exec` function that receives the `EditorHandle`:
|
||||
|
||||
```typescript
|
||||
interface EditorHotkey {
|
||||
name: string;
|
||||
key: string; // e.g., "Ctrl-Enter", "Alt-Shift-F"
|
||||
description?: string;
|
||||
exec: (handle: EditorHandle) => void;
|
||||
}
|
||||
```
|
||||
|
||||
Your editor must register these hotkeys with your editor framework and call `exec(handle)` when triggered.
|
||||
|
||||
## Keywords
|
||||
|
||||
Superset passes static autocomplete suggestions via the `keywords` prop. These include table names, column names, and SQL functions based on the current database context:
|
||||
|
||||
```typescript
|
||||
interface EditorKeyword {
|
||||
name: string;
|
||||
value?: string; // Text to insert (defaults to name)
|
||||
meta?: string; // Category like "table", "column", "function"
|
||||
score?: number; // Sorting priority
|
||||
}
|
||||
```
|
||||
|
||||
Your editor should convert these to your framework's completion format and register them for autocomplete.
|
||||
|
||||
## Completion Providers
|
||||
|
||||
For dynamic autocomplete (e.g., fetching suggestions as the user types), implement and register a `CompletionProvider` via the `EditorHandle`:
|
||||
|
||||
```typescript
|
||||
const provider: CompletionProvider = {
|
||||
id: 'my-sql-completions',
|
||||
triggerCharacters: ['.', ' '],
|
||||
provideCompletions: async (content, position, context) => {
|
||||
// Use context.metadata for database info
|
||||
// Return array of CompletionItem
|
||||
return [
|
||||
{ label: 'SELECT', insertText: 'SELECT', kind: 'keyword' },
|
||||
// ...
|
||||
];
|
||||
},
|
||||
};
|
||||
|
||||
// Register during editor initialization
|
||||
const disposable = handle.registerCompletionProvider(provider);
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[SQL Lab Extension Points](./sqllab)** - Learn about other SQL Lab customizations
|
||||
- **[Contribution Types](../contribution-types)** - Explore other contribution types
|
||||
- **[Development](../development)** - Set up your development environment
|
||||
221
docs/developer_docs/extensions/extension-points/sqllab.md
Normal file
221
docs/developer_docs/extensions/extension-points/sqllab.md
Normal file
@@ -0,0 +1,221 @@
|
||||
---
|
||||
title: SQL Lab
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# SQL Lab Extension Points
|
||||
|
||||
SQL Lab provides 4 extension points where extensions can contribute custom UI components. Each area serves a specific purpose and supports different types of customizations. These areas will evolve over time as new features are added to SQL Lab.
|
||||
|
||||
## Layout Overview
|
||||
|
||||
```
|
||||
┌──────────┬─────────────────────────────────────────┬─────────────┐
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
│ │ Editor │ │
|
||||
│ │ │ │
|
||||
│ Left │ │ Right │
|
||||
│ Sidebar ├─────────────────────────────────────────┤ Sidebar │
|
||||
│ │ │ │
|
||||
│ │ Panels │ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
│ │ │ │
|
||||
└──────────┴─────────────────────────────────────────┴─────────────┘
|
||||
```
|
||||
|
||||
| Extension Point | ID | Views | Menus | Description |
|
||||
| ----------------- | --------------------- | ----- | ----- | ---------------------------------------------- |
|
||||
| **Left Sidebar** | `sqllab.leftSidebar` | — | ✓ | Menu actions for the database explorer |
|
||||
| **Editor** | `sqllab.editor` | ✓\* | ✓ | Custom editors + toolbar actions |
|
||||
| **Right Sidebar** | `sqllab.rightSidebar` | ✓ | — | Custom panels (AI assistants, query analysis) |
|
||||
| **Panels** | `sqllab.panels` | ✓ | ✓ | Custom tabs + toolbar actions (data profiling) |
|
||||
|
||||
\*Editor views are contributed via [Editor Contributions](./editors), not standard view contributions.
|
||||
|
||||
## Customization Types
|
||||
|
||||
### Views
|
||||
|
||||
Extensions can add custom views (React components) to **Right Sidebar** and **Panels**. Views appear as new panels or tabs in their respective areas.
|
||||
|
||||
### Menus
|
||||
|
||||
Extensions can add toolbar actions to **Left Sidebar**, **Editor**, and **Panels**. Menu contributions support:
|
||||
|
||||
```
|
||||
┌───────────────────────────────────────────────────────────────┐
|
||||
│ [Button] [Button] [•••] │
|
||||
├───────────────────────────────────────────────────────────────┤
|
||||
│ Area Content │
|
||||
└───────────────────────────────────────────────────────────────┘
|
||||
```
|
||||
|
||||
| Action Type | Location | Use Case |
|
||||
| --------------------- | ---------------- | ----------------------------------------------------- |
|
||||
| **Primary Actions** | Toolbar buttons | Frequently used actions (e.g., run, refresh, add new) |
|
||||
| **Secondary Actions** | 3-dot menu (•••) | Less common actions (e.g., export, settings) |
|
||||
|
||||
### Custom Editors
|
||||
|
||||
Extensions can replace the default SQL editor with custom implementations (Monaco, CodeMirror, etc.). See [Editor Contributions](./editors) for details.
|
||||
|
||||
## Examples
|
||||
|
||||
### Adding a Panel
|
||||
|
||||
This example adds a "Data Profiler" panel to SQL Lab:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "data_profiler",
|
||||
"version": "1.0.0",
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"views": {
|
||||
"sqllab": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "data_profiler.main",
|
||||
"name": "Data Profiler"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { core } from '@apache-superset/core';
|
||||
import DataProfilerPanel from './DataProfilerPanel';
|
||||
|
||||
export function activate(context) {
|
||||
// Register the panel view with the ID declared in extension.json
|
||||
const disposable = core.registerView('data_profiler.main', <DataProfilerPanel />);
|
||||
context.subscriptions.push(disposable);
|
||||
}
|
||||
```
|
||||
|
||||
### Adding Actions to the Editor
|
||||
|
||||
This example adds primary, secondary, and context actions to the editor:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "query_tools",
|
||||
"version": "1.0.0",
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"commands": [
|
||||
{
|
||||
"command": "query_tools.format",
|
||||
"title": "Format Query",
|
||||
"icon": "FormatPainterOutlined"
|
||||
},
|
||||
{
|
||||
"command": "query_tools.explain",
|
||||
"title": "Explain Query"
|
||||
},
|
||||
{
|
||||
"command": "query_tools.copy_as_cte",
|
||||
"title": "Copy as CTE"
|
||||
}
|
||||
],
|
||||
"menus": {
|
||||
"sqllab": {
|
||||
"editor": {
|
||||
"primary": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "query_tools.format"
|
||||
}
|
||||
],
|
||||
"secondary": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "query_tools.explain"
|
||||
}
|
||||
],
|
||||
"context": [
|
||||
{
|
||||
"view": "builtin.editor",
|
||||
"command": "query_tools.copy_as_cte"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
import { commands, sqlLab } from '@apache-superset/core';
|
||||
|
||||
export function activate(context) {
|
||||
// Register the commands declared in extension.json
|
||||
const formatCommand = commands.registerCommand(
|
||||
'query_tools.format',
|
||||
async () => {
|
||||
const tab = sqlLab.getCurrentTab();
|
||||
if (tab) {
|
||||
const editor = await tab.getEditor();
|
||||
// Format the SQL query
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
const explainCommand = commands.registerCommand(
|
||||
'query_tools.explain',
|
||||
async () => {
|
||||
const tab = sqlLab.getCurrentTab();
|
||||
if (tab) {
|
||||
const editor = await tab.getEditor();
|
||||
// Show query explanation
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
const copyAsCteCommand = commands.registerCommand(
|
||||
'query_tools.copy_as_cte',
|
||||
async () => {
|
||||
const tab = sqlLab.getCurrentTab();
|
||||
if (tab) {
|
||||
const editor = await tab.getEditor();
|
||||
// Copy selected text as CTE
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
context.subscriptions.push(formatCommand, explainCommand, copyAsCteCommand);
|
||||
}
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[Contribution Types](../contribution-types)** - Learn about other contribution types (commands, menus)
|
||||
- **[Development](../development)** - Set up your development environment
|
||||
- **[Quick Start](../quick-start)** - Build a complete extension
|
||||
459
docs/developer_docs/extensions/mcp.md
Normal file
459
docs/developer_docs/extensions/mcp.md
Normal file
@@ -0,0 +1,459 @@
|
||||
---
|
||||
title: MCP Integration
|
||||
hide_title: true
|
||||
sidebar_position: 8
|
||||
version: 1
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# MCP Integration
|
||||
|
||||
Model Context Protocol (MCP) integration allows extensions to register custom AI agent capabilities that integrate seamlessly with Superset's MCP service. Extensions can provide both **tools** (executable functions) and **prompts** (interactive guidance) that AI agents can discover and use.
|
||||
|
||||
## What is MCP?
|
||||
|
||||
MCP enables extensions to extend Superset's AI capabilities in two ways:
|
||||
|
||||
### MCP Tools
|
||||
Tools are Python functions that AI agents can call to perform specific tasks. They provide executable functionality that extends Superset's capabilities.
|
||||
|
||||
**Examples of MCP tools:**
|
||||
- Data processing and transformation functions
|
||||
- Custom analytics calculations
|
||||
- Integration with external APIs
|
||||
- Specialized report generation
|
||||
- Business-specific operations
|
||||
|
||||
### MCP Prompts
|
||||
Prompts provide interactive guidance and context to AI agents. They help agents understand how to better assist users with specific workflows or domain knowledge.
|
||||
|
||||
**Examples of MCP prompts:**
|
||||
- Step-by-step workflow guidance
|
||||
- Domain-specific context and knowledge
|
||||
- Interactive troubleshooting assistance
|
||||
- Template generation helpers
|
||||
- Best practices recommendations
|
||||
|
||||
## Getting Started
|
||||
|
||||
## MCP Tools
|
||||
|
||||
### Basic Tool Registration
|
||||
|
||||
The simplest way to create an MCP tool is using the `@tool` decorator:
|
||||
|
||||
```python
|
||||
from superset_core.mcp import tool
|
||||
|
||||
@tool
|
||||
def hello_world() -> dict:
|
||||
"""A simple greeting tool."""
|
||||
return {"message": "Hello from my extension!"}
|
||||
```
|
||||
|
||||
This creates a tool that AI agents can call by name. The tool name defaults to the function name.
|
||||
|
||||
### Decorator Parameters
|
||||
|
||||
The `@tool` decorator accepts several optional parameters:
|
||||
|
||||
**Parameter details:**
|
||||
- **`name`**: Tool identifier (AI agents use this to call your tool)
|
||||
- **`description`**: Explains what the tool does (helps AI agents decide when to use it)
|
||||
- **`tags`**: Categories for organization and discovery
|
||||
- **`protect`**: Whether the tool requires user authentication (defaults to `True`)
|
||||
|
||||
### Naming Your Tools
|
||||
|
||||
For extensions, include your extension ID in tool names to avoid conflicts:
|
||||
|
||||
## Complete Example
|
||||
|
||||
Here's a more comprehensive example showing best practices:
|
||||
|
||||
```python
|
||||
# backend/mcp_tools.py
|
||||
import random
|
||||
from datetime import datetime, timezone
|
||||
from pydantic import BaseModel, Field
|
||||
from superset_core.mcp import tool
|
||||
|
||||
class RandomNumberRequest(BaseModel):
|
||||
"""Request schema for random number generation."""
|
||||
|
||||
min_value: int = Field(
|
||||
description="Minimum value (inclusive) for random number generation",
|
||||
ge=-2147483648,
|
||||
le=2147483647
|
||||
)
|
||||
max_value: int = Field(
|
||||
description="Maximum value (inclusive) for random number generation",
|
||||
ge=-2147483648,
|
||||
le=2147483647
|
||||
)
|
||||
|
||||
@tool(
|
||||
name="example_extension.random_number",
|
||||
tags=["extension", "utility", "random", "generator"]
|
||||
)
|
||||
def random_number_generator(request: RandomNumberRequest) -> dict:
|
||||
"""
|
||||
Generate a random integer between specified bounds.
|
||||
|
||||
This tool validates input ranges and provides detailed error messages
|
||||
for invalid requests.
|
||||
"""
|
||||
|
||||
# Validate business logic (Pydantic handles type/range validation)
|
||||
if request.min_value > request.max_value:
|
||||
return {
|
||||
"status": "error",
|
||||
"error": f"min_value ({request.min_value}) cannot be greater than max_value ({request.max_value})",
|
||||
"timestamp": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
|
||||
# Generate random number
|
||||
result = random.randint(request.min_value, request.max_value)
|
||||
|
||||
return {
|
||||
"status": "success",
|
||||
"random_number": result,
|
||||
"min_value": request.min_value,
|
||||
"max_value": request.max_value,
|
||||
"range_size": request.max_value - request.min_value + 1,
|
||||
"timestamp": datetime.now(timezone.utc).isoformat()
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Response Format
|
||||
|
||||
Use consistent response structures:
|
||||
|
||||
```python
|
||||
# Success response
|
||||
{
|
||||
"status": "success",
|
||||
"result": "your_data_here",
|
||||
"timestamp": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
|
||||
# Error response
|
||||
{
|
||||
"status": "error",
|
||||
"error": "Clear error message",
|
||||
"timestamp": "2024-01-01T00:00:00Z"
|
||||
}
|
||||
```
|
||||
|
||||
### Documentation
|
||||
|
||||
Write clear descriptions and docstrings:
|
||||
|
||||
```python
|
||||
@tool(
|
||||
name="my_extension.process_data",
|
||||
description="Process customer data and generate insights. Requires valid customer ID and date range.",
|
||||
tags=["analytics", "customer", "reporting"]
|
||||
)
|
||||
def process_data(customer_id: int, start_date: str, end_date: str) -> dict:
|
||||
"""
|
||||
Process customer data for the specified date range.
|
||||
|
||||
This tool analyzes customer behavior patterns and generates
|
||||
actionable insights for business decision-making.
|
||||
|
||||
Args:
|
||||
customer_id: Unique customer identifier
|
||||
start_date: Analysis start date (YYYY-MM-DD format)
|
||||
end_date: Analysis end date (YYYY-MM-DD format)
|
||||
|
||||
Returns:
|
||||
Dictionary containing analysis results and recommendations
|
||||
"""
|
||||
# Implementation here
|
||||
pass
|
||||
```
|
||||
|
||||
### Tool Naming
|
||||
|
||||
- **Extension tools**: Use prefixed names like `my_extension.tool_name`
|
||||
- **Descriptive names**: `calculate_tax_amount` vs `calculate`
|
||||
- **Consistent naming**: Follow patterns within your extension
|
||||
|
||||
## How AI Agents Use Your Tools
|
||||
|
||||
Once registered, AI agents can discover and use your tools automatically:
|
||||
|
||||
```
|
||||
User: "Generate a random number between 1 and 100"
|
||||
Agent: I'll use the random number generator tool.
|
||||
→ Calls: example_extension.random_number(min_value=1, max_value=100)
|
||||
← Returns: {"status": "success", "random_number": 42, ...}
|
||||
Agent: I generated the number 42 for you.
|
||||
```
|
||||
|
||||
The AI agent sees your tool's:
|
||||
- **Name**: How to call it
|
||||
- **Description**: What it does and when to use it
|
||||
- **Parameters**: What inputs it expects (from Pydantic schema)
|
||||
- **Tags**: Categories for discovery
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Tool Not Available to AI Agents
|
||||
|
||||
1. **Check extension registration**: Verify your tool module is listed in extension entrypoints
|
||||
2. **Verify decorator**: Ensure `@tool` is correctly applied
|
||||
3. **Extension loading**: Confirm your extension is installed and enabled
|
||||
|
||||
### Input Validation Errors
|
||||
|
||||
1. **Pydantic models**: Ensure field types match expected inputs
|
||||
2. **Field constraints**: Check min/max values and string lengths are reasonable
|
||||
3. **Required fields**: Verify which parameters are required vs optional
|
||||
|
||||
### Runtime Issues
|
||||
|
||||
1. **Error handling**: Add try/catch blocks with clear error messages
|
||||
2. **Response format**: Use consistent status/error/timestamp structure
|
||||
3. **Testing**: Test your tools with various input scenarios
|
||||
|
||||
### Development Tips
|
||||
|
||||
1. **Start simple**: Begin with basic tools, add complexity gradually
|
||||
2. **Test locally**: Use MCP clients (like Claude Desktop) to test your tools
|
||||
3. **Clear descriptions**: Write tool descriptions as if explaining to a new user
|
||||
4. **Meaningful tags**: Use tags that help categorize and discover tools
|
||||
5. **Error messages**: Provide specific, actionable error messages
|
||||
|
||||
## MCP Prompts
|
||||
|
||||
### Basic Prompt Registration
|
||||
|
||||
Create interactive prompts using the `@prompt` decorator:
|
||||
|
||||
```python
|
||||
from superset_core.mcp import prompt
|
||||
from fastmcp import Context
|
||||
|
||||
@prompt("my_extension.workflow_guide")
|
||||
async def workflow_guide(ctx: Context) -> str:
|
||||
"""Interactive guide for data analysis workflows."""
|
||||
return """
|
||||
# Data Analysis Workflow Guide
|
||||
|
||||
Here's a step-by-step approach to effective data analysis in Superset:
|
||||
|
||||
## 1. Data Discovery
|
||||
- Start by exploring your datasets using the dataset browser
|
||||
- Check data quality and identify key metrics
|
||||
- Look for patterns and relationships in your data
|
||||
|
||||
## 2. Chart Creation
|
||||
- Choose appropriate visualizations for your data types
|
||||
- Apply filters to focus on relevant subsets
|
||||
- Configure proper aggregations and groupings
|
||||
|
||||
## 3. Dashboard Assembly
|
||||
- Combine related charts into coherent dashboards
|
||||
- Use filters and parameters for interactivity
|
||||
- Add markdown components for context and explanations
|
||||
|
||||
Would you like guidance on any specific step?
|
||||
"""
|
||||
```
|
||||
|
||||
### Advanced Prompt Examples
|
||||
|
||||
#### Domain-Specific Context
|
||||
|
||||
```python
|
||||
@prompt(
|
||||
"sales_extension.sales_analysis_guide",
|
||||
title="Sales Analysis Guide",
|
||||
description="Specialized guidance for sales data analysis workflows"
|
||||
)
|
||||
async def sales_analysis_guide(ctx: Context) -> str:
|
||||
"""Provides sales-specific analysis guidance and best practices."""
|
||||
return """
|
||||
# Sales Data Analysis Best Practices
|
||||
|
||||
## Key Metrics to Track
|
||||
- **Revenue Growth**: Month-over-month and year-over-year trends
|
||||
- **Conversion Rates**: Lead-to-opportunity-to-close ratios
|
||||
- **Customer Lifetime Value**: Total value per customer segment
|
||||
- **Sales Cycle Length**: Time from lead to close by product/region
|
||||
|
||||
## Recommended Chart Types
|
||||
- **Time Series**: Revenue trends, seasonal patterns
|
||||
- **Funnel Charts**: Conversion analysis across sales stages
|
||||
- **Geographic Maps**: Regional performance comparison
|
||||
- **Cohort Analysis**: Customer retention and growth patterns
|
||||
|
||||
## Common Pitfalls to Avoid
|
||||
- Don't mix different time granularities without proper context
|
||||
- Always normalize for business days when comparing periods
|
||||
- Consider external factors (holidays, market events) in analysis
|
||||
- Segment by relevant dimensions (product, region, channel)
|
||||
|
||||
## Next Steps
|
||||
1. Identify your primary sales KPIs
|
||||
2. Create baseline trend charts for each metric
|
||||
3. Build comparative views across segments
|
||||
4. Set up automated alerts for significant changes
|
||||
"""
|
||||
```
|
||||
|
||||
#### Interactive Troubleshooting
|
||||
|
||||
```python
|
||||
@prompt("support_extension.troubleshoot_charts")
|
||||
async def troubleshoot_charts(ctx: Context) -> str:
|
||||
"""Interactive troubleshooting assistant for chart issues."""
|
||||
return """
|
||||
# Chart Troubleshooting Assistant
|
||||
|
||||
Let's diagnose your chart issue step by step:
|
||||
|
||||
## Common Issues and Solutions
|
||||
|
||||
### 🚫 No Data Showing
|
||||
**Possible causes:**
|
||||
- Filters are too restrictive
|
||||
- Date range doesn't match your data
|
||||
- Database connection issues
|
||||
- Missing permissions
|
||||
|
||||
**Check:** Try removing all filters and expanding the date range first.
|
||||
|
||||
### 📊 Unexpected Aggregation Results
|
||||
**Possible causes:**
|
||||
- Incorrect grouping dimensions
|
||||
- Wrong aggregation function (SUM vs COUNT vs AVG)
|
||||
- Data quality issues (duplicates, nulls)
|
||||
- Time zone mismatches
|
||||
|
||||
**Check:** Verify your GROUP BY columns and aggregation logic.
|
||||
|
||||
### 🐌 Slow Performance
|
||||
**Possible causes:**
|
||||
- Large dataset without proper indexing
|
||||
- Complex joins or calculations
|
||||
- Missing query optimizations
|
||||
- Resource constraints
|
||||
|
||||
**Check:** Simplify the query and add appropriate filters first.
|
||||
|
||||
## Debug Steps
|
||||
1. **Start Simple**: Create a basic count query first
|
||||
2. **Add Gradually**: Introduce complexity step by step
|
||||
3. **Check SQL**: Review the generated SQL for issues
|
||||
4. **Test Data**: Verify with a small sample first
|
||||
|
||||
What specific issue are you experiencing?
|
||||
"""
|
||||
```
|
||||
|
||||
### Prompt Best Practices
|
||||
|
||||
#### Content Structure
|
||||
- **Use clear headings** and sections for easy navigation
|
||||
- **Provide actionable steps** rather than just theory
|
||||
- **Include examples** relevant to the user's domain
|
||||
- **Offer next steps** to continue the workflow
|
||||
|
||||
#### Interactive Design
|
||||
- **Ask questions** to engage the user
|
||||
- **Provide options** for different scenarios
|
||||
- **Reference specific Superset features** by name
|
||||
- **Link to related tools** when appropriate
|
||||
|
||||
#### Context Awareness
|
||||
```python
|
||||
@prompt("analytics_extension.context_aware_guide")
|
||||
async def context_aware_guide(ctx: Context) -> str:
|
||||
"""Provides guidance based on current user context."""
|
||||
# Access user information if available
|
||||
user_info = getattr(ctx, 'user', None)
|
||||
|
||||
guidance = """# Personalized Analytics Guide\n\n"""
|
||||
|
||||
if user_info:
|
||||
guidance += f"Welcome back! Here's guidance tailored for your role:\n\n"
|
||||
|
||||
guidance += """
|
||||
## Getting Started
|
||||
Based on your previous activity, here are recommended next steps:
|
||||
|
||||
1. **Review Recent Dashboards**: Check your most-used dashboards for updates
|
||||
2. **Explore New Data**: Look for recently added datasets in your domain
|
||||
3. **Share Insights**: Consider sharing successful analyses with your team
|
||||
|
||||
## Advanced Techniques
|
||||
- Set up automated alerts for key metrics
|
||||
- Create parameterized dashboards for different audiences
|
||||
- Use SQL Lab for complex custom analyses
|
||||
"""
|
||||
|
||||
return guidance
|
||||
```
|
||||
|
||||
## Combining Tools and Prompts
|
||||
|
||||
Extensions can provide both tools and prompts that work together:
|
||||
|
||||
```python
|
||||
# Tool for data processing
|
||||
@tool("analytics_extension.calculate_metrics")
|
||||
def calculate_metrics(data: dict) -> dict:
|
||||
"""Calculate advanced analytics metrics."""
|
||||
# Implementation here
|
||||
pass
|
||||
|
||||
# Prompt that guides users to the tool
|
||||
@prompt("analytics_extension.metrics_guide")
|
||||
async def metrics_guide(ctx: Context) -> str:
|
||||
"""Guide users through advanced metrics calculation."""
|
||||
return """
|
||||
# Advanced Metrics Calculation
|
||||
|
||||
Use the `calculate_metrics` tool to compute specialized analytics:
|
||||
|
||||
## Available Metrics
|
||||
- Customer Lifetime Value (CLV)
|
||||
- Cohort Retention Rates
|
||||
- Statistical Significance Tests
|
||||
- Predictive Trend Analysis
|
||||
|
||||
## Usage
|
||||
Call the tool with your dataset to get detailed calculations
|
||||
and recommendations for visualization approaches.
|
||||
|
||||
Would you like to calculate metrics for your current dataset?
|
||||
"""
|
||||
```
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[Development](./development)** - Project structure, APIs, and dev workflow
|
||||
- **[Security](./security)** - Security best practices for extensions
|
||||
55
docs/developer_docs/extensions/overview.md
Normal file
55
docs/developer_docs/extensions/overview.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: Overview
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Overview
|
||||
|
||||
Apache Superset's extension system enables organizations to build custom features without modifying the core codebase. Inspired by the [VS Code extension model](https://code.visualstudio.com/api), this architecture addresses a long-standing challenge: teams previously had to fork Superset or make invasive modifications to add capabilities like query optimizers, custom panels, or specialized integrations—resulting in maintenance overhead and codebase fragmentation.
|
||||
|
||||
The extension system introduces a modular, plugin-based architecture where both built-in features and external extensions use the same well-defined APIs. This "lean core" approach ensures that any capability available to Superset's internal features is equally accessible to community-developed extensions, fostering a vibrant ecosystem while reducing the maintenance burden on core contributors.
|
||||
|
||||
## What are Superset Extensions?
|
||||
|
||||
Superset extensions are self-contained `.supx` packages that extend the platform's capabilities through standardized contribution points. Each extension can include both frontend (React/TypeScript) and backend (Python) components, bundled together and loaded dynamically at runtime using Webpack Module Federation.
|
||||
|
||||
## Extension Capabilities
|
||||
|
||||
Extensions can provide:
|
||||
|
||||
- **Custom UI Components**: New panels, views, and interactive elements
|
||||
- **Commands and Menus**: Custom actions accessible via menus and keyboard shortcuts
|
||||
- **REST API Endpoints**: Backend services under the `/api/v1/extensions/` namespace
|
||||
- **MCP Tools and Prompts**: AI agent capabilities for enhanced user assistance
|
||||
|
||||
## Next Steps
|
||||
|
||||
- **[Quick Start](./quick-start)** - Build your first extension with a complete walkthrough
|
||||
- **[Architecture](./architecture)** - Design principles and system overview
|
||||
- **[Dependencies](./dependencies)** - Managing dependencies and understanding API stability
|
||||
- **[Contribution Types](./contribution-types)** - Available extension points
|
||||
- **[Development](./development)** - Project structure, APIs, and development workflow
|
||||
- **[Deployment](./deployment)** - Packaging and deploying extensions
|
||||
- **[MCP Integration](./mcp)** - Adding AI agent capabilities using extensions
|
||||
- **[Security](./security)** - Security considerations and best practices
|
||||
- **[Tasks](./tasks)** - Framework for creating and managing long running tasks
|
||||
- **[Community Extensions](./registry)** - Browse extensions shared by the community
|
||||
535
docs/developer_docs/extensions/quick-start.md
Normal file
535
docs/developer_docs/extensions/quick-start.md
Normal file
@@ -0,0 +1,535 @@
|
||||
---
|
||||
title: Quick Start
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Quick Start
|
||||
|
||||
This guide walks you through creating your first Superset extension - a simple "Hello World" panel that displays a message fetched from a backend API endpoint. You'll learn the essential structure and patterns for building full-stack Superset extensions.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Before starting, ensure you have:
|
||||
|
||||
- Node.js and npm compatible with your Superset version
|
||||
- Python compatible with your Superset version
|
||||
- A running Superset development environment
|
||||
- Basic knowledge of React, TypeScript, and Flask
|
||||
|
||||
## Step 1: Install the Extensions CLI
|
||||
|
||||
First, install the Apache Superset Extensions CLI:
|
||||
|
||||
```bash
|
||||
pip install apache-superset-extensions-cli
|
||||
```
|
||||
|
||||
## Step 2: Create a New Extension
|
||||
|
||||
Use the CLI to scaffold a new extension project. Extensions can include frontend functionality, backend functionality, or both, depending on your needs. This quickstart demonstrates a full-stack extension with both frontend UI components and backend API endpoints to show the complete integration pattern.
|
||||
|
||||
```bash
|
||||
superset-extensions init
|
||||
```
|
||||
|
||||
The CLI will prompt you for information using a three-step publisher workflow:
|
||||
|
||||
```
|
||||
Extension display name: Hello World
|
||||
Extension name (hello-world): hello-world
|
||||
Publisher (e.g., my-org): my-org
|
||||
Initial version [0.1.0]: 0.1.0
|
||||
License [Apache-2.0]: Apache-2.0
|
||||
Include frontend? [Y/n]: Y
|
||||
Include backend? [Y/n]: Y
|
||||
```
|
||||
|
||||
**Publisher Namespaces**: Extensions use organizational namespaces similar to VS Code extensions, providing collision-safe naming across organizations:
|
||||
- **NPM package**: `@my-org/hello-world` (scoped package for frontend distribution)
|
||||
- **Module Federation name**: `myOrg_helloWorld` (collision-safe JavaScript identifier)
|
||||
- **Backend package**: `my_org-hello_world` (collision-safe Python distribution name)
|
||||
- **Python namespace**: `superset_extensions.my_org.hello_world`
|
||||
|
||||
This approach ensures that extensions from different organizations cannot conflict, even if they use the same technical name (e.g., both `acme.dashboard-widgets` and `corp.dashboard-widgets` can coexist).
|
||||
|
||||
This creates a complete project structure:
|
||||
|
||||
```
|
||||
my-org.hello-world/
|
||||
├── extension.json # Extension metadata and configuration
|
||||
├── backend/ # Backend Python code
|
||||
│ ├── src/
|
||||
│ │ └── superset_extensions/
|
||||
│ │ └── my_org/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── hello_world/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── entrypoint.py # Backend registration
|
||||
│ └── pyproject.toml
|
||||
└── frontend/ # Frontend TypeScript/React code
|
||||
├── src/
|
||||
│ └── index.tsx # Frontend entry point
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── webpack.config.js
|
||||
```
|
||||
|
||||
## Step 3: Configure Extension Metadata
|
||||
|
||||
The generated `extension.json` contains basic metadata. Update it to register your panel in SQL Lab:
|
||||
|
||||
```json
|
||||
{
|
||||
"publisher": "my-org",
|
||||
"name": "hello-world",
|
||||
"displayName": "Hello World",
|
||||
"version": "0.1.0",
|
||||
"license": "Apache-2.0",
|
||||
"frontend": {
|
||||
"contributions": {
|
||||
"views": {
|
||||
"sqllab": {
|
||||
"panels": [
|
||||
{
|
||||
"id": "my-org.hello-world.main",
|
||||
"name": "Hello World"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"moduleFederation": {
|
||||
"exposes": ["./index"],
|
||||
"name": "myOrg_helloWorld"
|
||||
}
|
||||
},
|
||||
"backend": {
|
||||
"entryPoints": ["superset_extensions.my_org.hello_world.entrypoint"],
|
||||
"files": ["backend/src/superset_extensions/my_org/hello_world/**/*.py"]
|
||||
},
|
||||
"permissions": ["can_read"]
|
||||
}
|
||||
```
|
||||
|
||||
**Note**: The `moduleFederation.name` uses collision-safe naming (`myOrg_helloWorld`), and backend entry points use the full nested Python namespace (`superset_extensions.my_org.hello_world`).
|
||||
|
||||
**Key fields:**
|
||||
|
||||
- `publisher`: Organizational namespace for the extension
|
||||
- `name`: Technical identifier (kebab-case)
|
||||
- `displayName`: Human-readable name shown to users
|
||||
- `frontend.contributions.views.sqllab.panels`: Registers your panel in SQL Lab
|
||||
- `backend.entryPoints`: Python modules to load eagerly when extension starts
|
||||
|
||||
## Step 4: Create Backend API
|
||||
|
||||
The CLI generated a basic `backend/src/superset_extensions/my_org/hello_world/entrypoint.py`. We'll create an API endpoint.
|
||||
|
||||
**Create `backend/src/superset_extensions/my_org/hello_world/api.py`**
|
||||
|
||||
```python
|
||||
from flask import Response
|
||||
from flask_appbuilder.api import expose, protect, safe
|
||||
from superset_core.api.rest_api import RestApi
|
||||
|
||||
|
||||
class HelloWorldAPI(RestApi):
|
||||
resource_name = "hello_world"
|
||||
openapi_spec_tag = "Hello World"
|
||||
class_permission_name = "hello_world"
|
||||
|
||||
@expose("/message", methods=("GET",))
|
||||
@protect()
|
||||
@safe
|
||||
def get_message(self) -> Response:
|
||||
"""Gets a hello world message
|
||||
---
|
||||
get:
|
||||
description: >-
|
||||
Get a hello world message from the backend
|
||||
responses:
|
||||
200:
|
||||
description: Hello world message
|
||||
content:
|
||||
application/json:
|
||||
schema:
|
||||
type: object
|
||||
properties:
|
||||
result:
|
||||
type: object
|
||||
properties:
|
||||
message:
|
||||
type: string
|
||||
401:
|
||||
$ref: '#/components/responses/401'
|
||||
"""
|
||||
return self.response(
|
||||
200,
|
||||
result={"message": "Hello from the backend!"}
|
||||
)
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
|
||||
- Extends `RestApi` from `superset_core.api.types.rest_api`
|
||||
- Uses Flask-AppBuilder decorators (`@expose`, `@protect`, `@safe`)
|
||||
- Returns responses using `self.response(status_code, result=data)`
|
||||
- The endpoint will be accessible at `/extensions/my-org/hello-world/message`
|
||||
- OpenAPI docstrings are crucial - Flask-AppBuilder uses them to automatically generate interactive API documentation at `/swagger/v1`, allowing developers to explore endpoints, understand schemas, and test the API directly from the browser
|
||||
|
||||
**Update `backend/src/superset_extensions/my_org/hello_world/entrypoint.py`**
|
||||
|
||||
Replace the generated print statement with API registration:
|
||||
|
||||
```python
|
||||
from superset_core.api import rest_api
|
||||
|
||||
from .api import HelloWorldAPI
|
||||
|
||||
rest_api.add_extension_api(HelloWorldAPI)
|
||||
```
|
||||
|
||||
This registers your API with Superset when the extension loads.
|
||||
|
||||
## Step 5: Create Frontend Component
|
||||
|
||||
The CLI generates the frontend configuration files. Below are the key configurations that enable Module Federation integration with Superset.
|
||||
|
||||
**`frontend/package.json`**
|
||||
|
||||
The `@apache-superset/core` package must be listed in both `peerDependencies` (to declare runtime compatibility) and `devDependencies` (to provide TypeScript types during build):
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "@my-org/hello-world",
|
||||
"version": "0.1.0",
|
||||
"private": true,
|
||||
"license": "Apache-2.0",
|
||||
"scripts": {
|
||||
"start": "webpack serve --mode development",
|
||||
"build": "webpack --stats-error-details --mode production"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"@apache-superset/core": "^x.x.x",
|
||||
"react": "^x.x.x",
|
||||
"react-dom": "^x.x.x"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@apache-superset/core": "^x.x.x",
|
||||
"@types/react": "^x.x.x",
|
||||
"ts-loader": "^x.x.x",
|
||||
"typescript": "^x.x.x",
|
||||
"webpack": "^5.x.x",
|
||||
"webpack-cli": "^x.x.x",
|
||||
"webpack-dev-server": "^x.x.x"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**`frontend/webpack.config.js`**
|
||||
|
||||
The webpack configuration requires specific settings for Module Federation. Key settings include `externalsType: "window"` and `externals` to map `@apache-superset/core` to `window.superset` at runtime, `import: false` for shared modules to use the host's React instead of bundling a separate copy, and `remoteEntry.[contenthash].js` for cache busting:
|
||||
|
||||
```javascript
|
||||
const path = require("path");
|
||||
const { ModuleFederationPlugin } = require("webpack").container;
|
||||
const packageConfig = require("./package.json");
|
||||
|
||||
module.exports = (env, argv) => {
|
||||
const isProd = argv.mode === "production";
|
||||
|
||||
return {
|
||||
entry: isProd ? {} : "./src/index.tsx",
|
||||
mode: isProd ? "production" : "development",
|
||||
devServer: {
|
||||
port: 3001,
|
||||
headers: {
|
||||
"Access-Control-Allow-Origin": "*",
|
||||
},
|
||||
},
|
||||
output: {
|
||||
filename: isProd ? undefined : "[name].[contenthash].js",
|
||||
chunkFilename: "[name].[contenthash].js",
|
||||
clean: true,
|
||||
path: path.resolve(__dirname, "dist"),
|
||||
publicPath: `/api/v1/extensions/my-org/hello-world/`,
|
||||
},
|
||||
resolve: {
|
||||
extensions: [".ts", ".tsx", ".js", ".jsx"],
|
||||
},
|
||||
// Map @apache-superset/core imports to window.superset at runtime
|
||||
externalsType: "window",
|
||||
externals: {
|
||||
"@apache-superset/core": "superset",
|
||||
},
|
||||
module: {
|
||||
rules: [
|
||||
{
|
||||
test: /\.tsx?$/,
|
||||
use: "ts-loader",
|
||||
exclude: /node_modules/,
|
||||
},
|
||||
],
|
||||
},
|
||||
plugins: [
|
||||
new ModuleFederationPlugin({
|
||||
name: "myOrg_helloWorld",
|
||||
filename: "remoteEntry.[contenthash].js",
|
||||
exposes: {
|
||||
"./index": "./src/index.tsx",
|
||||
},
|
||||
shared: {
|
||||
react: {
|
||||
singleton: true,
|
||||
requiredVersion: packageConfig.peerDependencies.react,
|
||||
import: false, // Use host's React, don't bundle
|
||||
},
|
||||
"react-dom": {
|
||||
singleton: true,
|
||||
requiredVersion: packageConfig.peerDependencies["react-dom"],
|
||||
import: false,
|
||||
},
|
||||
},
|
||||
}),
|
||||
],
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
**`frontend/tsconfig.json`**
|
||||
|
||||
```json
|
||||
{
|
||||
"compilerOptions": {
|
||||
"baseUrl": ".",
|
||||
"moduleResolution": "node",
|
||||
"jsx": "react",
|
||||
"strict": true,
|
||||
"esModuleInterop": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true
|
||||
},
|
||||
"include": ["src"]
|
||||
}
|
||||
```
|
||||
|
||||
**Create `frontend/src/HelloWorldPanel.tsx`**
|
||||
|
||||
Create a new file for the component implementation:
|
||||
|
||||
```tsx
|
||||
import React, { useEffect, useState } from 'react';
|
||||
import { authentication } from '@apache-superset/core';
|
||||
|
||||
const HelloWorldPanel: React.FC = () => {
|
||||
const [message, setMessage] = useState<string>('');
|
||||
const [loading, setLoading] = useState(true);
|
||||
const [error, setError] = useState<string>('');
|
||||
|
||||
useEffect(() => {
|
||||
const fetchMessage = async () => {
|
||||
try {
|
||||
const csrfToken = await authentication.getCSRFToken();
|
||||
const response = await fetch('/extensions/my-org/hello-world/message', {
|
||||
method: 'GET',
|
||||
headers: {
|
||||
'Content-Type': 'application/json',
|
||||
'X-CSRFToken': csrfToken!,
|
||||
},
|
||||
});
|
||||
|
||||
if (!response.ok) {
|
||||
throw new Error(`Server returned ${response.status}`);
|
||||
}
|
||||
|
||||
const data = await response.json();
|
||||
setMessage(data.result.message);
|
||||
} catch (err) {
|
||||
setError(err instanceof Error ? err.message : 'An error occurred');
|
||||
} finally {
|
||||
setLoading(false);
|
||||
}
|
||||
};
|
||||
|
||||
fetchMessage();
|
||||
}, []);
|
||||
|
||||
if (loading) {
|
||||
return (
|
||||
<div style={{ padding: '20px', textAlign: 'center' }}>
|
||||
<p>Loading...</p>
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
if (error) {
|
||||
return (
|
||||
<div style={{ padding: '20px', color: 'red' }}>
|
||||
<strong>Error:</strong> {error}
|
||||
</div>
|
||||
);
|
||||
}
|
||||
|
||||
return (
|
||||
<div style={{ padding: '20px' }}>
|
||||
<h3>Hello World Extension</h3>
|
||||
<div
|
||||
style={{
|
||||
padding: '16px',
|
||||
backgroundColor: '#f6ffed',
|
||||
border: '1px solid #b7eb8f',
|
||||
borderRadius: '4px',
|
||||
marginBottom: '16px',
|
||||
}}
|
||||
>
|
||||
<strong>{message}</strong>
|
||||
</div>
|
||||
<p>This message was fetched from the backend API! 🎉</p>
|
||||
</div>
|
||||
);
|
||||
};
|
||||
|
||||
export default HelloWorldPanel;
|
||||
```
|
||||
|
||||
**Update `frontend/src/index.tsx`**
|
||||
|
||||
Replace the generated code with the extension entry point:
|
||||
|
||||
```tsx
|
||||
import React from 'react';
|
||||
import { core } from '@apache-superset/core';
|
||||
import HelloWorldPanel from './HelloWorldPanel';
|
||||
|
||||
export const activate = (context: core.ExtensionContext) => {
|
||||
context.disposables.push(
|
||||
core.registerViewProvider('my-org.hello-world.main', () => <HelloWorldPanel />),
|
||||
);
|
||||
};
|
||||
|
||||
export const deactivate = () => {};
|
||||
```
|
||||
|
||||
**Key patterns:**
|
||||
|
||||
- `activate` function is called when the extension loads
|
||||
- `core.registerViewProvider` registers the component with ID `my-org.hello-world.main` (matching `extension.json`)
|
||||
- `authentication.getCSRFToken()` retrieves the CSRF token for API calls
|
||||
- Fetch calls to `/extensions/{publisher}/{name}/{endpoint}` reach your backend API
|
||||
- `context.disposables.push()` ensures proper cleanup
|
||||
|
||||
## Step 6: Install Dependencies
|
||||
|
||||
Install the frontend dependencies:
|
||||
|
||||
```bash
|
||||
cd frontend
|
||||
npm install
|
||||
cd ..
|
||||
```
|
||||
|
||||
## Step 7: Package the Extension
|
||||
|
||||
Create a `.supx` bundle for deployment:
|
||||
|
||||
```bash
|
||||
superset-extensions bundle
|
||||
```
|
||||
|
||||
This command automatically:
|
||||
|
||||
- Builds frontend assets using Webpack with Module Federation
|
||||
- Collects backend Python source files
|
||||
- Creates a `dist/` directory with:
|
||||
- `manifest.json` - Build metadata and asset references
|
||||
- `frontend/dist/` - Built frontend assets (remoteEntry.js, chunks)
|
||||
- `backend/` - Python source files
|
||||
- Packages everything into `my-org.hello-world-0.1.0.supx` - a zip archive with the specific structure required by Superset
|
||||
|
||||
## Step 8: Deploy to Superset
|
||||
|
||||
To deploy your extension, you need to enable extensions support and configure where Superset should load them from.
|
||||
|
||||
**Configure Superset**
|
||||
|
||||
Add the following to your `superset_config.py`:
|
||||
|
||||
```python
|
||||
# Enable extensions feature
|
||||
FEATURE_FLAGS = {
|
||||
"ENABLE_EXTENSIONS": True,
|
||||
}
|
||||
|
||||
# Set the directory where extensions are stored
|
||||
EXTENSIONS_PATH = "/path/to/extensions/folder"
|
||||
```
|
||||
|
||||
**Copy Extension Bundle**
|
||||
|
||||
Copy your `.supx` file to the configured extensions path:
|
||||
|
||||
```bash
|
||||
cp my-org.hello-world-0.1.0.supx /path/to/extensions/folder/
|
||||
```
|
||||
|
||||
**Restart Superset**
|
||||
|
||||
Restart your Superset instance to load the extension:
|
||||
|
||||
```bash
|
||||
# Restart your Superset server
|
||||
superset run
|
||||
```
|
||||
|
||||
Superset will extract and validate the extension metadata, load the assets, register the extension with its capabilities, and make it available for use.
|
||||
|
||||
## Step 9: Test Your Extension
|
||||
|
||||
1. **Open SQL Lab** in Superset
|
||||
2. Look for the **"Hello World"** panel in the panels dropdown or sidebar
|
||||
3. Open the panel - it should display "Hello from the backend!"
|
||||
4. Check that the message was fetched from your API endpoint
|
||||
|
||||
## Understanding the Flow
|
||||
|
||||
Here's what happens when your extension loads:
|
||||
|
||||
1. **Superset starts**: Reads `extension.json` and loads backend entrypoint
|
||||
2. **Backend registration**: `entrypoint.py` registers your API via `rest_api.add_extension_api()`
|
||||
3. **Frontend loads**: When SQL Lab opens, Superset fetches the remote entry file
|
||||
4. **Module Federation**: Webpack loads your extension code and resolves `@apache-superset/core` to `window.superset`
|
||||
5. **Activation**: `activate()` is called, registering your view provider
|
||||
6. **Rendering**: When the user opens your panel, React renders `<HelloWorldPanel />`
|
||||
7. **API call**: Component fetches data from `/extensions/my-org/hello-world/message`
|
||||
8. **Backend response**: Your Flask API returns the hello world message
|
||||
9. **Display**: Component shows the message to the user
|
||||
|
||||
## Next Steps
|
||||
|
||||
Now that you have a working extension, explore:
|
||||
|
||||
- **[Development](./development)** - Project structure, APIs, and development workflow
|
||||
- **[Contribution Types](./contribution-types)** - Other contribution points beyond panels
|
||||
- **[Deployment](./deployment)** - Packaging and deploying your extension
|
||||
- **[Security](./security)** - Security best practices for extensions
|
||||
|
||||
For a complete real-world example, examine the query insights extension in the Superset codebase.
|
||||
55
docs/developer_docs/extensions/registry.md
Normal file
55
docs/developer_docs/extensions/registry.md
Normal file
@@ -0,0 +1,55 @@
|
||||
---
|
||||
title: Community Extensions
|
||||
sidebar_position: 11
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Community Extensions
|
||||
|
||||
This page serves as a registry of community-created Superset extensions. These extensions are developed and maintained by community members and are not officially supported or vetted by the Apache Superset project. **Before installing any community extension, administrators are responsible for evaluating the extension's source code for security vulnerabilities, performance impact, UI/UX quality, and compatibility with their Superset deployment.**
|
||||
|
||||
## Extensions
|
||||
|
||||
| Name | Description | Author | Preview |
|
||||
| ------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| [Extensions API Explorer](https://github.com/michael-s-molina/superset-extensions/tree/main/api_explorer) | A SQL Lab panel that demonstrates the Extensions API by providing an interactive explorer for testing commands like getTabs, getCurrentTab, and getDatabases. Useful for extension developers to understand and experiment with the available APIs. | Michael S. Molina | <a href="/img/extensions/api-explorer.png" target="_blank"><img src="/img/extensions/api-explorer.png" alt="Extensions API Explorer" width="120" /></a> |
|
||||
| [SQL Query Flow Visualizer](https://github.com/msyavuz/superset-sql-visualizer) | A SQL Lab panel that transforms SQL queries into interactive flow diagrams, helping developers and analysts understand query execution paths and data relationships. | Mehmet Salih Yavuz | <a href="/img/extensions/sql-flow-visualizer.png" target="_blank"><img src="/img/extensions/sql-flow-visualizer.png" alt="SQL Flow Visualizer" width="120" /></a> |
|
||||
| [SQL Lab Export to Google Sheets](https://github.com/michael-s-molina/superset-extensions/tree/main/sqllab_gsheets) | A Superset extension that allows users to export SQL Lab query results directly to Google Sheets. | Michael S. Molina | <a href="/img/extensions/gsheets-export.png" target="_blank"><img src="/img/extensions/gsheets-export.png" alt="SQL Lab Export to Google Sheets" width="120" /></a> |
|
||||
| [SQL Lab Export to Parquet](https://github.com/rusackas/superset-extensions/tree/main/sqllab_parquet) | Export SQL Lab query results directly to Apache Parquet format with Snappy compression. | Evan Rusackas | <a href="/img/extensions/parquet-export.png" target="_blank"><img src="/img/extensions/parquet-export.png" alt="SQL Lab Export to Parquet" width="120" /></a> |
|
||||
| [SQL Lab Query Comparison](https://github.com/michael-s-molina/superset-extensions/tree/main/query_comparison) | A SQL Lab extension that enables side-by-side comparison of query results across different tabs, with GitHub-style diff visualization showing added/removed rows and columns. | Michael S. Molina | <a href="/img/extensions/query-comparison.png" target="_blank"><img src="/img/extensions/query-comparison.png" alt="Query Comparison" width="120" /></a> |
|
||||
| [SQL Lab Result Stats](https://github.com/michael-s-molina/superset-extensions/tree/main/result_stats) | A SQL Lab extension that automatically computes statistics for query results, providing type-aware analysis including numeric metrics (min, max, mean, median, std dev), string analysis (length, empty counts), and date range information. | Michael S. Molina | <a href="/img/extensions/result-stats.png" target="_blank"><img src="/img/extensions/result-stats.png" alt="Result Stats" width="120" /></a> |
|
||||
| [SQL Snippets](https://github.com/michael-s-molina/superset-extensions/tree/main/sql_snippets) | A SQL Lab extension that provides reusable SQL code snippets, enabling quick insertion of commonly used code blocks such as license headers, author information, and frequently used SQL patterns. | Michael S. Molina | <a href="/img/extensions/sql-snippets.png" target="_blank"><img src="/img/extensions/sql-snippets.png" alt="SQL Snippets" width="120" /></a> |
|
||||
| [SQL Lab Query Estimator](https://github.com/michael-s-molina/superset-extensions/tree/main/query_estimator) | A SQL Lab panel that analyzes query execution plans to estimate resource impact, detect performance issues like Cartesian products and high-cost operations, and visualize the query plan tree. | Michael S. Molina | <a href="/img/extensions/query-estimator.png" target="_blank"><img src="/img/extensions/query-estimator.png" alt="Query Estimator" width="120" /></a> |
|
||||
| [Editors Bundle](https://github.com/michael-s-molina/superset-extensions/tree/main/editors_bundle) | A Superset extension that demonstrates how to provide custom code editors for different languages. This extension showcases the editor contribution system by registering alternative editors that can replace Superset's default Ace editor. | Michael S. Molina | <a href="/img/extensions/editors-bundle.png" target="_blank"><img src="/img/extensions/editors-bundle.png" alt="Editors Bundle" width="120" /></a> |
|
||||
|
||||
## How to Add Your Extension
|
||||
|
||||
To add your extension to this registry, submit a pull request to the [Apache Superset repository](https://github.com/apache/superset) with the following changes:
|
||||
|
||||
1. Add a row to the **Extensions** table above using this format:
|
||||
|
||||
```markdown
|
||||
| [Your Extension](https://github.com/your-username/your-repo) | A brief description of your extension. | Your Name | <a href="/img/extensions/your-screenshot.png" target="_blank"><img src="/img/extensions/your-screenshot.png" alt="Your Extension" width="120" /></a> |
|
||||
```
|
||||
|
||||
2. Add a screenshot to `docs/static/img/extensions/` (recommended size: 800x450px, PNG or JPG format)
|
||||
|
||||
3. Submit your PR with a title like "docs: Add [Extension Name] to community extensions registry"
|
||||
35
docs/developer_docs/extensions/security.md
Normal file
35
docs/developer_docs/extensions/security.md
Normal file
@@ -0,0 +1,35 @@
|
||||
---
|
||||
title: Security
|
||||
sidebar_position: 9
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Security
|
||||
|
||||
By default, extensions are disabled and must be explicitly enabled by setting the `ENABLE_EXTENSIONS` feature flag. Built-in extensions are included as part of the Superset codebase and are held to the same security standards and review processes as the rest of the application.
|
||||
|
||||
For external extensions, administrators are responsible for evaluating and verifying the security of any extensions they choose to install, just as they would when installing third-party NPM or PyPI packages. At this stage, all extensions run in the same context as the host application, without additional sandboxing. This means that external extensions can impact the security and performance of a Superset environment in the same way as any other installed dependency.
|
||||
|
||||
We plan to introduce an optional sandboxed execution model for extensions in the future (as part of an additional SIP). Until then, administrators should exercise caution and follow best practices when selecting and deploying third-party extensions. A directory of community extensions is available in the [Community Extensions](./registry) page. Note that these extensions are not vetted by the Apache Superset project—administrators must evaluate each extension before installation.
|
||||
|
||||
**Any performance or security vulnerabilities introduced by external extensions should be reported directly to the extension author, not as Superset vulnerabilities.**
|
||||
|
||||
Any security concerns regarding built-in extensions (included in Superset's monorepo) should be reported to the Superset Security mailing list for triage and resolution by maintainers.
|
||||
440
docs/developer_docs/extensions/tasks.md
Normal file
440
docs/developer_docs/extensions/tasks.md
Normal file
@@ -0,0 +1,440 @@
|
||||
---
|
||||
title: Tasks
|
||||
sidebar_position: 10
|
||||
---
|
||||
|
||||
<!--
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# Global Task Framework
|
||||
|
||||
The Global Task Framework (GTF) provides a unified way to manage background tasks. It handles task execution, progress tracking, cancellation, and deduplication for both synchronous and asynchronous execution. The framework uses distributed locking internally to ensure race-free operations—you don't need to worry about concurrent task creation or cancellation conflicts.
|
||||
|
||||
## Enabling GTF
|
||||
|
||||
GTF is disabled by default and must be enabled via the `GLOBAL_TASK_FRAMEWORK` feature flag in your `superset_config.py`:
|
||||
|
||||
```python
|
||||
FEATURE_FLAGS = {
|
||||
"GLOBAL_TASK_FRAMEWORK": True,
|
||||
}
|
||||
```
|
||||
|
||||
When GTF is disabled:
|
||||
- The Task List UI menu item is hidden
|
||||
- The `/api/v1/task/*` endpoints return 404
|
||||
- Calling or scheduling a `@task`-decorated function raises `GlobalTaskFrameworkDisabledError`
|
||||
|
||||
:::note Future Migration
|
||||
When GTF is considered stable, it will replace legacy Celery tasks for built-in features like thumbnails and alerts & reports. Enabling this flag prepares your deployment for that migration.
|
||||
:::
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Define a Task
|
||||
|
||||
```python
|
||||
from superset_core.api.tasks import task, get_context
|
||||
|
||||
@task
|
||||
def process_data(dataset_id: int) -> None:
|
||||
ctx = get_context()
|
||||
|
||||
@ctx.on_cleanup
|
||||
def cleanup():
|
||||
logger.info("Processing complete")
|
||||
|
||||
data = fetch_dataset(dataset_id)
|
||||
process_and_cache(data)
|
||||
```
|
||||
|
||||
### Execute a Task
|
||||
|
||||
```python
|
||||
# Async execution - schedules on Celery worker
|
||||
task = process_data.schedule(dataset_id=123)
|
||||
print(task.status) # "pending"
|
||||
|
||||
# Sync execution - runs inline in current process
|
||||
task = process_data(dataset_id=123)
|
||||
# ... blocks until complete
|
||||
print(task.status) # "success"
|
||||
```
|
||||
|
||||
### Async vs Sync Execution
|
||||
|
||||
| Method | When to Use |
|
||||
|--------|-------------|
|
||||
| `.schedule()` | Long-running operations, background processing, when you need to return immediately |
|
||||
| Direct call | Short operations, when deduplication matters, when you need the result before responding |
|
||||
|
||||
Both execution modes provide the same task features: deduplication, progress tracking, cancellation, and visibility in the Task List UI. The difference is whether execution happens in a Celery worker (async) or inline (sync).
|
||||
|
||||
## Task Lifecycle
|
||||
|
||||
```
|
||||
PENDING ──→ IN_PROGRESS ────→ SUCCESS
|
||||
│ │
|
||||
│ ├──────────→ FAILURE
|
||||
│ ↓ ↑
|
||||
│ ABORTING ────────────┘
|
||||
│ │
|
||||
│ ├──────────→ TIMED_OUT (timeout)
|
||||
│ │
|
||||
└─────────────┴──────────→ ABORTED (user cancel)
|
||||
```
|
||||
|
||||
| Status | Description |
|
||||
|--------|-------------|
|
||||
| `PENDING` | Queued, awaiting execution |
|
||||
| `IN_PROGRESS` | Executing |
|
||||
| `ABORTING` | Abort/timeout triggered, abort handlers running |
|
||||
| `SUCCESS` | Completed successfully |
|
||||
| `FAILURE` | Failed with error or abort/cleanup handler exception |
|
||||
| `ABORTED` | Cancelled by user/admin |
|
||||
| `TIMED_OUT` | Exceeded configured timeout |
|
||||
|
||||
## Context API
|
||||
|
||||
Access task context via `get_context()` from within any `@task` function. The context provides methods for updating task metadata and registering handlers.
|
||||
|
||||
### Updating Task Metadata
|
||||
|
||||
Use `update_task()` to report progress and store custom payload data:
|
||||
|
||||
```python
|
||||
@task
|
||||
def my_task(items: list[int]) -> None:
|
||||
ctx = get_context()
|
||||
|
||||
for i, item in enumerate(items):
|
||||
result = process(item)
|
||||
ctx.update_task(
|
||||
progress=(i + 1, len(items)),
|
||||
payload={"last_result": result}
|
||||
)
|
||||
```
|
||||
|
||||
:::tip
|
||||
Call `update_task()` once per iteration for best performance. Frequent DB writes are throttled to limit metastore load, so batching progress and payload updates together in a single call ensures both are persisted at the same time.
|
||||
:::
|
||||
|
||||
#### Progress Formats
|
||||
|
||||
The `progress` parameter accepts three formats:
|
||||
|
||||
| Format | Example | Display |
|
||||
|--------|---------|---------|
|
||||
| `tuple[int, int]` | `progress=(3, 100)` | 3 of 100 (3%) with ETA |
|
||||
| `float` (0.0-1.0) | `progress=0.5` | 50% with ETA |
|
||||
| `int` | `progress=42` | 42 processed |
|
||||
|
||||
:::tip
|
||||
Use the tuple format `(current, total)` whenever possible. It provides the richest information to users: showing both the count and percentage, while still computing ETA automatically.
|
||||
:::
|
||||
|
||||
#### Payload
|
||||
|
||||
The `payload` parameter stores custom metadata that can help users understand what the task is doing. Each call to `update_task()` replaces the previous payload completely.
|
||||
|
||||
In the Task List UI, when a payload is defined, an info icon appears in the **Details** column. Users can hover over it to see the JSON content.
|
||||
|
||||
### Handlers
|
||||
|
||||
Register handlers to run cleanup logic or respond to abort requests:
|
||||
|
||||
| Handler | When it runs | Use case |
|
||||
|---------|--------------|----------|
|
||||
| `on_cleanup` | Always (success, failure, abort) | Release resources, close connections |
|
||||
| `on_abort` | When task is aborted | Set stop flag, cancel external operations |
|
||||
|
||||
```python
|
||||
@task
|
||||
def my_task() -> None:
|
||||
ctx = get_context()
|
||||
|
||||
@ctx.on_cleanup
|
||||
def cleanup():
|
||||
logger.info("Task ended, cleaning up")
|
||||
|
||||
@ctx.on_abort
|
||||
def handle_abort():
|
||||
logger.info("Abort requested")
|
||||
|
||||
# ... task logic
|
||||
```
|
||||
|
||||
Multiple handlers of the same type execute in LIFO order (last registered runs first). Abort handlers run first when abort is detected, then cleanup handlers run when the task ends.
|
||||
|
||||
#### Best-Effort Execution
|
||||
|
||||
**All registered handlers will always be attempted, even if one fails.** This ensures that a failure in one handler doesn't prevent other handlers from running their cleanup logic.
|
||||
|
||||
For example, if you have three cleanup handlers and the second one throws an exception:
|
||||
1. Handler 3 runs ✓
|
||||
2. Handler 2 throws an exception ✗ (logged, but execution continues)
|
||||
3. Handler 1 runs ✓
|
||||
|
||||
If any handler fails, the task is marked as `FAILURE` with combined error details showing all handler failures.
|
||||
|
||||
:::tip
|
||||
Write handlers to be independent and self-contained. Don't assume previous handlers succeeded, and don't rely on shared state between handlers.
|
||||
:::
|
||||
|
||||
## Making Tasks Abortable
|
||||
|
||||
When users click **Cancel** in the Task List, the system decides whether to **abort** (stop) the task or **unsubscribe** (remove the user from a shared task). Abort occurs when:
|
||||
- It's a private or system task
|
||||
- It's a shared task and the user is the last subscriber
|
||||
- An admin checks **Force abort** to stop the task for all subscribers
|
||||
|
||||
Pending tasks can always be aborted: they simply won't start. In-progress tasks require an abort handler to be abortable:
|
||||
|
||||
```python
|
||||
@task
|
||||
def abortable_task(items: list[str]) -> None:
|
||||
ctx = get_context()
|
||||
should_stop = False
|
||||
|
||||
@ctx.on_abort
|
||||
def handle_abort():
|
||||
nonlocal should_stop
|
||||
should_stop = True
|
||||
logger.info("Abort signal received")
|
||||
|
||||
@ctx.on_cleanup
|
||||
def cleanup():
|
||||
logger.info("Task ended, cleaning up")
|
||||
|
||||
for item in items:
|
||||
if should_stop:
|
||||
return # Exit gracefully
|
||||
process(item)
|
||||
```
|
||||
|
||||
**Key points:**
|
||||
- Registering `on_abort` marks the task as abortable and starts the abort listener
|
||||
- The abort handler fires automatically when abort is triggered
|
||||
- Use a flag pattern to gracefully stop processing at safe points
|
||||
- Without an abort handler, in-progress tasks cannot be aborted: the Cancel button in the Task List UI will be disabled
|
||||
|
||||
The framework automatically skips execution if a task was aborted while pending: no manual check needed at task start.
|
||||
|
||||
:::tip
|
||||
Always implement an abort handler for long-running tasks. This allows users to cancel unneeded tasks and free up worker capacity for other operations.
|
||||
:::
|
||||
|
||||
## Timeouts
|
||||
|
||||
Set a timeout to automatically abort tasks that run too long:
|
||||
|
||||
```python
|
||||
from superset_core.api.tasks import task, get_context, TaskOptions
|
||||
|
||||
# Set default timeout in decorator
|
||||
@task(timeout=300) # 5 minutes
|
||||
def process_data(dataset_id: int) -> None:
|
||||
ctx = get_context()
|
||||
should_stop = False
|
||||
|
||||
@ctx.on_abort
|
||||
def handle_abort():
|
||||
nonlocal should_stop
|
||||
should_stop = True
|
||||
|
||||
for chunk in fetch_large_dataset(dataset_id):
|
||||
if should_stop:
|
||||
return
|
||||
process(chunk)
|
||||
|
||||
# Override timeout at call time
|
||||
task = process_data.schedule(
|
||||
dataset_id=123,
|
||||
options=TaskOptions(timeout=600) # Override to 10 minutes
|
||||
)
|
||||
```
|
||||
|
||||
### How Timeouts Work
|
||||
|
||||
The timeout timer starts when the task begins executing (status changes to `IN_PROGRESS`). When the timeout expires:
|
||||
|
||||
1. **With an abort handler registered:** The task transitions to `ABORTING`, abort handlers run, then cleanup handlers run. The final status depends on handler execution:
|
||||
- If handlers complete successfully → `TIMED_OUT` status
|
||||
- If handlers throw an exception → `FAILURE` status
|
||||
|
||||
2. **Without an abort handler:** The framework cannot forcibly terminate the task. A warning is logged, and the task continues running. The Task List UI shows a warning indicator (⚠️) in the Details column to alert users that the timeout cannot be enforced.
|
||||
|
||||
### Timeout Precedence
|
||||
|
||||
| Source | Priority | Example |
|
||||
|--------|----------|---------|
|
||||
| `TaskOptions.timeout` | Highest | `options=TaskOptions(timeout=600)` |
|
||||
| `@task(timeout=...)` | Default | `@task(timeout=300)` |
|
||||
| Not set | No timeout | Task runs indefinitely |
|
||||
|
||||
Call-time options always override decorator defaults, allowing tasks to have sensible defaults while permitting callers to extend or shorten the timeout for specific use cases.
|
||||
|
||||
:::warning
|
||||
Timeouts require an abort handler to be effective. Without one, the timeout triggers only a warning and the task continues running. Always implement an abort handler when using timeouts.
|
||||
:::
|
||||
|
||||
## Deduplication
|
||||
|
||||
Use `task_key` to prevent duplicate task execution:
|
||||
|
||||
```python
|
||||
from superset_core.api.tasks import TaskOptions
|
||||
|
||||
# Without key - creates new task each time (random UUID)
|
||||
task1 = my_task.schedule(x=1)
|
||||
task2 = my_task.schedule(x=1) # Different task
|
||||
|
||||
# With key - joins existing task if active
|
||||
task1 = my_task.schedule(x=1, options=TaskOptions(task_key="report_123"))
|
||||
task2 = my_task.schedule(x=1, options=TaskOptions(task_key="report_123")) # Returns same task
|
||||
```
|
||||
|
||||
When a task with matching key already exists, the user is added as a subscriber and the existing task is returned. This behavior is consistent across all scopes—private tasks naturally have only one subscriber since their deduplication key includes the user ID.
|
||||
|
||||
Deduplication only applies to active tasks (pending/in-progress). Once a task completes, a new task with the same key can be created.
|
||||
|
||||
### Sync Join-and-Wait
|
||||
|
||||
When a sync call joins an existing task, it blocks until the task completes:
|
||||
|
||||
```python
|
||||
# Schedule async task
|
||||
task = my_task.schedule(options=TaskOptions(task_key="report_123"))
|
||||
|
||||
# Later sync call with same key blocks until completion of the active task
|
||||
task2 = my_task(options=TaskOptions(task_key="report_123"))
|
||||
assert task.uuid == task2.uuid # True
|
||||
print(task2.status) # "success" (terminal status)
|
||||
```
|
||||
|
||||
## Task Scopes
|
||||
|
||||
```python
|
||||
from superset_core.api.tasks import task, TaskScope
|
||||
|
||||
@task # Private by default
|
||||
def private_task(): ...
|
||||
|
||||
@task(scope=TaskScope.SHARED) # Multiple users can subscribe
|
||||
def shared_task(): ...
|
||||
|
||||
@task(scope=TaskScope.SYSTEM) # Admin-only visibility
|
||||
def system_task(): ...
|
||||
```
|
||||
|
||||
| Scope | Visibility | Cancel Behavior |
|
||||
|-------|------------|-----------------|
|
||||
| `PRIVATE` | Creator only | Cancels immediately |
|
||||
| `SHARED` | All subscribers | Last subscriber cancels; others unsubscribe |
|
||||
| `SYSTEM` | Admins only | Admin cancels |
|
||||
|
||||
## Task Cleanup
|
||||
|
||||
Completed tasks accumulate in the database over time. Configure a scheduled prune job to automatically remove old tasks:
|
||||
|
||||
```python
|
||||
# In your superset_config.py, add to your Celery beat schedule:
|
||||
CELERY_CONFIG.beat_schedule["prune_tasks"] = {
|
||||
"task": "prune_tasks",
|
||||
"schedule": crontab(minute=0, hour=0), # Run daily at midnight
|
||||
"kwargs": {
|
||||
"retention_period_days": 90, # Keep tasks for 90 days
|
||||
"max_rows_per_run": 10000, # Limit deletions per run
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
The prune job only removes tasks in terminal states (`SUCCESS`, `FAILURE`, `ABORTED`, `TIMED_OUT`). Active tasks (`PENDING`, `IN_PROGRESS`, `ABORTING`) are never pruned.
|
||||
|
||||
See `superset/config.py` for a complete example configuration.
|
||||
|
||||
:::tip Signal Cache for Faster Notifications
|
||||
By default, abort detection and sync join-and-wait use database polling. Configure `SIGNAL_CACHE_CONFIG` to enable Redis pub/sub for real-time notifications. See [Signal Cache Backend](/admin-docs/configuration/cache#signal-cache-backend) for configuration details.
|
||||
:::
|
||||
|
||||
## API Reference
|
||||
|
||||
### @task Decorator
|
||||
|
||||
```python
|
||||
@task(
|
||||
name: str | None = None,
|
||||
scope: TaskScope = TaskScope.PRIVATE,
|
||||
timeout: int | None = None
|
||||
)
|
||||
```
|
||||
|
||||
- `name`: Task identifier (defaults to function name)
|
||||
- `scope`: `PRIVATE`, `SHARED`, or `SYSTEM`
|
||||
- `timeout`: Default timeout in seconds (can be overridden via `TaskOptions`)
|
||||
|
||||
### TaskContext Methods
|
||||
|
||||
| Method | Description |
|
||||
|--------|-------------|
|
||||
| `update_task(progress, payload)` | Update progress and/or custom payload |
|
||||
| `on_cleanup(handler)` | Register cleanup handler |
|
||||
| `on_abort(handler)` | Register abort handler (makes task abortable) |
|
||||
|
||||
### TaskOptions
|
||||
|
||||
```python
|
||||
TaskOptions(
|
||||
task_key: str | None = None,
|
||||
task_name: str | None = None,
|
||||
timeout: int | None = None
|
||||
)
|
||||
```
|
||||
|
||||
- `task_key`: Deduplication key (also used as display name if `task_name` is not set)
|
||||
- `task_name`: Human-readable display name for the Task List UI
|
||||
- `timeout`: Timeout in seconds (overrides decorator default)
|
||||
|
||||
:::tip
|
||||
Provide a descriptive `task_name` for better readability in the Task List UI. While `task_key` is used for deduplication and may be technical (e.g., `chart_export_123`), `task_name` can be user-friendly (e.g., `"Export Sales Chart 123"`).
|
||||
:::
|
||||
|
||||
## Error Handling
|
||||
|
||||
Let exceptions propagate: the framework captures them automatically and sets task status to `FAILURE`:
|
||||
|
||||
```python
|
||||
@task
|
||||
def risky_task() -> None:
|
||||
# No try/catch needed - framework handles it
|
||||
result = operation_that_might_fail()
|
||||
```
|
||||
|
||||
On failure, the framework records:
|
||||
- `error_message`: Exception message
|
||||
- `exception_type`: Exception class name
|
||||
- `stack_trace`: Full traceback (visible when `SHOW_STACKTRACE=True`)
|
||||
|
||||
In the Task List UI, failed tasks show error details when hovering over the status. When stack traces are enabled, a separate bug icon appears in the **Details** column for viewing the full traceback.
|
||||
|
||||
Cleanup handlers still run after an exception, so resources can be properly released as necessary.
|
||||
|
||||
:::tip
|
||||
Use descriptive exception messages. In environments where stack traces are hidden (`SHOW_STACKTRACE=False`), users see only the error message and exception type when hovering over failed tasks. Clear messages help users troubleshoot issues without administrator assistance.
|
||||
:::
|
||||
Reference in New Issue
Block a user