Compare commits

..

58 Commits

Author SHA1 Message Date
Maxime Beauchemin
6d7467bafd feat: add devcontainer profiles for MCP development
- Split devcontainer config into base + profile structure
  - default: Standard Superset development (port 9001)
  - with-mcp: Superset + MCP service (ports 9001, 5008)
- Add MCP service to docker-compose-light.yml as optional profile
- Update start-superset.sh to conditionally start MCP when ENABLE_MCP=true
- Add MCP case to docker-bootstrap.sh for starting MCP service
- Preserve original devcontainer.json as .old for migration reference

This allows developers to choose their development environment:
- Use default profile for standard Superset development
- Use with-mcp profile for MCP/LLM agent development
- MCP service runs on port 5008 when enabled via profile

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-07-30 18:59:27 -07:00
Amin Ghadersohi
71f294b7d3 docs: update MCP Chart Test Plan with accurate implementation details
- Add authentication setup and dataset discovery prerequisites
- Remove references to unsupported preview formats (base64, interactive, vega_lite)
- Fix schema examples: operator -> op, xlsx -> excel
- Add performance limits for preview formats
- Update error expectations for empty query results
- Enhance cache testing with cache_timeout parameter
- Add Important Schema Notes section for common gotchas
- Update response patterns to match actual implementation
2025-07-30 20:15:41 -04:00
Amin Ghadersohi
e839d0989a refactor: remove redundant docs folder from mcp_service 2025-07-30 19:41:05 -04:00
Amin Ghadersohi
714e21b3ec feat: enhance MCP documentation with Superset best practices
- Add Docusaurus admonitions (:::note, :::tip, :::warning) following Superset patterns
- Improve code block formatting and language tags
- Add important notes about authentication and installation
- Ensure consistency with Superset documentation style
- Maintain accuracy with FastMCP specifications
2025-07-30 19:26:08 -04:00
Amin Ghadersohi
bf78eb69ed fix: resolve Mermaid diagram syntax errors in development guide
Replace @ symbols in node names that cause Mermaid parsing errors
2025-07-30 19:21:45 -04:00
Amin Ghadersohi
a13c1ba8c2 fix: update broken README.md links in MCP documentation
Replace ./README.md references with proper Docusaurus links to intro page
2025-07-30 19:20:38 -04:00
Amin Ghadersohi
422c34a6ee feat: integrate MCP service documentation into Superset Docusaurus site
- Add comprehensive MCP service documentation to docs/docs/mcp-service/
- Create 7 new documentation files covering all aspects of MCP service
- Add MCP Service section to Docusaurus sidebar configuration
- Update README files to reference official documentation
- Remove outdated standalone architecture documentation
- Install missing docs dependencies to fix eslint-docs pre-commit hook

Documentation includes:
- Introduction and overview with getting started guide
- Complete API reference with all 16 tools and examples
- Authentication setup with JWT and identity provider integration
- Development guide with internal architecture and patterns
- System architecture overview with deployment considerations
- Preset integration guide for enterprise features
2025-07-30 19:17:18 -04:00
Amin Ghadersohi
cd213fc57d update: enhance MCP service README with comprehensive setup instructions
Improve README with official Superset installation instructions, detailed Claude Desktop
connection steps using FastMCP proxy, and troubleshooting guide. Adds verification steps
and clear instructions for HTTP-based MCP service deployment.
2025-07-30 18:24:54 -04:00
Amin Ghadersohi
0f20a88598 feat(mcp): major enhancements to chart generation with validation, pooled screenshots, and simplified tests
This commit introduces significant improvements to the MCP chart generation system:

## Core Features
- Enhanced chart generation with comprehensive validation and error handling
- Implemented WebDriver pooling for 10x faster screenshot generation
- Added detailed validation for chart configurations with helpful error messages
- Support for both table and XY (line/bar/area/scatter) chart types

## Validation & Error Handling
- Column existence validation with similarity-based suggestions
- Data type compatibility checks for aggregations (SUM/AVG for numbers only)
- Filter operator validation
- Detailed error responses with actionable suggestions and available columns/metrics

## Performance Improvements
- WebDriver connection pooling eliminates browser startup overhead
- Reuses browser instances across multiple screenshot requests
- Automatic health checking and recovery of WebDriver instances
- Configurable pool size and timeout settings

## Screenshot Enhancements
- New PooledChartScreenshot and PooledExploreScreenshot classes
- FastAPI endpoints for serving chart and explore screenshots as PNG images
- Proper caching with TTL support
- Clean chart-only screenshots for explore views (hides UI elements)

## Test Improvements
- Simplified test structure focusing on schema validation over complex mocking
- Removed brittle integration-style tests in favor of unit tests
- Fixed all test failures with cleaner, more maintainable approach
- Added comprehensive test plan for manual testing with Claude Desktop

## API Enhancements
- Centralized URL generation for screenshots and explore links
- Consistent error response format across all endpoints
- Support for UUID-based chart lookups alongside numeric IDs

## Code Quality
- Fixed all pre-commit issues (mypy, ruff, long lines, complexity)
- Extracted helper functions to reduce cyclomatic complexity
- Proper type annotations throughout
- Better separation of concerns

## Documentation
- Added comprehensive test plan for chart tools
- Detailed guides for table chart configuration
- WebDriver pooling architecture documentation
- Demo scripts for testing functionality

The changes maintain backward compatibility while significantly improving
performance, reliability, and developer experience for chart generation in the
MCP service.
2025-07-30 18:09:23 -04:00
Amin Ghadersohi
33d16eaca1 refactor: rename pydantic_schemas to schemas for MCP service
Simplified the directory structure by renaming `pydantic_schemas` to `schemas` throughout the MCP service codebase. This follows standard Python conventions and makes imports more concise.

Changes:
- Renamed directory: superset/mcp_service/pydantic_schemas/ → superset/mcp_service/schemas/
- Updated all import statements across 42 files
- Updated documentation references in README files
- All tests pass, pre-commit hooks clean
2025-07-30 14:20:42 -04:00
Amin Ghadersohi
5b56bc622b feat: major MCP service enhancements with cache control, testing infrastructure, and bug fixes
This comprehensive update adds significant functionality and improvements to the MCP service:

  New Features:
  - Cache control utilities for metadata operations with configurable TTL
  - Entity testing plan documentation for systematic API validation
  - Chart data retrieval tool with caching support
  - Comprehensive request schema validation framework
  - Sortable columns testing infrastructure

  Documentation Updates:
  - Expanded README with architecture details and usage examples
  - Phase 1 status tracking and implementation notes
  - Enhanced schema documentation with cache control patterns
  - LLMS.md reference added to CLAUDE.md

  Search Functionality Fixes:
  - Dashboard search: removed invalid columns (owners, roles, tags, published)
  - Dataset search: removed non-existent 'tags' and problematic 'catalog' columns
  - Chart search: maintained correct text-only columns (slice_name, description)
  - Updated corresponding unit tests to match implementations

  Infrastructure Improvements:
  - Added comprehensive cache control test coverage
  - Request schema validation testing
  - Sortable columns validation across all entity types
  - Enhanced error handling and logging

  Technical Debt:
  - Standardized import patterns across MCP tools
  - Improved type safety in Pydantic schemas
  - Better separation of concerns for cache operations
2025-07-30 14:20:42 -04:00
Amin Ghadersohi
8cbfe027b5 update: add chart update tools and centralized URL configuration
- Add update_chart and update_chart_preview tools for modifying saved and cached charts
  - Implement centralized URL configuration using SUPERSET_WEBSERVER_ADDRESS
  - Add comprehensive MCP audit logging with impersonation tracking and payload  sanitization
  - Refactor duplicate chart analysis functions to chart_utils.py
  - Update all README documentation with new tools and features
  - Add 194+ unit tests covering URL utils, audit logging, and chart updates
  - Fix all pre-commit issues and ensure test coverage
2025-07-30 14:20:42 -04:00
Amin Ghadersohi
c3b3edc6ba refactor: rename create_chart to generate_chart and standardize MCP test naming
- Rename create_chart function to generate_chart throughout MCP service
  - Update CreateChartRequest to GenerateChartRequest schema
  - Rename test files to follow test_*.py convention consistently
  - Fix test class and method names (TestCreateChart → TestGenerateChart)
  - Update all documentation and README files with new naming
  - Standardize auth scope mappings and tool references
  - Maintain backward compatibility for Superset CreateChartCommand references
2025-07-30 14:20:42 -04:00
Amin Ghadersohi
aa06bb9fda revert unnecessary config change 2025-07-30 14:20:41 -04:00
Amin Ghadersohi
0f222b9034 update: revert comment change 2025-07-30 14:20:41 -04:00
Amin Ghadersohi
7044153ca4 revert: switch webdriver back to Firefox for chart
screenshots

  Changed WEBDRIVER_TYPE from "chrome" to "firefox" and removed
   Chrome-specific
  binary path from WEBDRIVER_CONFIGURATION. Firefox provides
  more reliable
  screenshot generation for the MCP service chart preview
  functionality.

  - Set WEBDRIVER_TYPE = "firefox"
  - Cleared binary_location in WEBDRIVER_CONFIGURATION to use
  default Firefox
  - Chart preview tools now use geckodriver instead of Chrome
  headless mode

  This resolves webdriver timeout issues in chart screenshot
  generation.
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
ce82c35bb6 docs: clarify MCP service capabilities and available operations
Updated all MCP service READMEs to accurately reflect available operations:

- Removed overuse of 'comprehensive' and 'complete' where it overstated capabilities
- Added clear 'Available Operations' section showing:
  - Read operations available for all entities
  - Create operations only for charts and dashboards
  - No update or delete operations available
- Updated tool descriptions to be more specific about capabilities
- Changed 'complete workflow automation' to 'dashboard creation and chart addition'
- Clarified that we have 5 chart types, not 'comprehensive' support
- Updated test count references to be consistent (185+ tests)

This ensures documentation accurately represents Phase 1 capabilities without
overstating what operations are available.
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
9b25bd973f docs: update MCP service documentation to reflect Phase 1 completion
Updated all MCP service README files to reflect current status:

- README_ARCHITECTURE.md: Updated to 95% completion status, documented new
  dashboard generation and SQL Lab tools, corrected file references
- README_PHASE1_STATUS.md: Marked all core epics as complete, updated tool
  count to 16, reflected recent technical completions
- README_SCHEMAS.md: Updated test coverage numbers, documented new schema
  enhancements, marked Phase 1 features as complete

Changes reflect the successful completion of core MCP service functionality
including BaseDAO enhancements, comprehensive testing coverage, and all
planned Phase 1 tools and features.
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
5f7502b85c feat: comprehensive MCP service enhancements with type safety and new tools
Core Improvements:
- Enhanced BaseDAO with type-safe UUID handling and flexible column support
- Added comprehensive test coverage (185+ passing unit tests)
- Eliminated hardcoded string checks with SQLAlchemy type inspection
- Removed unnecessary async patterns for better performance consistency

New MCP Tools & Features:
- Dashboard generation: create_dashboard and add_chart_to_existing_dashboard
- Chart data/preview: get_chart_data and get_chart_preview with multiple formats
- SQL Lab integration: open_sql_lab_with_context with proper parameter handling
- Enhanced chart creation with comprehensive type support

Technical Enhancements:
- Fixed SQL Lab parameter naming (dbid) for frontend compatibility
- Improved error handling with graceful UUID conversion fallbacks
- Enhanced multi-identifier support (ID, UUID, slug) across all tools
- Comprehensive request schema pattern eliminating LLM validation issues

Testing & Quality:
- Added BaseDAO unit tests with edge cases and error scenarios
- Enhanced MCP tool testing with proper mocking patterns
- Fixed trailing whitespace and pre-commit compliance
- Professional error handling and type validation

Documentation:
- Updated all MCP service READMEs with current status (95% complete)
- Documented all 16 MCP tools with examples and usage patterns
- Enhanced architecture documentation with recent improvements
- Updated Phase 1 status reflecting completion of core features

This commit consolidates Phase 1 MCP service development with production-ready
authentication, comprehensive testing, and complete tool functionality.
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
1d5372210f update: read files 2025-07-30 14:20:41 -04:00
Amin Ghadersohi
64af53f6f6 update: enhance MCP service auth with factory pattern and code cleanup
- Implement configurable auth factory pattern as suggested by @dpgaspar
  - Add unit test coverage
  - Refactor auth.py for better pythonic patterns and error handling
  - Add proper type annotations and logging throughout
  - Split complex functions into focused helper functions
  - Add detailed architecture documentation with Mermaid diagrams
  - Improve JWT token extraction with robust fallback logic
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
56b308340e update: use real Superset user for MCP authentication instead of MCPUser
Replace MCPUser wrapper with actual Flask-AppBuilder User from database
  to enable proper RBAC permission filtering. Fixes MCP service returning
  0 counts due to empty permission queries.
2025-07-30 14:20:41 -04:00
Amin Ghadersohi
3b2b7609a8 update: add optional JWT Bearer authentication to MCP service
Implement comprehensive JWT authentication system for production deployments:

  • Add BearerAuthProvider integration with FastMCP for RS256 token validation
  • Support both static public keys and JWKS endpoints for enterprise key rotation
  • Implement scope-based authorization (dashboard:read, chart:write, etc.)
  • Extract user identity from JWT sub claim for proper audit trails
  • Maintain backward compatibility with admin fallback when auth disabled
  • Add comprehensive test coverage for authentication flows
  • Update documentation with human-friendly security setup guide

  Authentication is disabled by default for development convenience.
  Configure via environment variables for production use.
2025-07-30 14:20:40 -04:00
Amin Ghadersohi
61a91f80fe update deps 2025-07-30 14:20:40 -04:00
Amin Ghadersohi
ed3c5ecbc2 enhance MCP service with multi-identifier support and comprehensive UUID/slug functionality
- Add UUID field to ChartInfo and DatasetInfo Pydantic schemas for complete serialization
  - Include UUID in chart and dataset serialization functions (serialize_chart_object,
  serialize_dataset_object)
  - UUID and slug are now included in default response columns for better discoverability:
    * Dashboards: UUID and slug in DEFAULT_DASHBOARD_COLUMNS and returned by default
    * Charts: UUID in DEFAULT_CHART_COLUMNS and returned by default
    * Datasets: UUID in DEFAULT_DATASET_COLUMNS and returned by default
  - Search functionality enhanced to include UUID/slug fields across all relevant tools
  - Add comprehensive test coverage for UUID/slug functionality:
    * Default column verification tests ensuring UUID/slug are in default responses
    * Response data verification tests confirming UUID/slug values are returned
    * Custom column selection tests for explicit UUID/slug requests
    * Metadata accuracy tests verifying columns_requested/columns_loaded tracking
  - Update documentation to reflect enhanced multi-identifier capabilities
  - All 132 tests pass with comprehensive verification of UUID/slug support
2025-07-30 14:20:40 -04:00
Amin Ghadersohi
c8fbd4233c add multi-identifier support and fix columns metadata accuracy
**Multi-Identifier Support:**
  - Enhance ModelGetInfoTool to support ID, UUID, and slug lookup
  - Add intelligent identifier detection for get_*_info tools
  - Dashboards: support ID, UUID, and slug via id_or_slug_filter
  - Datasets/Charts: support ID and UUID via direct database queries
  - Add GetDashboardInfoRequest, GetDatasetInfoRequest, GetChartInfoRequest schemas

  **Enhanced Default Columns & Metadata:**
  - Add uuid to default columns for all list tools (dashboards, datasets, charts)
  - Include uuid/slug in search columns for better discoverability
  - Fix columns_requested to accurately reflect user input vs defaults
  - Fix columns_loaded to show actual DAO columns requested vs serialized fields

  **Testing & Code Quality:**
  - Add multi-identifier tests for all get_*_info tools (ID/UUID/slug scenarios)
  - Remove unused serialized_columns variable in ModelListTool
  - Fix linting issues (line length, docstrings)

  This provides flexible identifier support while ensuring accurate metadata
  tracking for better LLM compatibility and debugging.
2025-07-30 14:20:40 -04:00
Amin Ghadersohi
fc85f68585 update: implement FastMCP complex inputs pattern for MCP tools
- Replace individual parameters with structured request schemas for list_datasets, list_charts, and
  list_dashboards
  - Fix validation issues where LLMs passed arrays/objects as strings
  - Add ListDatasetsRequest, ListChartsRequest, ListDashboardsRequest schemas
  - Fix object serialization (charts, dashboards, datasets)
  - Add validation to prevent search+filters conflicts
  - Update all tests and fix linting issues

  Resolves string/array validation ambiguity that caused tool failures when LLMs sent complex
  parameters incorrectly.
2025-07-30 14:20:40 -04:00
Amin Ghadersohi
e825bbe1f4 update: generate_explore_link and create_chart working well 2025-07-30 14:20:40 -04:00
Amin Ghadersohi
a80b637a2a update: add a save_chart boolean parameter to create chart so that generate link to explore can use its code and just pass false 2025-07-30 14:20:40 -04:00
Amin Ghadersohi
541b3bd727 update: add docs to proxy 2025-07-30 14:20:40 -04:00
Amin Ghadersohi
a909799e5c update: simplify create chart a lot: support area, line, bar, scatter and table charts.
example prompt:
- can you use superset dataset 2 to plot popular baby names in 2024

- plot airline delay by day of week and group by airline use dataset 6
2025-07-30 14:20:39 -04:00
Amin Ghadersohi
01329f1c62 update: tests and deps 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
67f621d360 update: remove create_chart_simple tool 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
b552dbf4a1 update: fix all lint issues 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
364af98c04 update: create_chart: Remove x_axis from groupby if present, update docs and tests
- Updates create_chart logic to automatically remove x_axis from groupby for ECharts timeseries charts, preventing duplicate dimension usage.
- Updates and expands unit test to verify x_axis is excluded from groupby, using improved test mocks for accurate backend simulation.
- Updates documentation (README.md, README_ARCHITECTURE.md, README_PHASE1_STATUS.md, README_SCHEMAS.md) to clarify create_chart tool behavior and schema, including new groupby/x_axis handling.
- No breaking changes to tool signatures; behavior is now more robust and LLM-friendly.
2025-07-30 14:20:39 -04:00
Amin Ghadersohi
afdb8b38a6 update: Add columns and metrics to DatasetInfo/list_datasets, improve test coverage
- Updated DatasetInfo schema to include columns and metrics fields, with new TableColumnInfo and SqlMetricInfo models.
- Updated serialize_dataset_object to serialize columns and metrics for each dataset.
- Modified list_datasets tool to use serialize_dataset_object and include columns/metrics by default.
- Improved and fixed all related unit tests to use proper MagicMock objects for columns/metrics and to parse JSON responses.
- Ensured LLM/OpenAPI compatibility for dataset listing and info tools.
2025-07-30 14:20:39 -04:00
Amin Ghadersohi
fc7ea804bc update: chart create working with union of types 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
7c256ae9aa update: docs 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
1b190abc3b update: add new ModelGetAvailableFiltersTool 2025-07-30 14:20:39 -04:00
Amin Ghadersohi
c6c71bf835 update: update tool registration to use decorators, and update all tests to use mcp client directly in the tests as recommended 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
cd5ead7f11 update: update tool registration 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
e5eebe28f9 update: refactor auth and model tools 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
9eac6ef433 update: add tests for mcp list and info tools 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
d523d523e5 update: refactor tools directories 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
91a3214ed4 update: wip 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
95b787f024 update: wip 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
39121791e8 update: wip 2025-07-30 14:20:38 -04:00
Amin Ghadersohi
9d40fe913f update: wip 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
748ae49c8c update: wip 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
a9d543b6f4 update: unify FastMCP server, modularize tools, and document new DAO-based architecture
- Replace dual Flask/FastAPI setup with a single, unified FastMCP server (`server.py`)
- Introduce `MCPDAOWrapper` for secure, context-aware DAO access (`dao_wrapper.py`)
- Refactor all MCP tools to be modular and domain-organized (`tools/dashboard/`, `tools/chart/`, `tools/dataset/`, `tools/system/`)
- Strongly type all tool contracts with Pydantic v2 models, including full field documentation for LLM/OpenAPI compatibility
- Refactor and extend `BaseDAO` for robust, generic CRUD/list operations
- Add and update documentation:
  - Architecture and flow diagrams (`README_ARCHITECTURE.md`)
  - Tool schema reference and usage instructions (`README.md`, `README_SCHEMAS.md`)
  - Phase 1 status and roadmap (`README_PHASE1_STATUS.md`)
- Implement and test all core list/info tools for dashboards, datasets, and charts, with full search and filter support
- Add chart creation tool (`create_chart_simple`)
- Provide extension points for Preset-specific auth, RBAC, and logging (stubbed in Phase 1)
- Prepare for LLM/agent workflows and future command-based mutations (create/update/delete)
- Expand and update unit/integration test coverage for all tools
2025-07-30 14:20:37 -04:00
Amin Ghadersohi
55d6130fc4 update: docs and deps 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
b98e3eb309 Superset MCP Service - Adding get_instance_info tool 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
b469077e0e update: add list function to BaseDAO and use it to get and filter the dashboards 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
397b4e450b Superset MCP Service - Initial Commit DRAFT 2025-07-30 14:20:37 -04:00
Amin Ghadersohi
0f97002520 update: Refactor BaseDAO: enhance column operator logic and expand test coverage
- Improved the BaseDAO class to robustly handle column operator logic, ensuring all supported operators (eq, ne, sw, ew, in, nin, gt, gte, lt, lte, like, ilike, is_null, is_not_null) are consistently applied via ColumnOperatorEnum.
- Refactored the apply_column_operators and list methods for clarity and reliability, including better handling of columns, relationships, and search.
- Removed 1 index base page handing from list
2025-07-30 14:20:37 -04:00
Amin Ghadersohi
2312250127 update: add idea of ColumnOperator 2025-07-30 14:20:36 -04:00
Amin Ghadersohi
cd52193869 update: add helper for applying base filter 2025-07-30 14:20:36 -04:00
Amin Ghadersohi
9ffe680aaa feat(dao): add robust generic list and count methods to BaseDAO with full test coverage
- Introduced generic `list` and `count` methods to BaseDAO for consistent querying and counting across all DAOs.
- Both methods support filtering (including IN queries), ordering, pagination, search across columns, custom FAB-style filters, and always-on base filters.
- Added comprehensive unit tests for `list` and `count` in `tests/unit_tests/dao/base_test.py`, covering:
  - Filtering (including boolean, None, and IN queries)
  - Ordering (asc/desc, multiple columns)
  - Pagination (including out-of-range)
  - Search across columns
  - Custom filter logic
  - Always-on base filter logic
  - Edge cases and skip_base_filter
- Moved common test fixtures to `conftest.py` for reuse.
2025-07-30 14:20:36 -04:00
2687 changed files with 54834 additions and 188481 deletions

View File

@@ -83,7 +83,6 @@ github:
- cypress-matrix (5, chrome)
- dependency-review
- frontend-build
- playwright-tests (chromium)
- pre-commit (current)
- pre-commit (previous)
- test-mysql

View File

@@ -1,10 +0,0 @@
# JavaScript to TypeScript Migration Command
## Usage
```
/js-to-ts <core-filename>
```
- `<core-filename>` - Path to CORE file relative to `superset-frontend/` (e.g., `src/utils/common.js`, `src/middleware/loggerMiddleware.js`)
## Agent Instructions
**See:** [../projects/js-to-ts/AGENT.md](../projects/js-to-ts/AGENT.md) for complete migration guide.

View File

@@ -1,684 +0,0 @@
# JavaScript to TypeScript Migration Agent Guide
**Complete technical reference for converting JavaScript/JSX files to TypeScript/TSX in Apache Superset frontend.**
**Agent Role:** Atomic migration unit - migrate the core file + ALL related tests/mocks as one cohesive unit. Use `git mv` to preserve history, NO `git commit`. NO global import changes. Report results upon completion.
---
## 🎯 Migration Principles
1. **Atomic migration units** - Core file + all related tests/mocks migrate together
2. **Zero `any` types** - Use proper TypeScript throughout
3. **Leverage existing types** - Reuse established definitions
4. **Type inheritance** - Derivatives extend base component types
5. **Strategic placement** - File types for maximum discoverability
6. **Surgical improvements** - Enhance existing types during migration
---
## Step 0: Dependency Check (MANDATORY)
**Command:**
```bash
grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" superset-frontend/{filename}
```
**Decision:**
- ✅ No matches → Proceed with atomic migration (core + tests + mocks)
- ❌ Matches found → EXIT with dependency report (see format below)
---
## Step 1: Identify Related Files (REQUIRED)
**Atomic Migration Scope:**
For core file `src/utils/example.js`, also migrate:
- `src/utils/example.test.js` / `src/utils/example.test.jsx`
- `src/utils/example.spec.js` / `src/utils/example.spec.jsx`
- `src/utils/__mocks__/example.js`
- Any other related test/mock files found by pattern matching
**Find all related test and mock files:**
```bash
# Pattern-based search for related files
basename=$(basename {filename} .js)
dirname=$(dirname superset-frontend/{filename})
# Find test files
find "$dirname" -name "${basename}.test.js" -o -name "${basename}.test.jsx"
find "$dirname" -name "${basename}.spec.js" -o -name "${basename}.spec.jsx"
# Find mock files
find "$dirname" -name "__mocks__/${basename}.js"
find "$dirname" -name "${basename}.mock.js"
```
**Migration Requirement:** All discovered related files MUST be migrated together as one atomic unit.
**Test File Creation:** If NO test files exist for the core file, CREATE a minimal test file using the following pattern:
- Location: Same directory as core file
- Name: `{basename}.test.ts` (e.g., `DebouncedMessageQueue.test.ts`)
- Content: Basic test structure importing and testing the main functionality
- Use proper TypeScript types in test file
---
## 🗺️ Type Reference Map
### From `@superset-ui/core`
```typescript
// Data & Query
QueryFormData, QueryData, JsonObject, AnnotationData, AdhocMetric
LatestQueryFormData, GenericDataType, DatasourceType, ExtraFormData
DataMaskStateWithId, NativeFilterScope, NativeFiltersState, NativeFilterTarget
// UI & Theme
FeatureFlagMap, LanguagePack, ColorSchemeConfig, SequentialSchemeConfig
```
### From `@superset-ui/chart-controls`
```typescript
Dataset, ColumnMeta, ControlStateMapping
```
### From Local Types (`src/types/`)
```typescript
// Authentication
User, UserWithPermissionsAndRoles, BootstrapUser, PermissionsAndRoles
// Dashboard
Dashboard, DashboardState, DashboardInfo, DashboardLayout, LayoutItem
ComponentType, ChartConfiguration, ActiveFilters
// Charts
Chart, ChartState, ChartStatus, ChartLinkedDashboard, Slice, SaveActionType
// Data
Datasource, Database, Owner, Role
// UI Components
TagType, FavoriteStatus, Filter, ImportResourceName
```
### From Domain Types
```typescript
// src/dashboard/types.ts
RootState, ChartsState, DatasourcesState, FilterBarOrientation
ChartCrossFiltersConfig, ActiveTabs, MenuKeys
// src/explore/types.ts
ExplorePageInitialData, ExplorePageState, ExploreResponsePayload, OptionSortType
// src/SqlLab/types.ts
[SQL Lab specific types]
```
---
## 🏗️ Type Organization Strategy
### Type Placement Hierarchy
1. **Component-Colocated** (90% of cases)
```typescript
// Same file as component
interface MyComponentProps {
title: string;
onClick: () => void;
}
```
2. **Feature-Shared**
```typescript
// src/[domain]/components/[Feature]/types.ts
export interface FilterConfiguration {
filterId: string;
targets: NativeFilterTarget[];
}
```
3. **Domain-Wide**
```typescript
// src/[domain]/types.ts
export interface ExploreFormData extends QueryFormData {
viz_type: string;
}
```
4. **Global**
```typescript
// src/types/[TypeName].ts
export interface ApiResponse<T> {
result: T;
count?: number;
}
```
### Type Discovery Commands
```bash
# Search existing types before creating
find superset-frontend/src -name "types.ts" -exec grep -l "[TypeConcept]" {} \;
grep -r "interface.*Props\|type.*Props" superset-frontend/src/
```
### Derivative Component Patterns
**Rule:** Components that extend others should extend their type interfaces.
```typescript
// ✅ Base component type
interface SelectProps {
value: string | number;
options: SelectOption[];
onChange: (value: string | number) => void;
disabled?: boolean;
}
// ✅ Derivative extends base
interface ChartSelectProps extends SelectProps {
charts: Chart[];
onChartSelect: (chart: Chart) => void;
}
// ✅ Derivative with modified props
interface DatabaseSelectProps extends Omit<SelectProps, 'value' | 'onChange'> {
value: number; // Narrowed type
onChange: (databaseId: number) => void; // Specific signature
}
```
**Common Patterns:**
- **Extension:** `extends BaseProps` - adds new props
- **Omission:** `Omit<BaseProps, 'prop'>` - removes props
- **Modification:** `Omit<BaseProps, 'prop'> & { prop: NewType }` - changes prop type
- **Restriction:** Override with narrower types (union → specific)
---
## 📋 Migration Recipe
### Step 2: File Conversion
```bash
# Use git mv to preserve history
git mv component.js component.ts
git mv Component.jsx Component.tsx
```
### Step 3: Import & Type Setup
```typescript
// Import order (enforced by linting)
import { FC, ReactNode } from 'react';
import { JsonObject, QueryFormData } from '@superset-ui/core';
import { Dataset } from '@superset-ui/chart-controls';
import type { Dashboard } from 'src/types/Dashboard';
```
### Step 4: Function & Component Typing
```typescript
// Functions with proper parameter/return types
export function processData(
data: Dataset[],
config: JsonObject
): ProcessedData[] {
// implementation
}
// Component props with inheritance
interface ComponentProps extends BaseProps {
data: Chart[];
onSelect: (id: number) => void;
}
const Component: FC<ComponentProps> = ({ data, onSelect }) => {
// implementation
};
```
### Step 5: State & Redux Typing
```typescript
// Hooks with specific types
const [data, setData] = useState<Chart[]>([]);
const [selected, setSelected] = useState<number | null>(null);
// Redux with existing RootState
const mapStateToProps = (state: RootState) => ({
charts: state.charts,
user: state.user,
});
```
---
## 🧠 Type Debugging Strategies (Real-World Learnings)
### The Evolution of Type Approaches
When you hit type errors, follow this debugging evolution:
#### 1. ❌ Idealized Union Types (First Attempt)
```typescript
// Looks clean but doesn't match reality
type DatasourceInput = Datasource | QueryEditor;
```
**Problem**: Real calling sites pass variations, not exact types.
#### 2. ❌ Overly Precise Types (Second Attempt)
```typescript
// Tried to match exact calling signatures
type DatasourceInput =
| IDatasource // From DatasourcePanel
| (QueryEditor & { columns: ColumnMeta[] }); // From SaveQuery
```
**Problem**: Too rigid, doesn't handle legacy variations.
#### 3. ✅ Flexible Interface (Final Solution)
```typescript
// Captures what the function actually needs
interface DatasourceInput {
name?: string | null; // Allow null for compatibility
datasource_name?: string | null; // Legacy variations
columns?: any[]; // Multiple column types accepted
database?: { id?: number };
// ... other optional properties
}
```
**Success**: Works with all calling sites, focuses on function needs.
### Type Debugging Process
1. **Start with compilation errors** - they show exact mismatches
2. **Examine actual usage** - look at calling sites, not idealized types
3. **Build flexible interfaces** - capture what functions need, not rigid contracts
4. **Iterate based on downstream validation** - let calling sites guide your types
---
## 🚨 Anti-Patterns to Avoid
```typescript
// ❌ Never use any
const obj: any = {};
// ✅ Use proper types
const obj: Record<string, JsonObject> = {};
// ❌ Don't recreate base component props
interface ChartSelectProps {
value: string; // Duplicated from SelectProps
onChange: () => void; // Duplicated from SelectProps
charts: Chart[]; // New prop
}
// ✅ Inherit and extend
interface ChartSelectProps extends SelectProps {
charts: Chart[]; // Only new props
}
// ❌ Don't create ad-hoc type variations
interface UserInfo {
name: string;
email: string;
}
// ✅ Extend existing types (DRY principle)
import { User } from 'src/types/bootstrapTypes';
type UserDisplayInfo = Pick<User, 'firstName' | 'lastName' | 'email'>;
// ❌ Don't create overly rigid unions
type StrictInput = ExactTypeA | ExactTypeB;
// ✅ Create flexible interfaces for function parameters
interface FlexibleInput {
// Focus on what the function actually needs
commonProperty: string;
optionalVariations?: any; // Allow for legacy variations
}
```
## 📍 DRY Type Guidelines (WHERE TYPES BELONG)
### Type Placement Rules
**CRITICAL**: Type variations must live close to where they belong, not scattered across files.
#### ✅ Proper Type Organization
```typescript
// ❌ Don't create one-off interfaces in utility files
// src/utils/datasourceUtils.ts
interface DatasourceInput { /* custom interface */ } // Wrong!
// ✅ Use existing types or extend them in their proper domain
// src/utils/datasourceUtils.ts
import { IDatasource } from 'src/explore/components/DatasourcePanel';
import { QueryEditor } from 'src/SqlLab/types';
// Create flexible interface that references existing types
interface FlexibleDatasourceInput {
// Properties that actually exist across variations
}
```
#### Type Location Hierarchy
1. **Domain Types**: `src/{domain}/types.ts` (dashboard, explore, SqlLab)
2. **Component Types**: Co-located with components
3. **Global Types**: `src/types/` directory
4. **Utility Types**: Only when they truly don't belong elsewhere
#### ✅ DRY Type Patterns
```typescript
// ✅ Extend existing domain types
interface SaveQueryData extends Pick<QueryEditor, 'sql' | 'dbId' | 'catalog'> {
columns: ColumnMeta[]; // Add what's needed
}
// ✅ Create flexible interfaces for cross-domain utilities
interface CrossDomainInput {
// Common properties that exist across different source types
name?: string | null; // Accommodate legacy null values
// Only include properties the function actually uses
}
```
---
## 🎯 PropTypes Auto-Generation (Elegant Approach)
**IMPORTANT**: Superset has `babel-plugin-typescript-to-proptypes` configured to automatically generate PropTypes from TypeScript interfaces. Use this instead of manual PropTypes duplication!
### ❌ Manual PropTypes Duplication (Avoid This)
```typescript
export interface MyComponentProps {
title: string;
count?: number;
}
// 8+ lines of manual PropTypes duplication 😱
const propTypes = PropTypes.shape({
title: PropTypes.string.isRequired,
count: PropTypes.number,
});
export default propTypes;
```
### ✅ Auto-Generated PropTypes (Use This)
```typescript
import { InferProps } from 'prop-types';
export interface MyComponentProps {
title: string;
count?: number;
}
// Single validator function - babel plugin auto-generates PropTypes! ✨
export default function MyComponentValidator(props: MyComponentProps) {
return null; // PropTypes auto-assigned by babel-plugin-typescript-to-proptypes
}
// Optional: For consumers needing PropTypes type inference
export type MyComponentPropsInferred = InferProps<typeof MyComponentValidator>;
```
### Migration Pattern for Type-Only Files
**When migrating type-only files with manual PropTypes:**
1. **Keep the TypeScript interfaces** (single source of truth)
2. **Replace manual PropTypes** with validator function
3. **Remove PropTypes imports** and manual shape definitions
4. **Add InferProps import** if type inference needed
**Example Migration:**
```typescript
// Before: 25+ lines with manual PropTypes duplication
export interface AdhocFilterType { /* ... */ }
const adhocFilterTypePropTypes = PropTypes.oneOfType([...]);
// After: 3 lines with auto-generation
export interface AdhocFilterType { /* ... */ }
export default function AdhocFilterValidator(props: { filter: AdhocFilterType }) {
return null; // Auto-generated PropTypes by babel plugin
}
```
### Component PropTypes Pattern
**For React components, the babel plugin works automatically:**
```typescript
interface ComponentProps {
title: string;
onClick: () => void;
}
const MyComponent: FC<ComponentProps> = ({ title, onClick }) => {
// Component implementation
};
// PropTypes automatically generated by babel plugin - no manual work needed!
export default MyComponent;
```
### Auto-Generation Benefits
- ✅ **Single source of truth**: TypeScript interfaces drive PropTypes
- ✅ **No duplication**: Eliminate 15-20 lines of manual PropTypes code
- ✅ **Automatic updates**: Changes to TypeScript automatically update PropTypes
- ✅ **Type safety**: Compile-time checking ensures PropTypes match interfaces
- ✅ **Backward compatibility**: Existing JavaScript components continue working
### Babel Plugin Configuration
The plugin is already configured in `babel.config.js`:
```javascript
['babel-plugin-typescript-to-proptypes', { loose: true }]
```
**No additional setup required** - just use TypeScript interfaces and the plugin handles the rest!
---
## 🧪 Test File Migration Patterns
### Test File Priority
- **Always migrate test files** alongside production files
- **Test files are often leaf nodes** - good starting candidates
- **Create tests if missing** - Leverage new TypeScript types for better test coverage
### Test-Specific Type Patterns
```typescript
// Mock interfaces for testing
interface MockStore {
getState: () => Partial<RootState>; // Partial allows minimal mocking
}
// Type-safe mocking for complex objects
const mockDashboardInfo: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
};
// Sinon stub typing
let postStub: sinon.SinonStub;
beforeEach(() => {
postStub = sinon.stub(SupersetClient, 'post');
});
// Use stub reference instead of original method
expect(postStub.callCount).toBe(1);
expect(postStub.getCall(0).args[0].endpoint).toMatch('/api/');
```
### Test Migration Recipe
1. **Migrate production file first** (if both need migration)
2. **Update test imports** to point to `.ts/.tsx` files
3. **Add proper mock typing** using `Partial<T> as T` pattern
4. **Fix stub typing** - Use stub references, not original methods
5. **Verify all tests pass** with TypeScript compilation
---
## 🔧 Type Conflict Resolution
### Multiple Type Definitions Issue
**Problem**: Same type name defined in multiple files causes compilation errors.
**Example**: `DashboardInfo` defined in both:
- `src/dashboard/reducers/types.ts` (minimal)
- `src/dashboard/components/Header/types.ts` (different shape)
- `src/dashboard/types.ts` (complete - used by RootState)
### Resolution Strategy
1. **Identify the authoritative type**:
```bash
# Find which type is used by RootState/main interfaces
grep -r "DashboardInfo" src/dashboard/types.ts
```
2. **Use import from authoritative source**:
```typescript
// ✅ Import from main domain types
import { RootState, DashboardInfo } from 'src/dashboard/types';
// ❌ Don't import from component-specific files
import { DashboardInfo } from 'src/dashboard/components/Header/types';
```
3. **Mock complex types in tests**:
```typescript
// For testing - provide minimal required fields
const mockInfo: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
// Only provide fields actually used in test
};
```
### Type Hierarchy Discovery Commands
```bash
# Find all definitions of a type
grep -r "interface.*TypeName\|type.*TypeName" src/
# Find import usage patterns
grep -r "import.*TypeName" src/
# Check what RootState uses
grep -A 10 -B 10 "TypeName" src/*/types.ts
```
---
## Agent Constraints (CRITICAL)
1. **Use git mv** - Run `git mv file.js file.ts` to preserve git history, but NO `git commit`
2. **NO global import changes** - Don't update imports across codebase
3. **Type files OK** - Can modify existing type files to improve/align types
4. **Single-File TypeScript Validation** (CRITICAL) - tsc has known issues with multi-file compilation:
- **Core Issue**: TypeScript's `tsc` has documented problems validating multiple files simultaneously in complex projects
- **Solution**: ALWAYS validate files one at a time using individual `tsc` calls
- **Command Pattern**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {single-file-path}`
- **Why**: Multi-file validation can produce false positives, miss real errors, and conflict during parallel agent execution
5. **Downstream Impact Validation** (CRITICAL) - Your migration affects calling sites:
- **Find downstream files**: `find superset-frontend/src -name "*.tsx" -o -name "*.ts" | xargs grep -l "your-core-filename" 2>/dev/null || echo "No files found"`
- **Validate each downstream file individually**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}`
- **Fix type mismatches** you introduced in calling sites
- **NEVER ignore downstream errors** - they indicate your types don't match reality
6. **Avoid Project-Wide Validation During Migration**:
- **NEVER use `npm run type`** during parallel agent execution - produces unreliable results
- **Single-file validation is authoritative** - trust individual file checks over project-wide scans
6. **ESLint validation** - Run `npm run eslint -- --fix {file}` for each migrated file to auto-fix formatting/linting issues
6. Zero `any` types - use proper TypeScript types
7. Search existing types before creating new ones
8. Follow patterns from this guide
---
## Success Report Format
```
SUCCESS: Atomic Migration of {core-filename}
## Files Migrated (Atomic Unit)
- Core: {core-filename} → {core-filename.ts/tsx}
- Tests: {list-of-test-files} → {list-of-test-files.ts/tsx} OR "CREATED: {basename}.test.ts"
- Mocks: {list-of-mock-files} → {list-of-mock-files.ts}
- Type files modified: {list-of-type-files}
## Types Created/Improved
- {TypeName}: {location} ({scope}) - {rationale}
- {ExistingType}: enhanced in {location} - {improvement-description}
## Documentation Recommendations
- ADD_TO_DIRECTORY: {TypeName} - {reason}
- NO_DOCUMENTATION: {TypeName} - {reason}
## Quality Validation
- **Single-File TypeScript Validation**: ✅ PASS - Core files individually validated
- Core file: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {core-file}`
- Test files: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {test-file}` (if exists)
- **Downstream Impact Check**: ✅ PASS - Found {N} files importing this module, all validate individually
- Downstream files: {list-of-files-that-import-your-module}
- Individual validation: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}`
- **ESLint validation**: ✅ PASS (using `npm run eslint -- --fix {files}` to auto-fix formatting)
- **Zero any types**: ✅ PASS
- **Local imports resolved**: ✅ PASS
- **Functionality preserved**: ✅ PASS
- **Tests pass** (if test file): ✅ PASS
- **Follow-up action required**: {YES/NO}
## Validation Strategy Notes
- **Single-file approach used**: Avoided multi-file tsc validation due to known TypeScript compilation issues
- **Project-wide validation skipped**: `npm run type` not used during parallel migration to prevent false positives
## Migration Learnings
- Type conflicts encountered: {describe any multiple type definitions}
- Mock patterns used: {describe test mocking approaches}
- Import hierarchy decisions: {note authoritative type sources used}
- PropTypes strategy: {AUTO_GENERATED via babel plugin | MANUAL_DUPLICATION_REMOVED | N/A}
## Improvement Suggestions for Documentation
- AGENT.md enhancement: {suggest additions to migration guide}
- Common pattern identified: {note reusable patterns for future migrations}
```
---
## Dependency Block Report Format
```
DEPENDENCY_BLOCK: Cannot migrate {filename}
## Blocking Dependencies
- {path}: {type} - {usage} - {priority}
## Impact Analysis
- Estimated types: {number}
- Expected locations: {list}
- Cross-domain: {YES/NO}
## Recommended Order
{ordered-list}
```
---
## 📚 Quick Reference
**Type Utilities:**
- `Record<K, V>` - Object with specific key/value types
- `Partial<T>` - All properties optional
- `Pick<T, K>` - Subset of properties
- `Omit<T, K>` - Exclude specific properties
- `NonNullable<T>` - Exclude null/undefined
**Event Types:**
- `MouseEvent<HTMLButtonElement>`
- `ChangeEvent<HTMLInputElement>`
- `FormEvent<HTMLFormElement>`
**React Types:**
- `FC<Props>` - Functional component
- `ReactNode` - Any renderable content
- `CSSProperties` - Style objects
---
**Remember:** Every type should add value and clarity. The goal is meaningful type safety that catches bugs and improves developer experience.

View File

@@ -1,199 +0,0 @@
# JS-to-TS Coordinator Workflow
**Role:** Strategic migration coordination - select leaf-node files, trigger agents, review results, handle integration, manage dependencies.
---
## 1. Core File Selection Strategy
**Target ONLY Core Files**: Coordinators identify core files (production code), agents handle related tests/mocks atomically.
**File Analysis Commands**:
```bash
# Find CORE files with no JS/JSX dependencies (exclude tests/mocks) - SIZE PRIORITIZED
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | xargs wc -l | sort -n | head -20
# Alternative: Get file sizes in lines with paths
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | while read file; do
lines=$(wc -l < "$file")
echo "$lines $file"
done | sort -n | head -20
# Check dependencies for core files only (start with smallest)
for file in <core-files-sorted-by-size>; do
echo "=== $file ($(wc -l < "$file") lines) ==="
grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" || echo "✅ LEAF CANDIDATE"
done
# Identify heavily imported files (migrate last)
grep -r "from.*utils/common" superset-frontend/src/ | wc -l
# Quick leaf analysis with size priority
find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | head -30 | while read file; do
deps=$(grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" | wc -l)
lines=$(wc -l < "$file")
if [ "$deps" -eq 0 ]; then
echo "✅ LEAF: $lines lines - $file"
fi
done | sort -n
```
**Priority Order** (Smallest files first for easier wins):
1. **Small leaf files** (<50 lines) - No JS/JSX imports, quick TypeScript conversion
2. **Medium leaf files** (50-200 lines) - Self-contained utilities and helpers
3. **Small dependency files** (<100 lines) - Import only already-migrated files
4. **Larger components** (200+ lines) - Complex but well-contained functionality
5. **Core foundational files** (utils/common.js, controls.jsx) - migrate last regardless of size
**Size-First Benefits**:
- Faster completion builds momentum
- Earlier validation of migration patterns
- Easier rollback if issues arise
- Better success rate for agent learning
**Migration Unit**: Each agent call migrates:
- 1 core file (primary target)
- All related `*.test.js/jsx` files
- All related `*.mock.js` files
- All related `__mocks__/` files
---
## 2. Task Creation & Agent Control
### Task Triggering
When triggering the `/js-to-ts` command:
- **Task Title**: Use the core filename as the task title (e.g., "DebouncedMessageQueue.js migration", "hostNamesConfig.js migration")
- **Task Description**: Include the full relative path to help agent locate the file
- **Reference**: Point agent to [AGENT.md](./AGENT.md) for technical instructions
### Post-Processing Workflow
After each agent completes:
1. **Review Agent Report**: Always read and analyze the complete agent report
2. **Share Summary**: Provide user with key highlights from agent's work:
- Files migrated (core + tests/mocks)
- Types created or improved
- Any validation issues or coordinator actions needed
3. **Quality Assessment**: Evaluate agent's TypeScript implementation against criteria:
-**Type Usage**: Proper types used, no `any` types
-**Type Filing**: Types placed in correct hierarchy (component → feature → domain → global)
-**Side Effects**: No unintended changes to other files
-**Import Alignment**: Proper .ts/.tsx import extensions
4. **Integration Decision**:
- **COMMIT**: If agent work is complete and high quality
- **FIX & COMMIT**: If minor issues need coordinator fixes
- **ROLLBACK**: If major issues require complete rework
5. **Next Action**: Ask user preference - commit this work or trigger next migration
---
## 3. Integration Decision Framework
**Automatic Integration** ✅:
- `npm run type` passes without errors
- Agent created clean TypeScript with proper types
- Types appropriately filed in hierarchy
**Coordinator Integration** (Fix Side-Effects) 🔧:
- `npm run type` fails BUT agent's work is high quality
- Good type usage, proper patterns, well-organized
- Side-effects are manageable TypeScript compilation errors
- **Coordinator Action**: Integrate the change, then fix global compilation issues
**Rollback Only** ❌:
- Agent introduced `any` types or poor type choices
- Types poorly organized or conflicting with existing patterns
- Fundamental approach issues requiring complete rework
**Integration Process**:
1. **Review**: Agent already used `git mv` to preserve history
2. **Fix Side-Effects**: Update dependent files with proper import extensions
3. **Resolve Types**: Fix any cascading type issues across codebase
4. **Validate**: Ensure `npm run type` passes after fixes
---
## 4. Common Integration Patterns
**Common Side-Effects (Expect These)**:
- **Type import conflicts**: Multiple definitions of same type name
- **Mock object typing**: Tests need complete type satisfaction
- **Stub method references**: Use stub vars instead of original methods
**Coordinator Fixes (Standard Process)**:
1. **Import Resolution**:
```bash
# Find authoritative type source
grep -r "TypeName" src/*/types.ts
# Import from domain types (src/dashboard/types.ts) not component types
```
2. **Test Mock Completion**:
```typescript
// Use Partial<T> as T pattern for minimal mocking
const mockDashboard: Partial<DashboardInfo> as DashboardInfo = {
id: 123,
json_metadata: '{}',
};
```
3. **Stub Reference Fixes**:
```typescript
// ✅ Use stub variable
expect(postStub.callCount).toBe(1);
// ❌ Don't use original method
expect(SupersetClient.post.callCount).toBe(1);
```
4. **Validation Commands**:
```bash
npm run type # TypeScript compilation
npm test -- filename # Test functionality
git status # Should show rename, not add/delete
```
---
## 5. File Categories for Planning
### Leaf Files (Start Here)
**Self-contained files with minimal JS/JSX dependencies**:
- Test files (80 files) - Usually only import the file being tested
- Utility files without internal dependencies
- Components importing only external libraries
### Heavily Imported Files (Migrate Last)
**Core files that many others depend on**:
- `utils/common.js` - Core utility functions
- `utils/reducerUtils.js` - Redux helpers
- `@superset-ui/core` equivalent files
- Major state management files (`explore/store.js`, `dashboard/actions/`)
### Complex Components (Middle Priority)
**Large files requiring careful type analysis**:
- `components/Datasource/DatasourceEditor.jsx` (1,809 lines)
- `explore/components/controls/AnnotationLayerControl/AnnotationLayer.jsx` (1,031 lines)
- `explore/components/ExploreViewContainer/index.jsx` (911 lines)
---
## 6. Success Metrics & Continuous Improvement
**Per-File Gates**:
- ✅ `npm run type` passes after each migration
- ✅ Zero `any` types introduced
- ✅ All imports properly typed
- ✅ Types filed in correct hierarchy
**Linear Scheduling**:
When agents report `DEPENDENCY_BLOCK`:
- Queue dependencies in linear order
- Process one file at a time to avoid conflicts
- Handle cascading type changes between files
**After Each Migration**:
1. **Update guides** with new patterns discovered
2. **Document coordinator fixes** that become common
3. **Enhance agent instructions** based on recurring issues
4. **Track success metrics** - automatic vs coordinator integration rates

View File

@@ -1,76 +0,0 @@
# JavaScript to TypeScript Migration Project
Progressive migration of 219 JS/JSX files to TypeScript in Apache Superset frontend.
## 📁 Project Documentation
- **[AGENT.md](./AGENT.md)** - Complete technical migration guide for agents (includes type reference, patterns, validation)
- **[COORDINATOR.md](./COORDINATOR.md)** - Strategic workflow for coordinators (file selection, task management, integration)
## 🎯 Quick Start
**For Agents:** Read [AGENT.md](./AGENT.md) for complete migration instructions
**For Coordinators:** Read [COORDINATOR.md](./COORDINATOR.md) for workflow and [AGENT.md](./AGENT.md) for supervision
**Command:** `/js-to-ts <filename>` - See [../../commands/js-to-ts.md](../../commands/js-to-ts.md)
## 📊 Migration Progress
**Scope**: 219 files total (112 JS + 107 JSX)
- Production files: 139 (63%)
- Test files: 80 (37%)
**Strategy**: Leaf-first migration with dependency-aware coordination
### Completed Migrations ✅
1. **roundDecimal** - `plugins/legacy-plugin-chart-map-box/src/utils/roundDecimal.js`
- Migrated core + test files
- Added proper TypeScript function signature with optional precision parameter
- All tests pass
2. **timeGrainSqlaAnimationOverrides** - `src/explore/controlPanels/timeGrainSqlaAnimationOverrides.js`
- Migrated to TypeScript with ControlPanelState and Dataset types
- Added TimeGrainOverrideState interface for return type
- Used type guards for safe property access
3. **DebouncedMessageQueue** - `src/utils/DebouncedMessageQueue.js`
- Migrated to TypeScript with proper generics
- Created DebouncedMessageQueueOptions interface
- **CREATED test file** with 4 comprehensive test cases
- Excellent class property typing with private/readonly modifiers
**Files Migrated**: 3/219 (1.4%)
**Tests Created**: 2 (roundDecimal had existing, DebouncedMessageQueue created)
### Next Candidates (Leaf Nodes) 🎯
**Identified leaf files with no JS/JSX dependencies:**
- `src/utils/hostNamesConfig.js` - Domain configuration utility
- `src/explore/controlPanels/Separator.js` - Control panel configuration
- `src/middleware/loggerMiddleware.js` - Logging middleware
**Migration Quality**: All completed migrations have:
- ✅ Zero `any` types
- ✅ Proper TypeScript compilation
- ✅ ESLint validation passed
- ✅ Test coverage (created where missing)
---
## 📈 Success Metrics
**Per-File Gates**:
-`npm run type` passes after each migration
- ✅ Zero `any` types introduced
- ✅ All imports properly typed
- ✅ Types filed in correct hierarchy
**Overall Progress**:
- **Automatic Integration Rate**: 100% (3/3 migrations required no coordinator fixes)
- **Test Coverage**: Improved (1 new test file created)
- **Type Safety**: Enhanced with proper interfaces and generics
---
*This is a claudette-managed progressive refactor. All documentation and coordination resources are organized under `.claude/projects/js-to-ts/`*

View File

@@ -1,20 +0,0 @@
# Keep this in sync with the base image in the main Dockerfile (ARG PY_VER)
FROM python:3.11.13-trixie AS base
# Install system dependencies that Superset needs
# This layer will be cached across Codespace sessions
RUN apt-get update && apt-get install -y \
libsasl2-dev \
libldap2-dev \
libpq-dev \
tmux \
gh \
&& rm -rf /var/lib/apt/lists/*
# Install uv for fast Python package management
# This will also be cached in the image
RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
echo 'export PATH="/root/.cargo/bin:$PATH"' >> /etc/bash.bashrc
# Set the cargo/bin directory in PATH for all users
ENV PATH="/root/.cargo/bin:${PATH}"

View File

@@ -1,62 +0,0 @@
# Superset Codespaces environment setup
# This file is appended to ~/.bashrc during Codespace setup
# Find the workspace directory (handles both 'superset' and 'superset-2' names)
WORKSPACE_DIR=$(find /workspaces -maxdepth 1 -name "superset*" -type d | head -1)
if [ -n "$WORKSPACE_DIR" ]; then
# Check if virtual environment exists
if [ -d "$WORKSPACE_DIR/.venv" ]; then
# Activate the virtual environment
source "$WORKSPACE_DIR/.venv/bin/activate"
echo "✅ Python virtual environment activated"
# Verify pre-commit is installed and set up
if command -v pre-commit &> /dev/null; then
echo "✅ pre-commit is available ($(pre-commit --version))"
# Install git hooks if not already installed
if [ -d "$WORKSPACE_DIR/.git" ] && [ ! -f "$WORKSPACE_DIR/.git/hooks/pre-commit" ]; then
echo "🪝 Installing pre-commit hooks..."
cd "$WORKSPACE_DIR" && pre-commit install
fi
else
echo "⚠️ pre-commit not found. Run: pip install pre-commit"
fi
else
echo "⚠️ Python virtual environment not found at $WORKSPACE_DIR/.venv"
echo " Run: cd $WORKSPACE_DIR && .devcontainer/setup-dev.sh"
fi
# Always cd to the workspace directory for convenience
cd "$WORKSPACE_DIR"
fi
# Add helpful aliases for Superset development
alias start-superset="$WORKSPACE_DIR/.devcontainer/start-superset.sh"
alias setup-dev="$WORKSPACE_DIR/.devcontainer/setup-dev.sh"
# Show helpful message on login
echo ""
echo "🚀 Superset Codespaces Environment"
echo "=================================="
# Check if Superset is running
if docker ps 2>/dev/null | grep -q "superset"; then
echo "✅ Superset is running!"
echo " - Check the 'Ports' tab for your live Superset URL"
echo " - Initial startup takes 10-20 minutes"
echo " - Login: admin/admin"
else
echo "⚠️ Superset is not running. Use: start-superset"
# Check if there's a startup log
if [ -f "/tmp/superset-startup.log" ]; then
echo " 📋 Startup log found: cat /tmp/superset-startup.log"
fi
fi
echo ""
echo "Quick commands:"
echo " start-superset - Start Superset with Docker Compose"
echo " setup-dev - Set up Python environment (if not already done)"
echo " pre-commit run - Run pre-commit checks on staged files"
echo ""

View File

@@ -1,20 +0,0 @@
#!/bin/bash
# Script to build and push the devcontainer image to GitHub Container Registry
# This allows caching the image between Codespace sessions
# You'll need to run this with appropriate GitHub permissions
# gh auth login --scopes write:packages
REGISTRY="ghcr.io"
OWNER="apache"
REPO="superset"
TAG="devcontainer-base"
echo "Building devcontainer image..."
docker build -t $REGISTRY/$OWNER/$REPO:$TAG .devcontainer/
echo "Pushing to GitHub Container Registry..."
docker push $REGISTRY/$OWNER/$REPO:$TAG
echo "Done! Update .devcontainer/devcontainer.json to use:"
echo " \"image\": \"$REGISTRY/$OWNER/$REPO:$TAG\""

View File

@@ -1,15 +1,8 @@
{
"name": "Apache Superset Development",
// Option 1: Use pre-built image directly
// "image": "ghcr.io/apache/superset:devcontainer-base",
// Option 2: Build from Dockerfile with cache (current approach)
"build": {
"dockerfile": "Dockerfile",
"context": ".",
// Cache from the Apache registry image
"cacheFrom": ["ghcr.io/apache/superset:devcontainer-base"]
},
// Keep this in sync with the base image in Dockerfile (ARG PY_VER)
// Using the same base as Dockerfile, but non-slim for dev tools
"image": "python:3.11.13-bookworm",
"features": {
"ghcr.io/devcontainers/features/docker-in-docker:2": {
@@ -39,17 +32,10 @@
},
// Run commands after container is created
"postCreateCommand": "bash .devcontainer/setup-dev.sh || echo '⚠️ Setup had issues - run .devcontainer/setup-dev.sh manually'",
"postCreateCommand": "chmod +x .devcontainer/setup-dev.sh && .devcontainer/setup-dev.sh",
// Auto-start Superset after ensuring Docker is ready
// Run in foreground to see any errors, but don't block on failures
"postStartCommand": "bash -c 'echo \"Waiting 30s for services to initialize...\"; sleep 30; .devcontainer/start-superset.sh || echo \"⚠️ Auto-start failed - run start-superset manually\"'",
// Set environment variables
"remoteEnv": {
// Removed automatic venv activation to prevent startup issues
// The setup script will handle this
},
// Auto-start Superset on Codespace resume
"postStartCommand": ".devcontainer/start-superset.sh",
// VS Code customizations
"customizations": {

14
.github/CODEOWNERS vendored
View File

@@ -2,7 +2,7 @@
# https://github.com/apache/superset/issues/13351
/superset/migrations/ @mistercrunch @michael-s-molina @betodealmeida @eschutho @sadpandajoe
/superset/migrations/ @mistercrunch @michael-s-molina @betodealmeida @eschutho
# Notify some committers of changes in the components
@@ -20,7 +20,7 @@
# Notify PMC members of changes to GitHub Actions
/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @craig-rueda @kgabryje @dpgaspar @sadpandajoe
/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @craig-rueda @kgabryje @dpgaspar
# Notify PMC members of changes to required GitHub Actions
@@ -30,13 +30,3 @@
**/*.geojson @villebro @rusackas
/superset-frontend/plugins/legacy-plugin-chart-country-map/ @villebro @rusackas
# Notify PMC members of changes to extension-related files
/superset-core/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset-extensions-cli/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset/core/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset/extensions/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset-frontend/src/packages/superset-core/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset-frontend/src/core/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje
/superset-frontend/src/extensions/ @michael-s-molina @villebro @geido @eschutho @rusackas @kgabryje

View File

@@ -1,27 +1,24 @@
name: Change Detector
description: Detects file changes for pull request and push events
name: 'Change Detector'
description: 'Detects file changes for pull request and push events'
inputs:
token:
description: GitHub token for authentication
description: 'GitHub token for authentication'
required: true
outputs:
python:
description: Whether Python-related files were changed
description: 'Whether Python-related files were changed'
value: ${{ steps.change-detector.outputs.python }}
frontend:
description: Whether frontend-related files were changed
description: 'Whether frontend-related files were changed'
value: ${{ steps.change-detector.outputs.frontend }}
docker:
description: Whether docker-related files were changed
description: 'Whether docker-related files were changed'
value: ${{ steps.change-detector.outputs.docker }}
docs:
description: Whether docs-related files were changed
description: 'Whether docs-related files were changed'
value: ${{ steps.change-detector.outputs.docs }}
superset-extensions-cli:
description: Whether superset-extensions-cli package-related files were changed
value: ${{ steps.change-detector.outputs.superset-extensions-cli }}
runs:
using: composite
using: 'composite'
steps:
- name: Detect file changes
id: change-detector

View File

@@ -1 +1 @@
../AGENTS.md
../LLMS.md

View File

@@ -5,7 +5,7 @@ updates:
- package-ecosystem: "github-actions"
directory: "/"
schedule:
interval: "daily"
interval: "monthly"
- package-ecosystem: "npm"
ignore:
@@ -18,7 +18,7 @@ updates:
- dependency-name: "jest-environment-jsdom"
directory: "/superset-frontend/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -40,21 +40,21 @@ updates:
- package-ecosystem: "npm"
directory: ".github/actions"
schedule:
interval: "daily"
interval: "monthly"
open-pull-requests-limit: 10
versioning-strategy: increase
- package-ecosystem: "npm"
directory: "/docs/"
schedule:
interval: "daily"
interval: "monthly"
open-pull-requests-limit: 10
versioning-strategy: increase
- package-ecosystem: "npm"
directory: "/superset-websocket/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -63,7 +63,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-websocket/utils/client-ws-app/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -75,7 +75,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-calendar/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -85,7 +85,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-histogram/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -95,7 +95,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-partition/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -105,7 +105,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-world-map/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -115,7 +115,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-pivot-table/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -125,7 +125,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-chord/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -135,7 +135,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-horizon/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -145,7 +145,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-rose/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -155,7 +155,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-preset-chart-deckgl/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -165,7 +165,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-table/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -175,7 +175,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-country-map/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -185,7 +185,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-map-box/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -195,7 +195,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-sankey/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -205,7 +205,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-preset-chart-nvd3/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -215,7 +215,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-word-cloud/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -225,7 +225,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-event-flow/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -235,7 +235,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-paired-t-test/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -245,7 +245,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-sankey-loop/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -255,7 +255,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-echarts/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -265,7 +265,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/preset-chart-xy/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -275,7 +275,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-heatmap/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -285,7 +285,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-parallel-coordinates/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -295,7 +295,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/legacy-plugin-chart-sunburst/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -305,7 +305,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/plugins/plugin-chart-handlebars/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -315,7 +315,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/generator-superset/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -325,7 +325,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-chart-controls/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -339,7 +339,7 @@ updates:
- dependency-name: "react-markdown"
- dependency-name: "remark-gfm"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -349,7 +349,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-demo/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot
@@ -359,7 +359,7 @@ updates:
- package-ecosystem: "npm"
directory: "/superset-frontend/packages/superset-ui-switchboard/"
schedule:
interval: "daily"
interval: "monthly"
labels:
- npm
- dependabot

View File

@@ -182,95 +182,6 @@ cypress-run-all() {
kill $flaskProcessId
}
playwright-install() {
cd "$GITHUB_WORKSPACE/superset-frontend"
say "::group::Install Playwright browsers"
npx playwright install --with-deps chromium
# Create output directories for test results and debugging
mkdir -p playwright-results
mkdir -p test-results
say "::endgroup::"
}
playwright-run() {
local APP_ROOT=$1
local TEST_PATH=$2
# Start Flask from the project root (same as Cypress)
cd "$GITHUB_WORKSPACE"
local flasklog="${HOME}/flask-playwright.log"
local port=8081
PLAYWRIGHT_BASE_URL="http://localhost:${port}"
if [ -n "$APP_ROOT" ]; then
export SUPERSET_APP_ROOT=$APP_ROOT
PLAYWRIGHT_BASE_URL=${PLAYWRIGHT_BASE_URL}${APP_ROOT}/
fi
export PLAYWRIGHT_BASE_URL
nohup flask run --no-debugger -p $port >"$flasklog" 2>&1 </dev/null &
local flaskProcessId=$!
# Ensure cleanup on exit
trap "kill $flaskProcessId 2>/dev/null || true" EXIT
# Wait for server to be ready with health check
local timeout=60
say "Waiting for Flask server to start on port $port..."
while [ $timeout -gt 0 ]; do
if curl -f ${PLAYWRIGHT_BASE_URL}/health >/dev/null 2>&1; then
say "Flask server is ready"
break
fi
sleep 1
timeout=$((timeout - 1))
done
if [ $timeout -eq 0 ]; then
echo "::error::Flask server failed to start within 60 seconds"
echo "::group::Flask startup log"
cat "$flasklog"
echo "::endgroup::"
return 1
fi
# Change to frontend directory for Playwright execution
cd "$GITHUB_WORKSPACE/superset-frontend"
say "::group::Run Playwright tests"
echo "Running Playwright with baseURL: ${PLAYWRIGHT_BASE_URL}"
if [ -n "$TEST_PATH" ]; then
# Check if there are any test files in the specified path
if ! find "playwright/tests/${TEST_PATH}" -name "*.spec.ts" -type f 2>/dev/null | grep -q .; then
echo "No test files found in ${TEST_PATH} - skipping test run"
say "::endgroup::"
kill $flaskProcessId
return 0
fi
echo "Running tests: ${TEST_PATH}"
# Set INCLUDE_EXPERIMENTAL=true to allow experimental tests to run
export INCLUDE_EXPERIMENTAL=true
npx playwright test "${TEST_PATH}" --output=playwright-results
local status=$?
# Unset to prevent leaking into subsequent commands
unset INCLUDE_EXPERIMENTAL
else
echo "Running all required tests (experimental/ excluded via playwright.config.ts)"
npx playwright test --output=playwright-results
local status=$?
fi
say "::endgroup::"
# After job is done, print out Flask log for debugging
echo "::group::Flask log for Playwright run"
cat "$flasklog"
echo "::endgroup::"
# make sure the program exits
kill $flaskProcessId
return $status
}
eyes-storybook-dependencies() {
say "::group::install eyes-storyook dependencies"
sudo apt-get update -y && sudo apt-get -y install gconf-service ca-certificates libxshmfence-dev fonts-liberation libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libglib2.0-0 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libnss3 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release xdg-utils libappindicator1

View File

@@ -32,7 +32,7 @@ jobs:
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: true
ref: master
@@ -41,7 +41,7 @@ jobs:
uses: ./.github/actions/setup-supersetbot/
- name: Set up Python ${{ inputs.python-version }}
uses: actions/setup-python@v6
uses: actions/setup-python@v5
with:
python-version: "3.10"

View File

@@ -31,7 +31,7 @@ jobs:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
if: steps.check_queued.outputs.count >= 20
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Cancel duplicate workflow runs
if: steps.check_queued.outputs.count >= 20

View File

@@ -18,7 +18,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -25,9 +25,9 @@ jobs:
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Check and notify
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{ github.token }}
script: |

View File

@@ -44,7 +44,7 @@ jobs:
pull-requests: write
steps:
- name: Comment access denied
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
script: |
const message = `👋 Hi @${{ github.event.comment.user.login || github.event.review.user.login || github.event.issue.user.login }}!
@@ -71,7 +71,7 @@ jobs:
id-token: write
steps:
- name: Checkout repository
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
fetch-depth: 1

View File

@@ -31,7 +31,7 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Check for file changes
id: check
@@ -41,7 +41,7 @@ jobs:
# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4
uses: github/codeql-action/init@v3
with:
languages: ${{ matrix.language }}
# If you wish to specify custom queries, you can do so here or in a config file.
@@ -53,6 +53,6 @@ jobs:
- name: Perform CodeQL Analysis
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: github/codeql-action/analyze@v4
uses: github/codeql-action/analyze@v3
with:
category: "/language:${{matrix.language}}"

View File

@@ -27,7 +27,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout Repository"
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: "Dependency Review"
uses: actions/dependency-review-action@v4
continue-on-error: true
@@ -53,7 +53,7 @@ jobs:
runs-on: ubuntu-22.04
steps:
- name: "Checkout Repository"
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Setup Python
uses: ./.github/actions/setup-backend/

View File

@@ -22,7 +22,7 @@ jobs:
steps:
- id: set_matrix
run: |
MATRIX_CONFIG=$(if [ "${{ github.event_name }}" == "pull_request" ]; then echo '["dev", "lean"]'; else echo '["dev", "lean", "py310", "websocket", "dockerize", "py311", "py312"]'; fi)
MATRIX_CONFIG=$(if [ "${{ github.event_name }}" == "pull_request" ]; then echo '["dev", "lean"]'; else echo '["dev", "lean", "py310", "websocket", "dockerize", "py311"]'; fi)
echo "matrix_config=${MATRIX_CONFIG}" >> $GITHUB_OUTPUT
echo $GITHUB_OUTPUT
@@ -42,7 +42,7 @@ jobs:
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
@@ -102,7 +102,7 @@ jobs:
docker history $IMAGE_TAG
- name: docker-compose sanity check
if: (steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker) && matrix.build_preset == 'dev'
if: (steps.check.outputs.python || steps.check.outputs.frontend || steps.check.outputs.docker) && (matrix.build_preset == 'dev' || matrix.build_preset == 'lean')
shell: bash
run: |
export SUPERSET_BUILD_TARGET=${{ matrix.build_preset }}
@@ -117,7 +117,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Check for file changes

View File

@@ -28,8 +28,8 @@ jobs:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v5
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: './superset-embedded-sdk/.nvmrc'
registry-url: 'https://registry.npmjs.org'

View File

@@ -18,8 +18,8 @@ jobs:
run:
working-directory: superset-embedded-sdk
steps:
- uses: actions/checkout@v5
- uses: actions/setup-node@v5
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version-file: './superset-embedded-sdk/.nvmrc'
registry-url: 'https://registry.npmjs.org'

View File

@@ -1,10 +1,4 @@
name: Cleanup ephemeral envs (PR close) [DEPRECATED]
# ⚠️ DEPRECATION NOTICE ⚠️
# This workflow is deprecated and will be removed in a future version.
# The new Superset Showtime workflow handles cleanup automatically.
# See .github/workflows/showtime.yml and showtime-cleanup.yml for replacements.
# Migration guide: https://github.com/mistercrunch/superset-showtime
name: Cleanup ephemeral envs (PR close)
on:
pull_request_target:
@@ -33,7 +27,7 @@ jobs:
pull-requests: write
steps:
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v5
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -69,7 +63,7 @@ jobs:
- name: Comment (success)
if: steps.describe-services.outputs.active == 'true'
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{github.token}}
script: |
@@ -77,5 +71,5 @@ jobs:
issue_number: ${{ github.event.number }},
owner: context.repo.owner,
repo: context.repo.repo,
body: '⚠️ **DEPRECATED WORKFLOW** - Ephemeral environment shutdown and build artifacts deleted. Please migrate to the new Superset Showtime system for future PRs.'
body: 'Ephemeral environment shutdown and build artifacts deleted.'
})

View File

@@ -1,12 +1,4 @@
name: Ephemeral env workflow [DEPRECATED]
# ⚠️ DEPRECATION NOTICE ⚠️
# This workflow is deprecated and will be removed in a future version.
# Please use the new Superset Showtime workflow instead:
# - Use label "🎪 trigger-start" instead of "testenv-up"
# - Showtime provides better reliability and easier management
# - See .github/workflows/showtime.yml for the replacement
# - Migration guide: https://github.com/mistercrunch/superset-showtime
name: Ephemeral env workflow
# Example manual trigger:
# gh workflow run ephemeral-env.yml --ref fix_ephemerals --field label_name="testenv-up" --field issue_number=666
@@ -63,7 +55,7 @@ jobs:
- name: Get event SHA
id: get-sha
if: steps.eval-label.outputs.result == 'up'
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
script: |
@@ -94,7 +86,7 @@ jobs:
core.setOutput("sha", prSha);
- name: Looking for feature flags in PR description
uses: actions/github-script@v8
uses: actions/github-script@v7
id: eval-feature-flags
if: steps.eval-label.outputs.result == 'up'
with:
@@ -116,7 +108,7 @@ jobs:
return results;
- name: Reply with confirmation comment
uses: actions/github-script@v8
uses: actions/github-script@v7
if: steps.eval-label.outputs.result == 'up'
with:
github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -134,11 +126,8 @@ jobs:
throw new Error("Issue number is not available.");
}
const body = `⚠️ **DEPRECATED WORKFLOW** ⚠️\n\n@${user} This workflow is deprecated! Please use the new **Superset Showtime** system instead:\n\n` +
`- Replace "testenv-up" label with "🎪 trigger-start"\n` +
`- Better reliability and easier management\n` +
`- See https://github.com/mistercrunch/superset-showtime for details\n\n` +
`Processing your ephemeral environment request [here](${workflowUrl}). Action: **${action}**.` +
const body = `@${user} Processing your ephemeral environment request [here](${workflowUrl}).` +
` Action: **${action}**.` +
` More information on [how to use or configure ephemeral environments]` +
`(https://superset.apache.org/docs/contributing/howtos/#github-ephemeral-environments)`;
@@ -160,7 +149,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ needs.ephemeral-env-label.outputs.sha }} : ${{steps.get-sha.outputs.sha}} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
ref: ${{ needs.ephemeral-env-label.outputs.sha }}
persist-credentials: false
@@ -189,7 +178,7 @@ jobs:
--extra-flags "--build-arg INCLUDE_CHROMIUM=false"
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v5
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -220,12 +209,12 @@ jobs:
pull-requests: write
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
with:
persist-credentials: false
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v5
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
@@ -248,7 +237,7 @@ jobs:
- name: Fail on missing container image
if: steps.check-image.outcome == 'failure'
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{ github.token }}
script: |
@@ -318,7 +307,7 @@ jobs:
echo "ip=$(aws ec2 describe-network-interfaces --network-interface-ids ${{ steps.get-eni.outputs.eni }} | jq -r '.NetworkInterfaces | first | .Association.PublicIp')" >> $GITHUB_OUTPUT
- name: Comment (success)
if: ${{ success() }}
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{github.token}}
script: |
@@ -331,7 +320,7 @@ jobs:
});
- name: Comment (failure)
if: ${{ failure() }}
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{github.token}}
script: |

View File

@@ -27,12 +27,12 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@v5
uses: actions/setup-java@v4
with:
distribution: "temurin"
java-version: "11"

View File

@@ -14,10 +14,10 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Checkout Repository
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version: '20'

View File

@@ -17,7 +17,7 @@ jobs:
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false

View File

@@ -9,7 +9,7 @@ jobs:
pull-requests: write
runs-on: ubuntu-24.04
steps:
- uses: actions/labeler@v6
- uses: actions/labeler@v5
with:
sync-labels: true

View File

@@ -12,7 +12,7 @@ jobs:
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -15,12 +15,12 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Setup Java
uses: actions/setup-java@v5
uses: actions/setup-java@v4
with:
distribution: 'temurin'
java-version: '11'

View File

@@ -14,7 +14,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Check for 'hold' label
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
github-token: ${{secrets.GITHUB_TOKEN}}
script: |

View File

@@ -16,7 +16,7 @@ jobs:
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -21,7 +21,7 @@ jobs:
python-version: ["current", "previous", "next"]
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -39,7 +39,7 @@ jobs:
echo "HOMEBREW_REPOSITORY=$HOMEBREW_REPOSITORY" >>"${GITHUB_ENV}"
brew install norwoodj/tap/helm-docs
- name: Setup Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version: '20'

View File

@@ -27,7 +27,7 @@ jobs:
pull-requests: write
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -26,7 +26,7 @@ jobs:
name: Bump version and publish package(s)
runs-on: ubuntu-24.04
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
with:
# pulls all commits (needed for lerna / semantic release to correctly version)
fetch-depth: 0
@@ -42,7 +42,7 @@ jobs:
- name: Install Node.js
if: env.HAS_TAGS
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'

View File

@@ -1,50 +0,0 @@
name: 🎪 Showtime Cleanup
# Scheduled cleanup of expired environments
on:
schedule:
- cron: '0 */6 * * *' # Every 6 hours
# Manual trigger for testing
workflow_dispatch:
inputs:
max_age_hours:
description: 'Maximum age in hours before cleanup'
required: false
default: '48'
type: string
# Common environment variables
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
AWS_REGION: ${{ vars.AWS_REGION || 'us-west-2' }}
GITHUB_ORG: ${{ github.repository_owner }}
GITHUB_REPO: ${{ github.event.repository.name }}
jobs:
cleanup-expired:
name: Clean up expired showtime environments
runs-on: ubuntu-latest
permissions:
contents: read
pull-requests: write
steps:
- name: Install Superset Showtime
run: pip install superset-showtime
- name: Cleanup expired environments
run: |
MAX_AGE="${{ github.event.inputs.max_age_hours || '48' }}"
# Validate max_age is numeric
if [[ ! "$MAX_AGE" =~ ^[0-9]+$ ]]; then
echo "❌ Invalid max_age_hours format: $MAX_AGE (must be numeric)"
exit 1
fi
echo "Cleaning up environments older than ${MAX_AGE}h"
python -m showtime cleanup --older-than "${MAX_AGE}h"

View File

@@ -1,179 +0,0 @@
name: 🎪 Superset Showtime
# Ultra-simple: just sync on any PR state change
on:
pull_request_target:
types: [labeled, unlabeled, synchronize, closed]
# Manual testing
workflow_dispatch:
inputs:
pr_number:
description: 'PR number to sync'
required: true
type: number
sha:
description: 'Specific SHA to deploy (optional, defaults to latest)'
required: false
type: string
# Common environment variables for all jobs (non-sensitive only)
env:
AWS_REGION: us-west-2
GITHUB_ORG: ${{ github.repository_owner }}
GITHUB_REPO: ${{ github.event.repository.name }}
GITHUB_ACTOR: ${{ github.actor }}
jobs:
sync:
name: 🎪 Sync PR to desired state
runs-on: ubuntu-latest
timeout-minutes: 90
permissions:
contents: read
pull-requests: write
steps:
- name: Security Check - Authorize Maintainers Only
id: auth
uses: actions/github-script@v8
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
with:
script: |
const actor = context.actor;
console.log(`🔍 Checking authorization for ${actor}`);
// Early exit for workflow_dispatch - assume authorized since it's manually triggered
if (context.eventName === 'workflow_dispatch') {
console.log(`✅ Workflow dispatch event - assuming authorized for ${actor}`);
core.setOutput('authorized', 'true');
return;
}
const { data: permission } = await github.rest.repos.getCollaboratorPermissionLevel({
owner: context.repo.owner,
repo: context.repo.repo,
username: actor
});
console.log(`📊 Permission level for ${actor}: ${permission.permission}`);
const authorized = ['write', 'admin'].includes(permission.permission);
// If this is a synchronize event from unauthorized user, check if Showtime is active and set blocked label
if (!authorized && context.eventName === 'pull_request_target' && context.payload.action === 'synchronize') {
console.log(`🔒 Synchronize event detected - checking if Showtime is active`);
// Check if PR has any circus tent labels (Showtime is in use)
const { data: issue } = await github.rest.issues.get({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number
});
const hasCircusLabels = issue.labels.some(label => label.name.startsWith('🎪 '));
if (hasCircusLabels) {
console.log(`🎪 Circus labels found - setting blocked label to prevent auto-deployment`);
await github.rest.issues.addLabels({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.payload.pull_request.number,
labels: ['🎪 🔒 showtime-blocked']
});
console.log(`✅ Blocked label set - Showtime will detect and skip operations`);
} else {
console.log(` No circus labels found - Showtime not in use, skipping block`);
}
}
if (!authorized) {
console.log(`🚨 Unauthorized user ${actor} - skipping all operations`);
core.setOutput('authorized', 'false');
return;
}
console.log(`✅ Authorized maintainer: ${actor}`);
core.setOutput('authorized', 'true');
- name: Install Superset Showtime
if: steps.auth.outputs.authorized == 'true'
run: |
echo "::notice::Maintainer ${{ github.actor }} triggered deploy for PR ${{ github.event.pull_request.number || github.event.inputs.pr_number }}"
pip install --upgrade superset-showtime
showtime version
- name: Check what actions are needed
if: steps.auth.outputs.authorized == 'true'
id: check
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
# Bulletproof PR number extraction
if [[ -n "${{ github.event.pull_request.number }}" ]]; then
PR_NUM="${{ github.event.pull_request.number }}"
elif [[ -n "${{ github.event.inputs.pr_number }}" ]]; then
PR_NUM="${{ github.event.inputs.pr_number }}"
else
echo "❌ No PR number found in event or inputs"
exit 1
fi
echo "Using PR number: $PR_NUM"
# Run sync check-only with optional SHA override
if [[ -n "${{ github.event.inputs.sha }}" ]]; then
OUTPUT=$(python -m showtime sync $PR_NUM --check-only --sha "${{ github.event.inputs.sha }}")
else
OUTPUT=$(python -m showtime sync $PR_NUM --check-only)
fi
echo "$OUTPUT"
# Extract the outputs we need for conditional steps
BUILD=$(echo "$OUTPUT" | grep "build_needed=" | cut -d'=' -f2)
SYNC=$(echo "$OUTPUT" | grep "sync_needed=" | cut -d'=' -f2)
PR_NUM_OUT=$(echo "$OUTPUT" | grep "pr_number=" | cut -d'=' -f2)
TARGET_SHA=$(echo "$OUTPUT" | grep "target_sha=" | cut -d'=' -f2)
echo "build_needed=$BUILD" >> $GITHUB_OUTPUT
echo "sync_needed=$SYNC" >> $GITHUB_OUTPUT
echo "pr_number=$PR_NUM_OUT" >> $GITHUB_OUTPUT
echo "target_sha=$TARGET_SHA" >> $GITHUB_OUTPUT
- name: Checkout PR code (only if build needed)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.build_needed == 'true'
uses: actions/checkout@v5
with:
ref: ${{ steps.check.outputs.target_sha }}
persist-credentials: false
- name: Setup Docker Environment (only if build needed)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.build_needed == 'true'
uses: ./.github/actions/setup-docker
with:
dockerhub-user: ${{ secrets.DOCKERHUB_USER }}
dockerhub-token: ${{ secrets.DOCKERHUB_TOKEN }}
build: "true"
install-docker-compose: "false"
- name: Execute sync (handles everything)
if: steps.auth.outputs.authorized == 'true' && steps.check.outputs.sync_needed == 'true'
env:
AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
run: |
PR_NUM="${{ steps.check.outputs.pr_number }}"
TARGET_SHA="${{ steps.check.outputs.target_sha }}"
if [[ -n "$TARGET_SHA" ]]; then
python -m showtime sync $PR_NUM --sha "$TARGET_SHA"
else
python -m showtime sync $PR_NUM
fi

View File

@@ -51,7 +51,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -63,7 +63,7 @@ jobs:
with:
run: testdata
- name: Setup Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies

View File

@@ -30,13 +30,13 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
ref: master
- name: Set up Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install eyes-storybook dependencies

View File

@@ -1,4 +1,4 @@
name: Superset App CLI tests
name: Superset CLI tests
on:
push:
@@ -37,7 +37,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -31,17 +31,17 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './docs/.nvmrc'
- name: Setup Python
uses: ./.github/actions/setup-backend/
- uses: actions/setup-java@v5
- uses: actions/setup-java@v4
with:
distribution: 'zulu'
java-version: '21'

View File

@@ -18,17 +18,15 @@ jobs:
name: Link Checking
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v5
- uses: actions/checkout@v4
# Do not bump this linkinator-action version without opening
# an ASF Infra ticket to allow the new version first!
- uses: JustinBeckwith/linkinator-action@3d5ba091319fa7b0ac14703761eebb7d100e6f6d # v1.11.0
- uses: JustinBeckwith/linkinator-action@v1.11.0
continue-on-error: true # This will make the job advisory (non-blocking, no red X)
with:
paths: "**/*.md, **/*.mdx"
paths: "**/*.md, **/*.mdx, !superset-frontend/CHANGELOG.md"
linksToSkip: >-
^https://github.com/apache/(superset|incubator-superset)/(pull|issues)/\d+,
^https://github.com/apache/(superset|incubator-superset)/commit/[a-f0-9]+,
superset-frontend/.*CHANGELOG\.md,
^https://github.com/apache/(superset|incubator-superset)/(pull|issue)/\d+,
http://localhost:8088/,
http://127.0.0.1:3000/,
http://localhost:9001/,
@@ -43,12 +41,12 @@ jobs:
http://theiconic.com.au/,
https://dev.mysql.com/doc/refman/5.7/en/innodb-limits.html,
^https://img\.shields\.io/.*,
https://vkusvill.ru/,
https://www.linkedin.com/in/mark-thomas-b16751158/,
https://theiconic.com.au/,
https://wattbewerb.de/,
https://timbr.ai/,
https://opensource.org/license/apache-2-0,
https://vkusvill.ru/
https://www.linkedin.com/in/mark-thomas-b16751158/
https://theiconic.com.au/
https://wattbewerb.de/
https://timbr.ai/
https://opensource.org/license/apache-2-0
https://www.plaidcloud.com/
build-deploy:
name: Build & Deploy
@@ -58,12 +56,12 @@ jobs:
working-directory: docs
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
- name: Set up Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './docs/.nvmrc'
- name: yarn install

View File

@@ -69,21 +69,21 @@ jobs:
# Conditional checkout based on context
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
@@ -109,7 +109,7 @@ jobs:
run: testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
@@ -151,118 +151,3 @@ jobs:
with:
path: ${{ github.workspace }}/superset-frontend/cypress-base/cypress/screenshots
name: cypress-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}-${{ matrix.parallel_id }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}
playwright-tests:
runs-on: ubuntu-22.04
permissions:
contents: read
pull-requests: read
strategy:
fail-fast: false
matrix:
browser: ["chromium"]
app_root: ["", "/app/prefix"]
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
# -------------------------------------------------------
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@v5
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@v5
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@v5
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python || steps.check.outputs.frontend
- name: Setup postgres
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@v5
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Install Playwright
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Required Tests)
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
with:
run: playwright-run "${{ matrix.app_root }}"
- name: Set safe app root
if: failure()
id: set-safe-app-root
run: |
APP_ROOT="${{ matrix.app_root }}"
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Playwright Artifacts
uses: actions/upload-artifact@v4
if: failure()
with:
path: |
${{ github.workspace }}/superset-frontend/playwright-results/
${{ github.workspace }}/superset-frontend/test-results/
name: playwright-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}

View File

@@ -1,64 +0,0 @@
name: Superset Extensions CLI Package Tests
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
# cancel previous workflow jobs for PRs
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
test-superset-extensions-cli-package:
runs-on: ubuntu-24.04
strategy:
matrix:
python-version: ["previous", "current", "next"]
defaults:
run:
working-directory: superset-extensions-cli
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
with:
persist-credentials: false
submodules: recursive
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
if: steps.check.outputs.superset-extensions-cli
uses: ./.github/actions/setup-backend/
with:
python-version: ${{ matrix.python-version }}
requirements-type: dev
- name: Run pytest with coverage
if: steps.check.outputs.superset-extensions-cli
run: |
pytest --cov=superset_extensions_cli --cov-report=xml --cov-report=term-missing --cov-report=html -v --tb=short
- name: Upload coverage reports to Codecov
if: steps.check.outputs.superset-extensions-cli
uses: codecov/codecov-action@v5
with:
file: ./coverage.xml
flags: superset-extensions-cli
name: superset-extensions-cli-coverage
fail_ci_if_error: false
- name: Upload HTML coverage report
if: steps.check.outputs.superset-extensions-cli
uses: actions/upload-artifact@v4
with:
name: superset-extensions-cli-coverage-html
path: htmlcov/

View File

@@ -23,7 +23,7 @@ jobs:
should-run: ${{ steps.check.outputs.frontend }}
steps:
- name: Checkout Code
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
fetch-depth: 0
@@ -47,7 +47,7 @@ jobs:
git show -s --format=raw HEAD
docker buildx build \
-t $TAG \
--cache-from=type=registry,ref=apache/superset-cache:3.10-slim-trixie \
--cache-from=type=registry,ref=apache/superset-cache:3.10-slim-bookworm \
--target superset-node-ci \
.
@@ -73,7 +73,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@v5
uses: actions/download-artifact@v4
with:
name: docker-image
@@ -101,7 +101,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Download Coverage Artifacts
uses: actions/download-artifact@v5
uses: actions/download-artifact@v4
with:
pattern: coverage-artifacts-*
path: coverage/
@@ -127,7 +127,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@v5
uses: actions/download-artifact@v4
with:
name: docker-image
@@ -135,15 +135,15 @@ jobs:
run: |
docker load < docker-image.tar.gz
- name: lint
- name: eslint
run: |
docker run --rm $TAG bash -c \
"npm i && npm run lint"
"npm i && npm run eslint -- . --quiet"
- name: tsc
run: |
docker run --rm $TAG bash -c \
"npm i && npm run plugins:build && npm run type"
"npm run type"
validate-frontend:
needs: frontend-build
@@ -151,7 +151,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: Download Docker Image Artifact
uses: actions/download-artifact@v5
uses: actions/download-artifact@v4
with:
name: docker-image

View File

@@ -16,7 +16,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -29,7 +29,7 @@ jobs:
steps:
- name: Checkout code
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
ref: ${{ inputs.ref || github.ref_name }}
persist-credentials: true
@@ -101,7 +101,7 @@ jobs:
CR_RELEASE_NAME_TEMPLATE: "superset-helm-chart-{{ .Version }}"
- name: Open Pull Request
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
script: |
const branchName = '${{ env.branch_name }}';

View File

@@ -1,142 +0,0 @@
name: Playwright Experimental Tests
on:
push:
branches:
- "master"
- "[0-9].[0-9]*"
pull_request:
types: [synchronize, opened, reopened, ready_for_review]
workflow_dispatch:
inputs:
ref:
description: 'The branch or tag to checkout'
required: false
default: ''
pr_id:
description: 'The pull request ID to checkout'
required: false
default: ''
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
cancel-in-progress: true
jobs:
# NOTE: Required Playwright tests are in superset-e2e.yml (E2E / playwright-tests)
# This workflow contains only experimental tests that run in shadow mode
playwright-tests-experimental:
runs-on: ubuntu-22.04
continue-on-error: true
permissions:
contents: read
pull-requests: read
strategy:
fail-fast: false
matrix:
browser: ["chromium"]
app_root: ["", "/app/prefix"]
env:
SUPERSET_ENV: development
SUPERSET_CONFIG: tests.integration_tests.superset_test_config
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset
PYTHONPATH: ${{ github.workspace }}
REDIS_PORT: 16379
GITHUB_TOKEN: ${{ github.token }}
services:
postgres:
image: postgres:16-alpine
env:
POSTGRES_USER: superset
POSTGRES_PASSWORD: superset
ports:
- 15432:5432
redis:
image: redis:7-alpine
ports:
- 16379:6379
steps:
# -------------------------------------------------------
# Conditional checkout based on context (same as Cypress workflow)
- name: Checkout for push or pull_request event
if: github.event_name == 'push' || github.event_name == 'pull_request'
uses: actions/checkout@v5
with:
persist-credentials: false
submodules: recursive
ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }}
- name: Checkout using ref (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != ''
uses: actions/checkout@v5
with:
persist-credentials: false
ref: ${{ github.event.inputs.ref }}
submodules: recursive
- name: Checkout using PR ID (workflow_dispatch)
if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != ''
uses: actions/checkout@v5
with:
persist-credentials: false
ref: refs/pull/${{ github.event.inputs.pr_id }}/merge
submodules: recursive
# -------------------------------------------------------
- name: Check for file changes
id: check
uses: ./.github/actions/change-detector/
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Setup Python
uses: ./.github/actions/setup-backend/
if: steps.check.outputs.python || steps.check.outputs.frontend
- name: Setup postgres
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: setup-postgres
- name: Import test data
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: testdata
- name: Setup Node.js
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: actions/setup-node@v5
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install npm dependencies
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: npm-install
- name: Build javascript packages
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: build-instrumented-assets
- name: Install Playwright
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
with:
run: playwright-install
- name: Run Playwright (Experimental Tests)
if: steps.check.outputs.python || steps.check.outputs.frontend
uses: ./.github/actions/cached-dependencies
env:
NODE_OPTIONS: "--max-old-space-size=4096"
with:
run: playwright-run "${{ matrix.app_root }}" experimental/
- name: Set safe app root
if: failure()
id: set-safe-app-root
run: |
APP_ROOT="${{ matrix.app_root }}"
SAFE_APP_ROOT=${APP_ROOT//\//_}
echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT
- name: Upload Playwright Artifacts
uses: actions/upload-artifact@v4
if: failure()
with:
path: |
${{ github.workspace }}/superset-frontend/playwright-results/
${{ github.workspace }}/superset-frontend/test-results/
name: playwright-experimental-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }}

View File

@@ -41,7 +41,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -99,7 +99,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -152,7 +152,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -48,7 +48,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -108,7 +108,7 @@ jobs:
- 16379:6379
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -24,7 +24,7 @@ jobs:
PYTHONPATH: ${{ github.workspace }}
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -18,7 +18,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive
@@ -31,7 +31,7 @@ jobs:
- name: Setup Node.js
if: steps.check.outputs.frontend
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'
- name: Install dependencies
@@ -49,7 +49,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
submodules: recursive

View File

@@ -21,7 +21,7 @@ jobs:
runs-on: ubuntu-24.04
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false
- name: Install dependencies

View File

@@ -26,7 +26,7 @@ jobs:
steps:
- name: Quickly add thumbs up!
if: github.event_name == 'issue_comment' && contains(github.event.comment.body, '@supersetbot')
uses: actions/github-script@v8
uses: actions/github-script@v7
with:
script: |
const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/')
@@ -38,7 +38,7 @@ jobs:
});
- name: "Checkout ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
persist-credentials: false

View File

@@ -42,12 +42,12 @@ jobs:
runs-on: ubuntu-24.04
strategy:
matrix:
build_preset: ["dev", "lean", "py310", "websocket", "dockerize", "py311", "py312"]
build_preset: ["dev", "lean", "py310", "websocket", "dockerize", "py311"]
fail-fast: false
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
fetch-depth: 0
@@ -60,7 +60,7 @@ jobs:
build: "true"
- name: Use Node.js 20
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version: 20
@@ -107,12 +107,12 @@ jobs:
steps:
- name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )"
uses: actions/checkout@v5
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Use Node.js 20
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version: 20

View File

@@ -27,10 +27,10 @@ jobs:
name: Generate Reports
steps:
- name: Checkout Repository
uses: actions/checkout@v5
uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v5
uses: actions/setup-node@v4
with:
node-version-file: './superset-frontend/.nvmrc'

View File

@@ -12,7 +12,7 @@ jobs:
steps:
- name: Welcome Message
uses: actions/first-interaction@v3
uses: actions/first-interaction@v1
continue-on-error: true
with:
repo-token: ${{ github.token }}

9
.gitignore vendored
View File

@@ -33,7 +33,6 @@ cover
.env
.envrc
.idea
.roo
.mypy_cache
.python-version
.tox
@@ -44,7 +43,7 @@ _modules
_static
build
app.db
*.egg-info/
apache_superset.egg-info/
changelog.sh
dist
dump.rdb
@@ -122,8 +121,6 @@ docker/requirements-local.txt
cache/
docker/*local*
docker/superset-websocket/config.json
docker-compose.override.yml
.temp_cache
@@ -133,8 +130,4 @@ superset/static/stats/statistics.html
# LLM-related
CLAUDE.local.md
PROJECT.md
.aider*
.claude_rc*
.env.local
oxc-custom-build/

View File

@@ -23,9 +23,7 @@ repos:
rev: v1.15.0
hooks:
- id: mypy
name: mypy (main)
args: [--check-untyped-defs]
exclude: ^superset-extensions-cli/
additional_dependencies: [
types-simplejson,
types-python-dateutil,
@@ -40,10 +38,6 @@ repos:
types-paramiko,
types-Markdown,
]
- id: mypy
name: mypy (superset-extensions-cli)
args: [--check-untyped-defs]
files: ^superset-extensions-cli/
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v5.0.0
hooks:
@@ -60,39 +54,25 @@ repos:
args: ["--markdown-linebreak-ext=md"]
- repo: local
hooks:
- id: prettier-frontend
name: prettier (frontend)
entry: bash -c 'cd superset-frontend && for file in "$@"; do npx prettier --write "${file#superset-frontend/}"; done'
language: system
pass_filenames: true
files: ^superset-frontend/.*\.(js|jsx|ts|tsx|css|scss|sass|json)$
- repo: local
hooks:
- id: oxlint-frontend
name: oxlint (frontend)
entry: ./scripts/oxlint.sh
language: system
pass_filenames: true
files: ^superset-frontend/.*\.(js|jsx|ts|tsx)$
- id: custom-rules-frontend
name: custom rules (frontend)
entry: ./scripts/check-custom-rules.sh
language: system
pass_filenames: true
files: ^superset-frontend/.*\.(js|jsx|ts|tsx)$
- id: eslint-docs
name: eslint (docs)
entry: bash -c 'cd docs && FILES=$(echo "$@" | sed "s|docs/||g") && yarn eslint --fix --quiet $FILES'
language: system
pass_filenames: true
files: ^docs/.*\.(js|jsx|ts|tsx)$
- id: type-checking-frontend
name: Type-Checking (Frontend)
entry: ./scripts/check-type.js package=superset-frontend excludeDeclarationDir=cypress-base
language: system
files: ^superset-frontend\/.*\.(js|jsx|ts|tsx)$
exclude: ^superset-frontend/cypress-base\/
require_serial: true
- id: eslint-frontend
name: eslint (frontend)
entry: ./scripts/eslint.sh
language: system
pass_filenames: true
files: ^superset-frontend/.*\.(js|jsx|ts|tsx)$
- id: eslint-docs
name: eslint (docs)
entry: bash -c 'cd docs && FILES=$(echo "$@" | sed "s|docs/||g") && yarn eslint --fix --ext .js,.jsx,.ts,.tsx --quiet $FILES'
language: system
pass_filenames: true
files: ^docs/.*\.(js|jsx|ts|tsx)$
- id: type-checking-frontend
name: Type-Checking (Frontend)
entry: ./scripts/check-type.js package=superset-frontend excludeDeclarationDir=cypress-base
language: system
files: ^superset-frontend\/.*\.(js|jsx|ts|tsx)$
exclude: ^superset-frontend/cypress-base\/
require_serial: true
# blacklist unsafe functions like make_url (see #19526)
- repo: https://github.com/skorokithakis/blacklist-pre-commit-hook
rev: e2f070289d8eddcaec0b580d3bde29437e7c8221
@@ -114,24 +94,21 @@ repos:
args: [--fix]
- repo: local
hooks:
- id: pylint
name: pylint with custom Superset plugins
entry: bash
language: system
types: [python]
exclude: ^(tests/|superset/migrations/|scripts/|RELEASING/|docker/)
args:
- -c
- |
TARGET_BRANCH=${GITHUB_BASE_REF:-master}
# Only fetch if we're not in CI (CI already has all refs)
if [ -z "$CI" ]; then
git fetch --no-recurse-submodules origin "$TARGET_BRANCH" 2>/dev/null || true
fi
BASE=$(git merge-base origin/"$TARGET_BRANCH" HEAD 2>/dev/null) || BASE="HEAD"
files=$(git diff --name-only --diff-filter=ACM "$BASE"..HEAD 2>/dev/null | grep '^superset/.*\.py$' || true)
if [ -n "$files" ]; then
pylint --rcfile=.pylintrc --load-plugins=superset.extensions.pylint --reports=no $files
else
echo "No Python files to lint."
fi
- id: pylint
name: pylint with custom Superset plugins
entry: bash
language: system
types: [python]
exclude: ^(tests/|superset/migrations/|scripts/|RELEASING/|docker/)
args:
- -c
- |
TARGET_BRANCH=${GITHUB_BASE_REF:-master}
git fetch origin "$TARGET_BRANCH"
BASE=$(git merge-base origin/"$TARGET_BRANCH" HEAD)
files=$(git diff --name-only --diff-filter=ACM "$BASE"..HEAD | grep '^superset/.*\.py$' || true)
if [ -n "$files" ]; then
pylint --rcfile=.pylintrc --load-plugins=superset.extensions.pylint --reports=no $files
else
echo "No Python files to lint."
fi

View File

@@ -53,7 +53,7 @@ extension-pkg-whitelist=pyarrow
[MESSAGES CONTROL]
disable=all
enable=json-import,disallowed-sql-import,consider-using-transaction
enable=disallowed-json-import,disallowed-sql-import,consider-using-transaction
[REPORTS]

View File

@@ -11,7 +11,6 @@
.nvmrc
.prettierrc
.rat-excludes
.swcrc
.*log
.*pyc
.*lock
@@ -33,8 +32,6 @@ apache_superset.egg-info
# json and csv in general cannot have comments
.*json
.*csv
# jinja templates often need to be as-is
.*j2
# Generated doc files
env/*
docs/.htaccess*
@@ -74,7 +71,6 @@ ibm-db2.svg
postgresql.svg
snowflake.svg
ydb.svg
loading.svg
# docs-related
erd.puml
@@ -83,7 +79,6 @@ intro_header.txt
# for LLMs
llm-context.md
AGENTS.md
LLMS.md
CLAUDE.md
CURSOR.md

View File

@@ -44,8 +44,4 @@ under the License.
- [4.0.1](./CHANGELOG/4.0.1.md)
- [4.0.2](./CHANGELOG/4.0.2.md)
- [4.1.0](./CHANGELOG/4.1.0.md)
- [4.1.1](./CHANGELOG/4.1.1.md)
- [4.1.2](./CHANGELOG/4.1.2.md)
- [4.1.3](./CHANGELOG/4.1.3.md)
- [4.1.4](./CHANGELOG/4.1.4.md)
- [5.0.0](./CHANGELOG/5.0.0.md)

View File

@@ -1,33 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
## Change Log
### 4.1.4 (Thu Jul 24 08:30:04 2025 -0300)
**Database Migrations**
**Features**
**Fixes**
- [#34289](https://github.com/apache/superset/pull/34289) fix: Saved queries list break if one query can't be parsed (@michael-s-molina)
- [#33059](https://github.com/apache/superset/pull/33059) fix: Adds missing __init__ file to commands/logs (@michael-s-molina)
**Others**
- [#32236](https://github.com/apache/superset/pull/32236) chore(deps): bump cryptography from 43.0.3 to 44.0.1 (@dependabot[bot])

215
CHART_METADATA_API.md Normal file
View File

@@ -0,0 +1,215 @@
# Chart Metadata API Reference
The Superset MCP service provides rich metadata alongside chart generation to enable better UI integration and user experiences.
## Background & Design Philosophy
Modern chart systems need to provide more than just visual output. Inspired by contemporary web standards and LLM integration patterns, this metadata system addresses several key needs:
**Accessibility-First Design**: Following WCAG guidelines and `aria-*` attribute patterns, charts include semantic descriptions and accessibility metadata to ensure inclusive experiences.
**Rich Context for AI Systems**: Similar to how platforms like social media generate rich previews (OpenGraph, Twitter Cards), charts provide semantic understanding beyond just visual representation - enabling AI agents to reason about and describe visualizations meaningfully.
**Performance-Aware Integration**: Modern web APIs emphasize performance transparency (Core Web Vitals, etc.). Charts include execution metrics and optimization suggestions to help UIs make informed decisions about rendering and user feedback.
**Capability-Driven UX**: Rather than requiring UIs to hardcode chart type behaviors, the system exposes what each chart can actually do - enabling dynamic, contextual interfaces that adapt to chart capabilities.
## Overview
When generating charts via `generate_chart`, the response includes structured metadata that helps UIs:
- Present appropriate controls and interactions
- Generate accessible descriptions
- Optimize rendering performance
- Guide user workflows
## Metadata Types
### ChartCapabilities
Describes what interactions and features the chart supports.
```python
{
"supports_interaction": bool, # User can interact (zoom, pan, hover)
"supports_real_time": bool, # Chart can update with live data
"supports_drill_down": bool, # Can navigate to more detailed views
"supports_export": bool, # Can be exported to other formats
"optimal_formats": [ # Recommended preview formats
"url", # Static image URL
"interactive", # HTML with JavaScript controls
"ascii", # Text-based representation
"vega_lite" # Vega-Lite specification
],
"data_types": [ # Types of data visualized
"time_series", # Time-based data
"categorical", # Discrete categories
"metric" # Numeric measurements
]
}
```
**UI Integration:**
- Show/hide interaction controls based on `supports_interaction`
- Enable real-time updates if `supports_real_time`
- Display drill-down options for `supports_drill_down`
- Choose optimal preview format from `optimal_formats`
### ChartSemantics
Provides semantic understanding of what the chart represents and reveals.
```python
{
"primary_insight": "Shows trends and changes over time",
"data_story": "This line chart analyzes sales, revenue over Q1-Q4",
"recommended_actions": [
"Review data patterns and trends",
"Consider filtering for more detail",
"Export chart for reporting"
],
"anomalies": [], # Notable outliers (future enhancement)
"statistical_summary": {} # Key statistics (future enhancement)
}
```
**UI Integration:**
- Display `primary_insight` as chart description
- Use `data_story` for accessibility and tooltips
- Show `recommended_actions` as suggested next steps
- Highlight `anomalies` in the visualization
### AccessibilityMetadata
Information for creating inclusive, accessible chart experiences.
```python
{
"color_blind_safe": bool, # Uses colorblind-friendly palette
"alt_text": "Chart showing Sales Data over time",
"high_contrast_available": bool # High contrast version available
}
```
**UI Integration:**
- Use `alt_text` for screen readers
- Show accessibility indicators if `color_blind_safe`
- Offer high contrast mode if available
### PerformanceMetadata
Performance information for optimization and user feedback.
```python
{
"query_duration_ms": 1250, # Time to generate chart data
"cache_status": "hit|miss|error", # Whether data came from cache
"optimization_suggestions": [ # Performance improvement tips
"Consider adding date filters to reduce data volume",
"Chart complexity may impact load time"
]
}
```
**UI Integration:**
- Show loading indicators based on `query_duration_ms`
- Display cache status for debugging
- Present `optimization_suggestions` to users
- Warn about slow queries
## Example Response
```json
{
"chart": {
"id": 123,
"slice_name": "Sales Trends Q1-Q4",
"viz_type": "echarts_timeseries_line",
"url": "/explore/?slice_id=123"
},
"capabilities": {
"supports_interaction": true,
"supports_real_time": false,
"supports_drill_down": false,
"supports_export": true,
"optimal_formats": ["url", "interactive", "ascii"],
"data_types": ["time_series", "metric"]
},
"semantics": {
"primary_insight": "Shows trends and changes over time",
"data_story": "This line chart analyzes sales over Q1-Q4",
"recommended_actions": [
"Review seasonal patterns",
"Export for quarterly report"
]
},
"accessibility": {
"color_blind_safe": true,
"alt_text": "Line chart showing sales trends from Q1 to Q4",
"high_contrast_available": false
},
"performance": {
"query_duration_ms": 450,
"cache_status": "miss",
"optimization_suggestions": []
}
}
```
## Usage Examples
### React Component Integration
```jsx
function ChartComponent({ chartData }) {
const { capabilities, semantics, accessibility, performance } = chartData;
return (
<div>
{/* Accessibility */}
<img
src={chartData.chart.url}
alt={accessibility.alt_text}
aria-describedby="chart-description"
/>
{/* Semantic description */}
<p id="chart-description">{semantics.primary_insight}</p>
{/* Conditional controls based on capabilities */}
{capabilities.supports_interaction && (
<InteractiveControls />
)}
{capabilities.supports_export && (
<ExportButton />
)}
{/* Performance feedback */}
{performance.query_duration_ms > 2000 && (
<SlowQueryWarning suggestions={performance.optimization_suggestions} />
)}
{/* Recommended actions */}
<ActionSuggestions actions={semantics.recommended_actions} />
</div>
);
}
```
## Chart Type Mapping
Different chart types provide different capabilities:
| Chart Type | Interaction | Real-time | Drill-down | Optimal Formats |
|------------|------------|-----------|------------|-----------------|
| `echarts_timeseries_line` | ✅ | ✅ | ❌ | url, interactive, ascii |
| `echarts_timeseries_bar` | ✅ | ✅ | ❌ | url, interactive, ascii |
| `table` | ❌ | ❌ | ✅ | url, table, ascii |
| `pie` | ✅ | ❌ | ❌ | url, interactive |
## Future Enhancements
- **Statistical Summary**: Automatic calculation of mean, median, trends
- **Anomaly Detection**: Identification of outliers and unusual patterns
- **Smart Recommendations**: ML-powered suggestions for chart improvements
- **Accessibility Scoring**: Automated accessibility compliance checking

View File

@@ -1 +1 @@
AGENTS.md
LLMS.md

View File

@@ -5,7 +5,7 @@
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
@@ -16,23 +16,9 @@
specific language governing permissions and limitations
under the License.
-->
# Contributing to Apache Superset
Contributions are welcome and are greatly appreciated! Every
little bit helps, and credit will always be given.
## Developer Portal
All developer and contribution documentation has moved to the Apache Superset Developer Portal:
**[📚 View the Developer Portal →](https://superset.apache.org/developer_portal/)**
The Developer Portal includes comprehensive guides for:
- [Contributing Overview](https://superset.apache.org/developer_portal/contributing/overview)
- [Development Setup](https://superset.apache.org/developer_portal/contributing/development-setup)
- [Submitting Pull Requests](https://superset.apache.org/developer_portal/contributing/submitting-pr)
- [Contribution Guidelines](https://superset.apache.org/developer_portal/contributing/guidelines)
- [Code Review Process](https://superset.apache.org/developer_portal/contributing/code-review)
- [Development How-tos](https://superset.apache.org/developer_portal/contributing/howtos)
Source for the Developer Portal documentation is [located here](https://github.com/apache/superset/tree/master/docs/developer_portal).
All matters related to contributions have moved to [this section of
the official Superset documentation](https://superset.apache.org/docs/contributing/). Source for the documentation is
[located here](https://github.com/apache/superset/tree/master/docs/docs).

View File

@@ -18,7 +18,7 @@
######################################################################
# Node stage to deal with static asset construction
######################################################################
ARG PY_VER=3.11.14-slim-trixie
ARG PY_VER=3.11.13-slim-bookworm
# If BUILDPLATFORM is null, set it to 'amd64' (or leave as is otherwise).
ARG BUILDPLATFORM=${BUILDPLATFORM:-amd64}
@@ -26,13 +26,10 @@ ARG BUILDPLATFORM=${BUILDPLATFORM:-amd64}
# Include translations in the final build
ARG BUILD_TRANSLATIONS="false"
# Build arg to pre-populate examples DuckDB file
ARG LOAD_EXAMPLES_DUCKDB="false"
######################################################################
# superset-node-ci used as a base for building frontend assets and CI
######################################################################
FROM --platform=${BUILDPLATFORM} node:20-trixie-slim AS superset-node-ci
FROM --platform=${BUILDPLATFORM} node:20-bookworm-slim AS superset-node-ci
ARG BUILD_TRANSLATIONS
ENV BUILD_TRANSLATIONS=${BUILD_TRANSLATIONS}
ARG DEV_MODE="false" # Skip frontend build in dev mode
@@ -67,7 +64,7 @@ RUN --mount=type=bind,source=./superset-frontend/package.json,target=./package.j
--mount=type=bind,source=./superset-frontend/package-lock.json,target=./package-lock.json \
--mount=type=cache,target=/root/.cache \
--mount=type=cache,target=/root/.npm \
if [ "${DEV_MODE}" = "false" ]; then \
if [ "$DEV_MODE" = "false" ]; then \
npm ci; \
else \
echo "Skipping 'npm ci' in dev mode"; \
@@ -83,7 +80,7 @@ FROM superset-node-ci AS superset-node
# Build the frontend if not in dev mode
RUN --mount=type=cache,target=/root/.npm \
if [ "${DEV_MODE}" = "false" ]; then \
if [ "$DEV_MODE" = "false" ]; then \
echo "Running 'npm run ${BUILD_CMD}'"; \
npm run ${BUILD_CMD}; \
else \
@@ -94,10 +91,11 @@ RUN --mount=type=cache,target=/root/.npm \
COPY superset/translations /app/superset/translations
# Build translations if enabled, then cleanup localization files
RUN if [ "${BUILD_TRANSLATIONS}" = "true" ]; then \
RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
npm run build-translation; \
fi; \
rm -rf /app/superset/translations/*/*/*.[po,mo];
rm -rf /app/superset/translations/*/*/*.po; \
rm -rf /app/superset/translations/*/*/*.mo;
######################################################################
@@ -108,10 +106,10 @@ FROM python:${PY_VER} AS python-base
ARG SUPERSET_HOME="/app/superset_home"
ENV SUPERSET_HOME=${SUPERSET_HOME}
RUN mkdir -p ${SUPERSET_HOME}
RUN mkdir -p $SUPERSET_HOME
RUN useradd --user-group -d ${SUPERSET_HOME} -m --no-log-init --shell /bin/bash superset \
&& chmod -R 1777 ${SUPERSET_HOME} \
&& chown -R superset:superset ${SUPERSET_HOME}
&& chmod -R 1777 $SUPERSET_HOME \
&& chown -R superset:superset $SUPERSET_HOME
# Some bash scripts needed throughout the layers
COPY --chmod=755 docker/*.sh /app/docker/
@@ -136,19 +134,17 @@ RUN --mount=type=cache,target=/root/.cache/uv \
. /app/.venv/bin/activate && /app/docker/pip-install.sh --requires-build-essential -r requirements/translations.txt
COPY superset/translations/ /app/translations_mo/
RUN if [ "${BUILD_TRANSLATIONS}" = "true" ]; then \
RUN if [ "$BUILD_TRANSLATIONS" = "true" ]; then \
pybabel compile -d /app/translations_mo | true; \
fi; \
rm -f /app/translations_mo/*/*/*.[po,json]
rm -f /app/translations_mo/*/*/*.po; \
rm -f /app/translations_mo/*/*/*.json;
######################################################################
# Python APP common layer
######################################################################
FROM python-base AS python-common
# Re-declare build arg to receive it in this stage
ARG LOAD_EXAMPLES_DUCKDB
ENV SUPERSET_HOME="/app/superset_home" \
HOME="/app/superset_home" \
SUPERSET_ENV="production" \
@@ -171,16 +167,14 @@ RUN mkdir -p \
&& touch superset/static/version_info.json
# Install Playwright and optionally setup headless browsers
ENV PLAYWRIGHT_BROWSERS_PATH=/usr/local/share/playwright-browsers
ARG INCLUDE_CHROMIUM="false"
ARG INCLUDE_FIREFOX="false"
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
if [ "${INCLUDE_CHROMIUM}" = "true" ] || [ "${INCLUDE_FIREFOX}" = "true" ]; then \
if [ "$INCLUDE_CHROMIUM" = "true" ] || [ "$INCLUDE_FIREFOX" = "true" ]; then \
uv pip install playwright && \
playwright install-deps && \
if [ "${INCLUDE_CHROMIUM}" = "true" ]; then playwright install chromium; fi && \
if [ "${INCLUDE_FIREFOX}" = "true" ]; then playwright install firefox; fi; \
if [ "$INCLUDE_CHROMIUM" = "true" ]; then playwright install chromium; fi && \
if [ "$INCLUDE_FIREFOX" = "true" ]; then playwright install firefox; fi; \
else \
echo "Skipping browser installation"; \
fi
@@ -202,18 +196,6 @@ RUN /app/docker/apt-install.sh \
libecpg-dev \
libldap2-dev
# Pre-load examples DuckDB file if requested
RUN if [ "$LOAD_EXAMPLES_DUCKDB" = "true" ]; then \
mkdir -p /app/data && \
echo "Downloading pre-built examples.duckdb..." && \
curl -L -o /app/data/examples.duckdb \
"https://raw.githubusercontent.com/apache-superset/examples-data/master/examples.duckdb" && \
chown -R superset:superset /app/data; \
else \
mkdir -p /app/data && \
chown -R superset:superset /app/data; \
fi
# Copy compiled things from previous stages
COPY --from=superset-node /app/superset/static/assets superset/static/assets
@@ -237,10 +219,6 @@ FROM python-common AS lean
# Install Python dependencies using docker/pip-install.sh
COPY requirements/base.txt requirements/
# Copy superset-core package needed for editable install in base.txt
COPY superset-core superset-core
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
/app/docker/pip-install.sh --requires-build-essential -r requirements/base.txt
# Install the superset package
@@ -263,11 +241,6 @@ RUN /app/docker/apt-install.sh \
# Copy development requirements and install them
COPY requirements/*.txt requirements/
# Copy local packages needed for editable installs in development.txt
COPY superset-core superset-core
COPY superset-extensions-cli superset-extensions-cli
# Install Python dependencies using docker/pip-install.sh
RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \
/app/docker/pip-install.sh --requires-build-essential -r requirements/development.txt
@@ -285,15 +258,6 @@ USER superset
######################################################################
FROM lean AS ci
USER root
RUN uv pip install .[postgres,duckdb]
USER superset
CMD ["/app/docker/entrypoints/docker-ci.sh"]
######################################################################
# Showtime image - lean + DuckDB for examples database
######################################################################
FROM lean AS showtime
USER root
RUN uv pip install .[duckdb]
RUN uv pip install .[postgres]
USER superset
CMD ["/app/docker/entrypoints/docker-ci.sh"]

View File

@@ -1 +1 @@
AGENTS.md
LLMS.md

2
GPT.md
View File

@@ -1 +1 @@
AGENTS.md
LLMS.md

View File

@@ -1,121 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# Superset Frontend Linting Architecture
## Overview
We use a hybrid linting approach combining OXC (fast, standard rules) with custom AST-based checks for Superset-specific patterns.
## Components
### 1. Primary Linter: OXC
- **What**: Oxidation Compiler's linter (oxlint)
- **Handles**: 95% of linting rules (standard ESLint rules, TypeScript, React, etc.)
- **Speed**: ~50-100x faster than ESLint
- **Config**: `oxlint.json`
### 2. Custom Rule Checker
- **What**: Node.js AST-based script
- **Handles**: Superset-specific rules:
- No literal colors (use theme)
- No FontAwesome icons (use Icons component)
- No template vars in i18n
- **Speed**: Fast enough for pre-commit
- **Script**: `scripts/check-custom-rules.js`
## Developer Workflow
### Local Development
```bash
# Fast linting (OXC only)
npm run lint
# Full linting (OXC + custom rules)
npm run lint:full
# Auto-fix what's possible
npm run lint-fix
```
### Pre-commit
1. OXC runs first (via `scripts/oxlint.sh`)
2. Custom rules check runs second (lightweight, AST-based)
3. Both must pass for commit to succeed
### CI Pipeline
```yaml
- name: Lint with OXC
run: npm run lint
- name: Check custom rules
run: npm run check:custom-rules
```
## Why This Architecture?
### ✅ Pros
1. **No binary distribution issues** - ASF compatible
2. **Fast performance** - OXC for bulk, lightweight script for custom
3. **Maintainable** - Custom rules in JavaScript, not Rust
4. **Flexible** - Can evolve as OXC adds plugin support
5. **Cacheable** - Both OXC and Node.js are standard tools
### ❌ Cons
1. **Two tools** - Slightly more complex than single linter
2. **Duplicate parsing** - Files parsed twice (once by each tool)
### 🔄 Migration Path
When OXC supports JavaScript plugins:
1. Convert `check-custom-rules.js` to OXC plugin format
2. Consolidate back to single tool
3. Keep same rules and developer experience
## Implementation Checklist
- [x] OXC for standard linting
- [x] Pre-commit integration
- [ ] Custom rules script
- [ ] Combine in npm scripts
- [ ] Update CI pipeline
- [ ] Developer documentation
## Performance Targets
| Operation | Target Time | Current |
|-----------|------------|---------|
| Pre-commit (changed files) | <2s | ✅ 1.5s |
| Full lint (all files) | <10s | ✅ 8s |
| Custom rules check | <5s | 🔄 TBD |
## Caching Strategy
### Local Development
- OXC: Built-in incremental checking
- Custom rules: Use file hash cache (similar to pytest cache)
### CI
- Cache `node_modules` (includes oxlint binary)
- Cache custom rules results by commit hash
- Skip unchanged files using git diff
## Future Improvements
1. **When OXC adds plugin support**: Migrate custom rules to OXC plugins
2. **Consider Biome**: Another Rust-based linter with plugin support
3. **AST sharing**: Investigate sharing AST between tools to avoid double parsing

View File

@@ -9,17 +9,13 @@ Apache Superset is a data visualization platform with Flask/Python backend and R
### Frontend Modernization
- **NO `any` types** - Use proper TypeScript types
- **NO JavaScript files** - Convert to TypeScript (.ts/.tsx)
- **Use @superset-ui/core** - Don't import Ant Design directly, prefer Ant Design component wrappers from @superset-ui/core/components
- **Use antd theming tokens** - Prefer antd tokens over legacy theming tokens
- **Avoid custom css and styles** - Follow antd best practices and avoid styling and custom CSS whenever possible
- **Use @superset-ui/core** - Don't import Ant Design directly
### Testing Strategy Migration
- **Prefer unit tests** over integration tests
- **Prefer integration tests** over end-to-end tests
- **Use Playwright for E2E tests** - Migrating from Cypress
- **Cypress is deprecated** - Will be removed once migration is completed
- **Prefer integration tests** over Cypress end-to-end tests
- **Cypress is last resort** - Actively moving away from Cypress
- **Use Jest + React Testing Library** for component testing
- **Use `test()` instead of `describe()`** - Follow [avoid nesting when testing](https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) principles
### Backend Type Safety
- **Add type hints** - All new Python code needs proper typing
@@ -68,11 +64,7 @@ superset/
### Apache License Headers
- **New files require ASF license headers** - When creating new code files, include the standard Apache Software Foundation license header
- **LLM instruction files are excluded** - Files like AGENTS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead
### Code Comments
- **Avoid time-specific language** - Don't use words like "now", "currently", "today" in code comments as they become outdated
- **Write timeless comments** - Comments should remain accurate regardless of when they're read
- **LLM instruction files are excluded** - Files like LLMS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead
## Documentation Requirements
@@ -112,18 +104,6 @@ superset/
npm run test # All tests
npm run test -- filename.test.tsx # Single file
# E2E Tests (Playwright - NEW)
npm run playwright:test # All Playwright tests
npm run playwright:ui # Interactive UI mode
npm run playwright:headed # See browser during tests
npx playwright test tests/auth/login.spec.ts # Single file
npm run playwright:debug tests/auth/login.spec.ts # Debug specific file
# E2E Tests (Cypress - DEPRECATED)
cd superset-frontend/cypress-base
npm run cypress-run-chrome # All Cypress tests (headless)
npm run cypress-debug # Interactive Cypress UI
# Backend
pytest # All tests
pytest tests/unit_tests/specific_test.py # Single file
@@ -153,19 +133,6 @@ curl -f http://localhost:8088/health || echo "❌ Setup required - see https://s
- **Use negation operator**: `~Model.field` instead of `== False` to avoid ruff E712 errors
- **Example**: `~Model.is_active` instead of `Model.is_active == False`
## Pull Request Guidelines
**When creating pull requests:**
1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format
2. **Use the template sections**: Include all sections from the template (SUMMARY, BEFORE/AFTER, TESTING INSTRUCTIONS, ADDITIONAL INFORMATION)
3. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/)
- Format: `type(scope): description`
- Example: `fix(dashboard): load charts correctly`
- Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore`
**Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time.
## Pre-commit Validation
**Use pre-commit hooks for quality validation:**
@@ -213,6 +180,7 @@ pre-commit run eslint # Frontend linting
## Platform-Specific Instructions
- **[LLMS.md](LLMS.md)** - General LLM development guide (READ THIS FIRST)
- **[CLAUDE.md](CLAUDE.md)** - For Claude/Anthropic tools
- **[.github/copilot-instructions.md](.github/copilot-instructions.md)** - For GitHub Copilot
- **[GEMINI.md](GEMINI.md)** - For Google Gemini tools

View File

@@ -91,7 +91,7 @@ js-format:
cd superset-frontend; npm run prettier
flask-app:
flask run -p 8088 --reload --debugger
flask run -p 8088 --with-threads --reload --debugger
node-app:
cd superset-frontend; npm run dev-server

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.10-slim-trixie
FROM python:3.10-slim-bookworm
RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.10-slim-trixie
FROM python:3.10-slim-bookworm
RUN useradd --user-group --create-home --no-log-init --shell /bin/bash superset

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.10-slim-trixie
FROM python:3.10-slim-bookworm
ARG VERSION
RUN git clone --depth 1 --branch ${VERSION} https://github.com/apache/superset.git /superset

View File

@@ -14,7 +14,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
FROM python:3.10-slim-trixie
FROM python:3.10-slim-bookworm
RUN apt-get update -y
RUN apt-get install -y \

View File

@@ -469,10 +469,6 @@ an account first if you don't have one, and reference your username
while requesting access to push packages.
```bash
# Run this first to make sure you are uploading the right version.
# Pypi does not allow you to delete or retract once uplaoded.
twine check dist/*
twine upload dist/*
```
@@ -522,8 +518,6 @@ takes the version (ie `3.1.1`), the git reference (any SHA, tag or branch
reference), and whether to force the `latest` Docker tag on the
generated images.
**NOTE:** If the docker image isn't built, you'll need to run this [GH action](https://github.com/apache/superset/actions/workflows/tag-release.yml) where you provide it the tag sha.
### Npm Release
You might want to publish the latest @superset-ui release to npm

View File

@@ -32,10 +32,11 @@ else
SUPERSET_VERSION="${1}"
SUPERSET_RC="${2}"
SUPERSET_PGP_FULLNAME="${3}"
SUPERSET_VERSION_RC="${SUPERSET_VERSION}rc${SUPERSET_RC}"
SUPERSET_RELEASE_RC_TARBALL="apache_superset-${SUPERSET_VERSION_RC}-source.tar.gz"
fi
SUPERSET_VERSION_RC="${SUPERSET_VERSION}rc${SUPERSET_RC}"
if [ -z "${SUPERSET_SVN_DEV_PATH}" ]; then
SUPERSET_SVN_DEV_PATH="$HOME/svn/superset_dev"
fi

View File

@@ -28,7 +28,6 @@ These features are considered **unfinished** and should only be used on developm
[//]: # "PLEASE KEEP THE LIST SORTED ALPHABETICALLY"
- ALERT_REPORT_TABS
- DATE_RANGE_TIMESHIFTS_ENABLED
- ENABLE_ADVANCED_DATA_TYPES
- PRESTO_EXPAND_DATA
- SHARE_QUERIES_VIA_KV_STORE

View File

@@ -94,9 +94,9 @@ under the License.
| can available domains on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can request access on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can dashboard on Superset |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|O|
| can post on TableSchemaView |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can expanded on TableSchemaView |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can delete on TableSchemaView |:heavy_check_mark:|O|O|:heavy_check_mark:|
| can post on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can expanded on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can delete on TableSchemaView |:heavy_check_mark:|:heavy_check_mark:|O|O|
| can get on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can post on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|
| can delete query on TabStateView |:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|:heavy_check_mark:|

View File

@@ -23,118 +23,6 @@ This file documents any backwards-incompatible changes in Superset and
assists people when migrating to a new version.
## Next
### MCP Service
The MCP (Model Context Protocol) service enables AI assistants and automation tools to interact programmatically with Superset.
#### New Features
- MCP service infrastructure with FastMCP framework
- Tools for dashboards, charts, datasets, SQL Lab, and instance metadata
- Optional dependency: install with `pip install apache-superset[fastmcp]`
- Runs as separate process from Superset web server
- JWT-based authentication for production deployments
#### New Configuration Options
**Development** (single-user, local testing):
```python
# superset_config.py
MCP_DEV_USERNAME = "admin" # User for MCP authentication
MCP_SERVICE_HOST = "localhost"
MCP_SERVICE_PORT = 5008
```
**Production** (JWT-based, multi-user):
```python
# superset_config.py
MCP_AUTH_ENABLED = True
MCP_JWT_ISSUER = "https://your-auth-provider.com"
MCP_JWT_AUDIENCE = "superset-mcp"
MCP_JWT_ALGORITHM = "RS256" # or "HS256" for shared secrets
# Option 1: Use JWKS endpoint (recommended for RS256)
MCP_JWKS_URI = "https://auth.example.com/.well-known/jwks.json"
# Option 2: Use static public key (RS256)
MCP_JWT_PUBLIC_KEY = "-----BEGIN PUBLIC KEY-----..."
# Option 3: Use shared secret (HS256)
MCP_JWT_ALGORITHM = "HS256"
MCP_JWT_SECRET = "your-shared-secret-key"
# Optional overrides
MCP_SERVICE_HOST = "0.0.0.0"
MCP_SERVICE_PORT = 5008
MCP_SESSION_CONFIG = {
"SESSION_COOKIE_SECURE": True,
"SESSION_COOKIE_HTTPONLY": True,
"SESSION_COOKIE_SAMESITE": "Strict",
}
```
#### Running the MCP Service
```bash
# Development
superset mcp run --port 5008 --debug
# Production
superset mcp run --port 5008
# With factory config
superset mcp run --port 5008 --use-factory-config
```
#### Deployment Considerations
The MCP service runs as a **separate process** from the Superset web server.
**Important**:
- Requires same Python environment and configuration as Superset
- Shares database connections with main Superset app
- Can be scaled independently from web server
- Requires `fastmcp` package (optional dependency)
**Installation**:
```bash
# Install with MCP support
pip install apache-superset[fastmcp]
# Or add to requirements.txt
apache-superset[fastmcp]>=X.Y.Z
```
**Process Management**:
Use systemd, supervisord, or Kubernetes to manage the MCP service process.
See `superset/mcp_service/PRODUCTION.md` for deployment guides.
**Security**:
- Development: Uses `MCP_DEV_USERNAME` for single-user access
- Production: **MUST** configure JWT authentication
- See `superset/mcp_service/SECURITY.md` for details
#### Documentation
- Architecture: `superset/mcp_service/ARCHITECTURE.md`
- Security: `superset/mcp_service/SECURITY.md`
- Production: `superset/mcp_service/PRODUCTION.md`
- Developer Guide: `superset/mcp_service/CLAUDE.md`
- Quick Start: `superset/mcp_service/README.md`
---
- [33055](https://github.com/apache/superset/pull/33055): Upgrades Flask-AppBuilder to 5.0.0. The AUTH_OID authentication type has been deprecated and is no longer available as an option in Flask-AppBuilder. OpenID (OID) is considered a deprecated authentication protocol - if you are using AUTH_OID, you will need to migrate to an alternative authentication method such as OAuth, LDAP, or database authentication before upgrading.
- [35062](https://github.com/apache/superset/pull/35062): Changed the function signature of `setupExtensions` to `setupCodeOverrides` with options as arguments.
- [34871](https://github.com/apache/superset/pull/34871): Fixed Jest test hanging issue from Ant Design v5 upgrade. MessageChannel is now mocked in test environment to prevent rc-overflow from causing Jest to hang. Test environment only - no production impact.
- [34782](https://github.com/apache/superset/pull/34782): Dataset exports now include the dataset ID in their file name (similar to charts and dashboards). If managing assets as code, make sure to rename existing dataset YAMLs to include the ID (and avoid duplicated files).
- [34536](https://github.com/apache/superset/pull/34536): The `ENVIRONMENT_TAG_CONFIG` color values have changed to support only Ant Design semantic colors. Update your `superset_config.py`:
- Change `"error.base"` to just `"error"` after this PR
- Change any hex color values to one of: `"success"`, `"processing"`, `"error"`, `"warning"`, `"default"`
- Custom colors are no longer supported to maintain consistency with Ant Design components
- [34561](https://github.com/apache/superset/pull/34561) Added tiled screenshot functionality for Playwright-based reports to handle large dashboards more efficiently. When enabled (default: `SCREENSHOT_TILED_ENABLED = True`), dashboards with 20+ charts or height exceeding 5000px will be captured using multiple viewport-sized tiles and combined into a single image. This improves report generation performance and reliability for large dashboards.
Note: Pillow is now a required dependency (previously optional) to support image processing for tiled screenshots.
`thumbnails` optional dependency is now deprecated and will be removed in the next major release (7.0).
- [33084](https://github.com/apache/superset/pull/33084) The DISALLOWED_SQL_FUNCTIONS configuration now includes additional potentially sensitive database functions across PostgreSQL, MySQL, SQLite, MS SQL Server, and ClickHouse. Existing queries using these functions may now be blocked. Review your SQL Lab queries and dashboards if you encounter "disallowed function" errors after upgrading
- [34235](https://github.com/apache/superset/pull/34235) CSV exports now use `utf-8-sig` encoding by default to include a UTF-8 BOM, improving compatibility with Excel.
- [34258](https://github.com/apache/superset/pull/34258) changing the default in Dockerfile to INCLUDE_CHROMIUM="false" (from "true") in the past. This ensures the `lean` layer is lean by default, and people can opt-in to the `chromium` layer by setting the build arg `INCLUDE_CHROMIUM=true`. This is a breaking change for anyone using the `lean` layer, as it will no longer include Chromium by default.
@@ -142,9 +30,8 @@ Note: Pillow is now a required dependency (previously optional) to support image
- [33116](https://github.com/apache/superset/pull/33116) In Echarts Series charts (e.g. Line, Area, Bar, etc.) charts, the `x_axis_sort_series` and `x_axis_sort_series_ascending` form data items have been renamed with `x_axis_sort` and `x_axis_sort_asc`.
There's a migration added that can potentially affect a significant number of existing charts.
- [32317](https://github.com/apache/superset/pull/32317) The horizontal filter bar feature is now out of testing/beta development and its feature flag `HORIZONTAL_FILTER_BAR` has been removed.
- [31590](https://github.com/apache/superset/pull/31590) Marks the beginning of intricate work around supporting dynamic Theming, and breaks support for [THEME_OVERRIDES](https://github.com/apache/superset/blob/732de4ac7fae88e29b7f123b6cbb2d7cd411b0e4/superset/config.py#L671) in favor of a new theming system based on AntD V5. Likely this will be in disrepair until settling over the 5.x lifecycle.
- [31590](https://github.com/apache/superset/pull/31590) Marks the begining of intricate work around supporting dynamic Theming, and breaks support for [THEME_OVERRIDES](https://github.com/apache/superset/blob/732de4ac7fae88e29b7f123b6cbb2d7cd411b0e4/superset/config.py#L671) in favor of a new theming system based on AntD V5. Likely this will be in disrepair until settling over the 5.x lifecycle.
- [32432](https://github.com/apache/superset/pull/31260) Moves the List Roles FAB view to the frontend and requires `FAB_ADD_SECURITY_API` to be enabled in the configuration and `superset init` to be executed.
- [34319](https://github.com/apache/superset/pull/34319) Drill to Detail and Drill By is now supported in Embedded mode, and also with the `DASHBOARD_RBAC` FF. If you don't want to expose these features in Embedded / `DASHBOARD_RBAC`, make sure the roles used for Embedded / `DASHBOARD_RBAC`don't have the required permissions to perform D2D actions.
## 5.0.0

View File

@@ -28,7 +28,6 @@ x-superset-image: &superset-image apachesuperset.docker.scarf.sh/apache/superset
x-superset-volumes:
&superset-volumes # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container
- ./docker:/app/docker
- ./superset-core:/app/superset-core
- superset_home:/app/superset_home
services:

View File

@@ -17,47 +17,22 @@
# -----------------------------------------------------------------------
# Lightweight docker-compose for running multiple Superset instances
# This includes only essential services: database and Superset app (no Redis)
# This includes only essential services: database, Redis, and Superset app
#
# RUNNING SUPERSET:
# 1. Start services: docker-compose -f docker-compose-light.yml up
# 2. Access at: http://localhost:9001 (or NODE_PORT if specified)
#
# RUNNING MULTIPLE INSTANCES:
# IMPORTANT: To run multiple instances in parallel:
# - Use different project names: docker-compose -p project1 -f docker-compose-light.yml up
# - Use different NODE_PORT values: NODE_PORT=9002 docker-compose -p project2 -f docker-compose-light.yml up
# - Volumes are isolated by project name (e.g., project1_db_home_light, project2_db_home_light)
# - Database name is intentionally different (superset_light) to prevent accidental cross-connections
#
# RUNNING TESTS WITH PYTEST:
# Tests run in an isolated environment with a separate test database.
# The pytest-runner service automatically creates and initializes the test database on first use.
# MCP Service (Model Context Protocol):
# - Optional service for LLM agent integration, available under 'mcp' profile
# - To include MCP: docker-compose -f docker-compose-light.yml --profile mcp up
# - MCP runs on port 5008 by default (customize with MCP_PORT=5009)
# - Enable SQL debugging with MCP_SQL_DEBUG=true
#
# Basic usage:
# docker-compose -f docker-compose-light.yml run --rm pytest-runner pytest tests/unit_tests/
#
# Run specific test file:
# docker-compose -f docker-compose-light.yml run --rm pytest-runner pytest tests/unit_tests/test_foo.py
#
# Run with pytest options:
# docker-compose -f docker-compose-light.yml run --rm pytest-runner pytest -v -s -x tests/
#
# Force reload test database and run tests (when tests are failing due to bad state):
# docker-compose -f docker-compose-light.yml run --rm -e FORCE_RELOAD=true pytest-runner pytest tests/
#
# Run any command in test environment:
# docker-compose -f docker-compose-light.yml run --rm pytest-runner bash
# docker-compose -f docker-compose-light.yml run --rm pytest-runner pytest --collect-only
#
# For parallel test execution with different projects:
# docker-compose -p project1 -f docker-compose-light.yml run --rm pytest-runner pytest tests/
#
# DEVELOPMENT TIPS:
# - First test run takes ~20-30 seconds (database creation + initialization)
# - Subsequent runs are fast (~2-3 seconds startup)
# - Use FORCE_RELOAD=true when you need a clean test database
# - Tests use SimpleCache instead of Redis (no Redis required)
# - Set SUPERSET_LOG_LEVEL=debug in docker/.env-local for detailed logs
# For verbose logging during development:
# - Set SUPERSET_LOG_LEVEL=debug in docker/.env-local for detailed Superset logs
# -----------------------------------------------------------------------
x-superset-user: &superset-user root
x-superset-volumes: &superset-volumes
@@ -67,18 +42,16 @@ x-superset-volumes: &superset-volumes
- ./superset-frontend:/app/superset-frontend
- superset_home_light:/app/superset_home
- ./tests:/app/tests
- ./extensions:/app/extensions
x-common-build: &common-build
context: .
target: ${SUPERSET_BUILD_TARGET:-dev} # can use `dev` (default) or `lean`
cache_from:
- apache/superset-cache:3.10-slim-trixie
- apache/superset-cache:3.10-slim-bookworm
args:
DEV_MODE: "true"
INCLUDE_CHROMIUM: ${INCLUDE_CHROMIUM:-false}
INCLUDE_FIREFOX: ${INCLUDE_FIREFOX:-false}
BUILD_TRANSLATIONS: ${BUILD_TRANSLATIONS:-false}
LOAD_EXAMPLES_DUCKDB: ${LOAD_EXAMPLES_DUCKDB:-true}
services:
db-light:
@@ -89,12 +62,13 @@ services:
required: false
image: postgres:16
restart: unless-stopped
# No host port mapping - only accessible within Docker network
volumes:
- db_home_light:/var/lib/postgresql/data
- ./docker/docker-entrypoint-initdb.d:/docker-entrypoint-initdb.d
environment:
# Override database name to avoid conflicts
POSTGRES_DB: superset_light
command: postgres -c max_connections=200
superset-light:
env_file:
@@ -106,6 +80,7 @@ services:
<<: *common-build
command: ["/app/docker/docker-bootstrap.sh", "app"]
restart: unless-stopped
# No host port mapping - accessed via webpack dev server proxy
extra_hosts:
- "host.docker.internal:host-gateway"
user: *superset-user
@@ -114,13 +89,16 @@ services:
condition: service_completed_successfully
volumes: *superset-volumes
environment:
# Override DB connection for light service
DATABASE_HOST: db-light
DATABASE_DB: superset_light
POSTGRES_DB: superset_light
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "duckdb:////app/data/examples.duckdb"
EXAMPLES_HOST: db-light
EXAMPLES_DB: superset_light
EXAMPLES_USER: superset
EXAMPLES_PASSWORD: superset
# Use light-specific config that disables Redis
SUPERSET_CONFIG_PATH: /app/docker/pythonpath_dev/superset_config_docker_light.py
GITHUB_HEAD_REF: ${GITHUB_HEAD_REF:-}
GITHUB_SHA: ${GITHUB_SHA:-}
superset-init-light:
build:
@@ -131,16 +109,21 @@ services:
required: true
- path: docker/.env-local # optional override
required: false
user: *superset-user
depends_on:
db-light:
condition: service_started
user: *superset-user
volumes: *superset-volumes
environment:
# Override DB connection for light service
DATABASE_HOST: db-light
DATABASE_DB: superset_light
POSTGRES_DB: superset_light
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "duckdb:////app/data/examples.duckdb"
EXAMPLES_HOST: db-light
EXAMPLES_DB: superset_light
EXAMPLES_USER: superset
EXAMPLES_PASSWORD: superset
# Use light-specific config that disables Redis
SUPERSET_CONFIG_PATH: /app/docker/pythonpath_dev/superset_config_docker_light.py
healthcheck:
disable: true
@@ -163,11 +146,8 @@ services:
SCARF_ANALYTICS: "${SCARF_ANALYTICS:-}"
# configuring the dev-server to use the host.docker.internal to connect to the backend
superset: "http://superset-light:8088"
# Webpack dev server configuration
WEBPACK_DEVSERVER_HOST: "${WEBPACK_DEVSERVER_HOST:-127.0.0.1}"
WEBPACK_DEVSERVER_PORT: "${WEBPACK_DEVSERVER_PORT:-9000}"
ports:
- "${NODE_PORT:-9001}:9000" # Parameterized port, accessible on all interfaces
- "127.0.0.1:${NODE_PORT:-9001}:9000" # Parameterized port
command: ["/app/docker/docker-frontend.sh"]
env_file:
- path: docker/.env # default
@@ -176,30 +156,36 @@ services:
required: false
volumes: *superset-volumes
pytest-runner:
superset-mcp-light:
profiles:
- mcp
build:
<<: *common-build
entrypoint: ["/app/docker/docker-pytest-entrypoint.sh"]
command: ["/app/docker/docker-bootstrap.sh", "mcp"]
restart: unless-stopped
ports:
- "127.0.0.1:${MCP_PORT:-5008}:5008" # Parameterized port
extra_hosts:
- "host.docker.internal:host-gateway"
user: *superset-user
depends_on:
superset-init-light:
condition: service_completed_successfully
volumes: *superset-volumes
env_file:
- path: docker/.env # default
required: true
- path: docker/.env-local # optional override
required: false
profiles:
- test # Only starts when --profile test is used
depends_on:
db-light:
condition: service_started
user: *superset-user
volumes: *superset-volumes
environment:
# Override DB connection for light service
DATABASE_HOST: db-light
DATABASE_DB: test
POSTGRES_DB: test
SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@db-light:5432/test
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "duckdb:////app/data/examples.duckdb"
SUPERSET_CONFIG: superset_test_config_light
PYTHONPATH: /app/pythonpath:/app/docker/pythonpath_dev:/app
DATABASE_DB: superset_light
POSTGRES_DB: superset_light
# Use light-specific config that disables Redis
SUPERSET_CONFIG_PATH: /app/docker/pythonpath_dev/superset_config_docker_light.py
# Enable SQL debugging for MCP if needed
SQLALCHEMY_DEBUG: ${MCP_SQL_DEBUG:-false}
volumes:
superset_home_light:

View File

@@ -33,7 +33,7 @@ x-common-build: &common-build
context: .
target: dev
cache_from:
- apache/superset-cache:3.10-slim-trixie
- apache/superset-cache:3.10-slim-bookworm
services:
redis:

View File

@@ -29,22 +29,19 @@ x-superset-volumes: &superset-volumes
# /app/pythonpath_docker will be appended to the PYTHONPATH in the final container
- ./docker:/app/docker
- ./superset:/app/superset
- ./superset-core:/app/superset-core
- ./superset-frontend:/app/superset-frontend
- superset_home:/app/superset_home
- ./tests:/app/tests
- superset_data:/app/data
x-common-build: &common-build
context: .
target: ${SUPERSET_BUILD_TARGET:-dev} # can use `dev` (default) or `lean`
cache_from:
- apache/superset-cache:3.10-slim-trixie
- apache/superset-cache:3.10-slim-bookworm
args:
DEV_MODE: "true"
INCLUDE_CHROMIUM: ${INCLUDE_CHROMIUM:-false}
INCLUDE_FIREFOX: ${INCLUDE_FIREFOX:-false}
BUILD_TRANSLATIONS: ${BUILD_TRANSLATIONS:-false}
LOAD_EXAMPLES_DUCKDB: ${LOAD_EXAMPLES_DUCKDB:-true}
services:
nginx:
@@ -110,8 +107,6 @@ services:
superset-init:
condition: service_completed_successfully
volumes: *superset-volumes
environment:
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "duckdb:////app/data/examples.duckdb"
superset-websocket:
container_name: superset_websocket
@@ -137,9 +132,9 @@ services:
- /home/superset-websocket/node_modules
- /home/superset-websocket/dist
# Mount config file. Create your own docker/superset-websocket/config.json
# for custom settings, then point to it here. Do not use this example in production.
- ./docker/superset-websocket/config.example.json:/home/superset-websocket/config.json:ro
# Mounting a config file that contains a dummy secret required to boot up.
# do not use this docker compose in production
- ./docker/superset-websocket/config.json:/home/superset-websocket/config.json
environment:
- PORT=8080
- REDIS_HOST=redis
@@ -163,8 +158,6 @@ services:
condition: service_started
user: *superset-user
volumes: *superset-volumes
environment:
SUPERSET__SQLALCHEMY_EXAMPLES_URI: "duckdb:////app/data/examples.duckdb"
healthcheck:
disable: true
@@ -276,5 +269,3 @@ volumes:
external: false
redis:
external: false
superset_data:
external: false

View File

@@ -34,24 +34,8 @@ intended for use with local development.
### Local overrides
#### Environment Variables
To override environment variables locally, create a `./docker/.env-local` file (git-ignored). This file will be loaded after `.env` and can override any settings.
#### Python Configuration
In order to override configuration settings locally, simply make a copy of [`./docker/pythonpath_dev/superset_config_local.example`](./pythonpath_dev/superset_config_local.example)
into `./docker/pythonpath_dev/superset_config_docker.py` (git-ignored) and fill in your overrides.
#### WebSocket Configuration
To customize the WebSocket server configuration, create `./docker/superset-websocket/config.json` (git-ignored) based on [`./docker/superset-websocket/config.example.json`](./superset-websocket/config.example.json).
Then update the `superset-websocket`.`volumes` config to mount it.
#### Docker Compose Overrides
For advanced Docker Compose customization, create a `docker-compose-override.yml` file (git-ignored) to override or extend services without modifying the main compose file.
into `./docker/pythonpath_dev/superset_config_docker.py` (git ignored) and fill in your overrides.
### Local packages

View File

@@ -18,7 +18,7 @@
set -euo pipefail
# Ensure this script is run as root
if [[ ${EUID} -ne 0 ]]; then
if [[ $EUID -ne 0 ]]; then
echo "This script must be run as root" >&2
exit 1
fi
@@ -42,7 +42,7 @@ echo -e "${GREEN}Installing packages: $@${RESET}"
apt-get install -yqq --no-install-recommends "$@"
echo -e "${GREEN}Autoremoving unnecessary packages...${RESET}"
apt-get autoremove -yqq --purge
apt-get autoremove -y
echo -e "${GREEN}Cleaning up package cache and metadata...${RESET}"
apt-get clean

View File

@@ -21,15 +21,8 @@ set -eo pipefail
# Make python interactive
if [ "$DEV_MODE" == "true" ]; then
if [ "$(whoami)" = "root" ] && command -v uv > /dev/null 2>&1; then
# Always ensure superset-core is available
echo "Installing superset-core in editable mode"
uv pip install --no-deps -e /app/superset-core
# Only reinstall the main app for non-worker processes
if [ "$1" != "worker" ] && [ "$1" != "beat" ]; then
echo "Reinstalling the app in editable mode"
uv pip install -e .
fi
echo "Reinstalling the app in editable mode"
uv pip install -e .
fi
fi
REQUIREMENTS_LOCAL="/app/docker/requirements-local.txt"
@@ -41,8 +34,7 @@ if [ "$CYPRESS_CONFIG" == "true" ]; then
export SUPERSET__SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://superset:superset@db:5432/superset_cypress
PORT=8081
fi
# Skip postgres requirements installation for workers to avoid conflicts
if [[ "$DATABASE_DIALECT" == postgres* ]] && [ "$(whoami)" = "root" ] && [ "$1" != "worker" ] && [ "$1" != "beat" ]; then
if [[ "$DATABASE_DIALECT" == postgres* ]] && [ "$(whoami)" = "root" ]; then
# older images may not have the postgres dev requirements installed
echo "Installing postgres requirements"
if command -v uv > /dev/null 2>&1; then
@@ -80,7 +72,7 @@ case "${1}" in
;;
app)
echo "Starting web app (using development server)..."
flask run -p $PORT --reload --debugger --without-threads --host=0.0.0.0 --exclude-patterns "*/node_modules/*:*/.venv/*:*/build/*:*/__pycache__/*"
flask run -p $PORT --with-threads --reload --debugger --host=0.0.0.0
;;
app-gunicorn)
echo "Starting web app..."

View File

@@ -69,8 +69,6 @@ echo_step "3" "Complete" "Setting up roles and perms"
if [ "$SUPERSET_LOAD_EXAMPLES" = "yes" ]; then
# Load some data to play with
echo_step "4" "Starting" "Loading examples"
# If Cypress run which consumes superset_test_config load required data for tests
if [ "$CYPRESS_CONFIG" == "true" ]; then
superset load_examples --load-test-data

View File

@@ -1,152 +0,0 @@
#!/bin/bash
#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
set -e
# Wait for PostgreSQL to be ready
echo "Waiting for database to be ready..."
for i in {1..30}; do
if python3 -c "
import psycopg2
try:
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='superset_light')
conn.close()
print('Database is ready!')
except:
exit(1)
" 2>/dev/null; then
echo "Database connection established!"
break
fi
echo "Waiting for database... ($i/30)"
if [ $i -eq 30 ]; then
echo "Database connection timeout after 30 seconds"
exit 1
fi
sleep 1
done
# Handle database setup based on FORCE_RELOAD
if [ "${FORCE_RELOAD}" = "true" ]; then
echo "Force reload requested - resetting test database"
# Drop and recreate the test database using Python
python3 -c "
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
# Connect to default database
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='superset_light')
conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
# Drop and recreate test database
try:
cur.execute('DROP DATABASE IF EXISTS test')
except:
pass
cur.execute('CREATE DATABASE test')
conn.close()
# Connect to test database to create schemas
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='test')
conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
cur.execute('CREATE SCHEMA sqllab_test_db')
cur.execute('CREATE SCHEMA admin_database')
cur.close()
conn.close()
print('Test database reset successfully')
"
# Use --no-reset-db since we already reset it
FLAGS="--no-reset-db"
else
echo "Using existing test database (set FORCE_RELOAD=true to reset)"
FLAGS="--no-reset-db"
# Ensure test database exists using Python
python3 -c "
import psycopg2
from psycopg2.extensions import ISOLATION_LEVEL_AUTOCOMMIT
# Check if test database exists
try:
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='test')
conn.close()
print('Test database already exists')
except:
print('Creating test database...')
# Connect to default database to create test database
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='superset_light')
conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
# Create test database
cur.execute('CREATE DATABASE test')
conn.close()
# Connect to test database to create schemas
conn = psycopg2.connect(host='db-light', user='superset', password='superset', database='test')
conn.set_isolation_level(ISOLATION_LEVEL_AUTOCOMMIT)
cur = conn.cursor()
cur.execute('CREATE SCHEMA IF NOT EXISTS sqllab_test_db')
cur.execute('CREATE SCHEMA IF NOT EXISTS admin_database')
cur.close()
conn.close()
print('Test database created successfully')
"
fi
# Always run database migrations to ensure schema is up to date
echo "Running database migrations..."
cd /app
superset db upgrade
# Initialize test environment if needed
if [ "${FORCE_RELOAD}" = "true" ] || [ ! -f "/app/superset_home/.test_initialized" ]; then
echo "Initializing test environment..."
# Run initialization commands
superset init
echo "Loading test users..."
superset load-test-users
# Mark as initialized
touch /app/superset_home/.test_initialized
else
echo "Test environment already initialized (skipping init and load-test-users)"
echo "Tip: Use FORCE_RELOAD=true to reinitialize the test database"
fi
# Create missing scripts needed for tests
if [ ! -f "/app/scripts/tag_latest_release.sh" ]; then
echo "Creating missing tag_latest_release.sh script for tests..."
cp /app/docker/tag_latest_release.sh /app/scripts/tag_latest_release.sh 2>/dev/null || true
fi
# Install pip module for Shillelagh compatibility (aligns with CI environment)
echo "Installing pip module for Shillelagh compatibility..."
uv pip install pip
# If arguments provided, execute them
if [ $# -gt 0 ]; then
exec "$@"
fi

View File

@@ -26,7 +26,7 @@ gunicorn \
--workers ${SERVER_WORKER_AMOUNT:-1} \
--worker-class ${SERVER_WORKER_CLASS:-gthread} \
--threads ${SERVER_THREADS_AMOUNT:-20} \
--log-level "${GUNICORN_LOGLEVEL:-info}" \
--log-level "${GUNICORN_LOGLEVEL:info}" \
--timeout ${GUNICORN_TIMEOUT:-60} \
--keep-alive ${GUNICORN_KEEPALIVE:-2} \
--max-requests ${WORKER_MAX_REQUESTS:-0} \

View File

@@ -23,57 +23,25 @@ MIN_MEM_FREE_GB=3
MIN_MEM_FREE_KB=$(($MIN_MEM_FREE_GB*1000000))
echo_mem_warn() {
# Check if running in Codespaces first
if [[ -n "${CODESPACES}" ]]; then
echo "Memory available: Codespaces managed"
return
fi
MEM_FREE_KB=$(awk '/MemFree/ { printf "%s \n", $2 }' /proc/meminfo)
MEM_FREE_GB=$(awk '/MemFree/ { printf "%s \n", $2/1024/1024 }' /proc/meminfo)
# Check platform and get memory accordingly
if [[ -f /proc/meminfo ]]; then
# Linux
if grep -q MemAvailable /proc/meminfo; then
MEM_AVAIL_KB=$(awk '/MemAvailable/ { printf "%s \n", $2 }' /proc/meminfo)
MEM_AVAIL_GB=$(awk '/MemAvailable/ { printf "%s \n", $2/1024/1024 }' /proc/meminfo)
else
MEM_AVAIL_KB=$(awk '/MemFree/ { printf "%s \n", $2 }' /proc/meminfo)
MEM_AVAIL_GB=$(awk '/MemFree/ { printf "%s \n", $2/1024/1024 }' /proc/meminfo)
fi
elif [[ "$(uname)" == "Darwin" ]]; then
# macOS - use vm_stat to get free memory
# vm_stat reports in pages, typically 4096 bytes per page
PAGE_SIZE=$(pagesize)
FREE_PAGES=$(vm_stat | awk '/Pages free:/ {print $3}' | tr -d '.')
INACTIVE_PAGES=$(vm_stat | awk '/Pages inactive:/ {print $3}' | tr -d '.')
# Free + inactive pages give us available memory (similar to MemAvailable on Linux)
AVAIL_PAGES=$((FREE_PAGES + INACTIVE_PAGES))
MEM_AVAIL_KB=$((AVAIL_PAGES * PAGE_SIZE / 1024))
MEM_AVAIL_GB=$(echo "scale=2; $MEM_AVAIL_KB / 1024 / 1024" | bc)
else
# Other platforms
echo "Memory available: Unable to determine"
return
fi
if [[ "${MEM_AVAIL_KB}" -lt "${MIN_MEM_FREE_KB}" ]]; then
if [[ "${MEM_FREE_KB}" -lt "${MIN_MEM_FREE_KB}" ]]; then
cat <<EOF
===============================================
======== Memory Insufficient Warning =========
===============================================
It looks like you only have ${MEM_AVAIL_GB}GB of
memory ${MEM_TYPE}. Please increase your Docker
It looks like you only have ${MEM_FREE_GB}GB of
memory free. Please increase your Docker
resources to at least ${MIN_MEM_FREE_GB}GB
Note: During builds, available memory may be
temporarily low due to caching and compilation.
===============================================
======== Memory Insufficient Warning =========
===============================================
EOF
else
echo "Memory available: ${MEM_AVAIL_GB} GB"
echo "Memory check Ok [${MEM_FREE_GB}GB free]"
fi
}

View File

@@ -38,14 +38,14 @@ for arg in "$@"; do
done
# Install build-essential if required
if ${REQUIRES_BUILD_ESSENTIAL}; then
if $REQUIRES_BUILD_ESSENTIAL; then
echo "Installing build-essential for package builds..."
apt-get update -qq \
&& apt-get install -yqq --no-install-recommends build-essential
fi
# Choose whether to use pip cache
if ${USE_CACHE}; then
if $USE_CACHE; then
echo "Using pip cache..."
uv pip install "${ARGS[@]}"
else
@@ -54,7 +54,7 @@ else
fi
# Remove build-essential if it was installed
if ${REQUIRES_BUILD_ESSENTIAL}; then
if $REQUIRES_BUILD_ESSENTIAL; then
echo "Removing build-essential to keep the image lean..."
apt-get autoremove -yqq --purge build-essential \
&& apt-get clean \

View File

@@ -49,18 +49,12 @@ SQLALCHEMY_DATABASE_URI = (
f"{DATABASE_HOST}:{DATABASE_PORT}/{DATABASE_DB}"
)
# Use environment variable if set, otherwise construct from components
# This MUST take precedence over any other configuration
SQLALCHEMY_EXAMPLES_URI = os.getenv(
"SUPERSET__SQLALCHEMY_EXAMPLES_URI",
(
f"{DATABASE_DIALECT}://"
f"{EXAMPLES_USER}:{EXAMPLES_PASSWORD}@"
f"{EXAMPLES_HOST}:{EXAMPLES_PORT}/{EXAMPLES_DB}"
),
SQLALCHEMY_EXAMPLES_URI = (
f"{DATABASE_DIALECT}://"
f"{EXAMPLES_USER}:{EXAMPLES_PASSWORD}@"
f"{EXAMPLES_HOST}:{EXAMPLES_PORT}/{EXAMPLES_DB}"
)
REDIS_HOST = os.getenv("REDIS_HOST", "redis")
REDIS_PORT = os.getenv("REDIS_PORT", "6379")
REDIS_CELERY_DB = os.getenv("REDIS_CELERY_DB", "0")
@@ -105,15 +99,7 @@ class CeleryConfig:
CELERY_CONFIG = CeleryConfig
# Extensions configuration
# For local development, point to the extensions directory
# Note: If running in Docker, this path needs to be accessible from inside the container
EXTENSIONS_PATH = os.getenv("EXTENSIONS_PATH", "/app/extensions")
FEATURE_FLAGS = {
"ALERT_REPORTS": True,
"ENABLE_EXTENSIONS": True,
}
FEATURE_FLAGS = {"ALERT_REPORTS": True}
ALERT_REPORTS_NOTIFICATION_DRY_RUN = True
WEBDRIVER_BASEURL = f"http://superset_app{os.environ.get('SUPERSET_APP_ROOT', '/')}/" # When using docker compose baseurl should be http://superset_nginx{ENV{BASEPATH}}/ # noqa: E501
# The base URL for the email report hyperlinks.
@@ -146,7 +132,7 @@ try:
from superset_config_docker import * # noqa: F403
logger.info(
"Loaded your Docker configuration at [%s]", superset_config_docker.__file__
f"Loaded your Docker configuration at [{superset_config_docker.__file__}]"
)
except ImportError:
logger.info("Using default Docker config...")

View File

@@ -1,55 +0,0 @@
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Test configuration for docker-compose-light.yml - uses SimpleCache instead of Redis
# Import all settings from the main test config first
import os
import sys
# Add the tests directory to the path to import the test config
sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..", ".."))
from tests.integration_tests.superset_test_config import * # noqa: F403
# Override Redis-based caching to use simple in-memory cache
CACHE_CONFIG = {
"CACHE_TYPE": "SimpleCache",
"CACHE_DEFAULT_TIMEOUT": 300,
"CACHE_KEY_PREFIX": "superset_test_",
}
DATA_CACHE_CONFIG = {
**CACHE_CONFIG,
"CACHE_DEFAULT_TIMEOUT": 30,
"CACHE_KEY_PREFIX": "superset_test_data_",
}
# Keep SimpleCache for these as they're already using it
# FILTER_STATE_CACHE_CONFIG - already SimpleCache in parent
# EXPLORE_FORM_DATA_CACHE_CONFIG - already SimpleCache in parent
# Disable Celery for lightweight testing
CELERY_CONFIG = None
# Use FileSystemCache for SQL Lab results instead of Redis
from flask_caching.backends.filesystemcache import FileSystemCache # noqa: E402
RESULTS_BACKEND = FileSystemCache("/app/superset_home/sqllab_test")
# Override WEBDRIVER_BASEURL for tests to match expected values
WEBDRIVER_BASEURL = "http://0.0.0.0:8080/"
WEBDRIVER_BASEURL_USER_FRIENDLY = WEBDRIVER_BASEURL

View File

@@ -1,22 +0,0 @@
{
"port": 8080,
"logLevel": "info",
"logToFile": false,
"logFilename": "app.log",
"statsd": {
"host": "127.0.0.1",
"port": 8125,
"globalTags": []
},
"redis": {
"port": 6379,
"host": "127.0.0.1",
"password": "",
"db": 0,
"ssl": false
},
"redisStreamPrefix": "async-events-",
"jwtAlgorithms": ["HS256"],
"jwtSecret": "CHANGE-ME-IN-PRODUCTION-GOTTA-BE-LONG-AND-SECRET",
"jwtCookieName": "async-token"
}

View File

@@ -1,190 +0,0 @@
#! /bin/bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
run_git_tag () {
if [[ "$DRY_RUN" == "false" ]] && [[ "$SKIP_TAG" == "false" ]]
then
git tag -a -f latest "${GITHUB_TAG_NAME}" -m "latest tag"
echo "${GITHUB_TAG_NAME} has been tagged 'latest'"
fi
exit 0
}
###
# separating out git commands into functions so they can be mocked in unit tests
###
git_show_ref () {
if [[ "$TEST_ENV" == "true" ]]
then
if [[ "$GITHUB_TAG_NAME" == "does_not_exist" ]]
# mock return for testing only
then
echo ""
else
echo "2817aebd69dc7d199ec45d973a2079f35e5658b6 refs/tags/${GITHUB_TAG_NAME}"
fi
fi
result=$(git show-ref "${GITHUB_TAG_NAME}")
echo "${result}"
}
get_latest_tag_list () {
if [[ "$TEST_ENV" == "true" ]]
then
echo "(tag: 2.1.0, apache/2.1test)"
else
result=$(git show-ref --tags --dereference latest | awk '{print $2}' | xargs git show --pretty=tformat:%d -s | grep tag:)
echo "${result}"
fi
}
###
split_string () {
local version="$1"
local delimiter="$2"
local components=()
local tmp=""
for (( i=0; i<${#version}; i++ )); do
local char="${version:$i:1}"
if [[ "$char" != "$delimiter" ]]; then
tmp="$tmp$char"
elif [[ -n "$tmp" ]]; then
components+=("$tmp")
tmp=""
fi
done
if [[ -n "$tmp" ]]; then
components+=("$tmp")
fi
echo "${components[@]}"
}
DRY_RUN=false
# get params passed in with script when it was run
# --dry-run is optional and returns the value of SKIP_TAG, but does not run the git tag statement
# A tag name is required as a param. A SHA won't work. You must first tag a sha with a release number
# and then run this script
while [[ $# -gt 0 ]]
do
key="$1"
case $key in
--dry-run)
DRY_RUN=true
shift # past value
;;
*) # this should be the tag name
GITHUB_TAG_NAME=$key
shift # past value
;;
esac
done
if [ -z "${GITHUB_TAG_NAME}" ]; then
echo "Missing tag parameter, usage: ./scripts/tag_latest_release.sh <GITHUB_TAG_NAME>"
echo "SKIP_TAG=true" >> $GITHUB_OUTPUT
exit 1
fi
if [ -z "$(git_show_ref)" ]; then
echo "The tag ${GITHUB_TAG_NAME} does not exist. Please use a different tag."
echo "SKIP_TAG=true" >> $GITHUB_OUTPUT
exit 0
fi
# check that this tag only contains a proper semantic version
if ! [[ ${GITHUB_TAG_NAME} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
then
echo "This tag ${GITHUB_TAG_NAME} is not a valid release version. Not tagging."
echo "SKIP_TAG=true" >> $GITHUB_OUTPUT
exit 1
fi
## split the current GITHUB_TAG_NAME into an array at the dot
THIS_TAG_NAME=$(split_string "${GITHUB_TAG_NAME}" ".")
# look up the 'latest' tag on git
LATEST_TAG_LIST=$(get_latest_tag_list) || echo 'not found'
# if 'latest' tag doesn't exist, then set this commit to latest
if [[ -z "$LATEST_TAG_LIST" ]]
then
echo "there are no latest tags yet, so I'm going to start by tagging this sha as the latest"
run_git_tag
exit 0
fi
# remove parenthesis and tag: from the list of tags
LATEST_TAGS_STRINGS=$(echo "$LATEST_TAG_LIST" | sed 's/tag: \([^,]*\)/\1/g' | tr -d '()')
LATEST_TAGS=$(split_string "$LATEST_TAGS_STRINGS" ",")
TAGS=($(split_string "$LATEST_TAGS" " "))
# Initialize a flag for comparison result
compare_result=""
# Iterate through the tags of the latest release
for tag in $TAGS
do
if [[ $tag == "latest" ]]; then
continue
else
## extract just the version from this tag
LATEST_RELEASE_TAG="$tag"
echo "LATEST_RELEASE_TAG: ${LATEST_RELEASE_TAG}"
# check that this only contains a proper semantic version
if ! [[ ${LATEST_RELEASE_TAG} =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]]
then
echo "'Latest' has been associated with tag ${LATEST_RELEASE_TAG} which is not a valid release version. Looking for another."
continue
fi
echo "The current release with the latest tag is version ${LATEST_RELEASE_TAG}"
# Split the version strings into arrays
THIS_TAG_NAME_ARRAY=($(split_string "$THIS_TAG_NAME" "."))
LATEST_RELEASE_TAG_ARRAY=($(split_string "$LATEST_RELEASE_TAG" "."))
# Iterate through the components of the version strings
for (( j=0; j<${#THIS_TAG_NAME_ARRAY[@]}; j++ )); do
echo "Comparing ${THIS_TAG_NAME_ARRAY[$j]} to ${LATEST_RELEASE_TAG_ARRAY[$j]}"
if [[ $((THIS_TAG_NAME_ARRAY[$j])) > $((LATEST_RELEASE_TAG_ARRAY[$j])) ]]; then
compare_result="greater"
break
elif [[ $((THIS_TAG_NAME_ARRAY[$j])) < $((LATEST_RELEASE_TAG_ARRAY[$j])) ]]; then
compare_result="lesser"
break
fi
done
fi
done
# Determine the result based on the comparison
if [[ -z "$compare_result" ]]; then
echo "Versions are equal"
echo "SKIP_TAG=true" >> $GITHUB_OUTPUT
elif [[ "$compare_result" == "greater" ]]; then
echo "This release tag ${GITHUB_TAG_NAME} is newer than the latest."
echo "SKIP_TAG=false" >> $GITHUB_OUTPUT
# Add other actions you want to perform for a newer version
elif [[ "$compare_result" == "lesser" ]]; then
echo "This release tag ${GITHUB_TAG_NAME} is older than the latest."
echo "This release tag ${GITHUB_TAG_NAME} is not the latest. Not tagging."
# if you've gotten this far, then we don't want to run any tags in the next step
echo "SKIP_TAG=true" >> $GITHUB_OUTPUT
fi

View File

@@ -1,642 +0,0 @@
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
# LLM Context Guide for Apache Superset Documentation
This guide helps LLMs work with the Apache Superset documentation site built with Docusaurus 3.
## 📍 Current Directory Context
You are currently in the `/docs` subdirectory of the Apache Superset repository. When referencing files from the main codebase, use `../` to access the parent directory.
```
/Users/evan_1/GitHub/superset/ # Main repository root
├── superset/ # Python backend code
├── superset-frontend/ # React/TypeScript frontend
└── docs/ # Documentation site (YOU ARE HERE)
├── docs/ # Main documentation content
├── developer_portal/ # Developer guides (currently disabled)
├── components/ # Component playground (currently disabled)
└── docusaurus.config.ts # Site configuration
```
## 🚀 Quick Commands
```bash
# Development
yarn start # Start dev server on http://localhost:3000
yarn stop # Stop running dev server
yarn build # Build production site
yarn serve # Serve built site locally
# Version Management (USE THESE, NOT docusaurus commands)
yarn version:add:docs <version> # Add new docs version
yarn version:add:developer_portal <version> # Add developer portal version
yarn version:add:components <version> # Add components version
yarn version:remove:docs <version> # Remove docs version
yarn version:remove:developer_portal <version> # Remove developer portal version
yarn version:remove:components <version> # Remove components version
# Quality Checks
yarn typecheck # TypeScript validation
yarn eslint # Lint TypeScript/JavaScript files
```
## 📁 Documentation Structure
### Main Documentation (`/docs`)
The primary documentation lives in `/docs` with this structure:
```
docs/
├── intro.md # Auto-generated from ../README.md
├── quickstart.mdx # Getting started guide
├── api.mdx # API reference with Swagger UI
├── faq.mdx # Frequently asked questions
├── installation/ # Installation guides
│ ├── installation-methods.mdx
│ ├── docker-compose.mdx
│ ├── docker-builds.mdx
│ ├── kubernetes.mdx
│ ├── pypi.mdx
│ └── architecture.mdx
├── configuration/ # Configuration guides
│ ├── configuring-superset.mdx
│ ├── alerts-reports.mdx
│ ├── caching.mdx
│ ├── databases.mdx
│ └── [more config docs]
├── using-superset/ # User guides
│ ├── creating-your-first-dashboard.md
│ ├── exploring-data.mdx
│ └── [more user docs]
├── contributing/ # Contributor guides
│ ├── development.mdx
│ ├── testing-locally.mdx
│ └── [more contributor docs]
└── security/ # Security documentation
├── security.mdx
└── [security guides]
```
### Developer Portal (`/developer_portal`) - Currently Disabled
When enabled, contains developer-focused content:
- API documentation
- Architecture guides
- CLI tools
- Code examples
### Component Playground (`/components`) - Currently Disabled
When enabled, provides interactive component examples for UI development.
## 📝 Documentation Standards
### File Types
- **`.md` files**: Basic Markdown documents
- **`.mdx` files**: Markdown with JSX - can include React components
- **`.tsx` files in `/src`**: Custom React components and pages
### Frontmatter Structure
Every documentation page should have frontmatter:
```yaml
---
title: Page Title
description: Brief description for SEO
sidebar_position: 1 # Optional: controls order in sidebar
---
```
### MDX Component Usage
MDX files can import and use React components:
```mdx
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
<Tabs>
<TabItem value="npm" label="npm" default>
```bash
npm install superset
```
</TabItem>
<TabItem value="yarn" label="yarn">
```bash
yarn add superset
```
</TabItem>
</Tabs>
```
### Code Blocks
Use triple backticks with language identifiers:
````markdown
```python
def hello_world():
print("Hello, Superset!")
```
```sql title="Example Query"
SELECT * FROM users WHERE active = true;
```
```bash
# Installation command
pip install apache-superset
```
````
### Admonitions
Docusaurus supports various admonition types:
```markdown
:::note
This is a note
:::
:::tip
This is a tip
:::
:::warning
This is a warning
:::
:::danger
This is a danger warning
:::
:::info
This is an info box
:::
```
## 🔄 Version Management
### Version Configuration
Versions are managed through `versions-config.json`:
```json
{
"docs": {
"disabled": false,
"lastVersion": "6.0.0", // Default version shown
"includeCurrentVersion": true, // Show "Next" version
"onlyIncludeVersions": ["current", "6.0.0"],
"versions": {
"current": {
"label": "Next",
"path": "",
"banner": "unreleased" // Shows warning banner
},
"6.0.0": {
"label": "6.0.0",
"path": "6.0.0",
"banner": "none"
}
}
}
}
```
### Creating New Versions
**IMPORTANT**: Always use the custom scripts, NOT native Docusaurus commands:
```bash
# ✅ CORRECT - Updates both Docusaurus and versions-config.json
yarn version:add:docs 6.1.0
# ❌ WRONG - Only updates Docusaurus, breaks version dropdown
yarn docusaurus docs:version 6.1.0
```
### Version Files Created
When versioning, these files are created:
- `versioned_docs/version-X.X.X/` - Snapshot of current docs
- `versioned_sidebars/version-X.X.X-sidebars.json` - Sidebar config
- `versions.json` - List of all versions
## 🎨 Styling and Theming
### Custom CSS
Add custom styles in `/src/css/custom.css`:
```css
:root {
--ifm-color-primary: #20a7c9;
--ifm-code-font-size: 95%;
}
```
### Custom Components
Create React components in `/src/components/`:
```tsx
// src/components/FeatureCard.tsx
import React from 'react';
export default function FeatureCard({title, description}) {
return (
<div className="card">
<h3>{title}</h3>
<p>{description}</p>
</div>
);
}
```
Use in MDX:
```mdx
import FeatureCard from '@site/src/components/FeatureCard';
<FeatureCard
title="Fast"
description="Lightning fast queries"
/>
```
## 📦 Key Dependencies
- **Docusaurus 3.8.1**: Static site generator
- **React 18.3**: UI framework
- **Ant Design 5.26**: Component library
- **@superset-ui/core**: Superset UI components
- **Swagger UI React**: API documentation
- **Prism**: Syntax highlighting
## 🔗 Linking Strategies
### Internal Links
Use relative paths for internal documentation:
```markdown
[Installation Guide](./installation/docker-compose)
[Configuration](../configuration/configuring-superset)
```
### External Links
Always use full URLs:
```markdown
[Apache Superset GitHub](https://github.com/apache/superset)
```
### Linking to Code
Reference code in the main repository:
```markdown
See the [main configuration file](https://github.com/apache/superset/blob/master/superset/config.py)
```
## 🛠️ Common Documentation Tasks
### Adding a New Guide
1. Create the `.mdx` file in the appropriate directory
2. Add frontmatter with title and description
3. Update sidebar if needed (for manual sidebar configs)
### Adding API Documentation
The API docs use Swagger UI embedded in `/docs/api.mdx`:
```mdx
import SwaggerUI from "swagger-ui-react";
import "swagger-ui-react/swagger-ui.css";
<SwaggerUI url="/api/v1/openapi.json" />
```
### Adding Interactive Examples
Use MDX to create interactive documentation:
```mdx
import CodeBlock from '@theme/CodeBlock';
import MyComponentExample from '!!raw-loader!../examples/MyComponent.tsx';
<CodeBlock language="tsx">{MyComponentExample}</CodeBlock>
```
## 📋 Documentation Checklist
When creating or updating documentation:
- [ ] Clear, descriptive title in frontmatter
- [ ] Description for SEO in frontmatter
- [ ] Proper heading hierarchy (h1 -> h2 -> h3)
- [ ] Code examples with language identifiers
- [ ] Links verified (internal and external)
- [ ] Images have alt text
- [ ] Admonitions used for important notes
- [ ] Tested locally with `yarn start`
- [ ] No broken links (check with `yarn build`)
## 🔍 Searching and Navigation
### Sidebar Configuration
Sidebars are configured in `/sidebars.js`:
```javascript
module.exports = {
CustomSidebar: [
{
type: 'doc',
label: 'Introduction',
id: 'intro',
},
{
type: 'category',
label: 'Installation',
items: [
{
type: 'autogenerated',
dirName: 'installation',
},
],
},
],
};
```
### Search
Docusaurus includes Algolia DocSearch integration configured in `docusaurus.config.ts`.
## 🚫 Common Pitfalls to Avoid
1. **Never use `yarn docusaurus docs:version`** - Use `yarn version:add:docs` instead
2. **Don't edit versioned docs directly** - Edit current docs and create new version
3. **Avoid absolute paths in links** - Use relative paths for maintainability
4. **Don't forget frontmatter** - Every doc needs title and description
5. **Test builds locally** - Run `yarn build` before committing
## 🔧 Troubleshooting
### Dev Server Issues
```bash
yarn stop # Kill any running servers
yarn clear # Clear cache
yarn start # Restart
```
### Build Failures
```bash
# Check for broken links
yarn build
# TypeScript issues
yarn typecheck
# Linting issues
yarn eslint
```
### Version Issues
If versions don't appear in dropdown:
1. Check `versions-config.json` includes the version
2. Verify version files exist in `versioned_docs/`
3. Restart dev server
## 📚 Resources
- [Docusaurus Documentation](https://docusaurus.io/docs)
- [MDX Documentation](https://mdxjs.com/)
- [Superset Contributing Guide](../CONTRIBUTING.md)
- [Main Superset Documentation](https://superset.apache.org/docs/intro)
## 📖 Real Examples and Patterns
### Example: Configuration Documentation Pattern
From `docs/configuration/configuring-superset.mdx`:
```mdx
---
title: Configuring Superset
hide_title: true
sidebar_position: 1
version: 1
---
# Configuring Superset
## superset_config.py
Superset exposes hundreds of configurable parameters through its
[config.py module](https://github.com/apache/superset/blob/master/superset/config.py).
```bash
export SUPERSET_CONFIG_PATH=/app/superset_config.py
```
```
**Key patterns:**
- Links to source code for reference
- Code blocks with bash/python examples
- Environment variable documentation
- Step-by-step configuration instructions
### Example: Tutorial Documentation Pattern
From `docs/using-superset/creating-your-first-dashboard.mdx`:
```mdx
import useBaseUrl from "@docusaurus/useBaseUrl";
## Creating Your First Dashboard
:::tip
In addition to this site, [Preset.io](http://preset.io/) maintains an updated set of end-user
documentation at [docs.preset.io](https://docs.preset.io/).
:::
### Connecting to a new database
<img src={useBaseUrl("/img/tutorial/tutorial_01_add_database_connection.png")} width="600" />
```
**Key patterns:**
- Import Docusaurus hooks for dynamic URLs
- Use of admonitions (:::tip) for helpful information
- Screenshots with useBaseUrl for proper path resolution
- Clear section hierarchy with ### subheadings
- Step-by-step visual guides
### Example: API Documentation Pattern
From `docs/api.mdx`:
```mdx
import SwaggerUI from "swagger-ui-react";
import "swagger-ui-react/swagger-ui.css";
## API Documentation
<SwaggerUI url="/api/v1/openapi.json" />
```
**Key patterns:**
- Embedding interactive Swagger UI
- Importing necessary CSS
- Direct API spec integration
### Common Image Patterns
```mdx
// For images in static folder
import useBaseUrl from "@docusaurus/useBaseUrl";
<img src={useBaseUrl("/img/feature-screenshot.png")} width="600" />
// With caption
<figure>
<img src={useBaseUrl("/img/dashboard.png")} alt="Dashboard view" />
<figcaption>Superset Dashboard Interface</figcaption>
</figure>
```
### Multi-Tab Code Examples
```mdx
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
<Tabs
defaultValue="docker"
values={[
{label: 'Docker', value: 'docker'},
{label: 'Kubernetes', value: 'k8s'},
{label: 'PyPI', value: 'pypi'},
]}>
<TabItem value="docker">
```bash
docker-compose up
```
</TabItem>
<TabItem value="k8s">
```bash
kubectl apply -f superset.yaml
```
</TabItem>
<TabItem value="pypi">
```bash
pip install apache-superset
```
</TabItem>
</Tabs>
```
### Configuration File Examples
```mdx
```python title="superset_config.py"
# Database connection example
SQLALCHEMY_DATABASE_URI = 'postgresql://user:password@localhost/superset'
# Security configuration
SECRET_KEY = 'YOUR_SECRET_KEY_HERE'
WTF_CSRF_ENABLED = True
# Feature flags
FEATURE_FLAGS = {
'ENABLE_TEMPLATE_PROCESSING': True,
'DASHBOARD_NATIVE_FILTERS': True,
}
```
```
### Cross-Referencing Pattern
```mdx
For detailed configuration options, see:
- [Configuring Superset](./configuration/configuring-superset)
- [Database Connections](./configuration/databases)
- [Security Settings](./security/security)
External resources:
- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/)
- [Flask Configuration](https://flask.palletsprojects.com/config/)
```
### Writing Installation Guides
```mdx
## Prerequisites
:::warning
Ensure you have Python 3.9+ and Node.js 16+ installed before proceeding.
:::
## Installation Steps
1. **Clone the repository**
```bash
git clone https://github.com/apache/superset.git
cd superset
```
2. **Install Python dependencies**
```bash
pip install -e .
```
3. **Initialize the database**
```bash
superset db upgrade
superset init
```
:::tip Success Check
Navigate to http://localhost:8088 and login with admin/admin
:::
```
### Documenting API Endpoints
```mdx
## Chart API
### GET /api/v1/chart/
Returns a list of charts.
**Parameters:**
- `page` (optional): Page number
- `page_size` (optional): Number of items per page
**Example Request:**
```bash
curl -X GET "http://localhost:8088/api/v1/chart/" \
-H "Authorization: Bearer YOUR_ACCESS_TOKEN"
```
**Example Response:**
```json
{
"count": 42,
"result": [
{
"id": 1,
"slice_name": "Sales Dashboard",
"viz_type": "line"
}
]
}
```
```
---
**Note**: This documentation site serves as the primary resource for Superset users, administrators, and contributors. Always prioritize clarity, accuracy, and completeness when creating or updating documentation.

Some files were not shown because too many files have changed in this diff Show More