diff --git a/.asf.yaml b/.asf.yaml index 39292a6e85c..90f75523f67 100644 --- a/.asf.yaml +++ b/.asf.yaml @@ -83,6 +83,7 @@ github: - cypress-matrix (5, chrome) - dependency-review - frontend-build + - playwright-tests (chromium) - pre-commit (current) - pre-commit (previous) - test-mysql diff --git a/.claude/commands/js-to-ts.md b/.claude/commands/js-to-ts.md new file mode 100644 index 00000000000..b3e0048983d --- /dev/null +++ b/.claude/commands/js-to-ts.md @@ -0,0 +1,10 @@ +# JavaScript to TypeScript Migration Command + +## Usage +``` +/js-to-ts +``` +- `` - Path to CORE file relative to `superset-frontend/` (e.g., `src/utils/common.js`, `src/middleware/loggerMiddleware.js`) + +## Agent Instructions +**See:** [../projects/js-to-ts/AGENT.md](../projects/js-to-ts/AGENT.md) for complete migration guide. diff --git a/.claude/projects/js-to-ts/AGENT.md b/.claude/projects/js-to-ts/AGENT.md new file mode 100644 index 00000000000..47e6cac7cf1 --- /dev/null +++ b/.claude/projects/js-to-ts/AGENT.md @@ -0,0 +1,684 @@ +# JavaScript to TypeScript Migration Agent Guide + +**Complete technical reference for converting JavaScript/JSX files to TypeScript/TSX in Apache Superset frontend.** + +**Agent Role:** Atomic migration unit - migrate the core file + ALL related tests/mocks as one cohesive unit. Use `git mv` to preserve history, NO `git commit`. NO global import changes. Report results upon completion. + +--- + +## 🎯 Migration Principles + +1. **Atomic migration units** - Core file + all related tests/mocks migrate together +2. **Zero `any` types** - Use proper TypeScript throughout +3. **Leverage existing types** - Reuse established definitions +4. **Type inheritance** - Derivatives extend base component types +5. **Strategic placement** - File types for maximum discoverability +6. **Surgical improvements** - Enhance existing types during migration + +--- + +## Step 0: Dependency Check (MANDATORY) + +**Command:** +```bash +grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" superset-frontend/{filename} +``` + +**Decision:** +- ✅ No matches → Proceed with atomic migration (core + tests + mocks) +- ❌ Matches found → EXIT with dependency report (see format below) + +--- + +## Step 1: Identify Related Files (REQUIRED) + +**Atomic Migration Scope:** +For core file `src/utils/example.js`, also migrate: +- `src/utils/example.test.js` / `src/utils/example.test.jsx` +- `src/utils/example.spec.js` / `src/utils/example.spec.jsx` +- `src/utils/__mocks__/example.js` +- Any other related test/mock files found by pattern matching + +**Find all related test and mock files:** +```bash +# Pattern-based search for related files +basename=$(basename {filename} .js) +dirname=$(dirname superset-frontend/{filename}) + +# Find test files +find "$dirname" -name "${basename}.test.js" -o -name "${basename}.test.jsx" +find "$dirname" -name "${basename}.spec.js" -o -name "${basename}.spec.jsx" + +# Find mock files +find "$dirname" -name "__mocks__/${basename}.js" +find "$dirname" -name "${basename}.mock.js" +``` + +**Migration Requirement:** All discovered related files MUST be migrated together as one atomic unit. + +**Test File Creation:** If NO test files exist for the core file, CREATE a minimal test file using the following pattern: +- Location: Same directory as core file +- Name: `{basename}.test.ts` (e.g., `DebouncedMessageQueue.test.ts`) +- Content: Basic test structure importing and testing the main functionality +- Use proper TypeScript types in test file + +--- + +## 🗺️ Type Reference Map + +### From `@superset-ui/core` +```typescript +// Data & Query +QueryFormData, QueryData, JsonObject, AnnotationData, AdhocMetric +LatestQueryFormData, GenericDataType, DatasourceType, ExtraFormData +DataMaskStateWithId, NativeFilterScope, NativeFiltersState, NativeFilterTarget + +// UI & Theme +FeatureFlagMap, LanguagePack, ColorSchemeConfig, SequentialSchemeConfig +``` + +### From `@superset-ui/chart-controls` +```typescript +Dataset, ColumnMeta, ControlStateMapping +``` + +### From Local Types (`src/types/`) +```typescript +// Authentication +User, UserWithPermissionsAndRoles, BootstrapUser, PermissionsAndRoles + +// Dashboard +Dashboard, DashboardState, DashboardInfo, DashboardLayout, LayoutItem +ComponentType, ChartConfiguration, ActiveFilters + +// Charts +Chart, ChartState, ChartStatus, ChartLinkedDashboard, Slice, SaveActionType + +// Data +Datasource, Database, Owner, Role + +// UI Components +TagType, FavoriteStatus, Filter, ImportResourceName +``` + +### From Domain Types +```typescript +// src/dashboard/types.ts +RootState, ChartsState, DatasourcesState, FilterBarOrientation +ChartCrossFiltersConfig, ActiveTabs, MenuKeys + +// src/explore/types.ts +ExplorePageInitialData, ExplorePageState, ExploreResponsePayload, OptionSortType + +// src/SqlLab/types.ts +[SQL Lab specific types] +``` + +--- + +## 🏗️ Type Organization Strategy + +### Type Placement Hierarchy + +1. **Component-Colocated** (90% of cases) + ```typescript + // Same file as component + interface MyComponentProps { + title: string; + onClick: () => void; + } + ``` + +2. **Feature-Shared** + ```typescript + // src/[domain]/components/[Feature]/types.ts + export interface FilterConfiguration { + filterId: string; + targets: NativeFilterTarget[]; + } + ``` + +3. **Domain-Wide** + ```typescript + // src/[domain]/types.ts + export interface ExploreFormData extends QueryFormData { + viz_type: string; + } + ``` + +4. **Global** + ```typescript + // src/types/[TypeName].ts + export interface ApiResponse { + result: T; + count?: number; + } + ``` + +### Type Discovery Commands +```bash +# Search existing types before creating +find superset-frontend/src -name "types.ts" -exec grep -l "[TypeConcept]" {} \; +grep -r "interface.*Props\|type.*Props" superset-frontend/src/ +``` + +### Derivative Component Patterns + +**Rule:** Components that extend others should extend their type interfaces. + +```typescript +// ✅ Base component type +interface SelectProps { + value: string | number; + options: SelectOption[]; + onChange: (value: string | number) => void; + disabled?: boolean; +} + +// ✅ Derivative extends base +interface ChartSelectProps extends SelectProps { + charts: Chart[]; + onChartSelect: (chart: Chart) => void; +} + +// ✅ Derivative with modified props +interface DatabaseSelectProps extends Omit { + value: number; // Narrowed type + onChange: (databaseId: number) => void; // Specific signature +} +``` + +**Common Patterns:** +- **Extension:** `extends BaseProps` - adds new props +- **Omission:** `Omit` - removes props +- **Modification:** `Omit & { prop: NewType }` - changes prop type +- **Restriction:** Override with narrower types (union → specific) + +--- + +## 📋 Migration Recipe + +### Step 2: File Conversion +```bash +# Use git mv to preserve history +git mv component.js component.ts +git mv Component.jsx Component.tsx +``` + +### Step 3: Import & Type Setup +```typescript +// Import order (enforced by linting) +import { FC, ReactNode } from 'react'; +import { JsonObject, QueryFormData } from '@superset-ui/core'; +import { Dataset } from '@superset-ui/chart-controls'; +import type { Dashboard } from 'src/types/Dashboard'; +``` + +### Step 4: Function & Component Typing +```typescript +// Functions with proper parameter/return types +export function processData( + data: Dataset[], + config: JsonObject +): ProcessedData[] { + // implementation +} + +// Component props with inheritance +interface ComponentProps extends BaseProps { + data: Chart[]; + onSelect: (id: number) => void; +} + +const Component: FC = ({ data, onSelect }) => { + // implementation +}; +``` + +### Step 5: State & Redux Typing +```typescript +// Hooks with specific types +const [data, setData] = useState([]); +const [selected, setSelected] = useState(null); + +// Redux with existing RootState +const mapStateToProps = (state: RootState) => ({ + charts: state.charts, + user: state.user, +}); +``` + +--- + +## 🧠 Type Debugging Strategies (Real-World Learnings) + +### The Evolution of Type Approaches +When you hit type errors, follow this debugging evolution: + +#### 1. ❌ Idealized Union Types (First Attempt) +```typescript +// Looks clean but doesn't match reality +type DatasourceInput = Datasource | QueryEditor; +``` +**Problem**: Real calling sites pass variations, not exact types. + +#### 2. ❌ Overly Precise Types (Second Attempt) +```typescript +// Tried to match exact calling signatures +type DatasourceInput = + | IDatasource // From DatasourcePanel + | (QueryEditor & { columns: ColumnMeta[] }); // From SaveQuery +``` +**Problem**: Too rigid, doesn't handle legacy variations. + +#### 3. ✅ Flexible Interface (Final Solution) +```typescript +// Captures what the function actually needs +interface DatasourceInput { + name?: string | null; // Allow null for compatibility + datasource_name?: string | null; // Legacy variations + columns?: any[]; // Multiple column types accepted + database?: { id?: number }; + // ... other optional properties +} +``` +**Success**: Works with all calling sites, focuses on function needs. + +### Type Debugging Process +1. **Start with compilation errors** - they show exact mismatches +2. **Examine actual usage** - look at calling sites, not idealized types +3. **Build flexible interfaces** - capture what functions need, not rigid contracts +4. **Iterate based on downstream validation** - let calling sites guide your types + +--- + +## 🚨 Anti-Patterns to Avoid + +```typescript +// ❌ Never use any +const obj: any = {}; + +// ✅ Use proper types +const obj: Record = {}; + +// ❌ Don't recreate base component props +interface ChartSelectProps { + value: string; // Duplicated from SelectProps + onChange: () => void; // Duplicated from SelectProps + charts: Chart[]; // New prop +} + +// ✅ Inherit and extend +interface ChartSelectProps extends SelectProps { + charts: Chart[]; // Only new props +} + +// ❌ Don't create ad-hoc type variations +interface UserInfo { + name: string; + email: string; +} + +// ✅ Extend existing types (DRY principle) +import { User } from 'src/types/bootstrapTypes'; +type UserDisplayInfo = Pick; + +// ❌ Don't create overly rigid unions +type StrictInput = ExactTypeA | ExactTypeB; + +// ✅ Create flexible interfaces for function parameters +interface FlexibleInput { + // Focus on what the function actually needs + commonProperty: string; + optionalVariations?: any; // Allow for legacy variations +} +``` + +## 📍 DRY Type Guidelines (WHERE TYPES BELONG) + +### Type Placement Rules +**CRITICAL**: Type variations must live close to where they belong, not scattered across files. + +#### ✅ Proper Type Organization +```typescript +// ❌ Don't create one-off interfaces in utility files +// src/utils/datasourceUtils.ts +interface DatasourceInput { /* custom interface */ } // Wrong! + +// ✅ Use existing types or extend them in their proper domain +// src/utils/datasourceUtils.ts +import { IDatasource } from 'src/explore/components/DatasourcePanel'; +import { QueryEditor } from 'src/SqlLab/types'; + +// Create flexible interface that references existing types +interface FlexibleDatasourceInput { + // Properties that actually exist across variations +} +``` + +#### Type Location Hierarchy +1. **Domain Types**: `src/{domain}/types.ts` (dashboard, explore, SqlLab) +2. **Component Types**: Co-located with components +3. **Global Types**: `src/types/` directory +4. **Utility Types**: Only when they truly don't belong elsewhere + +#### ✅ DRY Type Patterns +```typescript +// ✅ Extend existing domain types +interface SaveQueryData extends Pick { + columns: ColumnMeta[]; // Add what's needed +} + +// ✅ Create flexible interfaces for cross-domain utilities +interface CrossDomainInput { + // Common properties that exist across different source types + name?: string | null; // Accommodate legacy null values + // Only include properties the function actually uses +} +``` + +--- + +## 🎯 PropTypes Auto-Generation (Elegant Approach) + +**IMPORTANT**: Superset has `babel-plugin-typescript-to-proptypes` configured to automatically generate PropTypes from TypeScript interfaces. Use this instead of manual PropTypes duplication! + +### ❌ Manual PropTypes Duplication (Avoid This) +```typescript +export interface MyComponentProps { + title: string; + count?: number; +} + +// 8+ lines of manual PropTypes duplication 😱 +const propTypes = PropTypes.shape({ + title: PropTypes.string.isRequired, + count: PropTypes.number, +}); + +export default propTypes; +``` + +### ✅ Auto-Generated PropTypes (Use This) +```typescript +import { InferProps } from 'prop-types'; + +export interface MyComponentProps { + title: string; + count?: number; +} + +// Single validator function - babel plugin auto-generates PropTypes! ✨ +export default function MyComponentValidator(props: MyComponentProps) { + return null; // PropTypes auto-assigned by babel-plugin-typescript-to-proptypes +} + +// Optional: For consumers needing PropTypes type inference +export type MyComponentPropsInferred = InferProps; +``` + +### Migration Pattern for Type-Only Files + +**When migrating type-only files with manual PropTypes:** + +1. **Keep the TypeScript interfaces** (single source of truth) +2. **Replace manual PropTypes** with validator function +3. **Remove PropTypes imports** and manual shape definitions +4. **Add InferProps import** if type inference needed + +**Example Migration:** +```typescript +// Before: 25+ lines with manual PropTypes duplication +export interface AdhocFilterType { /* ... */ } +const adhocFilterTypePropTypes = PropTypes.oneOfType([...]); + +// After: 3 lines with auto-generation +export interface AdhocFilterType { /* ... */ } +export default function AdhocFilterValidator(props: { filter: AdhocFilterType }) { + return null; // Auto-generated PropTypes by babel plugin +} +``` + +### Component PropTypes Pattern + +**For React components, the babel plugin works automatically:** + +```typescript +interface ComponentProps { + title: string; + onClick: () => void; +} + +const MyComponent: FC = ({ title, onClick }) => { + // Component implementation +}; + +// PropTypes automatically generated by babel plugin - no manual work needed! +export default MyComponent; +``` + +### Auto-Generation Benefits + +- ✅ **Single source of truth**: TypeScript interfaces drive PropTypes +- ✅ **No duplication**: Eliminate 15-20 lines of manual PropTypes code +- ✅ **Automatic updates**: Changes to TypeScript automatically update PropTypes +- ✅ **Type safety**: Compile-time checking ensures PropTypes match interfaces +- ✅ **Backward compatibility**: Existing JavaScript components continue working + +### Babel Plugin Configuration + +The plugin is already configured in `babel.config.js`: +```javascript +['babel-plugin-typescript-to-proptypes', { loose: true }] +``` + +**No additional setup required** - just use TypeScript interfaces and the plugin handles the rest! + +--- + +## 🧪 Test File Migration Patterns + +### Test File Priority +- **Always migrate test files** alongside production files +- **Test files are often leaf nodes** - good starting candidates +- **Create tests if missing** - Leverage new TypeScript types for better test coverage + +### Test-Specific Type Patterns +```typescript +// Mock interfaces for testing +interface MockStore { + getState: () => Partial; // Partial allows minimal mocking +} + +// Type-safe mocking for complex objects +const mockDashboardInfo: Partial as DashboardInfo = { + id: 123, + json_metadata: '{}', +}; + +// Sinon stub typing +let postStub: sinon.SinonStub; +beforeEach(() => { + postStub = sinon.stub(SupersetClient, 'post'); +}); + +// Use stub reference instead of original method +expect(postStub.callCount).toBe(1); +expect(postStub.getCall(0).args[0].endpoint).toMatch('/api/'); +``` + +### Test Migration Recipe +1. **Migrate production file first** (if both need migration) +2. **Update test imports** to point to `.ts/.tsx` files +3. **Add proper mock typing** using `Partial as T` pattern +4. **Fix stub typing** - Use stub references, not original methods +5. **Verify all tests pass** with TypeScript compilation + +--- + +## 🔧 Type Conflict Resolution + +### Multiple Type Definitions Issue +**Problem**: Same type name defined in multiple files causes compilation errors. + +**Example**: `DashboardInfo` defined in both: +- `src/dashboard/reducers/types.ts` (minimal) +- `src/dashboard/components/Header/types.ts` (different shape) +- `src/dashboard/types.ts` (complete - used by RootState) + +### Resolution Strategy +1. **Identify the authoritative type**: + ```bash + # Find which type is used by RootState/main interfaces + grep -r "DashboardInfo" src/dashboard/types.ts + ``` + +2. **Use import from authoritative source**: + ```typescript + // ✅ Import from main domain types + import { RootState, DashboardInfo } from 'src/dashboard/types'; + + // ❌ Don't import from component-specific files + import { DashboardInfo } from 'src/dashboard/components/Header/types'; + ``` + +3. **Mock complex types in tests**: + ```typescript + // For testing - provide minimal required fields + const mockInfo: Partial as DashboardInfo = { + id: 123, + json_metadata: '{}', + // Only provide fields actually used in test + }; + ``` + +### Type Hierarchy Discovery Commands +```bash +# Find all definitions of a type +grep -r "interface.*TypeName\|type.*TypeName" src/ + +# Find import usage patterns +grep -r "import.*TypeName" src/ + +# Check what RootState uses +grep -A 10 -B 10 "TypeName" src/*/types.ts +``` + +--- + +## Agent Constraints (CRITICAL) + +1. **Use git mv** - Run `git mv file.js file.ts` to preserve git history, but NO `git commit` +2. **NO global import changes** - Don't update imports across codebase +3. **Type files OK** - Can modify existing type files to improve/align types +4. **Single-File TypeScript Validation** (CRITICAL) - tsc has known issues with multi-file compilation: + - **Core Issue**: TypeScript's `tsc` has documented problems validating multiple files simultaneously in complex projects + - **Solution**: ALWAYS validate files one at a time using individual `tsc` calls + - **Command Pattern**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {single-file-path}` + - **Why**: Multi-file validation can produce false positives, miss real errors, and conflict during parallel agent execution +5. **Downstream Impact Validation** (CRITICAL) - Your migration affects calling sites: + - **Find downstream files**: `find superset-frontend/src -name "*.tsx" -o -name "*.ts" | xargs grep -l "your-core-filename" 2>/dev/null || echo "No files found"` + - **Validate each downstream file individually**: `cd superset-frontend && npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}` + - **Fix type mismatches** you introduced in calling sites + - **NEVER ignore downstream errors** - they indicate your types don't match reality +6. **Avoid Project-Wide Validation During Migration**: + - **NEVER use `npm run type`** during parallel agent execution - produces unreliable results + - **Single-file validation is authoritative** - trust individual file checks over project-wide scans +6. **ESLint validation** - Run `npm run eslint -- --fix {file}` for each migrated file to auto-fix formatting/linting issues +6. Zero `any` types - use proper TypeScript types +7. Search existing types before creating new ones +8. Follow patterns from this guide + +--- + +## Success Report Format + +``` +SUCCESS: Atomic Migration of {core-filename} + +## Files Migrated (Atomic Unit) +- Core: {core-filename} → {core-filename.ts/tsx} +- Tests: {list-of-test-files} → {list-of-test-files.ts/tsx} OR "CREATED: {basename}.test.ts" +- Mocks: {list-of-mock-files} → {list-of-mock-files.ts} +- Type files modified: {list-of-type-files} + +## Types Created/Improved +- {TypeName}: {location} ({scope}) - {rationale} +- {ExistingType}: enhanced in {location} - {improvement-description} + +## Documentation Recommendations +- ADD_TO_DIRECTORY: {TypeName} - {reason} +- NO_DOCUMENTATION: {TypeName} - {reason} + +## Quality Validation +- **Single-File TypeScript Validation**: ✅ PASS - Core files individually validated + - Core file: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {core-file}` + - Test files: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {test-file}` (if exists) +- **Downstream Impact Check**: ✅ PASS - Found {N} files importing this module, all validate individually + - Downstream files: {list-of-files-that-import-your-module} + - Individual validation: `npx tscw --noEmit --allowJs --composite false --project tsconfig.json {each-downstream-file}` +- **ESLint validation**: ✅ PASS (using `npm run eslint -- --fix {files}` to auto-fix formatting) +- **Zero any types**: ✅ PASS +- **Local imports resolved**: ✅ PASS +- **Functionality preserved**: ✅ PASS +- **Tests pass** (if test file): ✅ PASS +- **Follow-up action required**: {YES/NO} + +## Validation Strategy Notes +- **Single-file approach used**: Avoided multi-file tsc validation due to known TypeScript compilation issues +- **Project-wide validation skipped**: `npm run type` not used during parallel migration to prevent false positives + +## Migration Learnings +- Type conflicts encountered: {describe any multiple type definitions} +- Mock patterns used: {describe test mocking approaches} +- Import hierarchy decisions: {note authoritative type sources used} +- PropTypes strategy: {AUTO_GENERATED via babel plugin | MANUAL_DUPLICATION_REMOVED | N/A} + +## Improvement Suggestions for Documentation +- AGENT.md enhancement: {suggest additions to migration guide} +- Common pattern identified: {note reusable patterns for future migrations} +``` + +--- + +## Dependency Block Report Format + +``` +DEPENDENCY_BLOCK: Cannot migrate {filename} + +## Blocking Dependencies +- {path}: {type} - {usage} - {priority} + +## Impact Analysis +- Estimated types: {number} +- Expected locations: {list} +- Cross-domain: {YES/NO} + +## Recommended Order +{ordered-list} +``` + +--- + +## 📚 Quick Reference + +**Type Utilities:** +- `Record` - Object with specific key/value types +- `Partial` - All properties optional +- `Pick` - Subset of properties +- `Omit` - Exclude specific properties +- `NonNullable` - Exclude null/undefined + +**Event Types:** +- `MouseEvent` +- `ChangeEvent` +- `FormEvent` + +**React Types:** +- `FC` - Functional component +- `ReactNode` - Any renderable content +- `CSSProperties` - Style objects + +--- + +**Remember:** Every type should add value and clarity. The goal is meaningful type safety that catches bugs and improves developer experience. diff --git a/.claude/projects/js-to-ts/COORDINATOR.md b/.claude/projects/js-to-ts/COORDINATOR.md new file mode 100644 index 00000000000..4d225a26ddb --- /dev/null +++ b/.claude/projects/js-to-ts/COORDINATOR.md @@ -0,0 +1,199 @@ +# JS-to-TS Coordinator Workflow + +**Role:** Strategic migration coordination - select leaf-node files, trigger agents, review results, handle integration, manage dependencies. + +--- + +## 1. Core File Selection Strategy + +**Target ONLY Core Files**: Coordinators identify core files (production code), agents handle related tests/mocks atomically. + +**File Analysis Commands**: +```bash +# Find CORE files with no JS/JSX dependencies (exclude tests/mocks) - SIZE PRIORITIZED +find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | xargs wc -l | sort -n | head -20 + +# Alternative: Get file sizes in lines with paths +find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | while read file; do + lines=$(wc -l < "$file") + echo "$lines $file" +done | sort -n | head -20 + +# Check dependencies for core files only (start with smallest) +for file in ; do + echo "=== $file ($(wc -l < "$file") lines) ===" + grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" || echo "✅ LEAF CANDIDATE" +done + +# Identify heavily imported files (migrate last) +grep -r "from.*utils/common" superset-frontend/src/ | wc -l + +# Quick leaf analysis with size priority +find superset-frontend/src -name "*.js" -o -name "*.jsx" | grep -v "test\|spec\|mock" | head -30 | while read file; do + deps=$(grep -E "from '\.\./.*\.jsx?'|from '\./.*\.jsx?'|from 'src/.*\.jsx?'" "$file" | wc -l) + lines=$(wc -l < "$file") + if [ "$deps" -eq 0 ]; then + echo "✅ LEAF: $lines lines - $file" + fi +done | sort -n +``` + +**Priority Order** (Smallest files first for easier wins): +1. **Small leaf files** (<50 lines) - No JS/JSX imports, quick TypeScript conversion +2. **Medium leaf files** (50-200 lines) - Self-contained utilities and helpers +3. **Small dependency files** (<100 lines) - Import only already-migrated files +4. **Larger components** (200+ lines) - Complex but well-contained functionality +5. **Core foundational files** (utils/common.js, controls.jsx) - migrate last regardless of size + +**Size-First Benefits**: +- Faster completion builds momentum +- Earlier validation of migration patterns +- Easier rollback if issues arise +- Better success rate for agent learning + +**Migration Unit**: Each agent call migrates: +- 1 core file (primary target) +- All related `*.test.js/jsx` files +- All related `*.mock.js` files +- All related `__mocks__/` files + +--- + +## 2. Task Creation & Agent Control + +### Task Triggering +When triggering the `/js-to-ts` command: +- **Task Title**: Use the core filename as the task title (e.g., "DebouncedMessageQueue.js migration", "hostNamesConfig.js migration") +- **Task Description**: Include the full relative path to help agent locate the file +- **Reference**: Point agent to [AGENT.md](./AGENT.md) for technical instructions + +### Post-Processing Workflow +After each agent completes: + +1. **Review Agent Report**: Always read and analyze the complete agent report +2. **Share Summary**: Provide user with key highlights from agent's work: + - Files migrated (core + tests/mocks) + - Types created or improved + - Any validation issues or coordinator actions needed +3. **Quality Assessment**: Evaluate agent's TypeScript implementation against criteria: + - ✅ **Type Usage**: Proper types used, no `any` types + - ✅ **Type Filing**: Types placed in correct hierarchy (component → feature → domain → global) + - ✅ **Side Effects**: No unintended changes to other files + - ✅ **Import Alignment**: Proper .ts/.tsx import extensions +4. **Integration Decision**: + - **COMMIT**: If agent work is complete and high quality + - **FIX & COMMIT**: If minor issues need coordinator fixes + - **ROLLBACK**: If major issues require complete rework +5. **Next Action**: Ask user preference - commit this work or trigger next migration + +--- + +## 3. Integration Decision Framework + +**Automatic Integration** ✅: +- `npm run type` passes without errors +- Agent created clean TypeScript with proper types +- Types appropriately filed in hierarchy + +**Coordinator Integration** (Fix Side-Effects) 🔧: +- `npm run type` fails BUT agent's work is high quality +- Good type usage, proper patterns, well-organized +- Side-effects are manageable TypeScript compilation errors +- **Coordinator Action**: Integrate the change, then fix global compilation issues + +**Rollback Only** ❌: +- Agent introduced `any` types or poor type choices +- Types poorly organized or conflicting with existing patterns +- Fundamental approach issues requiring complete rework + +**Integration Process**: +1. **Review**: Agent already used `git mv` to preserve history +2. **Fix Side-Effects**: Update dependent files with proper import extensions +3. **Resolve Types**: Fix any cascading type issues across codebase +4. **Validate**: Ensure `npm run type` passes after fixes + +--- + +## 4. Common Integration Patterns + +**Common Side-Effects (Expect These)**: +- **Type import conflicts**: Multiple definitions of same type name +- **Mock object typing**: Tests need complete type satisfaction +- **Stub method references**: Use stub vars instead of original methods + +**Coordinator Fixes (Standard Process)**: +1. **Import Resolution**: + ```bash + # Find authoritative type source + grep -r "TypeName" src/*/types.ts + # Import from domain types (src/dashboard/types.ts) not component types + ``` + +2. **Test Mock Completion**: + ```typescript + // Use Partial as T pattern for minimal mocking + const mockDashboard: Partial as DashboardInfo = { + id: 123, + json_metadata: '{}', + }; + ``` + +3. **Stub Reference Fixes**: + ```typescript + // ✅ Use stub variable + expect(postStub.callCount).toBe(1); + // ❌ Don't use original method + expect(SupersetClient.post.callCount).toBe(1); + ``` + +4. **Validation Commands**: + ```bash + npm run type # TypeScript compilation + npm test -- filename # Test functionality + git status # Should show rename, not add/delete + ``` + +--- + +## 5. File Categories for Planning + +### Leaf Files (Start Here) +**Self-contained files with minimal JS/JSX dependencies**: +- Test files (80 files) - Usually only import the file being tested +- Utility files without internal dependencies +- Components importing only external libraries + +### Heavily Imported Files (Migrate Last) +**Core files that many others depend on**: +- `utils/common.js` - Core utility functions +- `utils/reducerUtils.js` - Redux helpers +- `@superset-ui/core` equivalent files +- Major state management files (`explore/store.js`, `dashboard/actions/`) + +### Complex Components (Middle Priority) +**Large files requiring careful type analysis**: +- `components/Datasource/DatasourceEditor.jsx` (1,809 lines) +- `explore/components/controls/AnnotationLayerControl/AnnotationLayer.jsx` (1,031 lines) +- `explore/components/ExploreViewContainer/index.jsx` (911 lines) + +--- + +## 6. Success Metrics & Continuous Improvement + +**Per-File Gates**: +- ✅ `npm run type` passes after each migration +- ✅ Zero `any` types introduced +- ✅ All imports properly typed +- ✅ Types filed in correct hierarchy + +**Linear Scheduling**: +When agents report `DEPENDENCY_BLOCK`: +- Queue dependencies in linear order +- Process one file at a time to avoid conflicts +- Handle cascading type changes between files + +**After Each Migration**: +1. **Update guides** with new patterns discovered +2. **Document coordinator fixes** that become common +3. **Enhance agent instructions** based on recurring issues +4. **Track success metrics** - automatic vs coordinator integration rates diff --git a/.claude/projects/js-to-ts/PROJECT.md b/.claude/projects/js-to-ts/PROJECT.md new file mode 100644 index 00000000000..c8a7b80e62a --- /dev/null +++ b/.claude/projects/js-to-ts/PROJECT.md @@ -0,0 +1,76 @@ +# JavaScript to TypeScript Migration Project + +Progressive migration of 219 JS/JSX files to TypeScript in Apache Superset frontend. + +## 📁 Project Documentation + +- **[AGENT.md](./AGENT.md)** - Complete technical migration guide for agents (includes type reference, patterns, validation) +- **[COORDINATOR.md](./COORDINATOR.md)** - Strategic workflow for coordinators (file selection, task management, integration) + +## 🎯 Quick Start + +**For Agents:** Read [AGENT.md](./AGENT.md) for complete migration instructions +**For Coordinators:** Read [COORDINATOR.md](./COORDINATOR.md) for workflow and [AGENT.md](./AGENT.md) for supervision + +**Command:** `/js-to-ts ` - See [../../commands/js-to-ts.md](../../commands/js-to-ts.md) + +## 📊 Migration Progress + +**Scope**: 219 files total (112 JS + 107 JSX) +- Production files: 139 (63%) +- Test files: 80 (37%) + +**Strategy**: Leaf-first migration with dependency-aware coordination + +### Completed Migrations ✅ + +1. **roundDecimal** - `plugins/legacy-plugin-chart-map-box/src/utils/roundDecimal.js` + - Migrated core + test files + - Added proper TypeScript function signature with optional precision parameter + - All tests pass + +2. **timeGrainSqlaAnimationOverrides** - `src/explore/controlPanels/timeGrainSqlaAnimationOverrides.js` + - Migrated to TypeScript with ControlPanelState and Dataset types + - Added TimeGrainOverrideState interface for return type + - Used type guards for safe property access + +3. **DebouncedMessageQueue** - `src/utils/DebouncedMessageQueue.js` + - Migrated to TypeScript with proper generics + - Created DebouncedMessageQueueOptions interface + - **CREATED test file** with 4 comprehensive test cases + - Excellent class property typing with private/readonly modifiers + +**Files Migrated**: 3/219 (1.4%) +**Tests Created**: 2 (roundDecimal had existing, DebouncedMessageQueue created) + +### Next Candidates (Leaf Nodes) 🎯 + +**Identified leaf files with no JS/JSX dependencies:** +- `src/utils/hostNamesConfig.js` - Domain configuration utility +- `src/explore/controlPanels/Separator.js` - Control panel configuration +- `src/middleware/loggerMiddleware.js` - Logging middleware + +**Migration Quality**: All completed migrations have: +- ✅ Zero `any` types +- ✅ Proper TypeScript compilation +- ✅ ESLint validation passed +- ✅ Test coverage (created where missing) + +--- + +## 📈 Success Metrics + +**Per-File Gates**: +- ✅ `npm run type` passes after each migration +- ✅ Zero `any` types introduced +- ✅ All imports properly typed +- ✅ Types filed in correct hierarchy + +**Overall Progress**: +- **Automatic Integration Rate**: 100% (3/3 migrations required no coordinator fixes) +- **Test Coverage**: Improved (1 new test file created) +- **Type Safety**: Enhanced with proper interfaces and generics + +--- + +*This is a claudette-managed progressive refactor. All documentation and coordination resources are organized under `.claude/projects/js-to-ts/`* diff --git a/.devcontainer/README.md b/.devcontainer/README.md index e5dda78fe30..6b24183edc5 100644 --- a/.devcontainer/README.md +++ b/.devcontainer/README.md @@ -3,14 +3,3 @@ For complete documentation on using GitHub Codespaces with Apache Superset, please see: **[Setting up a Development Environment - GitHub Codespaces](https://superset.apache.org/docs/contributing/development#github-codespaces-cloud-development)** - -## Pre-installed Development Environment - -When you create a new Codespace from this repository, it automatically: - -1. **Creates a Python virtual environment** using `uv venv` -2. **Installs all development dependencies** via `uv pip install -r requirements/development.txt` -3. **Sets up pre-commit hooks** with `pre-commit install` -4. **Activates the virtual environment** automatically in all terminals - -The virtual environment is located at `/workspaces/{repository-name}/.venv` and is automatically activated through environment variables set in the devcontainer configuration. diff --git a/.devcontainer/default/devcontainer.json b/.devcontainer/default/devcontainer.json new file mode 100644 index 00000000000..d0988367947 --- /dev/null +++ b/.devcontainer/default/devcontainer.json @@ -0,0 +1,19 @@ +{ + // Extend the base configuration + "extends": "../devcontainer-base.json", + + "name": "Apache Superset Development (Default)", + + // Forward ports for development + "forwardPorts": [9001], + "portsAttributes": { + "9001": { + "label": "Superset (via Webpack Dev Server)", + "onAutoForward": "notify", + "visibility": "public" + } + }, + + // Auto-start Superset on Codespace resume + "postStartCommand": ".devcontainer/start-superset.sh" +} diff --git a/.devcontainer/devcontainer-base.json b/.devcontainer/devcontainer-base.json new file mode 100644 index 00000000000..59ed6ee1d2f --- /dev/null +++ b/.devcontainer/devcontainer-base.json @@ -0,0 +1,39 @@ +{ + "name": "Apache Superset Development", + // Keep this in sync with the base image in Dockerfile (ARG PY_VER) + // Using the same base as Dockerfile, but non-slim for dev tools + "image": "python:3.11.13-bookworm", + + "features": { + "ghcr.io/devcontainers/features/docker-in-docker:2": { + "moby": true, + "dockerDashComposeVersion": "v2" + }, + "ghcr.io/devcontainers/features/node:1": { + "version": "20" + }, + "ghcr.io/devcontainers/features/git:1": {}, + "ghcr.io/devcontainers/features/common-utils:2": { + "configureZshAsDefaultShell": true + }, + "ghcr.io/devcontainers/features/sshd:1": { + "version": "latest" + } + }, + + // Run commands after container is created + "postCreateCommand": "chmod +x .devcontainer/setup-dev.sh && .devcontainer/setup-dev.sh", + + // VS Code customizations + "customizations": { + "vscode": { + "extensions": [ + "ms-python.python", + "ms-python.vscode-pylance", + "charliermarsh.ruff", + "dbaeumer.vscode-eslint", + "esbenp.prettier-vscode" + ] + } + } +} diff --git a/.devcontainer/setup-dev.sh b/.devcontainer/setup-dev.sh index 91482551bee..f8521189009 100755 --- a/.devcontainer/setup-dev.sh +++ b/.devcontainer/setup-dev.sh @@ -3,76 +3,30 @@ echo "🔧 Setting up Superset development environment..." -# System dependencies and uv are now pre-installed in the Docker image -# This speeds up Codespace creation significantly! +# The universal image has most tools, just need Superset-specific libs +echo "📦 Installing Superset-specific dependencies..." +sudo apt-get update +sudo apt-get install -y \ + libsasl2-dev \ + libldap2-dev \ + libpq-dev \ + tmux \ + gh -# Create virtual environment using uv -echo "🐍 Creating Python virtual environment..." -if ! uv venv; then - echo "❌ Failed to create virtual environment" - exit 1 -fi +# Install uv for fast Python package management +echo "📦 Installing uv..." +curl -LsSf https://astral.sh/uv/install.sh | sh -# Install Python dependencies -echo "📦 Installing Python dependencies..." -if ! uv pip install -r requirements/development.txt; then - echo "❌ Failed to install Python dependencies" - echo "💡 You may need to run this manually after the Codespace starts" - exit 1 -fi - -# Install pre-commit hooks -echo "🪝 Installing pre-commit hooks..." -if source .venv/bin/activate && pre-commit install; then - echo "✅ Pre-commit hooks installed" -else - echo "⚠️ Pre-commit hooks installation failed (non-critical)" -fi +# Add cargo/bin to PATH for uv +echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.bashrc +echo 'export PATH="$HOME/.cargo/bin:$PATH"' >> ~/.zshrc # Install Claude Code CLI via npm echo "🤖 Installing Claude Code..." -if npm install -g @anthropic-ai/claude-code; then - echo "✅ Claude Code installed" -else - echo "⚠️ Claude Code installation failed (non-critical)" -fi +npm install -g @anthropic-ai/claude-code # Make the start script executable chmod +x .devcontainer/start-superset.sh -# Add bashrc additions for automatic venv activation -echo "🔧 Setting up automatic environment activation..." -if [ -f ~/.bashrc ]; then - # Check if we've already added our additions - if ! grep -q "Superset Codespaces environment setup" ~/.bashrc; then - echo "" >> ~/.bashrc - cat .devcontainer/bashrc-additions >> ~/.bashrc - echo "✅ Added automatic venv activation to ~/.bashrc" - else - echo "✅ Bashrc additions already present" - fi -else - # Create bashrc if it doesn't exist - cat .devcontainer/bashrc-additions > ~/.bashrc - echo "✅ Created ~/.bashrc with automatic venv activation" -fi - -# Also add to zshrc since that's the default shell -if [ -f ~/.zshrc ] || [ -n "$ZSH_VERSION" ]; then - if ! grep -q "Superset Codespaces environment setup" ~/.zshrc; then - echo "" >> ~/.zshrc - cat .devcontainer/bashrc-additions >> ~/.zshrc - echo "✅ Added automatic venv activation to ~/.zshrc" - fi -fi - echo "✅ Development environment setup complete!" -echo "" -echo "📝 The virtual environment will be automatically activated in new terminals" -echo "" -echo "🔄 To activate in this terminal, run:" -echo " source ~/.bashrc" -echo "" -echo "🚀 To start Superset:" -echo " start-superset" -echo "" +echo "🚀 Run '.devcontainer/start-superset.sh' to start Superset" diff --git a/.devcontainer/start-superset.sh b/.devcontainer/start-superset.sh index 6ba990cae10..b480b04aacb 100755 --- a/.devcontainer/start-superset.sh +++ b/.devcontainer/start-superset.sh @@ -1,14 +1,14 @@ #!/bin/bash # Startup script for Superset in Codespaces -# Log to a file for debugging -LOG_FILE="/tmp/superset-startup.log" -echo "[$(date)] Starting Superset startup script" >> "$LOG_FILE" -echo "[$(date)] User: $(whoami), PWD: $(pwd)" >> "$LOG_FILE" - echo "🚀 Starting Superset in Codespaces..." echo "🌐 Frontend will be available at port 9001" +# Check if MCP is enabled +if [ "$ENABLE_MCP" = "true" ]; then + echo "🤖 MCP Service will be available at port 5008" +fi + # Find the workspace directory (Codespaces clones as 'superset', not 'superset-2') WORKSPACE_DIR=$(find /workspaces -maxdepth 1 -name "superset*" -type d | head -1) if [ -n "$WORKSPACE_DIR" ]; then @@ -18,71 +18,32 @@ else echo "📁 Using current directory: $(pwd)" fi -# Wait for Docker to be available -echo "⏳ Waiting for Docker to start..." -echo "[$(date)] Waiting for Docker..." >> "$LOG_FILE" -max_attempts=30 -attempt=0 -while ! docker info > /dev/null 2>&1; do - if [ $attempt -eq $max_attempts ]; then - echo "❌ Docker failed to start after $max_attempts attempts" - echo "[$(date)] Docker failed to start after $max_attempts attempts" >> "$LOG_FILE" - echo "🔄 Please restart the Codespace or run this script manually later" - exit 1 - fi - echo " Attempt $((attempt + 1))/$max_attempts..." - echo "[$(date)] Docker check attempt $((attempt + 1))/$max_attempts" >> "$LOG_FILE" - sleep 2 - attempt=$((attempt + 1)) -done -echo "✅ Docker is ready!" -echo "[$(date)] Docker is ready" >> "$LOG_FILE" - -# Check if Superset containers are already running -if docker ps | grep -q "superset"; then - echo "✅ Superset containers are already running!" - echo "" - echo "🌐 To access Superset:" - echo " 1. Click the 'Ports' tab at the bottom of VS Code" - echo " 2. Find port 9001 and click the globe icon to open" - echo " 3. Wait 10-20 minutes for initial startup" - echo "" - echo "📝 Login credentials: admin/admin" - exit 0 +# Check if docker is running +if ! docker info > /dev/null 2>&1; then + echo "⏳ Waiting for Docker to start..." + sleep 5 fi # Clean up any existing containers echo "🧹 Cleaning up existing containers..." -docker-compose -f docker-compose-light.yml down +docker-compose -f docker-compose-light.yml --profile mcp down # Start services -echo "🏗️ Starting Superset in background (daemon mode)..." +echo "🏗️ Building and starting services..." echo "" +echo "📝 Once started, login with:" +echo " Username: admin" +echo " Password: admin" +echo "" +echo "📋 Running in foreground with live logs (Ctrl+C to stop)..." -# Start in detached mode -docker-compose -f docker-compose-light.yml up -d - -echo "" -echo "✅ Docker Compose started successfully!" -echo "" -echo "📋 Important information:" -echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -echo "⏱️ Initial startup takes 10-20 minutes" -echo "🌐 Check the 'Ports' tab for your Superset URL (port 9001)" -echo "👤 Login: admin / admin" -echo "" -echo "📊 Useful commands:" -echo " docker-compose -f docker-compose-light.yml logs -f # Follow logs" -echo " docker-compose -f docker-compose-light.yml ps # Check status" -echo " docker-compose -f docker-compose-light.yml down # Stop services" -echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━" -echo "" -echo "💤 Keeping terminal open for 60 seconds to test persistence..." -sleep 60 -echo "✅ Test complete - check if this terminal is still visible!" - -# Show final status -docker-compose -f docker-compose-light.yml ps +# Run docker-compose and capture exit code +if [ "$ENABLE_MCP" = "true" ]; then + echo "🤖 Starting with MCP Service enabled..." + docker-compose -f docker-compose-light.yml --profile mcp up +else + docker-compose -f docker-compose-light.yml up +fi EXIT_CODE=$? # If it failed, provide helpful instructions diff --git a/.devcontainer/with-mcp/devcontainer.json b/.devcontainer/with-mcp/devcontainer.json new file mode 100644 index 00000000000..c3f8b654ebc --- /dev/null +++ b/.devcontainer/with-mcp/devcontainer.json @@ -0,0 +1,29 @@ +{ + // Extend the base configuration + "extends": "../devcontainer-base.json", + + "name": "Apache Superset Development with MCP", + + // Forward ports for development + "forwardPorts": [9001, 5008], + "portsAttributes": { + "9001": { + "label": "Superset (via Webpack Dev Server)", + "onAutoForward": "notify", + "visibility": "public" + }, + "5008": { + "label": "MCP Service (Model Context Protocol)", + "onAutoForward": "notify", + "visibility": "private" + } + }, + + // Auto-start Superset with MCP on Codespace resume + "postStartCommand": "ENABLE_MCP=true .devcontainer/start-superset.sh", + + // Environment variables + "containerEnv": { + "ENABLE_MCP": "true" + } +} diff --git a/.github/CODEOWNERS b/.github/CODEOWNERS index fc6d3a84395..237907f74d3 100644 --- a/.github/CODEOWNERS +++ b/.github/CODEOWNERS @@ -20,7 +20,7 @@ # Notify PMC members of changes to GitHub Actions -/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @craig-rueda @kgabryje @dpgaspar +/.github/ @villebro @geido @eschutho @rusackas @betodealmeida @nytai @mistercrunch @craig-rueda @kgabryje @dpgaspar @sadpandajoe # Notify PMC members of changes to required GitHub Actions diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 3395d566a13..be77ac83a18 120000 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -1 +1 @@ -../LLMS.md \ No newline at end of file +../AGENTS.md \ No newline at end of file diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 40a5c18be56..ae523f1ac53 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -5,7 +5,7 @@ updates: - package-ecosystem: "github-actions" directory: "/" schedule: - interval: "monthly" + interval: "daily" - package-ecosystem: "npm" ignore: @@ -18,7 +18,7 @@ updates: - dependency-name: "jest-environment-jsdom" directory: "/superset-frontend/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -40,21 +40,21 @@ updates: - package-ecosystem: "npm" directory: ".github/actions" schedule: - interval: "monthly" + interval: "daily" open-pull-requests-limit: 10 versioning-strategy: increase - package-ecosystem: "npm" directory: "/docs/" schedule: - interval: "monthly" + interval: "daily" open-pull-requests-limit: 10 versioning-strategy: increase - package-ecosystem: "npm" directory: "/superset-websocket/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -63,7 +63,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-websocket/utils/client-ws-app/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -75,7 +75,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-calendar/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -85,7 +85,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-histogram/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -95,7 +95,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-partition/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -105,7 +105,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-world-map/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -115,7 +115,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/plugin-chart-pivot-table/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -125,7 +125,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-chord/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -135,7 +135,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-horizon/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -145,7 +145,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-rose/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -155,7 +155,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-preset-chart-deckgl/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -165,7 +165,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/plugin-chart-table/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -175,7 +175,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-country-map/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -185,7 +185,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-map-box/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -195,7 +195,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-sankey/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -205,7 +205,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-preset-chart-nvd3/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -215,7 +215,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/plugin-chart-word-cloud/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -225,7 +225,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-event-flow/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -235,7 +235,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-paired-t-test/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -245,7 +245,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-sankey-loop/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -255,7 +255,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/plugin-chart-echarts/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -265,7 +265,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/preset-chart-xy/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -275,7 +275,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-heatmap/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -285,7 +285,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-parallel-coordinates/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -295,7 +295,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/legacy-plugin-chart-sunburst/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -305,7 +305,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/plugins/plugin-chart-handlebars/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -315,7 +315,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/packages/generator-superset/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -325,7 +325,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/packages/superset-ui-chart-controls/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -339,7 +339,7 @@ updates: - dependency-name: "react-markdown" - dependency-name: "remark-gfm" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -349,7 +349,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/packages/superset-ui-demo/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot @@ -359,7 +359,7 @@ updates: - package-ecosystem: "npm" directory: "/superset-frontend/packages/superset-ui-switchboard/" schedule: - interval: "monthly" + interval: "daily" labels: - npm - dependabot diff --git a/.github/workflows/bashlib.sh b/.github/workflows/bashlib.sh index deb56ab649e..1289d07259e 100644 --- a/.github/workflows/bashlib.sh +++ b/.github/workflows/bashlib.sh @@ -182,6 +182,95 @@ cypress-run-all() { kill $flaskProcessId } +playwright-install() { + cd "$GITHUB_WORKSPACE/superset-frontend" + + say "::group::Install Playwright browsers" + npx playwright install --with-deps chromium + # Create output directories for test results and debugging + mkdir -p playwright-results + mkdir -p test-results + say "::endgroup::" +} + +playwright-run() { + local APP_ROOT=$1 + local TEST_PATH=$2 + + # Start Flask from the project root (same as Cypress) + cd "$GITHUB_WORKSPACE" + local flasklog="${HOME}/flask-playwright.log" + local port=8081 + PLAYWRIGHT_BASE_URL="http://localhost:${port}" + if [ -n "$APP_ROOT" ]; then + export SUPERSET_APP_ROOT=$APP_ROOT + PLAYWRIGHT_BASE_URL=${PLAYWRIGHT_BASE_URL}${APP_ROOT}/ + fi + export PLAYWRIGHT_BASE_URL + + nohup flask run --no-debugger -p $port >"$flasklog" 2>&1 /dev/null || true" EXIT + + # Wait for server to be ready with health check + local timeout=60 + say "Waiting for Flask server to start on port $port..." + while [ $timeout -gt 0 ]; do + if curl -f ${PLAYWRIGHT_BASE_URL}/health >/dev/null 2>&1; then + say "Flask server is ready" + break + fi + sleep 1 + timeout=$((timeout - 1)) + done + + if [ $timeout -eq 0 ]; then + echo "::error::Flask server failed to start within 60 seconds" + echo "::group::Flask startup log" + cat "$flasklog" + echo "::endgroup::" + return 1 + fi + + # Change to frontend directory for Playwright execution + cd "$GITHUB_WORKSPACE/superset-frontend" + + say "::group::Run Playwright tests" + echo "Running Playwright with baseURL: ${PLAYWRIGHT_BASE_URL}" + if [ -n "$TEST_PATH" ]; then + # Check if there are any test files in the specified path + if ! find "playwright/tests/${TEST_PATH}" -name "*.spec.ts" -type f 2>/dev/null | grep -q .; then + echo "No test files found in ${TEST_PATH} - skipping test run" + say "::endgroup::" + kill $flaskProcessId + return 0 + fi + echo "Running tests: ${TEST_PATH}" + # Set INCLUDE_EXPERIMENTAL=true to allow experimental tests to run + export INCLUDE_EXPERIMENTAL=true + npx playwright test "${TEST_PATH}" --output=playwright-results + local status=$? + # Unset to prevent leaking into subsequent commands + unset INCLUDE_EXPERIMENTAL + else + echo "Running all required tests (experimental/ excluded via playwright.config.ts)" + npx playwright test --output=playwright-results + local status=$? + fi + say "::endgroup::" + + # After job is done, print out Flask log for debugging + echo "::group::Flask log for Playwright run" + cat "$flasklog" + echo "::endgroup::" + # make sure the program exits + kill $flaskProcessId + + return $status +} + eyes-storybook-dependencies() { say "::group::install eyes-storyook dependencies" sudo apt-get update -y && sudo apt-get -y install gconf-service ca-certificates libxshmfence-dev fonts-liberation libappindicator3-1 libasound2 libatk-bridge2.0-0 libatk1.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgbm1 libgcc1 libgconf-2-4 libglib2.0-0 libgdk-pixbuf2.0-0 libgtk-3-0 libnspr4 libnss3 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 lsb-release xdg-utils libappindicator1 diff --git a/.github/workflows/bump-python-package.yml b/.github/workflows/bump-python-package.yml index 0be3cc53eeb..0bd6fb606a6 100644 --- a/.github/workflows/bump-python-package.yml +++ b/.github/workflows/bump-python-package.yml @@ -41,7 +41,7 @@ jobs: uses: ./.github/actions/setup-supersetbot/ - name: Set up Python ${{ inputs.python-version }} - uses: actions/setup-python@v5 + uses: actions/setup-python@v6 with: python-version: "3.10" diff --git a/.github/workflows/check_db_migration_confict.yml b/.github/workflows/check_db_migration_confict.yml index bc99b764bd2..228c33cdced 100644 --- a/.github/workflows/check_db_migration_confict.yml +++ b/.github/workflows/check_db_migration_confict.yml @@ -27,7 +27,7 @@ jobs: - name: "Checkout ${{ github.ref }} ( ${{ github.sha }} )" uses: actions/checkout@v5 - name: Check and notify - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{ github.token }} script: | diff --git a/.github/workflows/claude.yml b/.github/workflows/claude.yml index 7949aebaba4..fb1efc07ac7 100644 --- a/.github/workflows/claude.yml +++ b/.github/workflows/claude.yml @@ -44,7 +44,7 @@ jobs: pull-requests: write steps: - name: Comment access denied - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: script: | const message = `👋 Hi @${{ github.event.comment.user.login || github.event.review.user.login || github.event.issue.user.login }}! diff --git a/.github/workflows/codeql-analysis.yml b/.github/workflows/codeql-analysis.yml index 363a7cde447..aecda9b612c 100644 --- a/.github/workflows/codeql-analysis.yml +++ b/.github/workflows/codeql-analysis.yml @@ -41,7 +41,7 @@ jobs: # Initializes the CodeQL tools for scanning. - name: Initialize CodeQL - uses: github/codeql-action/init@v3 + uses: github/codeql-action/init@v4 with: languages: ${{ matrix.language }} # If you wish to specify custom queries, you can do so here or in a config file. @@ -53,6 +53,6 @@ jobs: - name: Perform CodeQL Analysis if: steps.check.outputs.python || steps.check.outputs.frontend - uses: github/codeql-action/analyze@v3 + uses: github/codeql-action/analyze@v4 with: category: "/language:${{matrix.language}}" diff --git a/.github/workflows/embedded-sdk-release.yml b/.github/workflows/embedded-sdk-release.yml index d357de5be07..82425e706b3 100644 --- a/.github/workflows/embedded-sdk-release.yml +++ b/.github/workflows/embedded-sdk-release.yml @@ -29,7 +29,7 @@ jobs: working-directory: superset-embedded-sdk steps: - uses: actions/checkout@v5 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v5 with: node-version-file: './superset-embedded-sdk/.nvmrc' registry-url: 'https://registry.npmjs.org' diff --git a/.github/workflows/embedded-sdk-test.yml b/.github/workflows/embedded-sdk-test.yml index a5309f7b6da..4483656446b 100644 --- a/.github/workflows/embedded-sdk-test.yml +++ b/.github/workflows/embedded-sdk-test.yml @@ -19,7 +19,7 @@ jobs: working-directory: superset-embedded-sdk steps: - uses: actions/checkout@v5 - - uses: actions/setup-node@v4 + - uses: actions/setup-node@v5 with: node-version-file: './superset-embedded-sdk/.nvmrc' registry-url: 'https://registry.npmjs.org' diff --git a/.github/workflows/ephemeral-env-pr-close.yml b/.github/workflows/ephemeral-env-pr-close.yml index fe65c6fa64a..98a62575d07 100644 --- a/.github/workflows/ephemeral-env-pr-close.yml +++ b/.github/workflows/ephemeral-env-pr-close.yml @@ -33,7 +33,7 @@ jobs: pull-requests: write steps: - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v4 + uses: aws-actions/configure-aws-credentials@v5 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} @@ -69,7 +69,7 @@ jobs: - name: Comment (success) if: steps.describe-services.outputs.active == 'true' - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{github.token}} script: | diff --git a/.github/workflows/ephemeral-env.yml b/.github/workflows/ephemeral-env.yml index d34cf46608b..75583baf61f 100644 --- a/.github/workflows/ephemeral-env.yml +++ b/.github/workflows/ephemeral-env.yml @@ -63,7 +63,7 @@ jobs: - name: Get event SHA id: get-sha if: steps.eval-label.outputs.result == 'up' - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{ secrets.GITHUB_TOKEN }} script: | @@ -94,7 +94,7 @@ jobs: core.setOutput("sha", prSha); - name: Looking for feature flags in PR description - uses: actions/github-script@v7 + uses: actions/github-script@v8 id: eval-feature-flags if: steps.eval-label.outputs.result == 'up' with: @@ -116,7 +116,7 @@ jobs: return results; - name: Reply with confirmation comment - uses: actions/github-script@v7 + uses: actions/github-script@v8 if: steps.eval-label.outputs.result == 'up' with: github-token: ${{ secrets.GITHUB_TOKEN }} @@ -189,7 +189,7 @@ jobs: --extra-flags "--build-arg INCLUDE_CHROMIUM=false" - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v4 + uses: aws-actions/configure-aws-credentials@v5 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} @@ -225,7 +225,7 @@ jobs: persist-credentials: false - name: Configure AWS credentials - uses: aws-actions/configure-aws-credentials@v4 + uses: aws-actions/configure-aws-credentials@v5 with: aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }} aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }} @@ -248,7 +248,7 @@ jobs: - name: Fail on missing container image if: steps.check-image.outcome == 'failure' - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{ github.token }} script: | @@ -318,7 +318,7 @@ jobs: echo "ip=$(aws ec2 describe-network-interfaces --network-interface-ids ${{ steps.get-eni.outputs.eni }} | jq -r '.NetworkInterfaces | first | .Association.PublicIp')" >> $GITHUB_OUTPUT - name: Comment (success) if: ${{ success() }} - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{github.token}} script: | @@ -331,7 +331,7 @@ jobs: }); - name: Comment (failure) if: ${{ failure() }} - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{github.token}} script: | diff --git a/.github/workflows/github-action-validator.yml b/.github/workflows/github-action-validator.yml index 28c5054f127..12587514df8 100644 --- a/.github/workflows/github-action-validator.yml +++ b/.github/workflows/github-action-validator.yml @@ -17,7 +17,7 @@ jobs: uses: actions/checkout@v5 - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version: '20' diff --git a/.github/workflows/labeler.yml b/.github/workflows/labeler.yml index c29387bcc85..3b85cecc030 100644 --- a/.github/workflows/labeler.yml +++ b/.github/workflows/labeler.yml @@ -9,7 +9,7 @@ jobs: pull-requests: write runs-on: ubuntu-24.04 steps: - - uses: actions/labeler@v5 + - uses: actions/labeler@v6 with: sync-labels: true diff --git a/.github/workflows/no-hold-label.yml b/.github/workflows/no-hold-label.yml index f5d739bad56..08df98eb4c9 100644 --- a/.github/workflows/no-hold-label.yml +++ b/.github/workflows/no-hold-label.yml @@ -14,7 +14,7 @@ jobs: runs-on: ubuntu-24.04 steps: - name: Check for 'hold' label - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: github-token: ${{secrets.GITHUB_TOKEN}} script: | diff --git a/.github/workflows/pre-commit.yml b/.github/workflows/pre-commit.yml index 05fc96da34e..d2095e8aec4 100644 --- a/.github/workflows/pre-commit.yml +++ b/.github/workflows/pre-commit.yml @@ -39,7 +39,7 @@ jobs: echo "HOMEBREW_REPOSITORY=$HOMEBREW_REPOSITORY" >>"${GITHUB_ENV}" brew install norwoodj/tap/helm-docs - name: Setup Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version: '20' diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml index 69e387db425..b01d6c80858 100644 --- a/.github/workflows/release.yml +++ b/.github/workflows/release.yml @@ -42,7 +42,7 @@ jobs: - name: Install Node.js if: env.HAS_TAGS - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' diff --git a/.github/workflows/showtime-trigger.yml b/.github/workflows/showtime-trigger.yml index 1bff3e92f9d..d7c3393b000 100644 --- a/.github/workflows/showtime-trigger.yml +++ b/.github/workflows/showtime-trigger.yml @@ -37,7 +37,7 @@ jobs: steps: - name: Security Check - Authorize Maintainers Only id: auth - uses: actions/github-script@v7 + uses: actions/github-script@v8 env: GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} with: @@ -61,17 +61,8 @@ jobs: console.log(`📊 Permission level for ${actor}: ${permission.permission}`); const authorized = ['write', 'admin'].includes(permission.permission); - if (!authorized) { - console.log(`🚨 Unauthorized user ${actor} - skipping all operations`); - core.setOutput('authorized', 'false'); - return; - } - - console.log(`✅ Authorized maintainer: ${actor}`); - core.setOutput('authorized', 'true'); - - // If this is a synchronize event, check if Showtime is active and set blocked label - if (context.eventName === 'pull_request_target' && context.payload.action === 'synchronize') { + // If this is a synchronize event from unauthorized user, check if Showtime is active and set blocked label + if (!authorized && context.eventName === 'pull_request_target' && context.payload.action === 'synchronize') { console.log(`🔒 Synchronize event detected - checking if Showtime is active`); // Check if PR has any circus tent labels (Showtime is in use) @@ -99,6 +90,15 @@ jobs: } } + if (!authorized) { + console.log(`🚨 Unauthorized user ${actor} - skipping all operations`); + core.setOutput('authorized', 'false'); + return; + } + + console.log(`✅ Authorized maintainer: ${actor}`); + core.setOutput('authorized', 'true'); + - name: Install Superset Showtime if: steps.auth.outputs.authorized == 'true' run: | diff --git a/.github/workflows/superset-applitool-cypress.yml b/.github/workflows/superset-applitool-cypress.yml index 071772f14a6..7aec63d9940 100644 --- a/.github/workflows/superset-applitool-cypress.yml +++ b/.github/workflows/superset-applitool-cypress.yml @@ -63,7 +63,7 @@ jobs: with: run: testdata - name: Setup Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' - name: Install npm dependencies diff --git a/.github/workflows/superset-applitools-storybook.yml b/.github/workflows/superset-applitools-storybook.yml index 1af39c115bf..cbf432569f5 100644 --- a/.github/workflows/superset-applitools-storybook.yml +++ b/.github/workflows/superset-applitools-storybook.yml @@ -36,7 +36,7 @@ jobs: submodules: recursive ref: master - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' - name: Install eyes-storybook dependencies diff --git a/.github/workflows/superset-docs-deploy.yml b/.github/workflows/superset-docs-deploy.yml index 148c0472985..55d6fe7232d 100644 --- a/.github/workflows/superset-docs-deploy.yml +++ b/.github/workflows/superset-docs-deploy.yml @@ -36,7 +36,7 @@ jobs: persist-credentials: false submodules: recursive - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './docs/.nvmrc' - name: Setup Python diff --git a/.github/workflows/superset-docs-verify.yml b/.github/workflows/superset-docs-verify.yml index 1bb81d55b55..72e8d049569 100644 --- a/.github/workflows/superset-docs-verify.yml +++ b/.github/workflows/superset-docs-verify.yml @@ -21,12 +21,14 @@ jobs: - uses: actions/checkout@v5 # Do not bump this linkinator-action version without opening # an ASF Infra ticket to allow the new version first! - - uses: JustinBeckwith/linkinator-action@v1.11.0 + - uses: JustinBeckwith/linkinator-action@3d5ba091319fa7b0ac14703761eebb7d100e6f6d # v1.11.0 continue-on-error: true # This will make the job advisory (non-blocking, no red X) with: - paths: "**/*.md, **/*.mdx, !superset-frontend/CHANGELOG.md" + paths: "**/*.md, **/*.mdx" linksToSkip: >- - ^https://github.com/apache/(superset|incubator-superset)/(pull|issue)/\d+, + ^https://github.com/apache/(superset|incubator-superset)/(pull|issues)/\d+, + ^https://github.com/apache/(superset|incubator-superset)/commit/[a-f0-9]+, + superset-frontend/.*CHANGELOG\.md, http://localhost:8088/, http://127.0.0.1:3000/, http://localhost:9001/, @@ -41,12 +43,12 @@ jobs: http://theiconic.com.au/, https://dev.mysql.com/doc/refman/5.7/en/innodb-limits.html, ^https://img\.shields\.io/.*, - https://vkusvill.ru/ - https://www.linkedin.com/in/mark-thomas-b16751158/ - https://theiconic.com.au/ - https://wattbewerb.de/ - https://timbr.ai/ - https://opensource.org/license/apache-2-0 + https://vkusvill.ru/, + https://www.linkedin.com/in/mark-thomas-b16751158/, + https://theiconic.com.au/, + https://wattbewerb.de/, + https://timbr.ai/, + https://opensource.org/license/apache-2-0, https://www.plaidcloud.com/ build-deploy: name: Build & Deploy @@ -61,7 +63,7 @@ jobs: persist-credentials: false submodules: recursive - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './docs/.nvmrc' - name: yarn install diff --git a/.github/workflows/superset-e2e.yml b/.github/workflows/superset-e2e.yml index cc1564b4f42..1e074980498 100644 --- a/.github/workflows/superset-e2e.yml +++ b/.github/workflows/superset-e2e.yml @@ -109,7 +109,7 @@ jobs: run: testdata - name: Setup Node.js if: steps.check.outputs.python || steps.check.outputs.frontend - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' - name: Install npm dependencies @@ -151,3 +151,118 @@ jobs: with: path: ${{ github.workspace }}/superset-frontend/cypress-base/cypress/screenshots name: cypress-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}-${{ matrix.parallel_id }}--${{ steps.set-safe-app-root.outputs.safe_app_root }} + + playwright-tests: + runs-on: ubuntu-22.04 + permissions: + contents: read + pull-requests: read + strategy: + fail-fast: false + matrix: + browser: ["chromium"] + app_root: ["", "/app/prefix"] + env: + SUPERSET_ENV: development + SUPERSET_CONFIG: tests.integration_tests.superset_test_config + SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset + PYTHONPATH: ${{ github.workspace }} + REDIS_PORT: 16379 + GITHUB_TOKEN: ${{ github.token }} + services: + postgres: + image: postgres:16-alpine + env: + POSTGRES_USER: superset + POSTGRES_PASSWORD: superset + ports: + - 15432:5432 + redis: + image: redis:7-alpine + ports: + - 16379:6379 + steps: + # ------------------------------------------------------- + # Conditional checkout based on context (same as Cypress workflow) + - name: Checkout for push or pull_request event + if: github.event_name == 'push' || github.event_name == 'pull_request' + uses: actions/checkout@v5 + with: + persist-credentials: false + submodules: recursive + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} + - name: Checkout using ref (workflow_dispatch) + if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != '' + uses: actions/checkout@v5 + with: + persist-credentials: false + ref: ${{ github.event.inputs.ref }} + submodules: recursive + - name: Checkout using PR ID (workflow_dispatch) + if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != '' + uses: actions/checkout@v5 + with: + persist-credentials: false + ref: refs/pull/${{ github.event.inputs.pr_id }}/merge + submodules: recursive + # ------------------------------------------------------- + - name: Check for file changes + id: check + uses: ./.github/actions/change-detector/ + with: + token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Python + uses: ./.github/actions/setup-backend/ + if: steps.check.outputs.python || steps.check.outputs.frontend + - name: Setup postgres + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: setup-postgres + - name: Import test data + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: testdata + - name: Setup Node.js + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: actions/setup-node@v5 + with: + node-version-file: './superset-frontend/.nvmrc' + - name: Install npm dependencies + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: npm-install + - name: Build javascript packages + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: build-instrumented-assets + - name: Install Playwright + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: playwright-install + - name: Run Playwright (Required Tests) + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + env: + NODE_OPTIONS: "--max-old-space-size=4096" + with: + run: playwright-run "${{ matrix.app_root }}" + - name: Set safe app root + if: failure() + id: set-safe-app-root + run: | + APP_ROOT="${{ matrix.app_root }}" + SAFE_APP_ROOT=${APP_ROOT//\//_} + echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT + - name: Upload Playwright Artifacts + uses: actions/upload-artifact@v4 + if: failure() + with: + path: | + ${{ github.workspace }}/superset-frontend/playwright-results/ + ${{ github.workspace }}/superset-frontend/test-results/ + name: playwright-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }} diff --git a/.github/workflows/superset-frontend.yml b/.github/workflows/superset-frontend.yml index 308d1f4d6bd..55ee117cea6 100644 --- a/.github/workflows/superset-frontend.yml +++ b/.github/workflows/superset-frontend.yml @@ -135,15 +135,15 @@ jobs: run: | docker load < docker-image.tar.gz - - name: eslint + - name: lint run: | docker run --rm $TAG bash -c \ - "npm i && npm run eslint -- . --quiet" + "npm i && npm run lint" - name: tsc run: | docker run --rm $TAG bash -c \ - "npm run type" + "npm i && npm run plugins:build && npm run type" validate-frontend: needs: frontend-build diff --git a/.github/workflows/superset-helm-release.yml b/.github/workflows/superset-helm-release.yml index ef89c37879d..b4551097db3 100644 --- a/.github/workflows/superset-helm-release.yml +++ b/.github/workflows/superset-helm-release.yml @@ -101,7 +101,7 @@ jobs: CR_RELEASE_NAME_TEMPLATE: "superset-helm-chart-{{ .Version }}" - name: Open Pull Request - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: script: | const branchName = '${{ env.branch_name }}'; diff --git a/.github/workflows/superset-playwright.yml b/.github/workflows/superset-playwright.yml new file mode 100644 index 00000000000..f9ca2c09aa9 --- /dev/null +++ b/.github/workflows/superset-playwright.yml @@ -0,0 +1,142 @@ +name: Playwright Experimental Tests + +on: + push: + branches: + - "master" + - "[0-9].[0-9]*" + pull_request: + types: [synchronize, opened, reopened, ready_for_review] + workflow_dispatch: + inputs: + ref: + description: 'The branch or tag to checkout' + required: false + default: '' + pr_id: + description: 'The pull request ID to checkout' + required: false + default: '' + +concurrency: + group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }} + cancel-in-progress: true + +jobs: + # NOTE: Required Playwright tests are in superset-e2e.yml (E2E / playwright-tests) + # This workflow contains only experimental tests that run in shadow mode + playwright-tests-experimental: + runs-on: ubuntu-22.04 + continue-on-error: true + permissions: + contents: read + pull-requests: read + strategy: + fail-fast: false + matrix: + browser: ["chromium"] + app_root: ["", "/app/prefix"] + env: + SUPERSET_ENV: development + SUPERSET_CONFIG: tests.integration_tests.superset_test_config + SUPERSET__SQLALCHEMY_DATABASE_URI: postgresql+psycopg2://superset:superset@127.0.0.1:15432/superset + PYTHONPATH: ${{ github.workspace }} + REDIS_PORT: 16379 + GITHUB_TOKEN: ${{ github.token }} + services: + postgres: + image: postgres:16-alpine + env: + POSTGRES_USER: superset + POSTGRES_PASSWORD: superset + ports: + - 15432:5432 + redis: + image: redis:7-alpine + ports: + - 16379:6379 + steps: + # ------------------------------------------------------- + # Conditional checkout based on context (same as Cypress workflow) + - name: Checkout for push or pull_request event + if: github.event_name == 'push' || github.event_name == 'pull_request' + uses: actions/checkout@v5 + with: + persist-credentials: false + submodules: recursive + ref: ${{ github.event_name == 'pull_request' && github.event.pull_request.head.sha || github.sha }} + - name: Checkout using ref (workflow_dispatch) + if: github.event_name == 'workflow_dispatch' && github.event.inputs.ref != '' + uses: actions/checkout@v5 + with: + persist-credentials: false + ref: ${{ github.event.inputs.ref }} + submodules: recursive + - name: Checkout using PR ID (workflow_dispatch) + if: github.event_name == 'workflow_dispatch' && github.event.inputs.pr_id != '' + uses: actions/checkout@v5 + with: + persist-credentials: false + ref: refs/pull/${{ github.event.inputs.pr_id }}/merge + submodules: recursive + # ------------------------------------------------------- + - name: Check for file changes + id: check + uses: ./.github/actions/change-detector/ + with: + token: ${{ secrets.GITHUB_TOKEN }} + - name: Setup Python + uses: ./.github/actions/setup-backend/ + if: steps.check.outputs.python || steps.check.outputs.frontend + - name: Setup postgres + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: setup-postgres + - name: Import test data + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: testdata + - name: Setup Node.js + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: actions/setup-node@v5 + with: + node-version-file: './superset-frontend/.nvmrc' + - name: Install npm dependencies + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: npm-install + - name: Build javascript packages + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: build-instrumented-assets + - name: Install Playwright + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + with: + run: playwright-install + - name: Run Playwright (Experimental Tests) + if: steps.check.outputs.python || steps.check.outputs.frontend + uses: ./.github/actions/cached-dependencies + env: + NODE_OPTIONS: "--max-old-space-size=4096" + with: + run: playwright-run "${{ matrix.app_root }}" experimental/ + - name: Set safe app root + if: failure() + id: set-safe-app-root + run: | + APP_ROOT="${{ matrix.app_root }}" + SAFE_APP_ROOT=${APP_ROOT//\//_} + echo "safe_app_root=$SAFE_APP_ROOT" >> $GITHUB_OUTPUT + - name: Upload Playwright Artifacts + uses: actions/upload-artifact@v4 + if: failure() + with: + path: | + ${{ github.workspace }}/superset-frontend/playwright-results/ + ${{ github.workspace }}/superset-frontend/test-results/ + name: playwright-experimental-artifact-${{ github.run_id }}-${{ github.job }}-${{ matrix.browser }}--${{ steps.set-safe-app-root.outputs.safe_app_root }} diff --git a/.github/workflows/superset-translations.yml b/.github/workflows/superset-translations.yml index 731365d3135..42a44397408 100644 --- a/.github/workflows/superset-translations.yml +++ b/.github/workflows/superset-translations.yml @@ -31,7 +31,7 @@ jobs: - name: Setup Node.js if: steps.check.outputs.frontend - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' - name: Install dependencies diff --git a/.github/workflows/supersetbot.yml b/.github/workflows/supersetbot.yml index 84076bb62f3..b6addbaa422 100644 --- a/.github/workflows/supersetbot.yml +++ b/.github/workflows/supersetbot.yml @@ -26,7 +26,7 @@ jobs: steps: - name: Quickly add thumbs up! if: github.event_name == 'issue_comment' && contains(github.event.comment.body, '@supersetbot') - uses: actions/github-script@v7 + uses: actions/github-script@v8 with: script: | const [owner, repo] = process.env.GITHUB_REPOSITORY.split('/') diff --git a/.github/workflows/tag-release.yml b/.github/workflows/tag-release.yml index a62f90b38ed..0d5be8137c8 100644 --- a/.github/workflows/tag-release.yml +++ b/.github/workflows/tag-release.yml @@ -60,7 +60,7 @@ jobs: build: "true" - name: Use Node.js 20 - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version: 20 @@ -112,7 +112,7 @@ jobs: fetch-depth: 0 - name: Use Node.js 20 - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version: 20 diff --git a/.github/workflows/tech-debt.yml b/.github/workflows/tech-debt.yml index 04dc907cf5d..c051d9d4ef2 100644 --- a/.github/workflows/tech-debt.yml +++ b/.github/workflows/tech-debt.yml @@ -30,7 +30,7 @@ jobs: uses: actions/checkout@v5 - name: Set up Node.js - uses: actions/setup-node@v4 + uses: actions/setup-node@v5 with: node-version-file: './superset-frontend/.nvmrc' diff --git a/.gitignore b/.gitignore index 9c9fc39d173..e43658f39a1 100644 --- a/.gitignore +++ b/.gitignore @@ -33,6 +33,7 @@ cover .env .envrc .idea +.roo .mypy_cache .python-version .tox @@ -121,6 +122,8 @@ docker/requirements-local.txt cache/ docker/*local* +docker/superset-websocket/config.json +docker-compose.override.yml .temp_cache @@ -134,3 +137,4 @@ PROJECT.md .aider* .claude_rc* .env.local +oxc-custom-build/ diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 1a2a799bb9e..4069fc6ebaf 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -60,15 +60,29 @@ repos: args: ["--markdown-linebreak-ext=md"] - repo: local hooks: - - id: eslint-frontend - name: eslint (frontend) - entry: ./scripts/eslint.sh + - id: prettier-frontend + name: prettier (frontend) + entry: bash -c 'cd superset-frontend && for file in "$@"; do npx prettier --write "${file#superset-frontend/}"; done' + language: system + pass_filenames: true + files: ^superset-frontend/.*\.(js|jsx|ts|tsx|css|scss|sass|json)$ + - repo: local + hooks: + - id: oxlint-frontend + name: oxlint (frontend) + entry: ./scripts/oxlint.sh + language: system + pass_filenames: true + files: ^superset-frontend/.*\.(js|jsx|ts|tsx)$ + - id: custom-rules-frontend + name: custom rules (frontend) + entry: ./scripts/check-custom-rules.sh language: system pass_filenames: true files: ^superset-frontend/.*\.(js|jsx|ts|tsx)$ - id: eslint-docs name: eslint (docs) - entry: bash -c 'cd docs && FILES=$(echo "$@" | sed "s|docs/||g") && yarn eslint --fix --ext .js,.jsx,.ts,.tsx --quiet $FILES' + entry: bash -c 'cd docs && FILES=$(echo "$@" | sed "s|docs/||g") && yarn eslint --fix --quiet $FILES' language: system pass_filenames: true files: ^docs/.*\.(js|jsx|ts|tsx)$ @@ -110,9 +124,12 @@ repos: - -c - | TARGET_BRANCH=${GITHUB_BASE_REF:-master} - git fetch origin "$TARGET_BRANCH" - BASE=$(git merge-base origin/"$TARGET_BRANCH" HEAD) - files=$(git diff --name-only --diff-filter=ACM "$BASE"..HEAD | grep '^superset/.*\.py$' || true) + # Only fetch if we're not in CI (CI already has all refs) + if [ -z "$CI" ]; then + git fetch --no-recurse-submodules origin "$TARGET_BRANCH" 2>/dev/null || true + fi + BASE=$(git merge-base origin/"$TARGET_BRANCH" HEAD 2>/dev/null) || BASE="HEAD" + files=$(git diff --name-only --diff-filter=ACM "$BASE"..HEAD 2>/dev/null | grep '^superset/.*\.py$' || true) if [ -n "$files" ]; then pylint --rcfile=.pylintrc --load-plugins=superset.extensions.pylint --reports=no $files else diff --git a/.rat-excludes b/.rat-excludes index 6d37fdd7a91..5cdbec0acf1 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -11,6 +11,7 @@ .nvmrc .prettierrc .rat-excludes +.swcrc .*log .*pyc .*lock @@ -82,6 +83,7 @@ intro_header.txt # for LLMs llm-context.md +AGENTS.md LLMS.md CLAUDE.md CURSOR.md diff --git a/LLMS.md b/AGENTS.md similarity index 81% rename from LLMS.md rename to AGENTS.md index ad3fbf22741..1b676b0157f 100644 --- a/LLMS.md +++ b/AGENTS.md @@ -15,8 +15,9 @@ Apache Superset is a data visualization platform with Flask/Python backend and R ### Testing Strategy Migration - **Prefer unit tests** over integration tests -- **Prefer integration tests** over Cypress end-to-end tests -- **Cypress is last resort** - Actively moving away from Cypress +- **Prefer integration tests** over end-to-end tests +- **Use Playwright for E2E tests** - Migrating from Cypress +- **Cypress is deprecated** - Will be removed once migration is completed - **Use Jest + React Testing Library** for component testing - **Use `test()` instead of `describe()`** - Follow [avoid nesting when testing](https://kentcdodds.com/blog/avoid-nesting-when-youre-testing) principles @@ -67,7 +68,11 @@ superset/ ### Apache License Headers - **New files require ASF license headers** - When creating new code files, include the standard Apache Software Foundation license header -- **LLM instruction files are excluded** - Files like LLMS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead +- **LLM instruction files are excluded** - Files like AGENTS.md, CLAUDE.md, etc. are in `.rat-excludes` to avoid header token overhead + +### Code Comments +- **Avoid time-specific language** - Don't use words like "now", "currently", "today" in code comments as they become outdated +- **Write timeless comments** - Comments should remain accurate regardless of when they're read ## Documentation Requirements @@ -107,6 +112,18 @@ superset/ npm run test # All tests npm run test -- filename.test.tsx # Single file +# E2E Tests (Playwright - NEW) +npm run playwright:test # All Playwright tests +npm run playwright:ui # Interactive UI mode +npm run playwright:headed # See browser during tests +npx playwright test tests/auth/login.spec.ts # Single file +npm run playwright:debug tests/auth/login.spec.ts # Debug specific file + +# E2E Tests (Cypress - DEPRECATED) +cd superset-frontend/cypress-base +npm run cypress-run-chrome # All Cypress tests (headless) +npm run cypress-debug # Interactive Cypress UI + # Backend pytest # All tests pytest tests/unit_tests/specific_test.py # Single file @@ -136,6 +153,19 @@ curl -f http://localhost:8088/health || echo "❌ Setup required - see https://s - **Use negation operator**: `~Model.field` instead of `== False` to avoid ruff E712 errors - **Example**: `~Model.is_active` instead of `Model.is_active == False` +## Pull Request Guidelines + +**When creating pull requests:** + +1. **Read the current PR template**: Always check `.github/PULL_REQUEST_TEMPLATE.md` for the latest format +2. **Use the template sections**: Include all sections from the template (SUMMARY, BEFORE/AFTER, TESTING INSTRUCTIONS, ADDITIONAL INFORMATION) +3. **Follow PR title conventions**: Use [Conventional Commits](https://www.conventionalcommits.org/en/v1.0.0/) + - Format: `type(scope): description` + - Example: `fix(dashboard): load charts correctly` + - Types: `fix`, `feat`, `docs`, `style`, `refactor`, `perf`, `test`, `chore` + +**Important**: Always reference the actual template file at `.github/PULL_REQUEST_TEMPLATE.md` instead of using cached content, as the template may be updated over time. + ## Pre-commit Validation **Use pre-commit hooks for quality validation:** diff --git a/CLAUDE.md b/CLAUDE.md index f811a97b81a..47dc3e3d863 120000 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1 +1 @@ -LLMS.md \ No newline at end of file +AGENTS.md \ No newline at end of file diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 06ed3b146c5..0b42658b5de 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -5,7 +5,7 @@ regarding copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance - with the License. You may obtain a copy of the License at + with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 @@ -16,9 +16,23 @@ specific language governing permissions and limitations under the License. --> +# Contributing to Apache Superset + Contributions are welcome and are greatly appreciated! Every little bit helps, and credit will always be given. -All matters related to contributions have moved to [this section of -the official Superset documentation](https://superset.apache.org/docs/contributing/). Source for the documentation is -[located here](https://github.com/apache/superset/tree/master/docs/docs). +## Developer Portal + +All developer and contribution documentation has moved to the Apache Superset Developer Portal: + +**[📚 View the Developer Portal →](https://superset.apache.org/developer_portal/)** + +The Developer Portal includes comprehensive guides for: +- [Contributing Overview](https://superset.apache.org/developer_portal/contributing/overview) +- [Development Setup](https://superset.apache.org/developer_portal/contributing/development-setup) +- [Submitting Pull Requests](https://superset.apache.org/developer_portal/contributing/submitting-pr) +- [Contribution Guidelines](https://superset.apache.org/developer_portal/contributing/guidelines) +- [Code Review Process](https://superset.apache.org/developer_portal/contributing/code-review) +- [Development How-tos](https://superset.apache.org/developer_portal/contributing/howtos) + +Source for the Developer Portal documentation is [located here](https://github.com/apache/superset/tree/master/docs/developer_portal). diff --git a/Dockerfile b/Dockerfile index 1b689c2319c..b496164069a 100644 --- a/Dockerfile +++ b/Dockerfile @@ -18,7 +18,7 @@ ###################################################################### # Node stage to deal with static asset construction ###################################################################### -ARG PY_VER=3.11.13-slim-trixie +ARG PY_VER=3.11.14-slim-trixie # If BUILDPLATFORM is null, set it to 'amd64' (or leave as is otherwise). ARG BUILDPLATFORM=${BUILDPLATFORM:-amd64} @@ -26,6 +26,9 @@ ARG BUILDPLATFORM=${BUILDPLATFORM:-amd64} # Include translations in the final build ARG BUILD_TRANSLATIONS="false" +# Build arg to pre-populate examples DuckDB file +ARG LOAD_EXAMPLES_DUCKDB="false" + ###################################################################### # superset-node-ci used as a base for building frontend assets and CI ###################################################################### @@ -143,8 +146,8 @@ RUN if [ "${BUILD_TRANSLATIONS}" = "true" ]; then \ ###################################################################### FROM python-base AS python-common -# Build arg to pre-populate examples DuckDB file -ARG LOAD_EXAMPLES_DUCKDB="false" +# Re-declare build arg to receive it in this stage +ARG LOAD_EXAMPLES_DUCKDB ENV SUPERSET_HOME="/app/superset_home" \ HOME="/app/superset_home" \ @@ -168,6 +171,8 @@ RUN mkdir -p \ && touch superset/static/version_info.json # Install Playwright and optionally setup headless browsers +ENV PLAYWRIGHT_BROWSERS_PATH=/usr/local/share/playwright-browsers + ARG INCLUDE_CHROMIUM="false" ARG INCLUDE_FIREFOX="false" RUN --mount=type=cache,target=${SUPERSET_HOME}/.cache/uv \ diff --git a/GEMINI.md b/GEMINI.md index f811a97b81a..47dc3e3d863 120000 --- a/GEMINI.md +++ b/GEMINI.md @@ -1 +1 @@ -LLMS.md \ No newline at end of file +AGENTS.md \ No newline at end of file diff --git a/GPT.md b/GPT.md index f811a97b81a..47dc3e3d863 120000 --- a/GPT.md +++ b/GPT.md @@ -1 +1 @@ -LLMS.md \ No newline at end of file +AGENTS.md \ No newline at end of file diff --git a/LINTING_ARCHITECTURE.md b/LINTING_ARCHITECTURE.md new file mode 100644 index 00000000000..274f0c24674 --- /dev/null +++ b/LINTING_ARCHITECTURE.md @@ -0,0 +1,121 @@ + + +# Superset Frontend Linting Architecture + +## Overview +We use a hybrid linting approach combining OXC (fast, standard rules) with custom AST-based checks for Superset-specific patterns. + +## Components + +### 1. Primary Linter: OXC +- **What**: Oxidation Compiler's linter (oxlint) +- **Handles**: 95% of linting rules (standard ESLint rules, TypeScript, React, etc.) +- **Speed**: ~50-100x faster than ESLint +- **Config**: `oxlint.json` + +### 2. Custom Rule Checker +- **What**: Node.js AST-based script +- **Handles**: Superset-specific rules: + - No literal colors (use theme) + - No FontAwesome icons (use Icons component) + - No template vars in i18n +- **Speed**: Fast enough for pre-commit +- **Script**: `scripts/check-custom-rules.js` + +## Developer Workflow + +### Local Development +```bash +# Fast linting (OXC only) +npm run lint + +# Full linting (OXC + custom rules) +npm run lint:full + +# Auto-fix what's possible +npm run lint-fix +``` + +### Pre-commit +1. OXC runs first (via `scripts/oxlint.sh`) +2. Custom rules check runs second (lightweight, AST-based) +3. Both must pass for commit to succeed + +### CI Pipeline +```yaml +- name: Lint with OXC + run: npm run lint + +- name: Check custom rules + run: npm run check:custom-rules +``` + +## Why This Architecture? + +### ✅ Pros +1. **No binary distribution issues** - ASF compatible +2. **Fast performance** - OXC for bulk, lightweight script for custom +3. **Maintainable** - Custom rules in JavaScript, not Rust +4. **Flexible** - Can evolve as OXC adds plugin support +5. **Cacheable** - Both OXC and Node.js are standard tools + +### ❌ Cons +1. **Two tools** - Slightly more complex than single linter +2. **Duplicate parsing** - Files parsed twice (once by each tool) + +### 🔄 Migration Path +When OXC supports JavaScript plugins: +1. Convert `check-custom-rules.js` to OXC plugin format +2. Consolidate back to single tool +3. Keep same rules and developer experience + +## Implementation Checklist + +- [x] OXC for standard linting +- [x] Pre-commit integration +- [ ] Custom rules script +- [ ] Combine in npm scripts +- [ ] Update CI pipeline +- [ ] Developer documentation + +## Performance Targets + +| Operation | Target Time | Current | +|-----------|------------|---------| +| Pre-commit (changed files) | <2s | ✅ 1.5s | +| Full lint (all files) | <10s | ✅ 8s | +| Custom rules check | <5s | 🔄 TBD | + +## Caching Strategy + +### Local Development +- OXC: Built-in incremental checking +- Custom rules: Use file hash cache (similar to pytest cache) + +### CI +- Cache `node_modules` (includes oxlint binary) +- Cache custom rules results by commit hash +- Skip unchanged files using git diff + +## Future Improvements + +1. **When OXC adds plugin support**: Migrate custom rules to OXC plugins +2. **Consider Biome**: Another Rust-based linter with plugin support +3. **AST sharing**: Investigate sharing AST between tools to avoid double parsing diff --git a/UPDATING.md b/UPDATING.md index f34c46debf0..384b7d22233 100644 --- a/UPDATING.md +++ b/UPDATING.md @@ -23,6 +23,107 @@ This file documents any backwards-incompatible changes in Superset and assists people when migrating to a new version. ## Next + +### MCP Service + +The MCP (Model Context Protocol) service enables AI assistants and automation tools to interact programmatically with Superset. + +#### New Features +- MCP service infrastructure with FastMCP framework +- Tools for dashboards, charts, datasets, SQL Lab, and instance metadata +- Optional dependency: install with `pip install apache-superset[fastmcp]` +- Runs as separate process from Superset web server +- JWT-based authentication for production deployments + +#### New Configuration Options + +**Development** (single-user, local testing): +```python +# superset_config.py +MCP_DEV_USERNAME = "admin" # User for MCP authentication +MCP_SERVICE_HOST = "localhost" +MCP_SERVICE_PORT = 5008 +``` + +**Production** (JWT-based, multi-user): +```python +# superset_config.py +MCP_AUTH_ENABLED = True +MCP_JWT_ISSUER = "https://your-auth-provider.com" +MCP_JWT_AUDIENCE = "superset-mcp" +MCP_JWT_ALGORITHM = "RS256" # or "HS256" for shared secrets + +# Option 1: Use JWKS endpoint (recommended for RS256) +MCP_JWKS_URI = "https://auth.example.com/.well-known/jwks.json" + +# Option 2: Use static public key (RS256) +MCP_JWT_PUBLIC_KEY = "-----BEGIN PUBLIC KEY-----..." + +# Option 3: Use shared secret (HS256) +MCP_JWT_ALGORITHM = "HS256" +MCP_JWT_SECRET = "your-shared-secret-key" + +# Optional overrides +MCP_SERVICE_HOST = "0.0.0.0" +MCP_SERVICE_PORT = 5008 +MCP_SESSION_CONFIG = { + "SESSION_COOKIE_SECURE": True, + "SESSION_COOKIE_HTTPONLY": True, + "SESSION_COOKIE_SAMESITE": "Strict", +} +``` + +#### Running the MCP Service + +```bash +# Development +superset mcp run --port 5008 --debug + +# Production +superset mcp run --port 5008 + +# With factory config +superset mcp run --port 5008 --use-factory-config +``` + +#### Deployment Considerations + +The MCP service runs as a **separate process** from the Superset web server. + +**Important**: +- Requires same Python environment and configuration as Superset +- Shares database connections with main Superset app +- Can be scaled independently from web server +- Requires `fastmcp` package (optional dependency) + +**Installation**: +```bash +# Install with MCP support +pip install apache-superset[fastmcp] + +# Or add to requirements.txt +apache-superset[fastmcp]>=X.Y.Z +``` + +**Process Management**: +Use systemd, supervisord, or Kubernetes to manage the MCP service process. +See `superset/mcp_service/PRODUCTION.md` for deployment guides. + +**Security**: +- Development: Uses `MCP_DEV_USERNAME` for single-user access +- Production: **MUST** configure JWT authentication +- See `superset/mcp_service/SECURITY.md` for details + +#### Documentation + +- Architecture: `superset/mcp_service/ARCHITECTURE.md` +- Security: `superset/mcp_service/SECURITY.md` +- Production: `superset/mcp_service/PRODUCTION.md` +- Developer Guide: `superset/mcp_service/CLAUDE.md` +- Quick Start: `superset/mcp_service/README.md` + +--- + - [33055](https://github.com/apache/superset/pull/33055): Upgrades Flask-AppBuilder to 5.0.0. The AUTH_OID authentication type has been deprecated and is no longer available as an option in Flask-AppBuilder. OpenID (OID) is considered a deprecated authentication protocol - if you are using AUTH_OID, you will need to migrate to an alternative authentication method such as OAuth, LDAP, or database authentication before upgrading. - [35062](https://github.com/apache/superset/pull/35062): Changed the function signature of `setupExtensions` to `setupCodeOverrides` with options as arguments. - [34871](https://github.com/apache/superset/pull/34871): Fixed Jest test hanging issue from Ant Design v5 upgrade. MessageChannel is now mocked in test environment to prevent rc-overflow from causing Jest to hang. Test environment only - no production impact. @@ -41,7 +142,7 @@ Note: Pillow is now a required dependency (previously optional) to support image - [33116](https://github.com/apache/superset/pull/33116) In Echarts Series charts (e.g. Line, Area, Bar, etc.) charts, the `x_axis_sort_series` and `x_axis_sort_series_ascending` form data items have been renamed with `x_axis_sort` and `x_axis_sort_asc`. There's a migration added that can potentially affect a significant number of existing charts. - [32317](https://github.com/apache/superset/pull/32317) The horizontal filter bar feature is now out of testing/beta development and its feature flag `HORIZONTAL_FILTER_BAR` has been removed. -- [31590](https://github.com/apache/superset/pull/31590) Marks the begining of intricate work around supporting dynamic Theming, and breaks support for [THEME_OVERRIDES](https://github.com/apache/superset/blob/732de4ac7fae88e29b7f123b6cbb2d7cd411b0e4/superset/config.py#L671) in favor of a new theming system based on AntD V5. Likely this will be in disrepair until settling over the 5.x lifecycle. +- [31590](https://github.com/apache/superset/pull/31590) Marks the beginning of intricate work around supporting dynamic Theming, and breaks support for [THEME_OVERRIDES](https://github.com/apache/superset/blob/732de4ac7fae88e29b7f123b6cbb2d7cd411b0e4/superset/config.py#L671) in favor of a new theming system based on AntD V5. Likely this will be in disrepair until settling over the 5.x lifecycle. - [32432](https://github.com/apache/superset/pull/31260) Moves the List Roles FAB view to the frontend and requires `FAB_ADD_SECURITY_API` to be enabled in the configuration and `superset init` to be executed. - [34319](https://github.com/apache/superset/pull/34319) Drill to Detail and Drill By is now supported in Embedded mode, and also with the `DASHBOARD_RBAC` FF. If you don't want to expose these features in Embedded / `DASHBOARD_RBAC`, make sure the roles used for Embedded / `DASHBOARD_RBAC`don't have the required permissions to perform D2D actions. diff --git a/docker-compose-image-tag.yml b/docker-compose-image-tag.yml index 5bc24189790..f9c486869a6 100644 --- a/docker-compose-image-tag.yml +++ b/docker-compose-image-tag.yml @@ -28,6 +28,7 @@ x-superset-image: &superset-image apachesuperset.docker.scarf.sh/apache/superset x-superset-volumes: &superset-volumes # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container - ./docker:/app/docker + - ./superset-core:/app/superset-core - superset_home:/app/superset_home services: diff --git a/docker-compose-light.yml b/docker-compose-light.yml index 1910699be4a..b06be681af3 100644 --- a/docker-compose-light.yml +++ b/docker-compose-light.yml @@ -162,8 +162,11 @@ services: SCARF_ANALYTICS: "${SCARF_ANALYTICS:-}" # configuring the dev-server to use the host.docker.internal to connect to the backend superset: "http://superset-light:8088" + # Webpack dev server configuration + WEBPACK_DEVSERVER_HOST: "${WEBPACK_DEVSERVER_HOST:-127.0.0.1}" + WEBPACK_DEVSERVER_PORT: "${WEBPACK_DEVSERVER_PORT:-9000}" ports: - - "127.0.0.1:${NODE_PORT:-9001}:9000" # Parameterized port + - "${NODE_PORT:-9001}:9000" # Parameterized port, accessible on all interfaces command: ["/app/docker/docker-frontend.sh"] env_file: - path: docker/.env # default diff --git a/docker-compose.yml b/docker-compose.yml index d58ed84488f..050c6b22ef3 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -29,9 +29,11 @@ x-superset-volumes: &superset-volumes # /app/pythonpath_docker will be appended to the PYTHONPATH in the final container - ./docker:/app/docker - ./superset:/app/superset + - ./superset-core:/app/superset-core - ./superset-frontend:/app/superset-frontend - superset_home:/app/superset_home - ./tests:/app/tests + - superset_data:/app/data x-common-build: &common-build context: . target: ${SUPERSET_BUILD_TARGET:-dev} # can use `dev` (default) or `lean` @@ -135,9 +137,9 @@ services: - /home/superset-websocket/node_modules - /home/superset-websocket/dist - # Mounting a config file that contains a dummy secret required to boot up. - # do not use this docker compose in production - - ./docker/superset-websocket/config.json:/home/superset-websocket/config.json + # Mount config file. Create your own docker/superset-websocket/config.json + # for custom settings, then point to it here. Do not use this example in production. + - ./docker/superset-websocket/config.example.json:/home/superset-websocket/config.json:ro environment: - PORT=8080 - REDIS_HOST=redis @@ -274,3 +276,5 @@ volumes: external: false redis: external: false + superset_data: + external: false diff --git a/docker/README.md b/docker/README.md index a61e5b5bc93..eeb2a8f3590 100644 --- a/docker/README.md +++ b/docker/README.md @@ -34,8 +34,24 @@ intended for use with local development. ### Local overrides +#### Environment Variables + +To override environment variables locally, create a `./docker/.env-local` file (git-ignored). This file will be loaded after `.env` and can override any settings. + +#### Python Configuration + In order to override configuration settings locally, simply make a copy of [`./docker/pythonpath_dev/superset_config_local.example`](./pythonpath_dev/superset_config_local.example) -into `./docker/pythonpath_dev/superset_config_docker.py` (git ignored) and fill in your overrides. +into `./docker/pythonpath_dev/superset_config_docker.py` (git-ignored) and fill in your overrides. + +#### WebSocket Configuration + +To customize the WebSocket server configuration, create `./docker/superset-websocket/config.json` (git-ignored) based on [`./docker/superset-websocket/config.example.json`](./superset-websocket/config.example.json). + +Then update the `superset-websocket`.`volumes` config to mount it. + +#### Docker Compose Overrides + +For advanced Docker Compose customization, create a `docker-compose-override.yml` file (git-ignored) to override or extend services without modifying the main compose file. ### Local packages diff --git a/docker/docker-bootstrap.sh b/docker/docker-bootstrap.sh index 04709396cf9..58d71d25c25 100755 --- a/docker/docker-bootstrap.sh +++ b/docker/docker-bootstrap.sh @@ -21,8 +21,15 @@ set -eo pipefail # Make python interactive if [ "$DEV_MODE" == "true" ]; then if [ "$(whoami)" = "root" ] && command -v uv > /dev/null 2>&1; then - echo "Reinstalling the app in editable mode" - uv pip install -e . + # Always ensure superset-core is available + echo "Installing superset-core in editable mode" + uv pip install --no-deps -e /app/superset-core + + # Only reinstall the main app for non-worker processes + if [ "$1" != "worker" ] && [ "$1" != "beat" ]; then + echo "Reinstalling the app in editable mode" + uv pip install -e . + fi fi fi REQUIREMENTS_LOCAL="/app/docker/requirements-local.txt" @@ -34,7 +41,8 @@ if [ "$CYPRESS_CONFIG" == "true" ]; then export SUPERSET__SQLALCHEMY_DATABASE_URI=postgresql+psycopg2://superset:superset@db:5432/superset_cypress PORT=8081 fi -if [[ "$DATABASE_DIALECT" == postgres* ]] && [ "$(whoami)" = "root" ]; then +# Skip postgres requirements installation for workers to avoid conflicts +if [[ "$DATABASE_DIALECT" == postgres* ]] && [ "$(whoami)" = "root" ] && [ "$1" != "worker" ] && [ "$1" != "beat" ]; then # older images may not have the postgres dev requirements installed echo "Installing postgres requirements" if command -v uv > /dev/null 2>&1; then @@ -72,12 +80,16 @@ case "${1}" in ;; app) echo "Starting web app (using development server)..." - flask run -p $PORT --reload --debugger --without-threads --host=0.0.0.0 + flask run -p $PORT --reload --debugger --without-threads --host=0.0.0.0 --exclude-patterns "*/node_modules/*:*/.venv/*:*/build/*:*/__pycache__/*" ;; app-gunicorn) echo "Starting web app..." /usr/bin/run-server.sh ;; + mcp) + echo "Starting MCP service..." + superset mcp run --host 0.0.0.0 --port ${MCP_PORT:-5008} --debug + ;; *) echo "Unknown Operation!!!" ;; diff --git a/docker/pythonpath_dev/superset_config.py b/docker/pythonpath_dev/superset_config.py index 6d80d254a6a..d88d9899c27 100644 --- a/docker/pythonpath_dev/superset_config.py +++ b/docker/pythonpath_dev/superset_config.py @@ -138,7 +138,7 @@ try: from superset_config_docker import * # noqa: F403 logger.info( - f"Loaded your Docker configuration at [{superset_config_docker.__file__}]" + "Loaded your Docker configuration at [%s]", superset_config_docker.__file__ ) except ImportError: logger.info("Using default Docker config...") diff --git a/docker/superset-websocket/config.example.json b/docker/superset-websocket/config.example.json new file mode 100644 index 00000000000..8f6afdcc22a --- /dev/null +++ b/docker/superset-websocket/config.example.json @@ -0,0 +1,22 @@ +{ + "port": 8080, + "logLevel": "info", + "logToFile": false, + "logFilename": "app.log", + "statsd": { + "host": "127.0.0.1", + "port": 8125, + "globalTags": [] + }, + "redis": { + "port": 6379, + "host": "127.0.0.1", + "password": "", + "db": 0, + "ssl": false + }, + "redisStreamPrefix": "async-events-", + "jwtAlgorithms": ["HS256"], + "jwtSecret": "CHANGE-ME-IN-PRODUCTION-GOTTA-BE-LONG-AND-SECRET", + "jwtCookieName": "async-token" +} diff --git a/docs/DOCS_CLAUDE.md b/docs/DOCS_CLAUDE.md new file mode 100644 index 00000000000..42fd0ad71a9 --- /dev/null +++ b/docs/DOCS_CLAUDE.md @@ -0,0 +1,642 @@ + + +# LLM Context Guide for Apache Superset Documentation + +This guide helps LLMs work with the Apache Superset documentation site built with Docusaurus 3. + +## 📍 Current Directory Context + +You are currently in the `/docs` subdirectory of the Apache Superset repository. When referencing files from the main codebase, use `../` to access the parent directory. + +``` +/Users/evan_1/GitHub/superset/ # Main repository root +├── superset/ # Python backend code +├── superset-frontend/ # React/TypeScript frontend +└── docs/ # Documentation site (YOU ARE HERE) + ├── docs/ # Main documentation content + ├── developer_portal/ # Developer guides (currently disabled) + ├── components/ # Component playground (currently disabled) + └── docusaurus.config.ts # Site configuration +``` + +## 🚀 Quick Commands + +```bash +# Development +yarn start # Start dev server on http://localhost:3000 +yarn stop # Stop running dev server +yarn build # Build production site +yarn serve # Serve built site locally + +# Version Management (USE THESE, NOT docusaurus commands) +yarn version:add:docs # Add new docs version +yarn version:add:developer_portal # Add developer portal version +yarn version:add:components # Add components version +yarn version:remove:docs # Remove docs version +yarn version:remove:developer_portal # Remove developer portal version +yarn version:remove:components # Remove components version + +# Quality Checks +yarn typecheck # TypeScript validation +yarn eslint # Lint TypeScript/JavaScript files +``` + +## 📁 Documentation Structure + +### Main Documentation (`/docs`) +The primary documentation lives in `/docs` with this structure: + +``` +docs/ +├── intro.md # Auto-generated from ../README.md +├── quickstart.mdx # Getting started guide +├── api.mdx # API reference with Swagger UI +├── faq.mdx # Frequently asked questions +├── installation/ # Installation guides +│ ├── installation-methods.mdx +│ ├── docker-compose.mdx +│ ├── docker-builds.mdx +│ ├── kubernetes.mdx +│ ├── pypi.mdx +│ └── architecture.mdx +├── configuration/ # Configuration guides +│ ├── configuring-superset.mdx +│ ├── alerts-reports.mdx +│ ├── caching.mdx +│ ├── databases.mdx +│ └── [more config docs] +├── using-superset/ # User guides +│ ├── creating-your-first-dashboard.md +│ ├── exploring-data.mdx +│ └── [more user docs] +├── contributing/ # Contributor guides +│ ├── development.mdx +│ ├── testing-locally.mdx +│ └── [more contributor docs] +└── security/ # Security documentation + ├── security.mdx + └── [security guides] +``` + +### Developer Portal (`/developer_portal`) - Currently Disabled +When enabled, contains developer-focused content: +- API documentation +- Architecture guides +- CLI tools +- Code examples + +### Component Playground (`/components`) - Currently Disabled +When enabled, provides interactive component examples for UI development. + +## 📝 Documentation Standards + +### File Types +- **`.md` files**: Basic Markdown documents +- **`.mdx` files**: Markdown with JSX - can include React components +- **`.tsx` files in `/src`**: Custom React components and pages + +### Frontmatter Structure +Every documentation page should have frontmatter: + +```yaml +--- +title: Page Title +description: Brief description for SEO +sidebar_position: 1 # Optional: controls order in sidebar +--- +``` + +### MDX Component Usage +MDX files can import and use React components: + +```mdx +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + + ```bash + npm install superset + ``` + + + ```bash + yarn add superset + ``` + + +``` + +### Code Blocks +Use triple backticks with language identifiers: + +````markdown +```python +def hello_world(): + print("Hello, Superset!") +``` + +```sql title="Example Query" +SELECT * FROM users WHERE active = true; +``` + +```bash +# Installation command +pip install apache-superset +``` +```` + +### Admonitions +Docusaurus supports various admonition types: + +```markdown +:::note +This is a note +::: + +:::tip +This is a tip +::: + +:::warning +This is a warning +::: + +:::danger +This is a danger warning +::: + +:::info +This is an info box +::: +``` + +## 🔄 Version Management + +### Version Configuration +Versions are managed through `versions-config.json`: + +```json +{ + "docs": { + "disabled": false, + "lastVersion": "6.0.0", // Default version shown + "includeCurrentVersion": true, // Show "Next" version + "onlyIncludeVersions": ["current", "6.0.0"], + "versions": { + "current": { + "label": "Next", + "path": "", + "banner": "unreleased" // Shows warning banner + }, + "6.0.0": { + "label": "6.0.0", + "path": "6.0.0", + "banner": "none" + } + } + } +} +``` + +### Creating New Versions +**IMPORTANT**: Always use the custom scripts, NOT native Docusaurus commands: + +```bash +# ✅ CORRECT - Updates both Docusaurus and versions-config.json +yarn version:add:docs 6.1.0 + +# ❌ WRONG - Only updates Docusaurus, breaks version dropdown +yarn docusaurus docs:version 6.1.0 +``` + +### Version Files Created +When versioning, these files are created: +- `versioned_docs/version-X.X.X/` - Snapshot of current docs +- `versioned_sidebars/version-X.X.X-sidebars.json` - Sidebar config +- `versions.json` - List of all versions + +## 🎨 Styling and Theming + +### Custom CSS +Add custom styles in `/src/css/custom.css`: + +```css +:root { + --ifm-color-primary: #20a7c9; + --ifm-code-font-size: 95%; +} +``` + +### Custom Components +Create React components in `/src/components/`: + +```tsx +// src/components/FeatureCard.tsx +import React from 'react'; + +export default function FeatureCard({title, description}) { + return ( +
+

{title}

+

{description}

+
+ ); +} +``` + +Use in MDX: + +```mdx +import FeatureCard from '@site/src/components/FeatureCard'; + + +``` + +## 📦 Key Dependencies + +- **Docusaurus 3.8.1**: Static site generator +- **React 18.3**: UI framework +- **Ant Design 5.26**: Component library +- **@superset-ui/core**: Superset UI components +- **Swagger UI React**: API documentation +- **Prism**: Syntax highlighting + +## 🔗 Linking Strategies + +### Internal Links +Use relative paths for internal documentation: + +```markdown +[Installation Guide](./installation/docker-compose) +[Configuration](../configuration/configuring-superset) +``` + +### External Links +Always use full URLs: + +```markdown +[Apache Superset GitHub](https://github.com/apache/superset) +``` + +### Linking to Code +Reference code in the main repository: + +```markdown +See the [main configuration file](https://github.com/apache/superset/blob/master/superset/config.py) +``` + +## 🛠️ Common Documentation Tasks + +### Adding a New Guide +1. Create the `.mdx` file in the appropriate directory +2. Add frontmatter with title and description +3. Update sidebar if needed (for manual sidebar configs) + +### Adding API Documentation +The API docs use Swagger UI embedded in `/docs/api.mdx`: + +```mdx +import SwaggerUI from "swagger-ui-react"; +import "swagger-ui-react/swagger-ui.css"; + + +``` + +### Adding Interactive Examples +Use MDX to create interactive documentation: + +```mdx +import CodeBlock from '@theme/CodeBlock'; +import MyComponentExample from '!!raw-loader!../examples/MyComponent.tsx'; + +{MyComponentExample} +``` + +## 📋 Documentation Checklist + +When creating or updating documentation: + +- [ ] Clear, descriptive title in frontmatter +- [ ] Description for SEO in frontmatter +- [ ] Proper heading hierarchy (h1 -> h2 -> h3) +- [ ] Code examples with language identifiers +- [ ] Links verified (internal and external) +- [ ] Images have alt text +- [ ] Admonitions used for important notes +- [ ] Tested locally with `yarn start` +- [ ] No broken links (check with `yarn build`) + +## 🔍 Searching and Navigation + +### Sidebar Configuration +Sidebars are configured in `/sidebars.js`: + +```javascript +module.exports = { + CustomSidebar: [ + { + type: 'doc', + label: 'Introduction', + id: 'intro', + }, + { + type: 'category', + label: 'Installation', + items: [ + { + type: 'autogenerated', + dirName: 'installation', + }, + ], + }, + ], +}; +``` + +### Search +Docusaurus includes Algolia DocSearch integration configured in `docusaurus.config.ts`. + +## 🚫 Common Pitfalls to Avoid + +1. **Never use `yarn docusaurus docs:version`** - Use `yarn version:add:docs` instead +2. **Don't edit versioned docs directly** - Edit current docs and create new version +3. **Avoid absolute paths in links** - Use relative paths for maintainability +4. **Don't forget frontmatter** - Every doc needs title and description +5. **Test builds locally** - Run `yarn build` before committing + +## 🔧 Troubleshooting + +### Dev Server Issues +```bash +yarn stop # Kill any running servers +yarn clear # Clear cache +yarn start # Restart +``` + +### Build Failures +```bash +# Check for broken links +yarn build + +# TypeScript issues +yarn typecheck + +# Linting issues +yarn eslint +``` + +### Version Issues +If versions don't appear in dropdown: +1. Check `versions-config.json` includes the version +2. Verify version files exist in `versioned_docs/` +3. Restart dev server + +## 📚 Resources + +- [Docusaurus Documentation](https://docusaurus.io/docs) +- [MDX Documentation](https://mdxjs.com/) +- [Superset Contributing Guide](../CONTRIBUTING.md) +- [Main Superset Documentation](https://superset.apache.org/docs/intro) + +## 📖 Real Examples and Patterns + +### Example: Configuration Documentation Pattern +From `docs/configuration/configuring-superset.mdx`: + +```mdx +--- +title: Configuring Superset +hide_title: true +sidebar_position: 1 +version: 1 +--- + +# Configuring Superset + +## superset_config.py + +Superset exposes hundreds of configurable parameters through its +[config.py module](https://github.com/apache/superset/blob/master/superset/config.py). + +```bash +export SUPERSET_CONFIG_PATH=/app/superset_config.py +``` +``` + +**Key patterns:** +- Links to source code for reference +- Code blocks with bash/python examples +- Environment variable documentation +- Step-by-step configuration instructions + +### Example: Tutorial Documentation Pattern +From `docs/using-superset/creating-your-first-dashboard.mdx`: + +```mdx +import useBaseUrl from "@docusaurus/useBaseUrl"; + +## Creating Your First Dashboard + +:::tip +In addition to this site, [Preset.io](http://preset.io/) maintains an updated set of end-user +documentation at [docs.preset.io](https://docs.preset.io/). +::: + +### Connecting to a new database + + +``` + +**Key patterns:** +- Import Docusaurus hooks for dynamic URLs +- Use of admonitions (:::tip) for helpful information +- Screenshots with useBaseUrl for proper path resolution +- Clear section hierarchy with ### subheadings +- Step-by-step visual guides + +### Example: API Documentation Pattern +From `docs/api.mdx`: + +```mdx +import SwaggerUI from "swagger-ui-react"; +import "swagger-ui-react/swagger-ui.css"; + +## API Documentation + + +``` + +**Key patterns:** +- Embedding interactive Swagger UI +- Importing necessary CSS +- Direct API spec integration + +### Common Image Patterns + +```mdx +// For images in static folder +import useBaseUrl from "@docusaurus/useBaseUrl"; + + + +// With caption +
+ Dashboard view +
Superset Dashboard Interface
+
+``` + +### Multi-Tab Code Examples + +```mdx +import Tabs from '@theme/Tabs'; +import TabItem from '@theme/TabItem'; + + + + ```bash + docker-compose up + ``` + + + ```bash + kubectl apply -f superset.yaml + ``` + + + ```bash + pip install apache-superset + ``` + + +``` + +### Configuration File Examples + +```mdx +```python title="superset_config.py" +# Database connection example +SQLALCHEMY_DATABASE_URI = 'postgresql://user:password@localhost/superset' + +# Security configuration +SECRET_KEY = 'YOUR_SECRET_KEY_HERE' +WTF_CSRF_ENABLED = True + +# Feature flags +FEATURE_FLAGS = { + 'ENABLE_TEMPLATE_PROCESSING': True, + 'DASHBOARD_NATIVE_FILTERS': True, +} +``` +``` + +### Cross-Referencing Pattern + +```mdx +For detailed configuration options, see: +- [Configuring Superset](./configuration/configuring-superset) +- [Database Connections](./configuration/databases) +- [Security Settings](./security/security) + +External resources: +- [SQLAlchemy Documentation](https://docs.sqlalchemy.org/) +- [Flask Configuration](https://flask.palletsprojects.com/config/) +``` + +### Writing Installation Guides + +```mdx +## Prerequisites + +:::warning +Ensure you have Python 3.9+ and Node.js 16+ installed before proceeding. +::: + +## Installation Steps + +1. **Clone the repository** + ```bash + git clone https://github.com/apache/superset.git + cd superset + ``` + +2. **Install Python dependencies** + ```bash + pip install -e . + ``` + +3. **Initialize the database** + ```bash + superset db upgrade + superset init + ``` + +:::tip Success Check +Navigate to http://localhost:8088 and login with admin/admin +::: +``` + +### Documenting API Endpoints + +```mdx +## Chart API + +### GET /api/v1/chart/ + +Returns a list of charts. + +**Parameters:** +- `page` (optional): Page number +- `page_size` (optional): Number of items per page + +**Example Request:** +```bash +curl -X GET "http://localhost:8088/api/v1/chart/" \ + -H "Authorization: Bearer YOUR_ACCESS_TOKEN" +``` + +**Example Response:** +```json +{ + "count": 42, + "result": [ + { + "id": 1, + "slice_name": "Sales Dashboard", + "viz_type": "line" + } + ] +} +``` +``` + +--- + +**Note**: This documentation site serves as the primary resource for Superset users, administrators, and contributors. Always prioritize clarity, accuracy, and completeness when creating or updating documentation. diff --git a/docs/developer_portal/api/frontend.md b/docs/developer_portal/api/frontend.md deleted file mode 100644 index e2d66ae9f0a..00000000000 --- a/docs/developer_portal/api/frontend.md +++ /dev/null @@ -1,477 +0,0 @@ ---- -title: Frontend API Reference -sidebar_position: 1 -hide_title: true ---- - - - -# Frontend API Reference - -The `@apache-superset/core` package provides comprehensive APIs for frontend extension development. All APIs are organized into logical namespaces for easy discovery and use. - -## Core API - -The core namespace provides fundamental extension functionality. - -### registerView - -Registers a new view or panel in the specified contribution point. - -```typescript -core.registerView( - id: string, - component: React.ComponentType -): Disposable -``` - -**Example:** -```typescript -const panel = context.core.registerView('my-extension.panel', () => ( - -)); -``` - -### getActiveView - -Gets the currently active view in a contribution area. - -```typescript -core.getActiveView(area: string): View | undefined -``` - -## Commands API - -Manages command registration and execution. - -### registerCommand - -Registers a new command that can be triggered by menus, shortcuts, or programmatically. - -```typescript -commands.registerCommand( - id: string, - handler: CommandHandler -): Disposable - -interface CommandHandler { - title: string; - icon?: string; - execute: (...args: any[]) => any; - isEnabled?: (...args: any[]) => boolean; -} -``` - -**Example:** -```typescript -const cmd = context.commands.registerCommand('my-extension.analyze', { - title: 'Analyze Query', - icon: 'BarChartOutlined', - execute: () => { - const query = context.sqlLab.getCurrentQuery(); - // Perform analysis - }, - isEnabled: () => { - return context.sqlLab.hasActiveEditor(); - } -}); -``` - -### executeCommand - -Executes a registered command by ID. - -```typescript -commands.executeCommand(id: string, ...args: any[]): Promise -``` - -## SQL Lab API - -Provides access to SQL Lab functionality and events. - -### Query Access - -```typescript -// Get current tab -sqlLab.getCurrentTab(): Tab | undefined - -// Get all tabs -sqlLab.getTabs(): Tab[] - -// Get current query -sqlLab.getCurrentQuery(): string - -// Get selected text -sqlLab.getSelectedText(): string | undefined -``` - -### Database Access - -```typescript -// Get available databases -sqlLab.getDatabases(): Database[] - -// Get database by ID -sqlLab.getDatabase(id: number): Database | undefined - -// Get schemas for database -sqlLab.getSchemas(databaseId: number): Promise - -// Get tables for schema -sqlLab.getTables( - databaseId: number, - schema: string -): Promise -``` - -### Events - -```typescript -// Query execution events -sqlLab.onDidQueryRun: Event -sqlLab.onDidQueryStop: Event -sqlLab.onDidQueryFail: Event - -// Editor events -sqlLab.onDidChangeEditorContent: Event -sqlLab.onDidChangeSelection: Event - -// Tab events -sqlLab.onDidChangeActiveTab: Event -sqlLab.onDidCloseTab: Event -sqlLab.onDidChangeTabTitle: Event<{tab: Tab, title: string}> - -// Panel events -sqlLab.onDidOpenPanel: Event -sqlLab.onDidClosePanel: Event -sqlLab.onDidChangeActivePanel: Event -``` - -**Event Usage Example:** -```typescript -const disposable = context.sqlLab.onDidQueryRun((result) => { - console.log('Query executed:', result.query); - console.log('Rows returned:', result.rowCount); - console.log('Execution time:', result.executionTime); -}); - -// Remember to dispose when done -context.subscriptions.push(disposable); -``` - -## Authentication API - -Handles authentication and security tokens. - -### getCSRFToken - -Gets the current CSRF token for API requests. - -```typescript -authentication.getCSRFToken(): Promise -``` - -### getCurrentUser - -Gets information about the current user. - -```typescript -authentication.getCurrentUser(): User - -interface User { - id: number; - username: string; - email: string; - roles: Role[]; - permissions: Permission[]; -} -``` - -### hasPermission - -Checks if the current user has a specific permission. - -```typescript -authentication.hasPermission(permission: string): boolean -``` - -## Extensions API - -Manages extension lifecycle and inter-extension communication. - -### getExtension - -Gets information about an installed extension. - -```typescript -extensions.getExtension(id: string): Extension | undefined - -interface Extension { - id: string; - name: string; - version: string; - isActive: boolean; - metadata: ExtensionMetadata; -} -``` - -### getActiveExtensions - -Gets all currently active extensions. - -```typescript -extensions.getActiveExtensions(): Extension[] -``` - -### Events - -```typescript -// Extension lifecycle events -extensions.onDidActivateExtension: Event -extensions.onDidDeactivateExtension: Event -``` - -## UI Components - -Import pre-built UI components from `@apache-superset/core`: - -```typescript -import { - Button, - Select, - Input, - Table, - Modal, - Alert, - Tabs, - Card, - Dropdown, - Menu, - Tooltip, - Icon, - // ... many more -} from '@apache-superset/core'; -``` - -### Example Component Usage - -```typescript -import { Button, Alert } from '@apache-superset/core'; - -function MyExtensionPanel() { - return ( -
- - -
- ); -} -``` - -## Storage API - -Provides persistent storage for extension data. - -### Local Storage - -```typescript -// Store data -storage.local.set(key: string, value: any): Promise - -// Retrieve data -storage.local.get(key: string): Promise - -// Remove data -storage.local.remove(key: string): Promise - -// Clear all extension data -storage.local.clear(): Promise -``` - -### Workspace Storage - -Workspace storage is shared across all users for collaborative features. - -```typescript -storage.workspace.set(key: string, value: any): Promise -storage.workspace.get(key: string): Promise -storage.workspace.remove(key: string): Promise -``` - -## Network API - -Utilities for making API calls to Superset. - -### fetch - -Enhanced fetch with CSRF token handling. - -```typescript -network.fetch(url: string, options?: RequestInit): Promise -``` - -### API Client - -Type-safe API client for Superset endpoints. - -```typescript -// Get chart data -network.api.charts.get(id: number): Promise - -// Query database -network.api.sqlLab.execute( - databaseId: number, - query: string -): Promise - -// Get datasets -network.api.datasets.list(): Promise -``` - -## Utility Functions - -### Formatting - -```typescript -// Format numbers -utils.formatNumber(value: number, format?: string): string - -// Format dates -utils.formatDate(date: Date, format?: string): string - -// Format SQL -utils.formatSQL(sql: string): string -``` - -### Validation - -```typescript -// Validate SQL syntax -utils.validateSQL(sql: string): ValidationResult - -// Check if valid database ID -utils.isValidDatabaseId(id: any): boolean -``` - -## TypeScript Types - -Import common types for type safety: - -```typescript -import type { - Database, - Dataset, - Chart, - Dashboard, - Query, - QueryResult, - Tab, - Panel, - User, - Role, - Permission, - ExtensionContext, - Disposable, - Event, - // ... more types -} from '@apache-superset/core'; -``` - -## Extension Context - -The context object passed to your extension's `activate` function: - -```typescript -interface ExtensionContext { - // Subscription management - subscriptions: Disposable[]; - - // Extension metadata - extensionId: string; - extensionPath: string; - - // API namespaces - core: CoreAPI; - commands: CommandsAPI; - sqlLab: SqlLabAPI; - authentication: AuthenticationAPI; - extensions: ExtensionsAPI; - storage: StorageAPI; - network: NetworkAPI; - utils: UtilsAPI; - - // Logging - logger: Logger; -} -``` - -## Event Handling - -Events follow the VS Code pattern with subscribe/dispose: - -```typescript -// Subscribe to event -const disposable = sqlLab.onDidQueryRun((result) => { - // Handle event -}); - -// Dispose when done -disposable.dispose(); - -// Or add to context for automatic cleanup -context.subscriptions.push(disposable); -``` - -## Best Practices - -1. **Always dispose subscriptions** to prevent memory leaks -2. **Use TypeScript** for better IDE support and type safety -3. **Handle errors gracefully** with try-catch blocks -4. **Check permissions** before sensitive operations -5. **Use provided UI components** for consistency -6. **Cache API responses** when appropriate -7. **Validate user input** before processing - -## Version Compatibility - -The frontend API follows semantic versioning: - -- **Major version**: Breaking changes -- **Minor version**: New features, backward compatible -- **Patch version**: Bug fixes - -Check compatibility in your `extension.json`: - -```json -{ - "engines": { - "@apache-superset/core": "^1.0.0" - } -} -``` diff --git a/docs/developer_portal/architecture/overview.md b/docs/developer_portal/architecture/overview.md deleted file mode 100644 index 452e8fe1688..00000000000 --- a/docs/developer_portal/architecture/overview.md +++ /dev/null @@ -1,348 +0,0 @@ ---- -title: Architecture Overview -sidebar_position: 1 -hide_title: true ---- - - - -# Extension Architecture Overview - -The Superset extension architecture is designed to be modular, secure, and performant. This document provides a comprehensive overview of how extensions work and interact with the Superset host application. - -## Core Principles - -### 1. Lean Core -Superset's core remains minimal, with features delegated to extensions wherever possible. Built-in features use the same APIs as external extensions, ensuring API quality through dogfooding. - -### 2. Explicit Contribution Points -All extension points are clearly defined and documented. Extensions declare their capabilities in metadata files, enabling predictable lifecycle management. - -### 3. Versioned APIs -Public interfaces follow semantic versioning, ensuring backward compatibility and safe evolution of the platform. - -### 4. Lazy Loading -Extensions load only when needed, minimizing performance impact and resource consumption. - -### 5. Composability -Architecture patterns and APIs are reusable across different Superset modules, promoting consistency. - -### 6. Community-Driven -The system evolves based on real-world feedback, with new extension points added as needs emerge. - -## System Architecture - -```mermaid -graph TB - subgraph "Superset Host Application" - Core[Core Application] - API[Extension APIs] - Loader[Extension Loader] - Manager[Extension Manager] - end - - subgraph "Core Packages" - FrontendCore["@apache-superset/core
(Frontend)"] - BackendCore["apache-superset-core
(Backend)"] - CLI["apache-superset-extensions-cli"] - end - - subgraph "Extension" - Metadata[extension.json] - Frontend[Frontend Code] - Backend[Backend Code] - Bundle[.supx Bundle] - end - - Core --> API - API --> FrontendCore - API --> BackendCore - Loader --> Manager - Manager --> Bundle - Frontend --> FrontendCore - Backend --> BackendCore - CLI --> Bundle -``` - -## Key Components - -### Host Application - -The Superset host application provides: - -- **Extension APIs**: Well-defined interfaces for extensions to interact with Superset -- **Extension Manager**: Handles lifecycle, activation, and deactivation -- **Module Loader**: Dynamically loads extension code using Webpack Module Federation -- **Security Context**: Manages permissions and sandboxing for extensions - -### Core Packages - -#### @apache-superset/core (Frontend) -- Shared UI components and utilities -- TypeScript type definitions -- Frontend API implementations -- Event system and command registry - -#### apache-superset-core (Backend) -- Python base classes and utilities -- Database access APIs -- Security and permission helpers -- REST API registration - -#### apache-superset-extensions-cli -- Project scaffolding -- Build and bundling tools -- Development server -- Package management - -### Extension Structure - -Each extension consists of: - -- **Metadata** (`extension.json`): Declares capabilities and requirements -- **Frontend**: React components and TypeScript code -- **Backend**: Python modules and API endpoints -- **Assets**: Styles, images, and other resources -- **Bundle** (`.supx`): Packaged distribution format - -## Module Federation - -Extensions use Webpack Module Federation for dynamic loading: - -```javascript -// Extension webpack.config.js -new ModuleFederationPlugin({ - name: 'my_extension', - filename: 'remoteEntry.[contenthash].js', - exposes: { - './index': './src/index.tsx', - }, - externals: { - '@apache-superset/core': 'superset', - }, - shared: { - react: { singleton: true }, - 'react-dom': { singleton: true }, - } -}) -``` - -This allows: -- **Independent builds**: Extensions compile separately from Superset -- **Shared dependencies**: Common libraries like React aren't duplicated -- **Dynamic loading**: Extensions load at runtime without rebuilding Superset -- **Version compatibility**: Extensions declare compatible core versions - -## Extension Lifecycle - -### 1. Registration -```typescript -// Extension registered with host -extensionManager.register({ - name: 'my-extension', - version: '1.0.0', - manifest: manifestData -}); -``` - -### 2. Activation -```typescript -// activate() called when extension loads -export function activate(context: ExtensionContext) { - // Register contributions - const disposables = []; - - // Add panel - disposables.push( - context.core.registerView('my-panel', MyPanel) - ); - - // Register command - disposables.push( - context.commands.registerCommand('my-command', { - execute: () => { /* ... */ } - }) - ); - - // Store for cleanup - context.subscriptions.push(...disposables); -} -``` - -### 3. Runtime -- Extension responds to events -- Provides UI components when requested -- Executes commands when triggered -- Accesses APIs as needed - -### 4. Deactivation -```typescript -// Automatic cleanup of registered items -export function deactivate() { - // context.subscriptions automatically disposed - // Additional cleanup if needed -} -``` - -## Contribution Types - -### Views -Extensions can add panels and UI components: - -```json -{ - "views": { - "sqllab.panels": [{ - "id": "my-panel", - "name": "My Panel", - "icon": "ToolOutlined" - }] - } -} -``` - -### Commands -Define executable actions: - -```json -{ - "commands": [{ - "command": "my-extension.run", - "title": "Run Analysis", - "icon": "PlayCircleOutlined" - }] -} -``` - -### Menus -Add items to existing menus: - -```json -{ - "menus": { - "sqllab.editor": { - "primary": [{ - "command": "my-extension.run", - "when": "editorHasSelection" - }] - } - } -} -``` - -### API Endpoints -Register backend REST endpoints: - -```python -from superset_core.api import rest_api - -@rest_api.route('/my-endpoint') -def my_endpoint(): - return {'data': 'value'} -``` - -## Security Model - -### Permissions -- Extensions run with user's permissions -- No elevation of privileges -- Access controlled by Superset's RBAC - -### Sandboxing -- Frontend code runs in browser context -- Backend code runs in Python process -- Future: Optional sandboxed execution - -### Validation -- Manifest validation on upload -- Signature verification (future) -- Dependency scanning - -## Performance Considerations - -### Lazy Loading -- Extensions load only when features are accessed -- Code splitting for large extensions -- Cached after first load - -### Bundle Optimization -- Tree shaking removes unused code -- Minification reduces size -- Compression for network transfer - -### Resource Management -- Automatic cleanup on deactivation -- Memory leak prevention -- Event listener management - -## Development vs Production - -### Development Mode -```python -# superset_config.py -ENABLE_EXTENSIONS = True -LOCAL_EXTENSIONS = ['/path/to/extension'] -``` -- Hot reloading -- Source maps -- Debug logging - -### Production Mode -- Optimized bundles -- Cached assets -- Performance monitoring - -## Future Enhancements - -### Planned Features -- Enhanced sandboxing -- Extension marketplace -- Inter-extension communication -- Theme contributions -- Chart type extensions - -### API Expansion -- Dashboard extensions -- Database connector API -- Security provider interface -- Workflow automation - -## Best Practices - -### Do's -- ✅ Use TypeScript for type safety -- ✅ Follow semantic versioning -- ✅ Handle errors gracefully -- ✅ Clean up resources properly -- ✅ Document your extension - -### Don'ts -- ❌ Access private APIs -- ❌ Modify global state directly -- ❌ Block the main thread -- ❌ Store sensitive data insecurely -- ❌ Assume API stability in 0.x versions - -## Learn More - -- [API Reference](../api/frontend) -- [Development Guide](../getting-started) -- [Security Guidelines](./security) -- [Performance Optimization](./performance) diff --git a/docs/developer_portal/cli/overview.md b/docs/developer_portal/cli/overview.md deleted file mode 100644 index 6aa43dfcf4a..00000000000 --- a/docs/developer_portal/cli/overview.md +++ /dev/null @@ -1,466 +0,0 @@ - ---- -title: CLI Documentation -sidebar_position: 1 -hide_title: true ---- - -# Superset Extensions CLI - -The `apache-superset-extensions-cli` provides command-line tools for creating, developing, and packaging Superset extensions. - -## Installation - -```bash -pip install apache-superset-extensions-cli -``` - -## Commands - -### init - -Creates a new extension project with the standard folder structure. - -```bash -superset-extensions init [options] -``` - -**Options:** -- `--template