mirror of
https://github.com/apache/superset.git
synced 2026-04-28 04:25:07 +00:00
Compare commits
3 Commits
fdf19db5e6
...
sidecar
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bc0e7b4984 | ||
|
|
d3c81fe6b4 | ||
|
|
df62bffadd |
348
SIDECAR_INTEGRATION.md
Normal file
348
SIDECAR_INTEGRATION.md
Normal file
@@ -0,0 +1,348 @@
|
||||
# Query Sidecar Service Integration
|
||||
|
||||
This document describes the Node.js Query Sidecar Service integration that eliminates stale QueryObject issues in Superset Alerts & Reports.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Previously, Superset stored QueryObjects in the database after chart visualization logic transformed `form_data` into QueryObject format. This approach had a critical flaw: when the JavaScript visualization code changed, the stored QueryObjects became stale, causing Alerts & Reports to use outdated query logic.
|
||||
|
||||
## Solution
|
||||
|
||||
The Query Sidecar Service provides a Node.js service that computes QueryObjects from `form_data` on-demand using the same logic as the frontend, ensuring:
|
||||
|
||||
- **No stale data**: QueryObjects are computed fresh every time
|
||||
- **Consistency**: Uses identical logic to the Superset frontend
|
||||
- **Backward compatibility**: Falls back to legacy screenshot method if sidecar is unavailable
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
A[Superset Frontend] --> B[form_data]
|
||||
B --> C[Chart Database Record]
|
||||
|
||||
D[Alerts & Reports] --> E{Sidecar Available?}
|
||||
E -->|Yes| F[Query Sidecar Service]
|
||||
E -->|No| G[Legacy Screenshot Method]
|
||||
|
||||
F --> H[buildQueryObject.ts Logic]
|
||||
H --> I[Fresh QueryObject]
|
||||
|
||||
C --> F
|
||||
I --> J[Chart Data API]
|
||||
G --> J
|
||||
```
|
||||
|
||||
## Components
|
||||
|
||||
### 1. Node.js Sidecar Service (`sidecar-node/`)
|
||||
|
||||
Located in `sidecar-node/`, this service provides:
|
||||
|
||||
- **REST API**: `POST /api/v1/query-object` to transform form_data
|
||||
- **Type Safety**: Full TypeScript implementation with Superset type definitions
|
||||
- **Frontend Compatibility**: Uses identical logic from `superset-ui-core`
|
||||
- **Health Checks**: `/health` endpoint for monitoring
|
||||
- **Docker Support**: Production-ready containerization
|
||||
|
||||
Key files:
|
||||
- `src/query/buildQueryObject.ts` - Main transformation logic
|
||||
- `src/types/index.ts` - TypeScript type definitions
|
||||
- `src/routes/queryObject.ts` - REST API endpoints
|
||||
- `Dockerfile` - Container configuration
|
||||
|
||||
### 2. Python Client (`superset/utils/query_sidecar.py`)
|
||||
|
||||
Python client library that:
|
||||
|
||||
- **HTTP Communication**: Handles requests to the sidecar service
|
||||
- **Error Handling**: Robust error handling with fallback mechanisms
|
||||
- **QueryObject Creation**: Converts responses to Superset QueryObject instances
|
||||
- **Configuration**: Configurable timeouts and URLs
|
||||
|
||||
### 3. Reports Integration (`superset/commands/report/execute.py`)
|
||||
|
||||
Updated report execution logic:
|
||||
|
||||
- **Primary Method**: Uses sidecar service to generate fresh QueryObjects
|
||||
- **Fallback Method**: Falls back to legacy screenshot method on failure
|
||||
- **Configuration**: Controlled by `QUERY_SIDECAR_ENABLED` setting
|
||||
|
||||
## Setup and Configuration
|
||||
|
||||
### 1. Deploy the Sidecar Service
|
||||
|
||||
#### Development
|
||||
```bash
|
||||
cd sidecar-node
|
||||
npm install
|
||||
npm run dev
|
||||
```
|
||||
|
||||
#### Production with Docker
|
||||
```bash
|
||||
cd sidecar-node
|
||||
docker build -t superset-query-sidecar .
|
||||
docker run -p 3001:3001 superset-query-sidecar
|
||||
```
|
||||
|
||||
#### Production with Docker Compose
|
||||
```bash
|
||||
docker-compose -f docker-compose.sidecar.yml up -d
|
||||
```
|
||||
|
||||
### 2. Configure Superset
|
||||
|
||||
Add to your Superset configuration:
|
||||
|
||||
```python
|
||||
# Enable sidecar service integration
|
||||
QUERY_SIDECAR_ENABLED = True
|
||||
|
||||
# Sidecar service URL
|
||||
QUERY_SIDECAR_BASE_URL = "http://localhost:3001"
|
||||
|
||||
# Request timeout (seconds)
|
||||
QUERY_SIDECAR_TIMEOUT = 10
|
||||
```
|
||||
|
||||
#### Production Configuration Example
|
||||
```python
|
||||
QUERY_SIDECAR_ENABLED = True
|
||||
QUERY_SIDECAR_BASE_URL = "http://superset-query-sidecar:3001"
|
||||
QUERY_SIDECAR_TIMEOUT = 30
|
||||
```
|
||||
|
||||
### 3. Verify Integration
|
||||
|
||||
#### Check Sidecar Health
|
||||
```bash
|
||||
curl http://localhost:3001/health
|
||||
```
|
||||
|
||||
#### Test API Endpoint
|
||||
```bash
|
||||
curl -X POST http://localhost:3001/api/v1/query-object \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"form_data": {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"]
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### POST /api/v1/query-object
|
||||
|
||||
Transforms `form_data` into a QueryObject.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"form_data": {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"time_range": "No filter"
|
||||
},
|
||||
"query_fields": {
|
||||
"x": "columns",
|
||||
"y": "metrics"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"query_object": {
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"time_range": "No filter",
|
||||
"filters": [],
|
||||
"extras": {},
|
||||
"row_limit": undefined,
|
||||
"order_desc": true
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Error Response:**
|
||||
```json
|
||||
{
|
||||
"error": "form_data must include datasource and viz_type"
|
||||
}
|
||||
```
|
||||
|
||||
### GET /health
|
||||
|
||||
Returns service health status.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"timestamp": "2023-12-01T12:00:00.000Z",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
The integration includes comprehensive error handling:
|
||||
|
||||
### Sidecar Service Errors
|
||||
- **Connection Errors**: Falls back to legacy screenshot method
|
||||
- **Timeout Errors**: Configurable timeout with fallback
|
||||
- **Service Errors**: Logs errors and uses fallback method
|
||||
|
||||
### Configuration
|
||||
```python
|
||||
# Disable sidecar to use legacy method only
|
||||
QUERY_SIDECAR_ENABLED = False
|
||||
|
||||
# Increase timeout for slow networks
|
||||
QUERY_SIDECAR_TIMEOUT = 30
|
||||
```
|
||||
|
||||
## Migration Strategy
|
||||
|
||||
### Phase 1: Parallel Running
|
||||
1. Deploy sidecar service alongside existing Superset
|
||||
2. Enable `QUERY_SIDECAR_ENABLED = True`
|
||||
3. Monitor logs for any fallback usage
|
||||
|
||||
### Phase 2: Validation
|
||||
1. Compare QueryObjects generated by sidecar vs stored versions
|
||||
2. Verify Alerts & Reports work correctly
|
||||
3. Monitor performance metrics
|
||||
|
||||
### Phase 3: Full Migration
|
||||
1. Remove dependency on stored query_context (future)
|
||||
2. Optimize sidecar performance
|
||||
3. Scale sidecar service as needed
|
||||
|
||||
## Testing
|
||||
|
||||
### Node.js Service Tests
|
||||
```bash
|
||||
cd sidecar-node
|
||||
npm test
|
||||
npm run test:watch
|
||||
```
|
||||
|
||||
### Python Integration Tests
|
||||
```bash
|
||||
pytest tests/unit_tests/utils/test_query_sidecar.py
|
||||
pytest tests/unit_tests/commands/report/test_execute_sidecar.py
|
||||
```
|
||||
|
||||
### Integration Testing
|
||||
1. Create test alerts/reports
|
||||
2. Verify they execute with fresh QueryObjects
|
||||
3. Test fallback behavior by stopping sidecar service
|
||||
|
||||
## Monitoring and Logging
|
||||
|
||||
### Service Monitoring
|
||||
- Health check endpoint: `GET /health`
|
||||
- Docker health checks included
|
||||
- Prometheus metrics (future enhancement)
|
||||
|
||||
### Superset Logging
|
||||
The integration adds detailed logging:
|
||||
|
||||
```python
|
||||
logger.info("Successfully generated query context via sidecar for chart %s", chart_id)
|
||||
logger.warning("Failed to generate query context via sidecar service: %s. Falling back to screenshot method.", error)
|
||||
```
|
||||
|
||||
### Log Levels
|
||||
- `INFO`: Successful sidecar operations
|
||||
- `WARNING`: Fallback to legacy method
|
||||
- `ERROR`: Critical failures
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Latency
|
||||
- Sidecar adds ~10-50ms per request
|
||||
- Network latency between services
|
||||
- Configurable timeout prevents blocking
|
||||
|
||||
### Scaling
|
||||
- Sidecar service is stateless and can be horizontally scaled
|
||||
- Consider load balancing for high-volume deployments
|
||||
- Cache QueryObjects if needed (future enhancement)
|
||||
|
||||
### Resource Usage
|
||||
- Node.js service: ~50-100MB RAM
|
||||
- CPU usage minimal for typical workloads
|
||||
- Network bandwidth: ~1-10KB per request
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
1. **Caching**: Add Redis/Memcached for QueryObject caching
|
||||
2. **Metrics**: Prometheus metrics for monitoring
|
||||
3. **Load Balancing**: Support multiple sidecar instances
|
||||
4. **Query Optimization**: Optimize query generation performance
|
||||
5. **Database Cleanup**: Remove stored query_context column (breaking change)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Sidecar Service Not Starting
|
||||
```bash
|
||||
# Check logs
|
||||
docker logs superset-query-sidecar
|
||||
|
||||
# Verify port availability
|
||||
netstat -tlnp | grep 3001
|
||||
```
|
||||
|
||||
#### Connection Errors
|
||||
```bash
|
||||
# Test connectivity
|
||||
curl http://localhost:3001/health
|
||||
|
||||
# Check Superset logs
|
||||
grep -i "sidecar" /var/log/superset/superset.log
|
||||
```
|
||||
|
||||
#### Fallback Behavior
|
||||
```bash
|
||||
# Check if sidecar is being bypassed
|
||||
grep -i "falling back" /var/log/superset/superset.log
|
||||
```
|
||||
|
||||
### Configuration Issues
|
||||
- Verify `QUERY_SIDECAR_BASE_URL` is correct
|
||||
- Check network connectivity between services
|
||||
- Ensure sidecar service is healthy
|
||||
|
||||
### Performance Issues
|
||||
- Increase `QUERY_SIDECAR_TIMEOUT` if needed
|
||||
- Monitor sidecar service resource usage
|
||||
- Consider scaling sidecar service
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Network Security
|
||||
- Run sidecar service on internal network
|
||||
- Use HTTPS in production environments
|
||||
- Configure proper firewall rules
|
||||
|
||||
### Input Validation
|
||||
- Sidecar service validates all inputs
|
||||
- Superset client validates responses
|
||||
- No raw SQL execution in sidecar service
|
||||
|
||||
### Access Control
|
||||
- Service-to-service authentication (future)
|
||||
- Network-level access controls
|
||||
- Audit logging of sidecar requests
|
||||
60
docker-compose.sidecar.yml
Normal file
60
docker-compose.sidecar.yml
Normal file
@@ -0,0 +1,60 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
superset-query-sidecar:
|
||||
build:
|
||||
context: ./sidecar-node
|
||||
dockerfile: Dockerfile
|
||||
container_name: superset-query-sidecar
|
||||
ports:
|
||||
- "3001:3001"
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
- HOST=0.0.0.0
|
||||
- PORT=3001
|
||||
- SUPERSET_ORIGINS=http://localhost:8088,http://superset:8088
|
||||
healthcheck:
|
||||
test: ["CMD", "node", "-e", "require('http').get('http://localhost:3001/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1) }).on('error', () => process.exit(1))"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
restart: unless-stopped
|
||||
networks:
|
||||
- superset
|
||||
|
||||
# Example integration with existing Superset service
|
||||
# Uncomment and modify according to your setup
|
||||
#
|
||||
# superset:
|
||||
# # ... your existing superset configuration
|
||||
# environment:
|
||||
# - QUERY_SIDECAR_ENABLED=true
|
||||
# - QUERY_SIDECAR_BASE_URL=http://superset-query-sidecar:3001
|
||||
# - QUERY_SIDECAR_TIMEOUT=30
|
||||
# depends_on:
|
||||
# superset-query-sidecar:
|
||||
# condition: service_healthy
|
||||
# networks:
|
||||
# - superset
|
||||
|
||||
networks:
|
||||
superset:
|
||||
external: true
|
||||
51
sidecar-node/Dockerfile
Normal file
51
sidecar-node/Dockerfile
Normal file
@@ -0,0 +1,51 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
FROM node:18-alpine
|
||||
|
||||
# Set working directory
|
||||
WORKDIR /app
|
||||
|
||||
# Copy package files
|
||||
COPY package*.json ./
|
||||
|
||||
# Install dependencies
|
||||
RUN npm ci --only=production
|
||||
|
||||
# Copy source code
|
||||
COPY . .
|
||||
|
||||
# Build the application
|
||||
RUN npm run build
|
||||
|
||||
# Create non-root user
|
||||
RUN addgroup -g 1001 -S nodejs && \
|
||||
adduser -S sidecar -u 1001
|
||||
|
||||
# Change ownership of the app directory
|
||||
RUN chown -R sidecar:nodejs /app
|
||||
USER sidecar
|
||||
|
||||
# Expose port
|
||||
EXPOSE 3001
|
||||
|
||||
# Health check
|
||||
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD node -e "require('http').get('http://localhost:3001/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1) }).on('error', () => process.exit(1))"
|
||||
|
||||
# Start the application
|
||||
CMD ["npm", "start"]
|
||||
143
sidecar-node/README.md
Normal file
143
sidecar-node/README.md
Normal file
@@ -0,0 +1,143 @@
|
||||
# Superset Query Sidecar Service
|
||||
|
||||
A Node.js sidecar service that computes Superset QueryObjects from form_data, eliminating the need to store stale query objects in the database for Alerts & Reports.
|
||||
|
||||
## Overview
|
||||
|
||||
This service provides a REST API that transforms Superset's `form_data` into `QueryObject` format using the same logic as the frontend, ensuring consistency and eliminating staleness issues in Alerts & Reports.
|
||||
|
||||
## Features
|
||||
|
||||
- **Real-time QueryObject generation**: No stale data from database
|
||||
- **Frontend-compatible logic**: Uses the same transformation logic as superset-ui-core
|
||||
- **TypeScript support**: Full type safety
|
||||
- **Docker support**: Easy deployment
|
||||
- **Health checks**: Built-in monitoring endpoints
|
||||
- **CORS support**: Configurable for Superset integration
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Development
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install
|
||||
|
||||
# Start development server
|
||||
npm run dev
|
||||
|
||||
# Run tests
|
||||
npm test
|
||||
```
|
||||
|
||||
### Production
|
||||
|
||||
```bash
|
||||
# Build the application
|
||||
npm run build
|
||||
|
||||
# Start production server
|
||||
npm start
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
```bash
|
||||
# Build Docker image
|
||||
docker build -t superset-query-sidecar .
|
||||
|
||||
# Run container
|
||||
docker run -p 3001:3001 superset-query-sidecar
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### POST /api/v1/query-object
|
||||
|
||||
Transforms form_data into a QueryObject.
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"form_data": {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"time_range": "No filter"
|
||||
},
|
||||
"query_fields": {
|
||||
"x": "columns",
|
||||
"y": "metrics"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"query_object": {
|
||||
"datasource": "1__table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"time_range": "No filter",
|
||||
"filters": [],
|
||||
"extras": {}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### GET /health
|
||||
|
||||
Health check endpoint.
|
||||
|
||||
**Response:**
|
||||
```json
|
||||
{
|
||||
"status": "healthy",
|
||||
"timestamp": "2023-12-01T12:00:00.000Z",
|
||||
"version": "1.0.0"
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Environment variables:
|
||||
|
||||
- `PORT`: Server port (default: 3001)
|
||||
- `HOST`: Server host (default: localhost)
|
||||
- `NODE_ENV`: Environment mode (development/production)
|
||||
- `SUPERSET_ORIGINS`: Allowed CORS origins (comma-separated)
|
||||
|
||||
## Integration with Superset
|
||||
|
||||
This service is designed to be called by Superset's Python backend to generate QueryObjects for Alerts & Reports, replacing the current approach of reading stale query objects from the database.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐
|
||||
│ Superset │ │ Query Sidecar │ │ Alerts & Reports │
|
||||
│ Frontend │ │ Service │ │ │
|
||||
│ │ │ │ │ │
|
||||
│ form_data ────┼────┼──► buildQueryObject │◄───┼─── Python Client │
|
||||
│ │ │ │ │ │
|
||||
└─────────────────┘ └──────────────────────┘ └─────────────────────┘
|
||||
```
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
# Run all tests
|
||||
npm test
|
||||
|
||||
# Run tests in watch mode
|
||||
npm run test:watch
|
||||
|
||||
# Run tests with coverage
|
||||
npm test -- --coverage
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Licensed under the Apache License, Version 2.0.
|
||||
16
sidecar-node/jest.config.js
Normal file
16
sidecar-node/jest.config.js
Normal file
@@ -0,0 +1,16 @@
|
||||
module.exports = {
|
||||
preset: 'ts-jest',
|
||||
testEnvironment: 'node',
|
||||
roots: ['<rootDir>/src'],
|
||||
testMatch: ['**/__tests__/**/*.test.ts', '**/?(*.)+(spec|test).ts'],
|
||||
transform: {
|
||||
'^.+\\.ts$': 'ts-jest',
|
||||
},
|
||||
collectCoverageFrom: [
|
||||
'src/**/*.ts',
|
||||
'!src/**/*.d.ts',
|
||||
'!src/**/*.test.ts',
|
||||
],
|
||||
coverageDirectory: 'coverage',
|
||||
coverageReporters: ['text', 'lcov', 'html'],
|
||||
};
|
||||
41
sidecar-node/package.json
Normal file
41
sidecar-node/package.json
Normal file
@@ -0,0 +1,41 @@
|
||||
{
|
||||
"name": "superset-query-sidecar",
|
||||
"version": "1.0.0",
|
||||
"description": "Node.js sidecar service for computing Superset QueryObjects from form_data",
|
||||
"main": "dist/index.js",
|
||||
"scripts": {
|
||||
"build": "tsc",
|
||||
"start": "node dist/index.js",
|
||||
"dev": "ts-node src/index.ts",
|
||||
"test": "jest",
|
||||
"test:watch": "jest --watch"
|
||||
},
|
||||
"dependencies": {
|
||||
"express": "^4.18.2",
|
||||
"cors": "^2.8.5",
|
||||
"helmet": "^7.1.0",
|
||||
"compression": "^1.7.4"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/express": "^4.17.21",
|
||||
"@types/node": "^20.10.0",
|
||||
"@types/cors": "^2.8.17",
|
||||
"@types/compression": "^1.7.5",
|
||||
"@types/jest": "^29.5.8",
|
||||
"typescript": "^5.3.2",
|
||||
"ts-node": "^10.9.1",
|
||||
"jest": "^29.7.0",
|
||||
"ts-jest": "^29.1.1"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=18.0.0"
|
||||
},
|
||||
"keywords": [
|
||||
"superset",
|
||||
"query",
|
||||
"sidecar",
|
||||
"analytics"
|
||||
],
|
||||
"author": "Apache Superset",
|
||||
"license": "Apache-2.0"
|
||||
}
|
||||
149
sidecar-node/src/__tests__/buildQueryObject.test.ts
Normal file
149
sidecar-node/src/__tests__/buildQueryObject.test.ts
Normal file
@@ -0,0 +1,149 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import buildQueryObject from '../query/buildQueryObject';
|
||||
import { QueryFormData } from '../types';
|
||||
|
||||
describe('buildQueryObject', () => {
|
||||
const baseFormData: QueryFormData = {
|
||||
datasource: '1__table',
|
||||
viz_type: 'table',
|
||||
time_range: 'No filter',
|
||||
metrics: ['count'],
|
||||
columns: ['name', 'category'],
|
||||
row_limit: 1000,
|
||||
order_desc: true,
|
||||
};
|
||||
|
||||
it('should build a basic query object', () => {
|
||||
const queryObject = buildQueryObject(baseFormData);
|
||||
|
||||
expect(queryObject).toEqual({
|
||||
time_range: 'No filter',
|
||||
since: undefined,
|
||||
until: undefined,
|
||||
granularity: undefined,
|
||||
columns: ['name', 'category'],
|
||||
metrics: ['count'],
|
||||
orderby: undefined,
|
||||
annotation_layers: [],
|
||||
row_limit: 1000,
|
||||
row_offset: undefined,
|
||||
series_columns: undefined,
|
||||
series_limit: 0,
|
||||
series_limit_metric: undefined,
|
||||
group_others_when_limit_reached: false,
|
||||
order_desc: true,
|
||||
url_params: undefined,
|
||||
custom_params: {},
|
||||
extras: {
|
||||
filters: [],
|
||||
},
|
||||
filters: [],
|
||||
custom_form_data: {},
|
||||
});
|
||||
});
|
||||
|
||||
it('should handle adhoc filters', () => {
|
||||
const formDataWithFilters: QueryFormData = {
|
||||
...baseFormData,
|
||||
adhoc_filters: [
|
||||
{
|
||||
clause: 'WHERE',
|
||||
expressionType: 'SIMPLE',
|
||||
subject: 'category',
|
||||
operator: '==',
|
||||
comparator: 'Electronics',
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const queryObject = buildQueryObject(formDataWithFilters);
|
||||
|
||||
expect(queryObject.filters).toContainEqual({
|
||||
col: 'category',
|
||||
op: '==',
|
||||
val: 'Electronics',
|
||||
});
|
||||
});
|
||||
|
||||
it('should handle SQL adhoc filters in extras', () => {
|
||||
const formDataWithSQLFilters: QueryFormData = {
|
||||
...baseFormData,
|
||||
adhoc_filters: [
|
||||
{
|
||||
clause: 'WHERE',
|
||||
expressionType: 'SQL',
|
||||
sqlExpression: 'price > 100',
|
||||
},
|
||||
],
|
||||
};
|
||||
|
||||
const queryObject = buildQueryObject(formDataWithSQLFilters);
|
||||
|
||||
expect(queryObject.extras?.where).toBe('(price > 100)');
|
||||
});
|
||||
|
||||
it('should handle extra_form_data overrides', () => {
|
||||
const formDataWithExtraFormData: QueryFormData = {
|
||||
...baseFormData,
|
||||
extra_form_data: {
|
||||
time_range: 'Last week',
|
||||
adhoc_filters: [
|
||||
{
|
||||
clause: 'WHERE',
|
||||
expressionType: 'SIMPLE',
|
||||
subject: 'status',
|
||||
operator: '==',
|
||||
comparator: 'active',
|
||||
},
|
||||
],
|
||||
},
|
||||
};
|
||||
|
||||
const queryObject = buildQueryObject(formDataWithExtraFormData);
|
||||
|
||||
expect(queryObject.time_range).toBe('Last week');
|
||||
expect(queryObject.filters).toContainEqual({
|
||||
col: 'status',
|
||||
op: '==',
|
||||
val: 'active',
|
||||
});
|
||||
});
|
||||
|
||||
it('should handle series_limit from limit field', () => {
|
||||
const formDataWithLimit: QueryFormData = {
|
||||
...baseFormData,
|
||||
limit: 5,
|
||||
};
|
||||
|
||||
const queryObject = buildQueryObject(formDataWithLimit);
|
||||
|
||||
expect(queryObject.series_limit).toBe(5);
|
||||
});
|
||||
|
||||
it('should handle granularity', () => {
|
||||
const formDataWithGranularity: QueryFormData = {
|
||||
...baseFormData,
|
||||
granularity: 'created_at',
|
||||
};
|
||||
|
||||
const queryObject = buildQueryObject(formDataWithGranularity);
|
||||
|
||||
expect(queryObject.granularity).toBe('created_at');
|
||||
});
|
||||
});
|
||||
93
sidecar-node/src/index.ts
Normal file
93
sidecar-node/src/index.ts
Normal file
@@ -0,0 +1,93 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import express from 'express';
|
||||
import cors from 'cors';
|
||||
import helmet from 'helmet';
|
||||
import compression from 'compression';
|
||||
import queryObjectRouter from './routes/queryObject';
|
||||
|
||||
const app = express();
|
||||
const PORT = process.env.PORT || 3001;
|
||||
const HOST = process.env.HOST || 'localhost';
|
||||
|
||||
// Security middleware
|
||||
app.use(helmet());
|
||||
|
||||
// Enable CORS for Superset backend
|
||||
app.use(cors({
|
||||
origin: process.env.SUPERSET_ORIGINS?.split(',') || ['http://localhost:8088'],
|
||||
credentials: true,
|
||||
}));
|
||||
|
||||
// Compression middleware
|
||||
app.use(compression());
|
||||
|
||||
// Body parsing middleware
|
||||
app.use(express.json({ limit: '10mb' }));
|
||||
app.use(express.urlencoded({ extended: true, limit: '10mb' }));
|
||||
|
||||
// Request logging middleware
|
||||
app.use((req, res, next) => {
|
||||
const timestamp = new Date().toISOString();
|
||||
console.log(`${timestamp} ${req.method} ${req.path}`);
|
||||
next();
|
||||
});
|
||||
|
||||
// Routes
|
||||
app.use(queryObjectRouter);
|
||||
|
||||
// Global error handler
|
||||
app.use((err: Error, req: express.Request, res: express.Response, next: express.NextFunction) => {
|
||||
console.error('Unhandled error:', err);
|
||||
res.status(500).json({
|
||||
error: 'Internal server error',
|
||||
message: process.env.NODE_ENV === 'development' ? err.message : undefined,
|
||||
});
|
||||
});
|
||||
|
||||
// 404 handler
|
||||
app.use((req, res) => {
|
||||
res.status(404).json({
|
||||
error: 'Not found',
|
||||
path: req.path,
|
||||
});
|
||||
});
|
||||
|
||||
// Start server
|
||||
const server = app.listen(PORT, HOST, () => {
|
||||
console.log(`🚀 Superset Query Sidecar Service running on http://${HOST}:${PORT}`);
|
||||
console.log(`📊 Health check available at http://${HOST}:${PORT}/health`);
|
||||
console.log(`🔧 Query object API available at http://${HOST}:${PORT}/api/v1/query-object`);
|
||||
});
|
||||
|
||||
// Graceful shutdown
|
||||
process.on('SIGTERM', () => {
|
||||
console.log('SIGTERM received, shutting down gracefully');
|
||||
server.close(() => {
|
||||
console.log('Process terminated');
|
||||
});
|
||||
});
|
||||
|
||||
process.on('SIGINT', () => {
|
||||
console.log('SIGINT received, shutting down gracefully');
|
||||
server.close(() => {
|
||||
console.log('Process terminated');
|
||||
});
|
||||
});
|
||||
|
||||
export default app;
|
||||
134
sidecar-node/src/query/buildQueryObject.ts
Normal file
134
sidecar-node/src/query/buildQueryObject.ts
Normal file
@@ -0,0 +1,134 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
/* eslint-disable camelcase */
|
||||
import {
|
||||
QueryObject,
|
||||
QueryFormData,
|
||||
QueryFieldAliases,
|
||||
isQueryFormMetric,
|
||||
isDefined,
|
||||
} from '../types';
|
||||
import processFilters from '../utils/processFilters';
|
||||
import extractExtras from '../utils/extractExtras';
|
||||
import extractQueryFields from '../utils/extractQueryFields';
|
||||
import { overrideExtraFormData } from '../utils/processExtraFormData';
|
||||
|
||||
/**
|
||||
* Build the common segments of all query objects (e.g. the granularity field derived from
|
||||
* SQLAlchemy). The segments specific to each viz type is constructed in the
|
||||
* buildQuery method for each viz type (see `wordcloud/buildQuery.ts` for an example).
|
||||
* Note the type of the formData argument passed in here is the type of the formData for a
|
||||
* specific viz, which is a subtype of the generic formData shared among all viz types.
|
||||
*/
|
||||
export default function buildQueryObject<T extends QueryFormData>(
|
||||
formData: T,
|
||||
queryFields?: QueryFieldAliases,
|
||||
): QueryObject {
|
||||
const {
|
||||
annotation_layers = [],
|
||||
extra_form_data,
|
||||
time_range,
|
||||
since,
|
||||
until,
|
||||
row_limit,
|
||||
row_offset,
|
||||
order_desc,
|
||||
limit,
|
||||
timeseries_limit_metric,
|
||||
granularity,
|
||||
url_params = {},
|
||||
custom_params = {},
|
||||
series_columns,
|
||||
series_limit,
|
||||
series_limit_metric,
|
||||
group_others_when_limit_reached,
|
||||
...residualFormData
|
||||
} = formData;
|
||||
|
||||
const {
|
||||
adhoc_filters: appendAdhocFilters = [],
|
||||
filters: appendFilters = [],
|
||||
custom_form_data = {},
|
||||
...overrides
|
||||
} = extra_form_data || {};
|
||||
|
||||
const numericRowLimit = Number(row_limit);
|
||||
const numericRowOffset = Number(row_offset);
|
||||
|
||||
const { metrics, columns, orderby } = extractQueryFields(
|
||||
residualFormData,
|
||||
queryFields,
|
||||
);
|
||||
|
||||
// collect all filters for conversion to simple filters/freeform clauses
|
||||
const extras = extractExtras(formData);
|
||||
const { filters: extraFilters } = extras;
|
||||
const filterFormData = {
|
||||
...formData,
|
||||
...extras,
|
||||
filters: [...extraFilters, ...appendFilters],
|
||||
adhoc_filters: [...(formData.adhoc_filters || []), ...appendAdhocFilters],
|
||||
};
|
||||
|
||||
const extrasAndfilters = processFilters(filterFormData);
|
||||
|
||||
const normalizeSeriesLimitMetric = (metric: any) => {
|
||||
if (isQueryFormMetric(metric)) {
|
||||
return metric;
|
||||
}
|
||||
return undefined;
|
||||
};
|
||||
|
||||
let queryObject: QueryObject = {
|
||||
// fallback `null` to `undefined` so they won't be sent to the backend
|
||||
// (JSON.stringify will ignore `undefined`.)
|
||||
time_range: time_range || undefined,
|
||||
since: since || undefined,
|
||||
until: until || undefined,
|
||||
granularity: granularity || undefined,
|
||||
...extras,
|
||||
...extrasAndfilters,
|
||||
columns,
|
||||
metrics,
|
||||
orderby,
|
||||
annotation_layers,
|
||||
row_limit:
|
||||
row_limit == null || Number.isNaN(numericRowLimit)
|
||||
? undefined
|
||||
: numericRowLimit,
|
||||
row_offset:
|
||||
row_offset == null || Number.isNaN(numericRowOffset)
|
||||
? undefined
|
||||
: numericRowOffset,
|
||||
series_columns,
|
||||
series_limit: series_limit ?? (isDefined(limit) ? Number(limit) : 0),
|
||||
series_limit_metric:
|
||||
normalizeSeriesLimitMetric(series_limit_metric) ??
|
||||
timeseries_limit_metric ??
|
||||
undefined,
|
||||
group_others_when_limit_reached: group_others_when_limit_reached ?? false,
|
||||
order_desc: typeof order_desc === 'undefined' ? true : order_desc,
|
||||
url_params: url_params || undefined,
|
||||
custom_params,
|
||||
};
|
||||
|
||||
// override extra form data used by native and cross filters
|
||||
queryObject = overrideExtraFormData(queryObject, overrides);
|
||||
|
||||
return { ...queryObject, custom_form_data };
|
||||
}
|
||||
91
sidecar-node/src/routes/queryObject.ts
Normal file
91
sidecar-node/src/routes/queryObject.ts
Normal file
@@ -0,0 +1,91 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import { Request, Response, Router } from 'express';
|
||||
import buildQueryObject from '../query/buildQueryObject';
|
||||
import { QueryFormData, QueryFieldAliases } from '../types';
|
||||
|
||||
const router = Router();
|
||||
|
||||
interface BuildQueryObjectRequest {
|
||||
form_data: QueryFormData;
|
||||
query_fields?: QueryFieldAliases;
|
||||
}
|
||||
|
||||
interface BuildQueryObjectResponse {
|
||||
query_object: any;
|
||||
error?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* POST /api/v1/query-object
|
||||
*
|
||||
* Build a QueryObject from form_data
|
||||
*
|
||||
* Request body:
|
||||
* - form_data: The form data from Superset frontend
|
||||
* - query_fields: Optional query field aliases for visualization-specific mappings
|
||||
*
|
||||
* Response:
|
||||
* - query_object: The computed QueryObject
|
||||
* - error: Error message if processing failed
|
||||
*/
|
||||
router.post('/api/v1/query-object', (req: Request, res: Response) => {
|
||||
try {
|
||||
const { form_data, query_fields }: BuildQueryObjectRequest = req.body;
|
||||
|
||||
if (!form_data) {
|
||||
return res.status(400).json({
|
||||
error: 'form_data is required',
|
||||
} as BuildQueryObjectResponse);
|
||||
}
|
||||
|
||||
// Validate required form_data fields
|
||||
if (!form_data.datasource || !form_data.viz_type) {
|
||||
return res.status(400).json({
|
||||
error: 'form_data must include datasource and viz_type',
|
||||
} as BuildQueryObjectResponse);
|
||||
}
|
||||
|
||||
const queryObject = buildQueryObject(form_data, query_fields);
|
||||
|
||||
res.json({
|
||||
query_object: queryObject,
|
||||
} as BuildQueryObjectResponse);
|
||||
|
||||
} catch (error: any) {
|
||||
console.error('Error building query object:', error);
|
||||
res.status(500).json({
|
||||
error: `Failed to build query object: ${error.message}`,
|
||||
} as BuildQueryObjectResponse);
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* GET /health
|
||||
*
|
||||
* Health check endpoint
|
||||
*/
|
||||
router.get('/health', (req: Request, res: Response) => {
|
||||
res.json({
|
||||
status: 'healthy',
|
||||
timestamp: new Date().toISOString(),
|
||||
version: process.env.npm_package_version || '1.0.0',
|
||||
});
|
||||
});
|
||||
|
||||
export default router;
|
||||
213
sidecar-node/src/types/index.ts
Normal file
213
sidecar-node/src/types/index.ts
Normal file
@@ -0,0 +1,213 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
/* eslint-disable camelcase */
|
||||
|
||||
// Basic types for form data and query objects
|
||||
export interface JsonObject {
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
// Metric types
|
||||
export interface SavedMetric {
|
||||
metric_name: string;
|
||||
expression?: string;
|
||||
label?: string;
|
||||
}
|
||||
|
||||
export interface AdhocMetric {
|
||||
aggregate: string;
|
||||
column?: any;
|
||||
expressionType: 'SIMPLE' | 'SQL';
|
||||
hasCustomLabel?: boolean;
|
||||
label: string;
|
||||
sqlExpression?: string;
|
||||
optionName?: string;
|
||||
}
|
||||
|
||||
export type QueryFormMetric = SavedMetric | AdhocMetric | string;
|
||||
|
||||
// Column types
|
||||
export interface PhysicalColumn {
|
||||
column_name: string;
|
||||
type?: string;
|
||||
}
|
||||
|
||||
export interface AdhocColumn {
|
||||
hasCustomLabel?: boolean;
|
||||
label: string;
|
||||
sqlExpression: string;
|
||||
expressionType: 'SQL';
|
||||
optionName?: string;
|
||||
}
|
||||
|
||||
export type QueryFormColumn = PhysicalColumn | AdhocColumn | string;
|
||||
|
||||
// Filter types
|
||||
export type BinaryOperator =
|
||||
| '==' | '!=' | '>' | '<' | '>=' | '<='
|
||||
| 'LIKE' | 'ILIKE' | 'REGEX' | 'NOT REGEX';
|
||||
|
||||
export type SetOperator = 'IN' | 'NOT IN';
|
||||
|
||||
export interface AdhocFilter {
|
||||
clause: 'WHERE' | 'HAVING';
|
||||
comparator?: any;
|
||||
expressionType: 'SIMPLE' | 'SQL';
|
||||
operator?: BinaryOperator | SetOperator;
|
||||
subject?: string | AdhocColumn;
|
||||
sqlExpression?: string;
|
||||
filterOptionName?: string;
|
||||
}
|
||||
|
||||
export interface QueryObjectFilterClause {
|
||||
col: string;
|
||||
op: BinaryOperator | SetOperator;
|
||||
val: any;
|
||||
}
|
||||
|
||||
// Order by types
|
||||
export type QueryFormOrderBy = [QueryFormColumn | QueryFormMetric | {}, boolean] | [];
|
||||
|
||||
// Annotation types
|
||||
export interface AnnotationLayer {
|
||||
annotationType: string;
|
||||
name: string;
|
||||
show: boolean;
|
||||
sourceType?: string;
|
||||
value?: string;
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
// Time range types
|
||||
export interface TimeRange {
|
||||
time_range?: string;
|
||||
since?: string;
|
||||
until?: string;
|
||||
}
|
||||
|
||||
// Extra form data types
|
||||
export interface ExtraFormDataAppend {
|
||||
adhoc_filters?: AdhocFilter[];
|
||||
filters?: QueryObjectFilterClause[];
|
||||
interactive_drilldown?: string[];
|
||||
interactive_groupby?: string[];
|
||||
interactive_highlight?: string[];
|
||||
custom_form_data?: JsonObject;
|
||||
}
|
||||
|
||||
export interface ExtraFormDataOverride {
|
||||
granularity_sqla?: string;
|
||||
granularity?: string;
|
||||
time_range?: string;
|
||||
time_column?: string;
|
||||
time_grain?: string;
|
||||
time_compare?: string[];
|
||||
relative_start?: string;
|
||||
relative_end?: string;
|
||||
time_grain_sqla?: string;
|
||||
}
|
||||
|
||||
export type ExtraFormData = ExtraFormDataAppend & ExtraFormDataOverride;
|
||||
|
||||
// Query extras interface
|
||||
export interface QueryObjectExtras {
|
||||
having?: string;
|
||||
where?: string;
|
||||
time_grain_sqla?: string;
|
||||
time_range_endpoints?: [string, string];
|
||||
relative_start?: string;
|
||||
relative_end?: string;
|
||||
time_compare?: string[];
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
// Main form data interface
|
||||
export interface BaseFormData extends TimeRange {
|
||||
datasource: string;
|
||||
viz_type: string;
|
||||
metrics?: QueryFormMetric[];
|
||||
where?: string;
|
||||
columns?: QueryFormColumn[];
|
||||
groupby?: QueryFormColumn[];
|
||||
all_columns?: QueryFormColumn[];
|
||||
adhoc_filters?: AdhocFilter[] | null;
|
||||
extra_form_data?: ExtraFormData;
|
||||
order_desc?: boolean;
|
||||
limit?: number;
|
||||
row_limit?: string | number | null;
|
||||
row_offset?: string | number | null;
|
||||
series_columns?: QueryFormColumn[];
|
||||
series_limit?: number;
|
||||
series_limit_metric?: QueryFormMetric;
|
||||
annotation_layers?: AnnotationLayer[];
|
||||
url_params?: Record<string, string>;
|
||||
custom_params?: Record<string, string>;
|
||||
[key: string]: any;
|
||||
}
|
||||
|
||||
export interface SqlaFormData extends BaseFormData {
|
||||
granularity?: string;
|
||||
granularity_sqla?: string;
|
||||
time_grain_sqla?: string;
|
||||
having?: string;
|
||||
}
|
||||
|
||||
export type QueryFormData = SqlaFormData;
|
||||
|
||||
// Query object interface
|
||||
export interface QueryObject {
|
||||
time_range?: string;
|
||||
since?: string;
|
||||
until?: string;
|
||||
granularity?: string;
|
||||
columns?: QueryFormColumn[];
|
||||
metrics?: QueryFormMetric[];
|
||||
orderby?: QueryFormOrderBy[];
|
||||
annotation_layers?: AnnotationLayer[];
|
||||
row_limit?: number;
|
||||
row_offset?: number;
|
||||
series_columns?: QueryFormColumn[];
|
||||
series_limit?: number;
|
||||
series_limit_metric?: QueryFormMetric;
|
||||
group_others_when_limit_reached?: boolean;
|
||||
order_desc?: boolean;
|
||||
url_params?: Record<string, string>;
|
||||
custom_params?: Record<string, string>;
|
||||
extras?: QueryObjectExtras;
|
||||
filters?: QueryObjectFilterClause[];
|
||||
custom_form_data?: JsonObject;
|
||||
}
|
||||
|
||||
// Query field aliases
|
||||
export type QueryFieldAliases = {
|
||||
[key: string]: 'metrics' | 'columns' | 'groupby';
|
||||
};
|
||||
|
||||
// Utility functions
|
||||
export function isAdhocMetric(metric: any): metric is AdhocMetric {
|
||||
return metric && typeof metric === 'object' && 'expressionType' in metric;
|
||||
}
|
||||
|
||||
export function isQueryFormMetric(metric: any): metric is QueryFormMetric {
|
||||
return typeof metric === 'string' || isAdhocMetric(metric) ||
|
||||
(metric && typeof metric === 'object' && 'metric_name' in metric);
|
||||
}
|
||||
|
||||
export function isDefined<T>(value: T | undefined | null): value is T {
|
||||
return value !== undefined && value !== null;
|
||||
}
|
||||
78
sidecar-node/src/utils/extractExtras.ts
Normal file
78
sidecar-node/src/utils/extractExtras.ts
Normal file
@@ -0,0 +1,78 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import {
|
||||
QueryFormData,
|
||||
QueryObjectExtras,
|
||||
QueryObjectFilterClause,
|
||||
} from '../types';
|
||||
|
||||
interface ExtrasResult extends QueryObjectExtras {
|
||||
filters: QueryObjectFilterClause[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract extras and filters from form data
|
||||
*/
|
||||
export default function extractExtras(formData: QueryFormData): ExtrasResult {
|
||||
const {
|
||||
where,
|
||||
having,
|
||||
time_grain_sqla,
|
||||
granularity_sqla,
|
||||
granularity,
|
||||
extra_filters = [],
|
||||
} = formData;
|
||||
|
||||
const extras: QueryObjectExtras = {};
|
||||
const filters: QueryObjectFilterClause[] = [];
|
||||
|
||||
// Add SQL clauses to extras
|
||||
if (where) {
|
||||
extras.where = where;
|
||||
}
|
||||
if (having) {
|
||||
extras.having = having;
|
||||
}
|
||||
if (time_grain_sqla) {
|
||||
extras.time_grain_sqla = time_grain_sqla;
|
||||
}
|
||||
|
||||
// Handle granularity - prefer granularity_sqla over granularity
|
||||
const timeColumn = granularity_sqla || granularity;
|
||||
if (timeColumn) {
|
||||
// Time column handling would go here if needed
|
||||
}
|
||||
|
||||
// Convert extra_filters to QueryObjectFilterClause format
|
||||
if (extra_filters && Array.isArray(extra_filters)) {
|
||||
for (const filter of extra_filters) {
|
||||
if (filter.col && filter.op && filter.val !== undefined) {
|
||||
filters.push({
|
||||
col: filter.col,
|
||||
op: filter.op,
|
||||
val: filter.val,
|
||||
});
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
...extras,
|
||||
filters,
|
||||
};
|
||||
}
|
||||
73
sidecar-node/src/utils/extractQueryFields.ts
Normal file
73
sidecar-node/src/utils/extractQueryFields.ts
Normal file
@@ -0,0 +1,73 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import {
|
||||
QueryFormData,
|
||||
QueryFormMetric,
|
||||
QueryFormColumn,
|
||||
QueryFormOrderBy,
|
||||
QueryFieldAliases,
|
||||
} from '../types';
|
||||
|
||||
interface QueryFieldsResult {
|
||||
metrics?: QueryFormMetric[];
|
||||
columns?: QueryFormColumn[];
|
||||
orderby?: QueryFormOrderBy[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Extract query fields (metrics, columns, orderby) from form data
|
||||
*/
|
||||
export default function extractQueryFields(
|
||||
formData: QueryFormData,
|
||||
queryFieldAliases?: QueryFieldAliases,
|
||||
): QueryFieldsResult {
|
||||
const result: QueryFieldsResult = {};
|
||||
|
||||
// Extract metrics
|
||||
if (formData.metrics && formData.metrics.length > 0) {
|
||||
result.metrics = formData.metrics;
|
||||
}
|
||||
|
||||
// Extract columns - prefer 'columns' over 'groupby'
|
||||
const columns = formData.columns || formData.groupby;
|
||||
if (columns && columns.length > 0) {
|
||||
result.columns = columns;
|
||||
}
|
||||
|
||||
// Handle query field aliases if provided
|
||||
if (queryFieldAliases) {
|
||||
for (const [formFieldName, queryField] of Object.entries(queryFieldAliases)) {
|
||||
const formValue = (formData as any)[formFieldName];
|
||||
if (formValue && Array.isArray(formValue) && formValue.length > 0) {
|
||||
if (queryField === 'metrics') {
|
||||
result.metrics = formValue;
|
||||
} else if (queryField === 'columns' || queryField === 'groupby') {
|
||||
result.columns = formValue;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Extract orderby - this can be complex as it depends on the form structure
|
||||
// For now, we'll handle basic cases
|
||||
if (formData.orderby && Array.isArray(formData.orderby)) {
|
||||
result.orderby = formData.orderby as QueryFormOrderBy[];
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
57
sidecar-node/src/utils/processExtraFormData.ts
Normal file
57
sidecar-node/src/utils/processExtraFormData.ts
Normal file
@@ -0,0 +1,57 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import { QueryObject, ExtraFormDataOverride } from '../types';
|
||||
|
||||
/**
|
||||
* Override extra form data used by native and cross filters
|
||||
*/
|
||||
export function overrideExtraFormData(
|
||||
queryObject: QueryObject,
|
||||
overrides: ExtraFormDataOverride,
|
||||
): QueryObject {
|
||||
const result = { ...queryObject };
|
||||
|
||||
// Override top-level properties
|
||||
if (overrides.time_range !== undefined) {
|
||||
result.time_range = overrides.time_range;
|
||||
}
|
||||
if (overrides.granularity !== undefined) {
|
||||
result.granularity = overrides.granularity;
|
||||
}
|
||||
if (overrides.granularity_sqla !== undefined) {
|
||||
result.granularity = overrides.granularity_sqla;
|
||||
}
|
||||
|
||||
// Override extras properties
|
||||
if (result.extras) {
|
||||
if (overrides.relative_start !== undefined) {
|
||||
result.extras.relative_start = overrides.relative_start;
|
||||
}
|
||||
if (overrides.relative_end !== undefined) {
|
||||
result.extras.relative_end = overrides.relative_end;
|
||||
}
|
||||
if (overrides.time_grain_sqla !== undefined) {
|
||||
result.extras.time_grain_sqla = overrides.time_grain_sqla;
|
||||
}
|
||||
if (overrides.time_compare !== undefined) {
|
||||
result.extras.time_compare = overrides.time_compare;
|
||||
}
|
||||
}
|
||||
|
||||
return result;
|
||||
}
|
||||
72
sidecar-node/src/utils/processFilters.ts
Normal file
72
sidecar-node/src/utils/processFilters.ts
Normal file
@@ -0,0 +1,72 @@
|
||||
// Licensed to the Apache Software Foundation (ASF) under one
|
||||
// or more contributor license agreements. See the NOTICE file
|
||||
// distributed with this work for additional information
|
||||
// regarding copyright ownership. The ASF licenses this file
|
||||
// to you under the Apache License, Version 2.0 (the
|
||||
// "License"); you may not use this file except in compliance
|
||||
// with the License. You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing,
|
||||
// software distributed under the License is distributed on an
|
||||
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
// KIND, either express or implied. See the License for the
|
||||
// specific language governing permissions and limitations
|
||||
// under the License.
|
||||
|
||||
import {
|
||||
AdhocFilter,
|
||||
QueryObjectFilterClause,
|
||||
QueryFormData,
|
||||
QueryObjectExtras,
|
||||
} from '../types';
|
||||
|
||||
interface FilterProcessorInput extends QueryFormData {
|
||||
extras: QueryObjectExtras;
|
||||
filters: QueryObjectFilterClause[];
|
||||
adhoc_filters: AdhocFilter[];
|
||||
}
|
||||
|
||||
interface FilterProcessorResult {
|
||||
extras: QueryObjectExtras;
|
||||
filters: QueryObjectFilterClause[];
|
||||
}
|
||||
|
||||
/**
|
||||
* Process filters from form data into QueryObject format
|
||||
*/
|
||||
export default function processFilters(
|
||||
formData: FilterProcessorInput,
|
||||
): FilterProcessorResult {
|
||||
const { filters = [], adhoc_filters = [], extras = {} } = formData;
|
||||
|
||||
// Convert adhoc filters to simple filters where possible
|
||||
const processedFilters: QueryObjectFilterClause[] = [...filters];
|
||||
const processedExtras: QueryObjectExtras = { ...extras };
|
||||
|
||||
// Process adhoc filters
|
||||
for (const adhocFilter of adhoc_filters) {
|
||||
if (adhocFilter.expressionType === 'SIMPLE' && adhocFilter.subject && adhocFilter.operator) {
|
||||
// Convert simple adhoc filters to QueryObjectFilterClause
|
||||
if (typeof adhocFilter.subject === 'string') {
|
||||
processedFilters.push({
|
||||
col: adhocFilter.subject,
|
||||
op: adhocFilter.operator as any,
|
||||
val: adhocFilter.comparator,
|
||||
});
|
||||
}
|
||||
} else if (adhocFilter.expressionType === 'SQL' && adhocFilter.sqlExpression) {
|
||||
// Add SQL filters to WHERE clause in extras
|
||||
const clause = adhocFilter.clause === 'HAVING' ? 'having' : 'where';
|
||||
const existingClause = processedExtras[clause] || '';
|
||||
const separator = existingClause ? ' AND ' : '';
|
||||
processedExtras[clause] = existingClause + separator + `(${adhocFilter.sqlExpression})`;
|
||||
}
|
||||
}
|
||||
|
||||
return {
|
||||
extras: processedExtras,
|
||||
filters: processedFilters,
|
||||
};
|
||||
}
|
||||
34
sidecar-node/tsconfig.json
Normal file
34
sidecar-node/tsconfig.json
Normal file
@@ -0,0 +1,34 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2020",
|
||||
"lib": ["ES2020"],
|
||||
"module": "commonjs",
|
||||
"declaration": true,
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"strict": true,
|
||||
"noUnusedLocals": true,
|
||||
"noUnusedParameters": true,
|
||||
"noImplicitReturns": true,
|
||||
"noFallthroughCasesInSwitch": true,
|
||||
"moduleResolution": "node",
|
||||
"baseUrl": "./",
|
||||
"paths": {
|
||||
"@/*": ["src/*"]
|
||||
},
|
||||
"allowSyntheticDefaultImports": true,
|
||||
"esModuleInterop": true,
|
||||
"experimentalDecorators": true,
|
||||
"emitDecoratorMetadata": true,
|
||||
"skipLibCheck": true,
|
||||
"forceConsistentCasingInFileNames": true
|
||||
},
|
||||
"include": [
|
||||
"src/**/*"
|
||||
],
|
||||
"exclude": [
|
||||
"node_modules",
|
||||
"dist",
|
||||
"**/*.test.ts"
|
||||
]
|
||||
}
|
||||
@@ -77,6 +77,7 @@ from superset.utils.core import HeaderDataType, override_user, recipients_string
|
||||
from superset.utils.csv import get_chart_csv_data, get_chart_dataframe
|
||||
from superset.utils.decorators import logs_context, transaction
|
||||
from superset.utils.pdf import build_pdf_from_screenshots
|
||||
from superset.utils.query_sidecar import get_query_sidecar_client, QuerySidecarException
|
||||
from superset.utils.screenshots import ChartScreenshot, DashboardScreenshot
|
||||
from superset.utils.slack import get_channels_with_search, SlackChannelTypes
|
||||
from superset.utils.urls import get_url_path
|
||||
@@ -388,9 +389,7 @@ class BaseReportState:
|
||||
user = security_manager.find_user(username)
|
||||
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
||||
|
||||
if self._report_schedule.chart.query_context is None:
|
||||
logger.warning("No query context found, taking a screenshot to generate it")
|
||||
self._update_query_context()
|
||||
self._ensure_query_context_available()
|
||||
|
||||
try:
|
||||
logger.info("Getting chart from %s as user %s", url, user.username)
|
||||
@@ -418,9 +417,7 @@ class BaseReportState:
|
||||
user = security_manager.find_user(username)
|
||||
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
||||
|
||||
if self._report_schedule.chart.query_context is None:
|
||||
logger.warning("No query context found, taking a screenshot to generate it")
|
||||
self._update_query_context()
|
||||
self._ensure_query_context_available()
|
||||
|
||||
try:
|
||||
logger.info("Getting chart from %s as user %s", url, user.username)
|
||||
@@ -435,9 +432,79 @@ class BaseReportState:
|
||||
raise ReportScheduleCsvFailedError()
|
||||
return dataframe
|
||||
|
||||
def _update_query_context(self) -> None:
|
||||
def _ensure_query_context_available(self) -> None:
|
||||
"""
|
||||
Update chart query context.
|
||||
Ensure query context is available for the chart.
|
||||
|
||||
First attempts to generate query context using the sidecar service
|
||||
from the chart's form_data. If that fails, falls back to the
|
||||
screenshot method for backward compatibility.
|
||||
"""
|
||||
# Try sidecar service first if available
|
||||
if app.config.get("QUERY_SIDECAR_ENABLED", True):
|
||||
try:
|
||||
self._generate_query_context_via_sidecar()
|
||||
return
|
||||
except QuerySidecarException as ex:
|
||||
logger.warning(
|
||||
"Failed to generate query context via sidecar service: %s. "
|
||||
"Falling back to screenshot method.",
|
||||
str(ex),
|
||||
)
|
||||
|
||||
# Fallback to legacy screenshot method
|
||||
if self._report_schedule.chart.query_context is None:
|
||||
logger.warning("No query context found, taking a screenshot to generate it")
|
||||
self._update_query_context_legacy()
|
||||
|
||||
@transaction()
|
||||
def _generate_query_context_via_sidecar(self) -> None:
|
||||
"""
|
||||
Generate and save query context using the sidecar service.
|
||||
|
||||
This method uses the chart's form_data to generate a fresh
|
||||
QueryObject via the sidecar service, eliminating staleness issues.
|
||||
"""
|
||||
if not self._report_schedule.chart:
|
||||
raise QuerySidecarException("No chart associated with report schedule")
|
||||
|
||||
try:
|
||||
# Get the chart's form_data
|
||||
form_data = self._report_schedule.chart.form_data
|
||||
|
||||
# Use sidecar service to generate QueryObject
|
||||
sidecar_client = get_query_sidecar_client()
|
||||
query_object_data = sidecar_client.build_query_object(form_data)
|
||||
|
||||
# Convert to JSON and save as query_context
|
||||
query_context = json.dumps(
|
||||
{
|
||||
"queries": [query_object_data],
|
||||
"form_data": form_data,
|
||||
"result_format": "json",
|
||||
"result_type": "full",
|
||||
}
|
||||
)
|
||||
|
||||
# Update the chart's query_context in the database
|
||||
self._report_schedule.chart.query_context = query_context
|
||||
from superset import db
|
||||
|
||||
db.session.add(self._report_schedule.chart)
|
||||
|
||||
logger.info(
|
||||
"Successfully generated query context via sidecar for chart %s",
|
||||
self._report_schedule.chart.id,
|
||||
)
|
||||
|
||||
except Exception as ex:
|
||||
raise QuerySidecarException(
|
||||
f"Failed to generate query context via sidecar: {ex}"
|
||||
) from ex
|
||||
|
||||
def _update_query_context_legacy(self) -> None:
|
||||
"""
|
||||
Update chart query context using the legacy screenshot method.
|
||||
|
||||
To load CSV data from the endpoint the chart must have been saved
|
||||
with its query context. For charts without saved query context we
|
||||
|
||||
54
superset/config_sidecar.py
Normal file
54
superset/config_sidecar.py
Normal file
@@ -0,0 +1,54 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""
|
||||
Configuration settings for the Query Sidecar Service integration.
|
||||
|
||||
This file contains configuration options for integrating Superset with
|
||||
the Node.js Query Sidecar Service that generates QueryObjects from form_data.
|
||||
"""
|
||||
|
||||
# Query Sidecar Service Configuration
|
||||
# Enable or disable the query sidecar service integration
|
||||
QUERY_SIDECAR_ENABLED = True
|
||||
|
||||
# Base URL for the Node.js query sidecar service
|
||||
# This service transforms form_data into QueryObjects for Alerts & Reports
|
||||
QUERY_SIDECAR_BASE_URL = "http://localhost:3001"
|
||||
|
||||
# Timeout for sidecar service requests in seconds
|
||||
QUERY_SIDECAR_TIMEOUT = 10
|
||||
|
||||
# Example production configuration:
|
||||
# QUERY_SIDECAR_BASE_URL = "http://superset-query-sidecar:3001"
|
||||
# QUERY_SIDECAR_ENABLED = True
|
||||
# QUERY_SIDECAR_TIMEOUT = 30
|
||||
|
||||
# Example Docker Compose setup:
|
||||
# services:
|
||||
# superset-query-sidecar:
|
||||
# image: superset-query-sidecar:latest
|
||||
# ports:
|
||||
# - "3001:3001"
|
||||
# environment:
|
||||
# - NODE_ENV=production
|
||||
# - SUPERSET_ORIGINS=http://localhost:8088
|
||||
#
|
||||
# superset:
|
||||
# # ... your superset config
|
||||
# environment:
|
||||
# - QUERY_SIDECAR_BASE_URL=http://superset-query-sidecar:3001
|
||||
259
superset/utils/query_sidecar.py
Normal file
259
superset/utils/query_sidecar.py
Normal file
@@ -0,0 +1,259 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""
|
||||
Query Sidecar Client
|
||||
|
||||
Provides functionality to communicate with the Node.js sidecar service
|
||||
that generates QueryObjects from form_data, eliminating staleness issues
|
||||
in Alerts & Reports.
|
||||
"""
|
||||
|
||||
import logging
|
||||
from typing import Any, Dict, Optional
|
||||
from urllib.parse import urljoin
|
||||
|
||||
import requests
|
||||
from flask import current_app
|
||||
|
||||
from superset.common.query_object import QueryObject
|
||||
from superset.exceptions import SupersetException
|
||||
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
class QuerySidecarException(SupersetException):
|
||||
"""Exception raised when the query sidecar service fails."""
|
||||
|
||||
|
||||
class QuerySidecarClient:
|
||||
"""
|
||||
Client for communicating with the Node.js query sidecar service.
|
||||
|
||||
This client transforms form_data into QueryObject format using the same
|
||||
logic as the frontend, ensuring consistency and eliminating staleness.
|
||||
"""
|
||||
|
||||
def __init__(self, base_url: Optional[str] = None, timeout: int = 10) -> None:
|
||||
"""
|
||||
Initialize the query sidecar client.
|
||||
|
||||
Args:
|
||||
base_url: Base URL of the sidecar service. If None, reads from config.
|
||||
timeout: Request timeout in seconds.
|
||||
"""
|
||||
self._base_url = base_url or current_app.config.get(
|
||||
"QUERY_SIDECAR_BASE_URL", "http://localhost:3001"
|
||||
)
|
||||
self._timeout = timeout
|
||||
self._session = requests.Session()
|
||||
|
||||
# Set default headers
|
||||
self._session.headers.update(
|
||||
{
|
||||
"Content-Type": "application/json",
|
||||
"Accept": "application/json",
|
||||
}
|
||||
)
|
||||
|
||||
def build_query_object(
|
||||
self, form_data: Dict[str, Any], query_fields: Optional[Dict[str, str]] = None
|
||||
) -> Dict[str, Any]:
|
||||
"""
|
||||
Build a QueryObject from form_data using the sidecar service.
|
||||
|
||||
Args:
|
||||
form_data: The form data from Superset frontend/chart configuration
|
||||
query_fields: Optional query field aliases for visualization-specific
|
||||
mappings
|
||||
|
||||
Returns:
|
||||
Dict containing the QueryObject data
|
||||
|
||||
Raises:
|
||||
QuerySidecarException: If the service is unavailable or returns an error
|
||||
"""
|
||||
if not form_data:
|
||||
raise QuerySidecarException("form_data is required")
|
||||
|
||||
# Validate required form_data fields
|
||||
if not form_data.get("datasource") or not form_data.get("viz_type"):
|
||||
raise QuerySidecarException(
|
||||
"form_data must include datasource and viz_type"
|
||||
)
|
||||
|
||||
url = urljoin(self._base_url, "/api/v1/query-object")
|
||||
payload = {
|
||||
"form_data": form_data,
|
||||
"query_fields": query_fields,
|
||||
}
|
||||
|
||||
try:
|
||||
logger.debug("Calling sidecar service at %s", url)
|
||||
response = self._session.post(url, json=payload, timeout=self._timeout)
|
||||
response.raise_for_status()
|
||||
|
||||
data = response.json()
|
||||
if "error" in data:
|
||||
raise QuerySidecarException(f"Sidecar service error: {data['error']}")
|
||||
|
||||
return data["query_object"]
|
||||
|
||||
except requests.exceptions.Timeout as ex:
|
||||
logger.error("Timeout calling query sidecar service")
|
||||
raise QuerySidecarException(
|
||||
"Query sidecar service timeout. Please check service availability."
|
||||
) from ex
|
||||
except requests.exceptions.ConnectionError as ex:
|
||||
logger.error("Connection error calling query sidecar service")
|
||||
raise QuerySidecarException(
|
||||
"Unable to connect to query sidecar service. "
|
||||
"Please check service availability."
|
||||
) from ex
|
||||
except requests.exceptions.HTTPError as ex:
|
||||
logger.error("HTTP error calling query sidecar service: %s", ex)
|
||||
if ex.response.status_code >= 500:
|
||||
raise QuerySidecarException(
|
||||
"Query sidecar service is experiencing issues. "
|
||||
"Please try again later."
|
||||
) from ex
|
||||
else:
|
||||
try:
|
||||
error_data = ex.response.json()
|
||||
error_message = error_data.get("error", str(ex))
|
||||
except (ValueError, KeyError):
|
||||
error_message = str(ex)
|
||||
raise QuerySidecarException(
|
||||
f"Query sidecar service error: {error_message}"
|
||||
) from ex
|
||||
except Exception as ex:
|
||||
logger.error("Unexpected error calling query sidecar service: %s", ex)
|
||||
raise QuerySidecarException(
|
||||
f"Unexpected error communicating with query sidecar service: {ex}"
|
||||
) from ex
|
||||
|
||||
def create_query_object_from_form_data(
|
||||
self,
|
||||
form_data: Dict[str, Any],
|
||||
query_fields: Optional[Dict[str, str]] = None,
|
||||
datasource: Optional[Any] = None,
|
||||
) -> QueryObject:
|
||||
"""
|
||||
Create a QueryObject instance from form_data using the sidecar service.
|
||||
|
||||
This is a convenience method that returns a QueryObject instance
|
||||
rather than raw dictionary data.
|
||||
|
||||
Args:
|
||||
form_data: The form data from Superset frontend/chart configuration
|
||||
query_fields: Optional query field aliases for visualization-specific
|
||||
mappings
|
||||
datasource: Optional datasource instance to attach to the QueryObject
|
||||
|
||||
Returns:
|
||||
QueryObject instance
|
||||
|
||||
Raises:
|
||||
QuerySidecarException: If the service is unavailable or returns an error
|
||||
"""
|
||||
query_data = self.build_query_object(form_data, query_fields)
|
||||
|
||||
# Create QueryObject instance from the returned data
|
||||
# Convert the dictionary to QueryObject constructor parameters
|
||||
query_object = QueryObject(
|
||||
datasource=datasource,
|
||||
columns=query_data.get("columns"),
|
||||
metrics=query_data.get("metrics"),
|
||||
filters=query_data.get("filters", []),
|
||||
granularity=query_data.get("granularity"),
|
||||
extras=query_data.get("extras", {}),
|
||||
orderby=query_data.get("orderby", []),
|
||||
annotation_layers=query_data.get("annotation_layers", []),
|
||||
row_limit=query_data.get("row_limit"),
|
||||
row_offset=query_data.get("row_offset"),
|
||||
series_columns=query_data.get("series_columns"),
|
||||
series_limit=query_data.get("series_limit", 0),
|
||||
series_limit_metric=query_data.get("series_limit_metric"),
|
||||
order_desc=query_data.get("order_desc", True),
|
||||
time_range=query_data.get("time_range"),
|
||||
)
|
||||
|
||||
return query_object
|
||||
|
||||
def health_check(self) -> bool:
|
||||
"""
|
||||
Check if the sidecar service is healthy.
|
||||
|
||||
Returns:
|
||||
True if service is healthy, False otherwise
|
||||
"""
|
||||
try:
|
||||
url = urljoin(self._base_url, "/health")
|
||||
response = self._session.get(url, timeout=5)
|
||||
response.raise_for_status()
|
||||
|
||||
data = response.json()
|
||||
return data.get("status") == "healthy"
|
||||
|
||||
except Exception as ex:
|
||||
logger.warning("Query sidecar health check failed: %s", ex)
|
||||
return False
|
||||
|
||||
|
||||
# Global client instance
|
||||
_sidecar_client: Optional[QuerySidecarClient] = None
|
||||
|
||||
|
||||
def get_query_sidecar_client() -> QuerySidecarClient:
|
||||
"""
|
||||
Get the global query sidecar client instance.
|
||||
|
||||
Returns:
|
||||
QuerySidecarClient instance
|
||||
"""
|
||||
global _sidecar_client
|
||||
|
||||
if _sidecar_client is None:
|
||||
_sidecar_client = QuerySidecarClient()
|
||||
|
||||
return _sidecar_client
|
||||
|
||||
|
||||
def build_query_object_from_form_data(
|
||||
form_data: Dict[str, Any],
|
||||
query_fields: Optional[Dict[str, str]] = None,
|
||||
datasource: Optional[Any] = None,
|
||||
) -> QueryObject:
|
||||
"""
|
||||
Convenience function to build a QueryObject from form_data using the sidecar
|
||||
service.
|
||||
|
||||
Args:
|
||||
form_data: The form data from Superset frontend/chart configuration
|
||||
query_fields: Optional query field aliases for visualization-specific mappings
|
||||
datasource: Optional datasource instance to attach to the QueryObject
|
||||
|
||||
Returns:
|
||||
QueryObject instance
|
||||
|
||||
Raises:
|
||||
QuerySidecarException: If the service is unavailable or returns an error
|
||||
"""
|
||||
client = get_query_sidecar_client()
|
||||
return client.create_query_object_from_form_data(
|
||||
form_data, query_fields, datasource
|
||||
)
|
||||
195
tests/unit_tests/commands/report/test_execute_sidecar.py
Normal file
195
tests/unit_tests/commands/report/test_execute_sidecar.py
Normal file
@@ -0,0 +1,195 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
"""Tests for sidecar integration in report execution."""
|
||||
|
||||
from datetime import datetime
|
||||
from unittest.mock import MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
import pytest
|
||||
|
||||
from superset.commands.report.execute import BaseReportState
|
||||
from superset.reports.models import ReportSchedule
|
||||
from superset.utils.query_sidecar import QuerySidecarException
|
||||
|
||||
|
||||
class TestReportExecutionSidecar:
|
||||
"""Tests for sidecar service integration in report execution."""
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
@patch("superset.commands.report.execute.get_query_sidecar_client")
|
||||
@patch("superset.commands.report.execute.db")
|
||||
def test_generate_query_context_via_sidecar_success(
|
||||
self, mock_db, mock_get_client, mock_app
|
||||
):
|
||||
"""Test successful query context generation via sidecar."""
|
||||
# Setup mocks
|
||||
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||
mock_sidecar_client = MagicMock()
|
||||
mock_get_client.return_value = mock_sidecar_client
|
||||
mock_sidecar_client.build_query_object.return_value = {
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"filters": [],
|
||||
}
|
||||
|
||||
# Create test report schedule with chart
|
||||
mock_chart = MagicMock()
|
||||
mock_chart.id = 1
|
||||
mock_chart.form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
}
|
||||
mock_chart.query_context = None
|
||||
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = mock_chart
|
||||
|
||||
# Create BaseReportState instance
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
# Test the method
|
||||
report_state._generate_query_context_via_sidecar()
|
||||
|
||||
# Verify sidecar was called with correct form_data
|
||||
mock_sidecar_client.build_query_object.assert_called_once_with(
|
||||
mock_chart.form_data
|
||||
)
|
||||
|
||||
# Verify query_context was updated
|
||||
assert mock_chart.query_context is not None
|
||||
mock_db.session.commit.assert_called_once()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
def test_generate_query_context_via_sidecar_no_chart(self, mock_app):
|
||||
"""Test error when no chart is associated with report schedule."""
|
||||
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = None
|
||||
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="No chart associated with report schedule"
|
||||
):
|
||||
report_state._generate_query_context_via_sidecar()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
@patch("superset.commands.report.execute.get_query_sidecar_client")
|
||||
def test_generate_query_context_via_sidecar_client_error(
|
||||
self, mock_get_client, mock_app
|
||||
):
|
||||
"""Test handling of sidecar client errors."""
|
||||
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||
mock_sidecar_client = MagicMock()
|
||||
mock_get_client.return_value = mock_sidecar_client
|
||||
mock_sidecar_client.build_query_object.side_effect = Exception("Sidecar error")
|
||||
|
||||
mock_chart = MagicMock()
|
||||
mock_chart.form_data = {"datasource": "1__table", "viz_type": "table"}
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = mock_chart
|
||||
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="Failed to generate query context via sidecar"
|
||||
):
|
||||
report_state._generate_query_context_via_sidecar()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
@patch(
|
||||
"superset.commands.report.execute.BaseReportState._generate_query_context_via_sidecar"
|
||||
)
|
||||
def test_ensure_query_context_available_sidecar_enabled(
|
||||
self, mock_generate_sidecar, mock_app
|
||||
):
|
||||
"""Test _ensure_query_context_available when sidecar is enabled."""
|
||||
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
report_state._ensure_query_context_available()
|
||||
|
||||
mock_generate_sidecar.assert_called_once()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
@patch(
|
||||
"superset.commands.report.execute.BaseReportState._generate_query_context_via_sidecar"
|
||||
)
|
||||
@patch(
|
||||
"superset.commands.report.execute.BaseReportState._update_query_context_legacy"
|
||||
)
|
||||
def test_ensure_query_context_available_fallback(
|
||||
self, mock_legacy, mock_generate_sidecar, mock_app
|
||||
):
|
||||
"""Test fallback to legacy method when sidecar fails."""
|
||||
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||
mock_generate_sidecar.side_effect = QuerySidecarException("Sidecar failed")
|
||||
|
||||
mock_chart = MagicMock()
|
||||
mock_chart.query_context = None
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = mock_chart
|
||||
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
report_state._ensure_query_context_available()
|
||||
|
||||
mock_generate_sidecar.assert_called_once()
|
||||
mock_legacy.assert_called_once()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
@patch(
|
||||
"superset.commands.report.execute.BaseReportState._update_query_context_legacy"
|
||||
)
|
||||
def test_ensure_query_context_available_sidecar_disabled(
|
||||
self, mock_legacy, mock_app
|
||||
):
|
||||
"""Test _ensure_query_context_available when sidecar is disabled."""
|
||||
mock_app.config.get.return_value = False # QUERY_SIDECAR_ENABLED = False
|
||||
|
||||
mock_chart = MagicMock()
|
||||
mock_chart.query_context = None
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = mock_chart
|
||||
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
report_state._ensure_query_context_available()
|
||||
|
||||
mock_legacy.assert_called_once()
|
||||
|
||||
@patch("superset.commands.report.execute.app")
|
||||
def test_ensure_query_context_available_existing_context(self, mock_app):
|
||||
"""Test _ensure_query_context_available when query_context already exists."""
|
||||
mock_app.config.get.return_value = False # QUERY_SIDECAR_ENABLED = False
|
||||
|
||||
mock_chart = MagicMock()
|
||||
mock_chart.query_context = '{"queries": []}' # Existing context
|
||||
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||
mock_report_schedule.chart = mock_chart
|
||||
|
||||
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||
|
||||
# Should not raise any errors or call legacy method
|
||||
report_state._ensure_query_context_available()
|
||||
246
tests/unit_tests/utils/test_query_sidecar.py
Normal file
246
tests/unit_tests/utils/test_query_sidecar.py
Normal file
@@ -0,0 +1,246 @@
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing,
|
||||
# software distributed under the License is distributed on an
|
||||
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
# KIND, either express or implied. See the License for the
|
||||
# specific language governing permissions and limitations
|
||||
# under the License.
|
||||
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
from requests.exceptions import ConnectionError, HTTPError, Timeout
|
||||
|
||||
from superset.utils.query_sidecar import (
|
||||
build_query_object_from_form_data,
|
||||
QuerySidecarClient,
|
||||
QuerySidecarException,
|
||||
)
|
||||
|
||||
|
||||
class TestQuerySidecarClient:
|
||||
"""Tests for the QuerySidecarClient class."""
|
||||
|
||||
def test_init_default_config(self):
|
||||
"""Test client initialization with default configuration."""
|
||||
with patch("superset.utils.query_sidecar.current_app") as mock_app:
|
||||
mock_app.config.get.return_value = "http://test:3001"
|
||||
client = QuerySidecarClient()
|
||||
assert client._base_url == "http://test:3001"
|
||||
assert client._timeout == 10
|
||||
|
||||
def test_init_custom_config(self):
|
||||
"""Test client initialization with custom configuration."""
|
||||
client = QuerySidecarClient("http://custom:3000", timeout=30)
|
||||
assert client._base_url == "http://custom:3000"
|
||||
assert client._timeout == 30
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_build_query_object_success(self, mock_session_class):
|
||||
"""Test successful query object building."""
|
||||
# Mock the session and response
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {
|
||||
"query_object": {
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"filters": [],
|
||||
"extras": {},
|
||||
}
|
||||
}
|
||||
mock_session.post.return_value = mock_response
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
}
|
||||
|
||||
result = client.build_query_object(form_data)
|
||||
|
||||
assert result["metrics"] == ["count"]
|
||||
assert result["columns"] == ["name"]
|
||||
mock_session.post.assert_called_once()
|
||||
|
||||
def test_build_query_object_missing_form_data(self):
|
||||
"""Test error when form_data is missing."""
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
|
||||
with pytest.raises(QuerySidecarException, match="form_data is required"):
|
||||
client.build_query_object(None)
|
||||
|
||||
def test_build_query_object_invalid_form_data(self):
|
||||
"""Test error when form_data is missing required fields."""
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {"viz_type": "table"} # Missing datasource
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="must include datasource and viz_type"
|
||||
):
|
||||
client.build_query_object(form_data)
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_build_query_object_service_error(self, mock_session_class):
|
||||
"""Test handling of service errors."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"error": "Service error"}
|
||||
mock_session.post.return_value = mock_response
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
}
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="Sidecar service error: Service error"
|
||||
):
|
||||
client.build_query_object(form_data)
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_build_query_object_timeout(self, mock_session_class):
|
||||
"""Test handling of timeout errors."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_session.post.side_effect = Timeout("Request timeout")
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
}
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="Query sidecar service timeout"
|
||||
):
|
||||
client.build_query_object(form_data)
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_build_query_object_connection_error(self, mock_session_class):
|
||||
"""Test handling of connection errors."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_session.post.side_effect = ConnectionError("Connection failed")
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
}
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="Unable to connect to query sidecar service"
|
||||
):
|
||||
client.build_query_object(form_data)
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_build_query_object_http_error(self, mock_session_class):
|
||||
"""Test handling of HTTP errors."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_response = MagicMock()
|
||||
mock_response.status_code = 500
|
||||
mock_response.json.return_value = {"error": "Internal server error"}
|
||||
mock_http_error = HTTPError("500 Server Error")
|
||||
mock_http_error.response = mock_response
|
||||
mock_session.post.side_effect = mock_http_error
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
}
|
||||
|
||||
with pytest.raises(
|
||||
QuerySidecarException, match="Query sidecar service is experiencing issues"
|
||||
):
|
||||
client.build_query_object(form_data)
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_create_query_object_from_form_data(self, mock_session_class):
|
||||
"""Test creating QueryObject instance from form_data."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {
|
||||
"query_object": {
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
"filters": [],
|
||||
"extras": {},
|
||||
"row_limit": 1000,
|
||||
}
|
||||
}
|
||||
mock_session.post.return_value = mock_response
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
form_data = {
|
||||
"datasource": "1__table",
|
||||
"viz_type": "table",
|
||||
"metrics": ["count"],
|
||||
"columns": ["name"],
|
||||
}
|
||||
|
||||
query_object = client.create_query_object_from_form_data(form_data)
|
||||
|
||||
assert query_object.metrics == ["count"]
|
||||
assert query_object.columns == ["name"]
|
||||
assert query_object.row_limit == 1000
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_health_check_healthy(self, mock_session_class):
|
||||
"""Test health check when service is healthy."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_response = MagicMock()
|
||||
mock_response.json.return_value = {"status": "healthy"}
|
||||
mock_session.get.return_value = mock_response
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
assert client.health_check() is True
|
||||
|
||||
@patch("superset.utils.query_sidecar.requests.Session")
|
||||
def test_health_check_unhealthy(self, mock_session_class):
|
||||
"""Test health check when service is unhealthy."""
|
||||
mock_session = MagicMock()
|
||||
mock_session_class.return_value = mock_session
|
||||
mock_session.get.side_effect = ConnectionError("Connection failed")
|
||||
|
||||
client = QuerySidecarClient("http://test:3001")
|
||||
assert client.health_check() is False
|
||||
|
||||
|
||||
class TestModuleFunctions:
|
||||
"""Tests for module-level functions."""
|
||||
|
||||
@patch("superset.utils.query_sidecar.get_query_sidecar_client")
|
||||
def test_build_query_object_from_form_data(self, mock_get_client):
|
||||
"""Test the convenience function for building query objects."""
|
||||
mock_client = MagicMock()
|
||||
mock_get_client.return_value = mock_client
|
||||
mock_query_object = MagicMock()
|
||||
mock_client.create_query_object_from_form_data.return_value = mock_query_object
|
||||
|
||||
form_data = {"datasource": "1__table", "viz_type": "table"}
|
||||
|
||||
result = build_query_object_from_form_data(form_data)
|
||||
|
||||
assert result == mock_query_object
|
||||
mock_client.create_query_object_from_form_data.assert_called_once_with(
|
||||
form_data, None, None
|
||||
)
|
||||
Reference in New Issue
Block a user