mirror of
https://github.com/apache/superset.git
synced 2026-05-01 22:14:23 +00:00
Compare commits
3 Commits
docs/testi
...
sidecar
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
bc0e7b4984 | ||
|
|
d3c81fe6b4 | ||
|
|
df62bffadd |
348
SIDECAR_INTEGRATION.md
Normal file
348
SIDECAR_INTEGRATION.md
Normal file
@@ -0,0 +1,348 @@
|
|||||||
|
# Query Sidecar Service Integration
|
||||||
|
|
||||||
|
This document describes the Node.js Query Sidecar Service integration that eliminates stale QueryObject issues in Superset Alerts & Reports.
|
||||||
|
|
||||||
|
## Problem Statement
|
||||||
|
|
||||||
|
Previously, Superset stored QueryObjects in the database after chart visualization logic transformed `form_data` into QueryObject format. This approach had a critical flaw: when the JavaScript visualization code changed, the stored QueryObjects became stale, causing Alerts & Reports to use outdated query logic.
|
||||||
|
|
||||||
|
## Solution
|
||||||
|
|
||||||
|
The Query Sidecar Service provides a Node.js service that computes QueryObjects from `form_data` on-demand using the same logic as the frontend, ensuring:
|
||||||
|
|
||||||
|
- **No stale data**: QueryObjects are computed fresh every time
|
||||||
|
- **Consistency**: Uses identical logic to the Superset frontend
|
||||||
|
- **Backward compatibility**: Falls back to legacy screenshot method if sidecar is unavailable
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
graph TB
|
||||||
|
A[Superset Frontend] --> B[form_data]
|
||||||
|
B --> C[Chart Database Record]
|
||||||
|
|
||||||
|
D[Alerts & Reports] --> E{Sidecar Available?}
|
||||||
|
E -->|Yes| F[Query Sidecar Service]
|
||||||
|
E -->|No| G[Legacy Screenshot Method]
|
||||||
|
|
||||||
|
F --> H[buildQueryObject.ts Logic]
|
||||||
|
H --> I[Fresh QueryObject]
|
||||||
|
|
||||||
|
C --> F
|
||||||
|
I --> J[Chart Data API]
|
||||||
|
G --> J
|
||||||
|
```
|
||||||
|
|
||||||
|
## Components
|
||||||
|
|
||||||
|
### 1. Node.js Sidecar Service (`sidecar-node/`)
|
||||||
|
|
||||||
|
Located in `sidecar-node/`, this service provides:
|
||||||
|
|
||||||
|
- **REST API**: `POST /api/v1/query-object` to transform form_data
|
||||||
|
- **Type Safety**: Full TypeScript implementation with Superset type definitions
|
||||||
|
- **Frontend Compatibility**: Uses identical logic from `superset-ui-core`
|
||||||
|
- **Health Checks**: `/health` endpoint for monitoring
|
||||||
|
- **Docker Support**: Production-ready containerization
|
||||||
|
|
||||||
|
Key files:
|
||||||
|
- `src/query/buildQueryObject.ts` - Main transformation logic
|
||||||
|
- `src/types/index.ts` - TypeScript type definitions
|
||||||
|
- `src/routes/queryObject.ts` - REST API endpoints
|
||||||
|
- `Dockerfile` - Container configuration
|
||||||
|
|
||||||
|
### 2. Python Client (`superset/utils/query_sidecar.py`)
|
||||||
|
|
||||||
|
Python client library that:
|
||||||
|
|
||||||
|
- **HTTP Communication**: Handles requests to the sidecar service
|
||||||
|
- **Error Handling**: Robust error handling with fallback mechanisms
|
||||||
|
- **QueryObject Creation**: Converts responses to Superset QueryObject instances
|
||||||
|
- **Configuration**: Configurable timeouts and URLs
|
||||||
|
|
||||||
|
### 3. Reports Integration (`superset/commands/report/execute.py`)
|
||||||
|
|
||||||
|
Updated report execution logic:
|
||||||
|
|
||||||
|
- **Primary Method**: Uses sidecar service to generate fresh QueryObjects
|
||||||
|
- **Fallback Method**: Falls back to legacy screenshot method on failure
|
||||||
|
- **Configuration**: Controlled by `QUERY_SIDECAR_ENABLED` setting
|
||||||
|
|
||||||
|
## Setup and Configuration
|
||||||
|
|
||||||
|
### 1. Deploy the Sidecar Service
|
||||||
|
|
||||||
|
#### Development
|
||||||
|
```bash
|
||||||
|
cd sidecar-node
|
||||||
|
npm install
|
||||||
|
npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Production with Docker
|
||||||
|
```bash
|
||||||
|
cd sidecar-node
|
||||||
|
docker build -t superset-query-sidecar .
|
||||||
|
docker run -p 3001:3001 superset-query-sidecar
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Production with Docker Compose
|
||||||
|
```bash
|
||||||
|
docker-compose -f docker-compose.sidecar.yml up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Configure Superset
|
||||||
|
|
||||||
|
Add to your Superset configuration:
|
||||||
|
|
||||||
|
```python
|
||||||
|
# Enable sidecar service integration
|
||||||
|
QUERY_SIDECAR_ENABLED = True
|
||||||
|
|
||||||
|
# Sidecar service URL
|
||||||
|
QUERY_SIDECAR_BASE_URL = "http://localhost:3001"
|
||||||
|
|
||||||
|
# Request timeout (seconds)
|
||||||
|
QUERY_SIDECAR_TIMEOUT = 10
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Production Configuration Example
|
||||||
|
```python
|
||||||
|
QUERY_SIDECAR_ENABLED = True
|
||||||
|
QUERY_SIDECAR_BASE_URL = "http://superset-query-sidecar:3001"
|
||||||
|
QUERY_SIDECAR_TIMEOUT = 30
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Verify Integration
|
||||||
|
|
||||||
|
#### Check Sidecar Health
|
||||||
|
```bash
|
||||||
|
curl http://localhost:3001/health
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Test API Endpoint
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:3001/api/v1/query-object \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"form_data": {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"]
|
||||||
|
}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
### POST /api/v1/query-object
|
||||||
|
|
||||||
|
Transforms `form_data` into a QueryObject.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"form_data": {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"time_range": "No filter"
|
||||||
|
},
|
||||||
|
"query_fields": {
|
||||||
|
"x": "columns",
|
||||||
|
"y": "metrics"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"query_object": {
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"time_range": "No filter",
|
||||||
|
"filters": [],
|
||||||
|
"extras": {},
|
||||||
|
"row_limit": undefined,
|
||||||
|
"order_desc": true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Error Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"error": "form_data must include datasource and viz_type"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /health
|
||||||
|
|
||||||
|
Returns service health status.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "healthy",
|
||||||
|
"timestamp": "2023-12-01T12:00:00.000Z",
|
||||||
|
"version": "1.0.0"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Error Handling
|
||||||
|
|
||||||
|
The integration includes comprehensive error handling:
|
||||||
|
|
||||||
|
### Sidecar Service Errors
|
||||||
|
- **Connection Errors**: Falls back to legacy screenshot method
|
||||||
|
- **Timeout Errors**: Configurable timeout with fallback
|
||||||
|
- **Service Errors**: Logs errors and uses fallback method
|
||||||
|
|
||||||
|
### Configuration
|
||||||
|
```python
|
||||||
|
# Disable sidecar to use legacy method only
|
||||||
|
QUERY_SIDECAR_ENABLED = False
|
||||||
|
|
||||||
|
# Increase timeout for slow networks
|
||||||
|
QUERY_SIDECAR_TIMEOUT = 30
|
||||||
|
```
|
||||||
|
|
||||||
|
## Migration Strategy
|
||||||
|
|
||||||
|
### Phase 1: Parallel Running
|
||||||
|
1. Deploy sidecar service alongside existing Superset
|
||||||
|
2. Enable `QUERY_SIDECAR_ENABLED = True`
|
||||||
|
3. Monitor logs for any fallback usage
|
||||||
|
|
||||||
|
### Phase 2: Validation
|
||||||
|
1. Compare QueryObjects generated by sidecar vs stored versions
|
||||||
|
2. Verify Alerts & Reports work correctly
|
||||||
|
3. Monitor performance metrics
|
||||||
|
|
||||||
|
### Phase 3: Full Migration
|
||||||
|
1. Remove dependency on stored query_context (future)
|
||||||
|
2. Optimize sidecar performance
|
||||||
|
3. Scale sidecar service as needed
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
### Node.js Service Tests
|
||||||
|
```bash
|
||||||
|
cd sidecar-node
|
||||||
|
npm test
|
||||||
|
npm run test:watch
|
||||||
|
```
|
||||||
|
|
||||||
|
### Python Integration Tests
|
||||||
|
```bash
|
||||||
|
pytest tests/unit_tests/utils/test_query_sidecar.py
|
||||||
|
pytest tests/unit_tests/commands/report/test_execute_sidecar.py
|
||||||
|
```
|
||||||
|
|
||||||
|
### Integration Testing
|
||||||
|
1. Create test alerts/reports
|
||||||
|
2. Verify they execute with fresh QueryObjects
|
||||||
|
3. Test fallback behavior by stopping sidecar service
|
||||||
|
|
||||||
|
## Monitoring and Logging
|
||||||
|
|
||||||
|
### Service Monitoring
|
||||||
|
- Health check endpoint: `GET /health`
|
||||||
|
- Docker health checks included
|
||||||
|
- Prometheus metrics (future enhancement)
|
||||||
|
|
||||||
|
### Superset Logging
|
||||||
|
The integration adds detailed logging:
|
||||||
|
|
||||||
|
```python
|
||||||
|
logger.info("Successfully generated query context via sidecar for chart %s", chart_id)
|
||||||
|
logger.warning("Failed to generate query context via sidecar service: %s. Falling back to screenshot method.", error)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Log Levels
|
||||||
|
- `INFO`: Successful sidecar operations
|
||||||
|
- `WARNING`: Fallback to legacy method
|
||||||
|
- `ERROR`: Critical failures
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
### Latency
|
||||||
|
- Sidecar adds ~10-50ms per request
|
||||||
|
- Network latency between services
|
||||||
|
- Configurable timeout prevents blocking
|
||||||
|
|
||||||
|
### Scaling
|
||||||
|
- Sidecar service is stateless and can be horizontally scaled
|
||||||
|
- Consider load balancing for high-volume deployments
|
||||||
|
- Cache QueryObjects if needed (future enhancement)
|
||||||
|
|
||||||
|
### Resource Usage
|
||||||
|
- Node.js service: ~50-100MB RAM
|
||||||
|
- CPU usage minimal for typical workloads
|
||||||
|
- Network bandwidth: ~1-10KB per request
|
||||||
|
|
||||||
|
## Future Enhancements
|
||||||
|
|
||||||
|
1. **Caching**: Add Redis/Memcached for QueryObject caching
|
||||||
|
2. **Metrics**: Prometheus metrics for monitoring
|
||||||
|
3. **Load Balancing**: Support multiple sidecar instances
|
||||||
|
4. **Query Optimization**: Optimize query generation performance
|
||||||
|
5. **Database Cleanup**: Remove stored query_context column (breaking change)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
#### Sidecar Service Not Starting
|
||||||
|
```bash
|
||||||
|
# Check logs
|
||||||
|
docker logs superset-query-sidecar
|
||||||
|
|
||||||
|
# Verify port availability
|
||||||
|
netstat -tlnp | grep 3001
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Connection Errors
|
||||||
|
```bash
|
||||||
|
# Test connectivity
|
||||||
|
curl http://localhost:3001/health
|
||||||
|
|
||||||
|
# Check Superset logs
|
||||||
|
grep -i "sidecar" /var/log/superset/superset.log
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Fallback Behavior
|
||||||
|
```bash
|
||||||
|
# Check if sidecar is being bypassed
|
||||||
|
grep -i "falling back" /var/log/superset/superset.log
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Issues
|
||||||
|
- Verify `QUERY_SIDECAR_BASE_URL` is correct
|
||||||
|
- Check network connectivity between services
|
||||||
|
- Ensure sidecar service is healthy
|
||||||
|
|
||||||
|
### Performance Issues
|
||||||
|
- Increase `QUERY_SIDECAR_TIMEOUT` if needed
|
||||||
|
- Monitor sidecar service resource usage
|
||||||
|
- Consider scaling sidecar service
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
### Network Security
|
||||||
|
- Run sidecar service on internal network
|
||||||
|
- Use HTTPS in production environments
|
||||||
|
- Configure proper firewall rules
|
||||||
|
|
||||||
|
### Input Validation
|
||||||
|
- Sidecar service validates all inputs
|
||||||
|
- Superset client validates responses
|
||||||
|
- No raw SQL execution in sidecar service
|
||||||
|
|
||||||
|
### Access Control
|
||||||
|
- Service-to-service authentication (future)
|
||||||
|
- Network-level access controls
|
||||||
|
- Audit logging of sidecar requests
|
||||||
60
docker-compose.sidecar.yml
Normal file
60
docker-compose.sidecar.yml
Normal file
@@ -0,0 +1,60 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
superset-query-sidecar:
|
||||||
|
build:
|
||||||
|
context: ./sidecar-node
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
container_name: superset-query-sidecar
|
||||||
|
ports:
|
||||||
|
- "3001:3001"
|
||||||
|
environment:
|
||||||
|
- NODE_ENV=production
|
||||||
|
- HOST=0.0.0.0
|
||||||
|
- PORT=3001
|
||||||
|
- SUPERSET_ORIGINS=http://localhost:8088,http://superset:8088
|
||||||
|
healthcheck:
|
||||||
|
test: ["CMD", "node", "-e", "require('http').get('http://localhost:3001/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1) }).on('error', () => process.exit(1))"]
|
||||||
|
interval: 30s
|
||||||
|
timeout: 10s
|
||||||
|
retries: 3
|
||||||
|
start_period: 10s
|
||||||
|
restart: unless-stopped
|
||||||
|
networks:
|
||||||
|
- superset
|
||||||
|
|
||||||
|
# Example integration with existing Superset service
|
||||||
|
# Uncomment and modify according to your setup
|
||||||
|
#
|
||||||
|
# superset:
|
||||||
|
# # ... your existing superset configuration
|
||||||
|
# environment:
|
||||||
|
# - QUERY_SIDECAR_ENABLED=true
|
||||||
|
# - QUERY_SIDECAR_BASE_URL=http://superset-query-sidecar:3001
|
||||||
|
# - QUERY_SIDECAR_TIMEOUT=30
|
||||||
|
# depends_on:
|
||||||
|
# superset-query-sidecar:
|
||||||
|
# condition: service_healthy
|
||||||
|
# networks:
|
||||||
|
# - superset
|
||||||
|
|
||||||
|
networks:
|
||||||
|
superset:
|
||||||
|
external: true
|
||||||
51
sidecar-node/Dockerfile
Normal file
51
sidecar-node/Dockerfile
Normal file
@@ -0,0 +1,51 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
FROM node:18-alpine
|
||||||
|
|
||||||
|
# Set working directory
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
|
# Copy package files
|
||||||
|
COPY package*.json ./
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
RUN npm ci --only=production
|
||||||
|
|
||||||
|
# Copy source code
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
# Build the application
|
||||||
|
RUN npm run build
|
||||||
|
|
||||||
|
# Create non-root user
|
||||||
|
RUN addgroup -g 1001 -S nodejs && \
|
||||||
|
adduser -S sidecar -u 1001
|
||||||
|
|
||||||
|
# Change ownership of the app directory
|
||||||
|
RUN chown -R sidecar:nodejs /app
|
||||||
|
USER sidecar
|
||||||
|
|
||||||
|
# Expose port
|
||||||
|
EXPOSE 3001
|
||||||
|
|
||||||
|
# Health check
|
||||||
|
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
|
||||||
|
CMD node -e "require('http').get('http://localhost:3001/health', (res) => { process.exit(res.statusCode === 200 ? 0 : 1) }).on('error', () => process.exit(1))"
|
||||||
|
|
||||||
|
# Start the application
|
||||||
|
CMD ["npm", "start"]
|
||||||
143
sidecar-node/README.md
Normal file
143
sidecar-node/README.md
Normal file
@@ -0,0 +1,143 @@
|
|||||||
|
# Superset Query Sidecar Service
|
||||||
|
|
||||||
|
A Node.js sidecar service that computes Superset QueryObjects from form_data, eliminating the need to store stale query objects in the database for Alerts & Reports.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
This service provides a REST API that transforms Superset's `form_data` into `QueryObject` format using the same logic as the frontend, ensuring consistency and eliminating staleness issues in Alerts & Reports.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- **Real-time QueryObject generation**: No stale data from database
|
||||||
|
- **Frontend-compatible logic**: Uses the same transformation logic as superset-ui-core
|
||||||
|
- **TypeScript support**: Full type safety
|
||||||
|
- **Docker support**: Easy deployment
|
||||||
|
- **Health checks**: Built-in monitoring endpoints
|
||||||
|
- **CORS support**: Configurable for Superset integration
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Development
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Install dependencies
|
||||||
|
npm install
|
||||||
|
|
||||||
|
# Start development server
|
||||||
|
npm run dev
|
||||||
|
|
||||||
|
# Run tests
|
||||||
|
npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
### Production
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build the application
|
||||||
|
npm run build
|
||||||
|
|
||||||
|
# Start production server
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
### Docker
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build Docker image
|
||||||
|
docker build -t superset-query-sidecar .
|
||||||
|
|
||||||
|
# Run container
|
||||||
|
docker run -p 3001:3001 superset-query-sidecar
|
||||||
|
```
|
||||||
|
|
||||||
|
## API Reference
|
||||||
|
|
||||||
|
### POST /api/v1/query-object
|
||||||
|
|
||||||
|
Transforms form_data into a QueryObject.
|
||||||
|
|
||||||
|
**Request:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"form_data": {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"time_range": "No filter"
|
||||||
|
},
|
||||||
|
"query_fields": {
|
||||||
|
"x": "columns",
|
||||||
|
"y": "metrics"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"query_object": {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"time_range": "No filter",
|
||||||
|
"filters": [],
|
||||||
|
"extras": {}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /health
|
||||||
|
|
||||||
|
Health check endpoint.
|
||||||
|
|
||||||
|
**Response:**
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"status": "healthy",
|
||||||
|
"timestamp": "2023-12-01T12:00:00.000Z",
|
||||||
|
"version": "1.0.0"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
Environment variables:
|
||||||
|
|
||||||
|
- `PORT`: Server port (default: 3001)
|
||||||
|
- `HOST`: Server host (default: localhost)
|
||||||
|
- `NODE_ENV`: Environment mode (development/production)
|
||||||
|
- `SUPERSET_ORIGINS`: Allowed CORS origins (comma-separated)
|
||||||
|
|
||||||
|
## Integration with Superset
|
||||||
|
|
||||||
|
This service is designed to be called by Superset's Python backend to generate QueryObjects for Alerts & Reports, replacing the current approach of reading stale query objects from the database.
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────┐ ┌──────────────────────┐ ┌─────────────────────┐
|
||||||
|
│ Superset │ │ Query Sidecar │ │ Alerts & Reports │
|
||||||
|
│ Frontend │ │ Service │ │ │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
│ form_data ────┼────┼──► buildQueryObject │◄───┼─── Python Client │
|
||||||
|
│ │ │ │ │ │
|
||||||
|
└─────────────────┘ └──────────────────────┘ └─────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run all tests
|
||||||
|
npm test
|
||||||
|
|
||||||
|
# Run tests in watch mode
|
||||||
|
npm run test:watch
|
||||||
|
|
||||||
|
# Run tests with coverage
|
||||||
|
npm test -- --coverage
|
||||||
|
```
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
Licensed under the Apache License, Version 2.0.
|
||||||
16
sidecar-node/jest.config.js
Normal file
16
sidecar-node/jest.config.js
Normal file
@@ -0,0 +1,16 @@
|
|||||||
|
module.exports = {
|
||||||
|
preset: 'ts-jest',
|
||||||
|
testEnvironment: 'node',
|
||||||
|
roots: ['<rootDir>/src'],
|
||||||
|
testMatch: ['**/__tests__/**/*.test.ts', '**/?(*.)+(spec|test).ts'],
|
||||||
|
transform: {
|
||||||
|
'^.+\\.ts$': 'ts-jest',
|
||||||
|
},
|
||||||
|
collectCoverageFrom: [
|
||||||
|
'src/**/*.ts',
|
||||||
|
'!src/**/*.d.ts',
|
||||||
|
'!src/**/*.test.ts',
|
||||||
|
],
|
||||||
|
coverageDirectory: 'coverage',
|
||||||
|
coverageReporters: ['text', 'lcov', 'html'],
|
||||||
|
};
|
||||||
41
sidecar-node/package.json
Normal file
41
sidecar-node/package.json
Normal file
@@ -0,0 +1,41 @@
|
|||||||
|
{
|
||||||
|
"name": "superset-query-sidecar",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"description": "Node.js sidecar service for computing Superset QueryObjects from form_data",
|
||||||
|
"main": "dist/index.js",
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsc",
|
||||||
|
"start": "node dist/index.js",
|
||||||
|
"dev": "ts-node src/index.ts",
|
||||||
|
"test": "jest",
|
||||||
|
"test:watch": "jest --watch"
|
||||||
|
},
|
||||||
|
"dependencies": {
|
||||||
|
"express": "^4.18.2",
|
||||||
|
"cors": "^2.8.5",
|
||||||
|
"helmet": "^7.1.0",
|
||||||
|
"compression": "^1.7.4"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/express": "^4.17.21",
|
||||||
|
"@types/node": "^20.10.0",
|
||||||
|
"@types/cors": "^2.8.17",
|
||||||
|
"@types/compression": "^1.7.5",
|
||||||
|
"@types/jest": "^29.5.8",
|
||||||
|
"typescript": "^5.3.2",
|
||||||
|
"ts-node": "^10.9.1",
|
||||||
|
"jest": "^29.7.0",
|
||||||
|
"ts-jest": "^29.1.1"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18.0.0"
|
||||||
|
},
|
||||||
|
"keywords": [
|
||||||
|
"superset",
|
||||||
|
"query",
|
||||||
|
"sidecar",
|
||||||
|
"analytics"
|
||||||
|
],
|
||||||
|
"author": "Apache Superset",
|
||||||
|
"license": "Apache-2.0"
|
||||||
|
}
|
||||||
149
sidecar-node/src/__tests__/buildQueryObject.test.ts
Normal file
149
sidecar-node/src/__tests__/buildQueryObject.test.ts
Normal file
@@ -0,0 +1,149 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import buildQueryObject from '../query/buildQueryObject';
|
||||||
|
import { QueryFormData } from '../types';
|
||||||
|
|
||||||
|
describe('buildQueryObject', () => {
|
||||||
|
const baseFormData: QueryFormData = {
|
||||||
|
datasource: '1__table',
|
||||||
|
viz_type: 'table',
|
||||||
|
time_range: 'No filter',
|
||||||
|
metrics: ['count'],
|
||||||
|
columns: ['name', 'category'],
|
||||||
|
row_limit: 1000,
|
||||||
|
order_desc: true,
|
||||||
|
};
|
||||||
|
|
||||||
|
it('should build a basic query object', () => {
|
||||||
|
const queryObject = buildQueryObject(baseFormData);
|
||||||
|
|
||||||
|
expect(queryObject).toEqual({
|
||||||
|
time_range: 'No filter',
|
||||||
|
since: undefined,
|
||||||
|
until: undefined,
|
||||||
|
granularity: undefined,
|
||||||
|
columns: ['name', 'category'],
|
||||||
|
metrics: ['count'],
|
||||||
|
orderby: undefined,
|
||||||
|
annotation_layers: [],
|
||||||
|
row_limit: 1000,
|
||||||
|
row_offset: undefined,
|
||||||
|
series_columns: undefined,
|
||||||
|
series_limit: 0,
|
||||||
|
series_limit_metric: undefined,
|
||||||
|
group_others_when_limit_reached: false,
|
||||||
|
order_desc: true,
|
||||||
|
url_params: undefined,
|
||||||
|
custom_params: {},
|
||||||
|
extras: {
|
||||||
|
filters: [],
|
||||||
|
},
|
||||||
|
filters: [],
|
||||||
|
custom_form_data: {},
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle adhoc filters', () => {
|
||||||
|
const formDataWithFilters: QueryFormData = {
|
||||||
|
...baseFormData,
|
||||||
|
adhoc_filters: [
|
||||||
|
{
|
||||||
|
clause: 'WHERE',
|
||||||
|
expressionType: 'SIMPLE',
|
||||||
|
subject: 'category',
|
||||||
|
operator: '==',
|
||||||
|
comparator: 'Electronics',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(formDataWithFilters);
|
||||||
|
|
||||||
|
expect(queryObject.filters).toContainEqual({
|
||||||
|
col: 'category',
|
||||||
|
op: '==',
|
||||||
|
val: 'Electronics',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle SQL adhoc filters in extras', () => {
|
||||||
|
const formDataWithSQLFilters: QueryFormData = {
|
||||||
|
...baseFormData,
|
||||||
|
adhoc_filters: [
|
||||||
|
{
|
||||||
|
clause: 'WHERE',
|
||||||
|
expressionType: 'SQL',
|
||||||
|
sqlExpression: 'price > 100',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
};
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(formDataWithSQLFilters);
|
||||||
|
|
||||||
|
expect(queryObject.extras?.where).toBe('(price > 100)');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle extra_form_data overrides', () => {
|
||||||
|
const formDataWithExtraFormData: QueryFormData = {
|
||||||
|
...baseFormData,
|
||||||
|
extra_form_data: {
|
||||||
|
time_range: 'Last week',
|
||||||
|
adhoc_filters: [
|
||||||
|
{
|
||||||
|
clause: 'WHERE',
|
||||||
|
expressionType: 'SIMPLE',
|
||||||
|
subject: 'status',
|
||||||
|
operator: '==',
|
||||||
|
comparator: 'active',
|
||||||
|
},
|
||||||
|
],
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(formDataWithExtraFormData);
|
||||||
|
|
||||||
|
expect(queryObject.time_range).toBe('Last week');
|
||||||
|
expect(queryObject.filters).toContainEqual({
|
||||||
|
col: 'status',
|
||||||
|
op: '==',
|
||||||
|
val: 'active',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle series_limit from limit field', () => {
|
||||||
|
const formDataWithLimit: QueryFormData = {
|
||||||
|
...baseFormData,
|
||||||
|
limit: 5,
|
||||||
|
};
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(formDataWithLimit);
|
||||||
|
|
||||||
|
expect(queryObject.series_limit).toBe(5);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('should handle granularity', () => {
|
||||||
|
const formDataWithGranularity: QueryFormData = {
|
||||||
|
...baseFormData,
|
||||||
|
granularity: 'created_at',
|
||||||
|
};
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(formDataWithGranularity);
|
||||||
|
|
||||||
|
expect(queryObject.granularity).toBe('created_at');
|
||||||
|
});
|
||||||
|
});
|
||||||
93
sidecar-node/src/index.ts
Normal file
93
sidecar-node/src/index.ts
Normal file
@@ -0,0 +1,93 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import express from 'express';
|
||||||
|
import cors from 'cors';
|
||||||
|
import helmet from 'helmet';
|
||||||
|
import compression from 'compression';
|
||||||
|
import queryObjectRouter from './routes/queryObject';
|
||||||
|
|
||||||
|
const app = express();
|
||||||
|
const PORT = process.env.PORT || 3001;
|
||||||
|
const HOST = process.env.HOST || 'localhost';
|
||||||
|
|
||||||
|
// Security middleware
|
||||||
|
app.use(helmet());
|
||||||
|
|
||||||
|
// Enable CORS for Superset backend
|
||||||
|
app.use(cors({
|
||||||
|
origin: process.env.SUPERSET_ORIGINS?.split(',') || ['http://localhost:8088'],
|
||||||
|
credentials: true,
|
||||||
|
}));
|
||||||
|
|
||||||
|
// Compression middleware
|
||||||
|
app.use(compression());
|
||||||
|
|
||||||
|
// Body parsing middleware
|
||||||
|
app.use(express.json({ limit: '10mb' }));
|
||||||
|
app.use(express.urlencoded({ extended: true, limit: '10mb' }));
|
||||||
|
|
||||||
|
// Request logging middleware
|
||||||
|
app.use((req, res, next) => {
|
||||||
|
const timestamp = new Date().toISOString();
|
||||||
|
console.log(`${timestamp} ${req.method} ${req.path}`);
|
||||||
|
next();
|
||||||
|
});
|
||||||
|
|
||||||
|
// Routes
|
||||||
|
app.use(queryObjectRouter);
|
||||||
|
|
||||||
|
// Global error handler
|
||||||
|
app.use((err: Error, req: express.Request, res: express.Response, next: express.NextFunction) => {
|
||||||
|
console.error('Unhandled error:', err);
|
||||||
|
res.status(500).json({
|
||||||
|
error: 'Internal server error',
|
||||||
|
message: process.env.NODE_ENV === 'development' ? err.message : undefined,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// 404 handler
|
||||||
|
app.use((req, res) => {
|
||||||
|
res.status(404).json({
|
||||||
|
error: 'Not found',
|
||||||
|
path: req.path,
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
// Start server
|
||||||
|
const server = app.listen(PORT, HOST, () => {
|
||||||
|
console.log(`🚀 Superset Query Sidecar Service running on http://${HOST}:${PORT}`);
|
||||||
|
console.log(`📊 Health check available at http://${HOST}:${PORT}/health`);
|
||||||
|
console.log(`🔧 Query object API available at http://${HOST}:${PORT}/api/v1/query-object`);
|
||||||
|
});
|
||||||
|
|
||||||
|
// Graceful shutdown
|
||||||
|
process.on('SIGTERM', () => {
|
||||||
|
console.log('SIGTERM received, shutting down gracefully');
|
||||||
|
server.close(() => {
|
||||||
|
console.log('Process terminated');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
process.on('SIGINT', () => {
|
||||||
|
console.log('SIGINT received, shutting down gracefully');
|
||||||
|
server.close(() => {
|
||||||
|
console.log('Process terminated');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
export default app;
|
||||||
134
sidecar-node/src/query/buildQueryObject.ts
Normal file
134
sidecar-node/src/query/buildQueryObject.ts
Normal file
@@ -0,0 +1,134 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
/* eslint-disable camelcase */
|
||||||
|
import {
|
||||||
|
QueryObject,
|
||||||
|
QueryFormData,
|
||||||
|
QueryFieldAliases,
|
||||||
|
isQueryFormMetric,
|
||||||
|
isDefined,
|
||||||
|
} from '../types';
|
||||||
|
import processFilters from '../utils/processFilters';
|
||||||
|
import extractExtras from '../utils/extractExtras';
|
||||||
|
import extractQueryFields from '../utils/extractQueryFields';
|
||||||
|
import { overrideExtraFormData } from '../utils/processExtraFormData';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build the common segments of all query objects (e.g. the granularity field derived from
|
||||||
|
* SQLAlchemy). The segments specific to each viz type is constructed in the
|
||||||
|
* buildQuery method for each viz type (see `wordcloud/buildQuery.ts` for an example).
|
||||||
|
* Note the type of the formData argument passed in here is the type of the formData for a
|
||||||
|
* specific viz, which is a subtype of the generic formData shared among all viz types.
|
||||||
|
*/
|
||||||
|
export default function buildQueryObject<T extends QueryFormData>(
|
||||||
|
formData: T,
|
||||||
|
queryFields?: QueryFieldAliases,
|
||||||
|
): QueryObject {
|
||||||
|
const {
|
||||||
|
annotation_layers = [],
|
||||||
|
extra_form_data,
|
||||||
|
time_range,
|
||||||
|
since,
|
||||||
|
until,
|
||||||
|
row_limit,
|
||||||
|
row_offset,
|
||||||
|
order_desc,
|
||||||
|
limit,
|
||||||
|
timeseries_limit_metric,
|
||||||
|
granularity,
|
||||||
|
url_params = {},
|
||||||
|
custom_params = {},
|
||||||
|
series_columns,
|
||||||
|
series_limit,
|
||||||
|
series_limit_metric,
|
||||||
|
group_others_when_limit_reached,
|
||||||
|
...residualFormData
|
||||||
|
} = formData;
|
||||||
|
|
||||||
|
const {
|
||||||
|
adhoc_filters: appendAdhocFilters = [],
|
||||||
|
filters: appendFilters = [],
|
||||||
|
custom_form_data = {},
|
||||||
|
...overrides
|
||||||
|
} = extra_form_data || {};
|
||||||
|
|
||||||
|
const numericRowLimit = Number(row_limit);
|
||||||
|
const numericRowOffset = Number(row_offset);
|
||||||
|
|
||||||
|
const { metrics, columns, orderby } = extractQueryFields(
|
||||||
|
residualFormData,
|
||||||
|
queryFields,
|
||||||
|
);
|
||||||
|
|
||||||
|
// collect all filters for conversion to simple filters/freeform clauses
|
||||||
|
const extras = extractExtras(formData);
|
||||||
|
const { filters: extraFilters } = extras;
|
||||||
|
const filterFormData = {
|
||||||
|
...formData,
|
||||||
|
...extras,
|
||||||
|
filters: [...extraFilters, ...appendFilters],
|
||||||
|
adhoc_filters: [...(formData.adhoc_filters || []), ...appendAdhocFilters],
|
||||||
|
};
|
||||||
|
|
||||||
|
const extrasAndfilters = processFilters(filterFormData);
|
||||||
|
|
||||||
|
const normalizeSeriesLimitMetric = (metric: any) => {
|
||||||
|
if (isQueryFormMetric(metric)) {
|
||||||
|
return metric;
|
||||||
|
}
|
||||||
|
return undefined;
|
||||||
|
};
|
||||||
|
|
||||||
|
let queryObject: QueryObject = {
|
||||||
|
// fallback `null` to `undefined` so they won't be sent to the backend
|
||||||
|
// (JSON.stringify will ignore `undefined`.)
|
||||||
|
time_range: time_range || undefined,
|
||||||
|
since: since || undefined,
|
||||||
|
until: until || undefined,
|
||||||
|
granularity: granularity || undefined,
|
||||||
|
...extras,
|
||||||
|
...extrasAndfilters,
|
||||||
|
columns,
|
||||||
|
metrics,
|
||||||
|
orderby,
|
||||||
|
annotation_layers,
|
||||||
|
row_limit:
|
||||||
|
row_limit == null || Number.isNaN(numericRowLimit)
|
||||||
|
? undefined
|
||||||
|
: numericRowLimit,
|
||||||
|
row_offset:
|
||||||
|
row_offset == null || Number.isNaN(numericRowOffset)
|
||||||
|
? undefined
|
||||||
|
: numericRowOffset,
|
||||||
|
series_columns,
|
||||||
|
series_limit: series_limit ?? (isDefined(limit) ? Number(limit) : 0),
|
||||||
|
series_limit_metric:
|
||||||
|
normalizeSeriesLimitMetric(series_limit_metric) ??
|
||||||
|
timeseries_limit_metric ??
|
||||||
|
undefined,
|
||||||
|
group_others_when_limit_reached: group_others_when_limit_reached ?? false,
|
||||||
|
order_desc: typeof order_desc === 'undefined' ? true : order_desc,
|
||||||
|
url_params: url_params || undefined,
|
||||||
|
custom_params,
|
||||||
|
};
|
||||||
|
|
||||||
|
// override extra form data used by native and cross filters
|
||||||
|
queryObject = overrideExtraFormData(queryObject, overrides);
|
||||||
|
|
||||||
|
return { ...queryObject, custom_form_data };
|
||||||
|
}
|
||||||
91
sidecar-node/src/routes/queryObject.ts
Normal file
91
sidecar-node/src/routes/queryObject.ts
Normal file
@@ -0,0 +1,91 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import { Request, Response, Router } from 'express';
|
||||||
|
import buildQueryObject from '../query/buildQueryObject';
|
||||||
|
import { QueryFormData, QueryFieldAliases } from '../types';
|
||||||
|
|
||||||
|
const router = Router();
|
||||||
|
|
||||||
|
interface BuildQueryObjectRequest {
|
||||||
|
form_data: QueryFormData;
|
||||||
|
query_fields?: QueryFieldAliases;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface BuildQueryObjectResponse {
|
||||||
|
query_object: any;
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* POST /api/v1/query-object
|
||||||
|
*
|
||||||
|
* Build a QueryObject from form_data
|
||||||
|
*
|
||||||
|
* Request body:
|
||||||
|
* - form_data: The form data from Superset frontend
|
||||||
|
* - query_fields: Optional query field aliases for visualization-specific mappings
|
||||||
|
*
|
||||||
|
* Response:
|
||||||
|
* - query_object: The computed QueryObject
|
||||||
|
* - error: Error message if processing failed
|
||||||
|
*/
|
||||||
|
router.post('/api/v1/query-object', (req: Request, res: Response) => {
|
||||||
|
try {
|
||||||
|
const { form_data, query_fields }: BuildQueryObjectRequest = req.body;
|
||||||
|
|
||||||
|
if (!form_data) {
|
||||||
|
return res.status(400).json({
|
||||||
|
error: 'form_data is required',
|
||||||
|
} as BuildQueryObjectResponse);
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate required form_data fields
|
||||||
|
if (!form_data.datasource || !form_data.viz_type) {
|
||||||
|
return res.status(400).json({
|
||||||
|
error: 'form_data must include datasource and viz_type',
|
||||||
|
} as BuildQueryObjectResponse);
|
||||||
|
}
|
||||||
|
|
||||||
|
const queryObject = buildQueryObject(form_data, query_fields);
|
||||||
|
|
||||||
|
res.json({
|
||||||
|
query_object: queryObject,
|
||||||
|
} as BuildQueryObjectResponse);
|
||||||
|
|
||||||
|
} catch (error: any) {
|
||||||
|
console.error('Error building query object:', error);
|
||||||
|
res.status(500).json({
|
||||||
|
error: `Failed to build query object: ${error.message}`,
|
||||||
|
} as BuildQueryObjectResponse);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
/**
|
||||||
|
* GET /health
|
||||||
|
*
|
||||||
|
* Health check endpoint
|
||||||
|
*/
|
||||||
|
router.get('/health', (req: Request, res: Response) => {
|
||||||
|
res.json({
|
||||||
|
status: 'healthy',
|
||||||
|
timestamp: new Date().toISOString(),
|
||||||
|
version: process.env.npm_package_version || '1.0.0',
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
export default router;
|
||||||
213
sidecar-node/src/types/index.ts
Normal file
213
sidecar-node/src/types/index.ts
Normal file
@@ -0,0 +1,213 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
/* eslint-disable camelcase */
|
||||||
|
|
||||||
|
// Basic types for form data and query objects
|
||||||
|
export interface JsonObject {
|
||||||
|
[key: string]: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Metric types
|
||||||
|
export interface SavedMetric {
|
||||||
|
metric_name: string;
|
||||||
|
expression?: string;
|
||||||
|
label?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AdhocMetric {
|
||||||
|
aggregate: string;
|
||||||
|
column?: any;
|
||||||
|
expressionType: 'SIMPLE' | 'SQL';
|
||||||
|
hasCustomLabel?: boolean;
|
||||||
|
label: string;
|
||||||
|
sqlExpression?: string;
|
||||||
|
optionName?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type QueryFormMetric = SavedMetric | AdhocMetric | string;
|
||||||
|
|
||||||
|
// Column types
|
||||||
|
export interface PhysicalColumn {
|
||||||
|
column_name: string;
|
||||||
|
type?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface AdhocColumn {
|
||||||
|
hasCustomLabel?: boolean;
|
||||||
|
label: string;
|
||||||
|
sqlExpression: string;
|
||||||
|
expressionType: 'SQL';
|
||||||
|
optionName?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type QueryFormColumn = PhysicalColumn | AdhocColumn | string;
|
||||||
|
|
||||||
|
// Filter types
|
||||||
|
export type BinaryOperator =
|
||||||
|
| '==' | '!=' | '>' | '<' | '>=' | '<='
|
||||||
|
| 'LIKE' | 'ILIKE' | 'REGEX' | 'NOT REGEX';
|
||||||
|
|
||||||
|
export type SetOperator = 'IN' | 'NOT IN';
|
||||||
|
|
||||||
|
export interface AdhocFilter {
|
||||||
|
clause: 'WHERE' | 'HAVING';
|
||||||
|
comparator?: any;
|
||||||
|
expressionType: 'SIMPLE' | 'SQL';
|
||||||
|
operator?: BinaryOperator | SetOperator;
|
||||||
|
subject?: string | AdhocColumn;
|
||||||
|
sqlExpression?: string;
|
||||||
|
filterOptionName?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface QueryObjectFilterClause {
|
||||||
|
col: string;
|
||||||
|
op: BinaryOperator | SetOperator;
|
||||||
|
val: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Order by types
|
||||||
|
export type QueryFormOrderBy = [QueryFormColumn | QueryFormMetric | {}, boolean] | [];
|
||||||
|
|
||||||
|
// Annotation types
|
||||||
|
export interface AnnotationLayer {
|
||||||
|
annotationType: string;
|
||||||
|
name: string;
|
||||||
|
show: boolean;
|
||||||
|
sourceType?: string;
|
||||||
|
value?: string;
|
||||||
|
[key: string]: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Time range types
|
||||||
|
export interface TimeRange {
|
||||||
|
time_range?: string;
|
||||||
|
since?: string;
|
||||||
|
until?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extra form data types
|
||||||
|
export interface ExtraFormDataAppend {
|
||||||
|
adhoc_filters?: AdhocFilter[];
|
||||||
|
filters?: QueryObjectFilterClause[];
|
||||||
|
interactive_drilldown?: string[];
|
||||||
|
interactive_groupby?: string[];
|
||||||
|
interactive_highlight?: string[];
|
||||||
|
custom_form_data?: JsonObject;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface ExtraFormDataOverride {
|
||||||
|
granularity_sqla?: string;
|
||||||
|
granularity?: string;
|
||||||
|
time_range?: string;
|
||||||
|
time_column?: string;
|
||||||
|
time_grain?: string;
|
||||||
|
time_compare?: string[];
|
||||||
|
relative_start?: string;
|
||||||
|
relative_end?: string;
|
||||||
|
time_grain_sqla?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type ExtraFormData = ExtraFormDataAppend & ExtraFormDataOverride;
|
||||||
|
|
||||||
|
// Query extras interface
|
||||||
|
export interface QueryObjectExtras {
|
||||||
|
having?: string;
|
||||||
|
where?: string;
|
||||||
|
time_grain_sqla?: string;
|
||||||
|
time_range_endpoints?: [string, string];
|
||||||
|
relative_start?: string;
|
||||||
|
relative_end?: string;
|
||||||
|
time_compare?: string[];
|
||||||
|
[key: string]: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Main form data interface
|
||||||
|
export interface BaseFormData extends TimeRange {
|
||||||
|
datasource: string;
|
||||||
|
viz_type: string;
|
||||||
|
metrics?: QueryFormMetric[];
|
||||||
|
where?: string;
|
||||||
|
columns?: QueryFormColumn[];
|
||||||
|
groupby?: QueryFormColumn[];
|
||||||
|
all_columns?: QueryFormColumn[];
|
||||||
|
adhoc_filters?: AdhocFilter[] | null;
|
||||||
|
extra_form_data?: ExtraFormData;
|
||||||
|
order_desc?: boolean;
|
||||||
|
limit?: number;
|
||||||
|
row_limit?: string | number | null;
|
||||||
|
row_offset?: string | number | null;
|
||||||
|
series_columns?: QueryFormColumn[];
|
||||||
|
series_limit?: number;
|
||||||
|
series_limit_metric?: QueryFormMetric;
|
||||||
|
annotation_layers?: AnnotationLayer[];
|
||||||
|
url_params?: Record<string, string>;
|
||||||
|
custom_params?: Record<string, string>;
|
||||||
|
[key: string]: any;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface SqlaFormData extends BaseFormData {
|
||||||
|
granularity?: string;
|
||||||
|
granularity_sqla?: string;
|
||||||
|
time_grain_sqla?: string;
|
||||||
|
having?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
export type QueryFormData = SqlaFormData;
|
||||||
|
|
||||||
|
// Query object interface
|
||||||
|
export interface QueryObject {
|
||||||
|
time_range?: string;
|
||||||
|
since?: string;
|
||||||
|
until?: string;
|
||||||
|
granularity?: string;
|
||||||
|
columns?: QueryFormColumn[];
|
||||||
|
metrics?: QueryFormMetric[];
|
||||||
|
orderby?: QueryFormOrderBy[];
|
||||||
|
annotation_layers?: AnnotationLayer[];
|
||||||
|
row_limit?: number;
|
||||||
|
row_offset?: number;
|
||||||
|
series_columns?: QueryFormColumn[];
|
||||||
|
series_limit?: number;
|
||||||
|
series_limit_metric?: QueryFormMetric;
|
||||||
|
group_others_when_limit_reached?: boolean;
|
||||||
|
order_desc?: boolean;
|
||||||
|
url_params?: Record<string, string>;
|
||||||
|
custom_params?: Record<string, string>;
|
||||||
|
extras?: QueryObjectExtras;
|
||||||
|
filters?: QueryObjectFilterClause[];
|
||||||
|
custom_form_data?: JsonObject;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Query field aliases
|
||||||
|
export type QueryFieldAliases = {
|
||||||
|
[key: string]: 'metrics' | 'columns' | 'groupby';
|
||||||
|
};
|
||||||
|
|
||||||
|
// Utility functions
|
||||||
|
export function isAdhocMetric(metric: any): metric is AdhocMetric {
|
||||||
|
return metric && typeof metric === 'object' && 'expressionType' in metric;
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isQueryFormMetric(metric: any): metric is QueryFormMetric {
|
||||||
|
return typeof metric === 'string' || isAdhocMetric(metric) ||
|
||||||
|
(metric && typeof metric === 'object' && 'metric_name' in metric);
|
||||||
|
}
|
||||||
|
|
||||||
|
export function isDefined<T>(value: T | undefined | null): value is T {
|
||||||
|
return value !== undefined && value !== null;
|
||||||
|
}
|
||||||
78
sidecar-node/src/utils/extractExtras.ts
Normal file
78
sidecar-node/src/utils/extractExtras.ts
Normal file
@@ -0,0 +1,78 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import {
|
||||||
|
QueryFormData,
|
||||||
|
QueryObjectExtras,
|
||||||
|
QueryObjectFilterClause,
|
||||||
|
} from '../types';
|
||||||
|
|
||||||
|
interface ExtrasResult extends QueryObjectExtras {
|
||||||
|
filters: QueryObjectFilterClause[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract extras and filters from form data
|
||||||
|
*/
|
||||||
|
export default function extractExtras(formData: QueryFormData): ExtrasResult {
|
||||||
|
const {
|
||||||
|
where,
|
||||||
|
having,
|
||||||
|
time_grain_sqla,
|
||||||
|
granularity_sqla,
|
||||||
|
granularity,
|
||||||
|
extra_filters = [],
|
||||||
|
} = formData;
|
||||||
|
|
||||||
|
const extras: QueryObjectExtras = {};
|
||||||
|
const filters: QueryObjectFilterClause[] = [];
|
||||||
|
|
||||||
|
// Add SQL clauses to extras
|
||||||
|
if (where) {
|
||||||
|
extras.where = where;
|
||||||
|
}
|
||||||
|
if (having) {
|
||||||
|
extras.having = having;
|
||||||
|
}
|
||||||
|
if (time_grain_sqla) {
|
||||||
|
extras.time_grain_sqla = time_grain_sqla;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle granularity - prefer granularity_sqla over granularity
|
||||||
|
const timeColumn = granularity_sqla || granularity;
|
||||||
|
if (timeColumn) {
|
||||||
|
// Time column handling would go here if needed
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert extra_filters to QueryObjectFilterClause format
|
||||||
|
if (extra_filters && Array.isArray(extra_filters)) {
|
||||||
|
for (const filter of extra_filters) {
|
||||||
|
if (filter.col && filter.op && filter.val !== undefined) {
|
||||||
|
filters.push({
|
||||||
|
col: filter.col,
|
||||||
|
op: filter.op,
|
||||||
|
val: filter.val,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
...extras,
|
||||||
|
filters,
|
||||||
|
};
|
||||||
|
}
|
||||||
73
sidecar-node/src/utils/extractQueryFields.ts
Normal file
73
sidecar-node/src/utils/extractQueryFields.ts
Normal file
@@ -0,0 +1,73 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import {
|
||||||
|
QueryFormData,
|
||||||
|
QueryFormMetric,
|
||||||
|
QueryFormColumn,
|
||||||
|
QueryFormOrderBy,
|
||||||
|
QueryFieldAliases,
|
||||||
|
} from '../types';
|
||||||
|
|
||||||
|
interface QueryFieldsResult {
|
||||||
|
metrics?: QueryFormMetric[];
|
||||||
|
columns?: QueryFormColumn[];
|
||||||
|
orderby?: QueryFormOrderBy[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Extract query fields (metrics, columns, orderby) from form data
|
||||||
|
*/
|
||||||
|
export default function extractQueryFields(
|
||||||
|
formData: QueryFormData,
|
||||||
|
queryFieldAliases?: QueryFieldAliases,
|
||||||
|
): QueryFieldsResult {
|
||||||
|
const result: QueryFieldsResult = {};
|
||||||
|
|
||||||
|
// Extract metrics
|
||||||
|
if (formData.metrics && formData.metrics.length > 0) {
|
||||||
|
result.metrics = formData.metrics;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract columns - prefer 'columns' over 'groupby'
|
||||||
|
const columns = formData.columns || formData.groupby;
|
||||||
|
if (columns && columns.length > 0) {
|
||||||
|
result.columns = columns;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Handle query field aliases if provided
|
||||||
|
if (queryFieldAliases) {
|
||||||
|
for (const [formFieldName, queryField] of Object.entries(queryFieldAliases)) {
|
||||||
|
const formValue = (formData as any)[formFieldName];
|
||||||
|
if (formValue && Array.isArray(formValue) && formValue.length > 0) {
|
||||||
|
if (queryField === 'metrics') {
|
||||||
|
result.metrics = formValue;
|
||||||
|
} else if (queryField === 'columns' || queryField === 'groupby') {
|
||||||
|
result.columns = formValue;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract orderby - this can be complex as it depends on the form structure
|
||||||
|
// For now, we'll handle basic cases
|
||||||
|
if (formData.orderby && Array.isArray(formData.orderby)) {
|
||||||
|
result.orderby = formData.orderby as QueryFormOrderBy[];
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
57
sidecar-node/src/utils/processExtraFormData.ts
Normal file
57
sidecar-node/src/utils/processExtraFormData.ts
Normal file
@@ -0,0 +1,57 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import { QueryObject, ExtraFormDataOverride } from '../types';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Override extra form data used by native and cross filters
|
||||||
|
*/
|
||||||
|
export function overrideExtraFormData(
|
||||||
|
queryObject: QueryObject,
|
||||||
|
overrides: ExtraFormDataOverride,
|
||||||
|
): QueryObject {
|
||||||
|
const result = { ...queryObject };
|
||||||
|
|
||||||
|
// Override top-level properties
|
||||||
|
if (overrides.time_range !== undefined) {
|
||||||
|
result.time_range = overrides.time_range;
|
||||||
|
}
|
||||||
|
if (overrides.granularity !== undefined) {
|
||||||
|
result.granularity = overrides.granularity;
|
||||||
|
}
|
||||||
|
if (overrides.granularity_sqla !== undefined) {
|
||||||
|
result.granularity = overrides.granularity_sqla;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Override extras properties
|
||||||
|
if (result.extras) {
|
||||||
|
if (overrides.relative_start !== undefined) {
|
||||||
|
result.extras.relative_start = overrides.relative_start;
|
||||||
|
}
|
||||||
|
if (overrides.relative_end !== undefined) {
|
||||||
|
result.extras.relative_end = overrides.relative_end;
|
||||||
|
}
|
||||||
|
if (overrides.time_grain_sqla !== undefined) {
|
||||||
|
result.extras.time_grain_sqla = overrides.time_grain_sqla;
|
||||||
|
}
|
||||||
|
if (overrides.time_compare !== undefined) {
|
||||||
|
result.extras.time_compare = overrides.time_compare;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result;
|
||||||
|
}
|
||||||
72
sidecar-node/src/utils/processFilters.ts
Normal file
72
sidecar-node/src/utils/processFilters.ts
Normal file
@@ -0,0 +1,72 @@
|
|||||||
|
// Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
// or more contributor license agreements. See the NOTICE file
|
||||||
|
// distributed with this work for additional information
|
||||||
|
// regarding copyright ownership. The ASF licenses this file
|
||||||
|
// to you under the Apache License, Version 2.0 (the
|
||||||
|
// "License"); you may not use this file except in compliance
|
||||||
|
// with the License. You may obtain a copy of the License at
|
||||||
|
//
|
||||||
|
// http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
//
|
||||||
|
// Unless required by applicable law or agreed to in writing,
|
||||||
|
// software distributed under the License is distributed on an
|
||||||
|
// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
// KIND, either express or implied. See the License for the
|
||||||
|
// specific language governing permissions and limitations
|
||||||
|
// under the License.
|
||||||
|
|
||||||
|
import {
|
||||||
|
AdhocFilter,
|
||||||
|
QueryObjectFilterClause,
|
||||||
|
QueryFormData,
|
||||||
|
QueryObjectExtras,
|
||||||
|
} from '../types';
|
||||||
|
|
||||||
|
interface FilterProcessorInput extends QueryFormData {
|
||||||
|
extras: QueryObjectExtras;
|
||||||
|
filters: QueryObjectFilterClause[];
|
||||||
|
adhoc_filters: AdhocFilter[];
|
||||||
|
}
|
||||||
|
|
||||||
|
interface FilterProcessorResult {
|
||||||
|
extras: QueryObjectExtras;
|
||||||
|
filters: QueryObjectFilterClause[];
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Process filters from form data into QueryObject format
|
||||||
|
*/
|
||||||
|
export default function processFilters(
|
||||||
|
formData: FilterProcessorInput,
|
||||||
|
): FilterProcessorResult {
|
||||||
|
const { filters = [], adhoc_filters = [], extras = {} } = formData;
|
||||||
|
|
||||||
|
// Convert adhoc filters to simple filters where possible
|
||||||
|
const processedFilters: QueryObjectFilterClause[] = [...filters];
|
||||||
|
const processedExtras: QueryObjectExtras = { ...extras };
|
||||||
|
|
||||||
|
// Process adhoc filters
|
||||||
|
for (const adhocFilter of adhoc_filters) {
|
||||||
|
if (adhocFilter.expressionType === 'SIMPLE' && adhocFilter.subject && adhocFilter.operator) {
|
||||||
|
// Convert simple adhoc filters to QueryObjectFilterClause
|
||||||
|
if (typeof adhocFilter.subject === 'string') {
|
||||||
|
processedFilters.push({
|
||||||
|
col: adhocFilter.subject,
|
||||||
|
op: adhocFilter.operator as any,
|
||||||
|
val: adhocFilter.comparator,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
} else if (adhocFilter.expressionType === 'SQL' && adhocFilter.sqlExpression) {
|
||||||
|
// Add SQL filters to WHERE clause in extras
|
||||||
|
const clause = adhocFilter.clause === 'HAVING' ? 'having' : 'where';
|
||||||
|
const existingClause = processedExtras[clause] || '';
|
||||||
|
const separator = existingClause ? ' AND ' : '';
|
||||||
|
processedExtras[clause] = existingClause + separator + `(${adhocFilter.sqlExpression})`;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
extras: processedExtras,
|
||||||
|
filters: processedFilters,
|
||||||
|
};
|
||||||
|
}
|
||||||
34
sidecar-node/tsconfig.json
Normal file
34
sidecar-node/tsconfig.json
Normal file
@@ -0,0 +1,34 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2020",
|
||||||
|
"lib": ["ES2020"],
|
||||||
|
"module": "commonjs",
|
||||||
|
"declaration": true,
|
||||||
|
"outDir": "./dist",
|
||||||
|
"rootDir": "./src",
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"noImplicitReturns": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"moduleResolution": "node",
|
||||||
|
"baseUrl": "./",
|
||||||
|
"paths": {
|
||||||
|
"@/*": ["src/*"]
|
||||||
|
},
|
||||||
|
"allowSyntheticDefaultImports": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"experimentalDecorators": true,
|
||||||
|
"emitDecoratorMetadata": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true
|
||||||
|
},
|
||||||
|
"include": [
|
||||||
|
"src/**/*"
|
||||||
|
],
|
||||||
|
"exclude": [
|
||||||
|
"node_modules",
|
||||||
|
"dist",
|
||||||
|
"**/*.test.ts"
|
||||||
|
]
|
||||||
|
}
|
||||||
@@ -77,6 +77,7 @@ from superset.utils.core import HeaderDataType, override_user, recipients_string
|
|||||||
from superset.utils.csv import get_chart_csv_data, get_chart_dataframe
|
from superset.utils.csv import get_chart_csv_data, get_chart_dataframe
|
||||||
from superset.utils.decorators import logs_context, transaction
|
from superset.utils.decorators import logs_context, transaction
|
||||||
from superset.utils.pdf import build_pdf_from_screenshots
|
from superset.utils.pdf import build_pdf_from_screenshots
|
||||||
|
from superset.utils.query_sidecar import get_query_sidecar_client, QuerySidecarException
|
||||||
from superset.utils.screenshots import ChartScreenshot, DashboardScreenshot
|
from superset.utils.screenshots import ChartScreenshot, DashboardScreenshot
|
||||||
from superset.utils.slack import get_channels_with_search, SlackChannelTypes
|
from superset.utils.slack import get_channels_with_search, SlackChannelTypes
|
||||||
from superset.utils.urls import get_url_path
|
from superset.utils.urls import get_url_path
|
||||||
@@ -388,9 +389,7 @@ class BaseReportState:
|
|||||||
user = security_manager.find_user(username)
|
user = security_manager.find_user(username)
|
||||||
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
||||||
|
|
||||||
if self._report_schedule.chart.query_context is None:
|
self._ensure_query_context_available()
|
||||||
logger.warning("No query context found, taking a screenshot to generate it")
|
|
||||||
self._update_query_context()
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
logger.info("Getting chart from %s as user %s", url, user.username)
|
logger.info("Getting chart from %s as user %s", url, user.username)
|
||||||
@@ -418,9 +417,7 @@ class BaseReportState:
|
|||||||
user = security_manager.find_user(username)
|
user = security_manager.find_user(username)
|
||||||
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
auth_cookies = machine_auth_provider_factory.instance.get_auth_cookies(user)
|
||||||
|
|
||||||
if self._report_schedule.chart.query_context is None:
|
self._ensure_query_context_available()
|
||||||
logger.warning("No query context found, taking a screenshot to generate it")
|
|
||||||
self._update_query_context()
|
|
||||||
|
|
||||||
try:
|
try:
|
||||||
logger.info("Getting chart from %s as user %s", url, user.username)
|
logger.info("Getting chart from %s as user %s", url, user.username)
|
||||||
@@ -435,9 +432,79 @@ class BaseReportState:
|
|||||||
raise ReportScheduleCsvFailedError()
|
raise ReportScheduleCsvFailedError()
|
||||||
return dataframe
|
return dataframe
|
||||||
|
|
||||||
def _update_query_context(self) -> None:
|
def _ensure_query_context_available(self) -> None:
|
||||||
"""
|
"""
|
||||||
Update chart query context.
|
Ensure query context is available for the chart.
|
||||||
|
|
||||||
|
First attempts to generate query context using the sidecar service
|
||||||
|
from the chart's form_data. If that fails, falls back to the
|
||||||
|
screenshot method for backward compatibility.
|
||||||
|
"""
|
||||||
|
# Try sidecar service first if available
|
||||||
|
if app.config.get("QUERY_SIDECAR_ENABLED", True):
|
||||||
|
try:
|
||||||
|
self._generate_query_context_via_sidecar()
|
||||||
|
return
|
||||||
|
except QuerySidecarException as ex:
|
||||||
|
logger.warning(
|
||||||
|
"Failed to generate query context via sidecar service: %s. "
|
||||||
|
"Falling back to screenshot method.",
|
||||||
|
str(ex),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Fallback to legacy screenshot method
|
||||||
|
if self._report_schedule.chart.query_context is None:
|
||||||
|
logger.warning("No query context found, taking a screenshot to generate it")
|
||||||
|
self._update_query_context_legacy()
|
||||||
|
|
||||||
|
@transaction()
|
||||||
|
def _generate_query_context_via_sidecar(self) -> None:
|
||||||
|
"""
|
||||||
|
Generate and save query context using the sidecar service.
|
||||||
|
|
||||||
|
This method uses the chart's form_data to generate a fresh
|
||||||
|
QueryObject via the sidecar service, eliminating staleness issues.
|
||||||
|
"""
|
||||||
|
if not self._report_schedule.chart:
|
||||||
|
raise QuerySidecarException("No chart associated with report schedule")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Get the chart's form_data
|
||||||
|
form_data = self._report_schedule.chart.form_data
|
||||||
|
|
||||||
|
# Use sidecar service to generate QueryObject
|
||||||
|
sidecar_client = get_query_sidecar_client()
|
||||||
|
query_object_data = sidecar_client.build_query_object(form_data)
|
||||||
|
|
||||||
|
# Convert to JSON and save as query_context
|
||||||
|
query_context = json.dumps(
|
||||||
|
{
|
||||||
|
"queries": [query_object_data],
|
||||||
|
"form_data": form_data,
|
||||||
|
"result_format": "json",
|
||||||
|
"result_type": "full",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update the chart's query_context in the database
|
||||||
|
self._report_schedule.chart.query_context = query_context
|
||||||
|
from superset import db
|
||||||
|
|
||||||
|
db.session.add(self._report_schedule.chart)
|
||||||
|
|
||||||
|
logger.info(
|
||||||
|
"Successfully generated query context via sidecar for chart %s",
|
||||||
|
self._report_schedule.chart.id,
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as ex:
|
||||||
|
raise QuerySidecarException(
|
||||||
|
f"Failed to generate query context via sidecar: {ex}"
|
||||||
|
) from ex
|
||||||
|
|
||||||
|
def _update_query_context_legacy(self) -> None:
|
||||||
|
"""
|
||||||
|
Update chart query context using the legacy screenshot method.
|
||||||
|
|
||||||
To load CSV data from the endpoint the chart must have been saved
|
To load CSV data from the endpoint the chart must have been saved
|
||||||
with its query context. For charts without saved query context we
|
with its query context. For charts without saved query context we
|
||||||
|
|||||||
54
superset/config_sidecar.py
Normal file
54
superset/config_sidecar.py
Normal file
@@ -0,0 +1,54 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Configuration settings for the Query Sidecar Service integration.
|
||||||
|
|
||||||
|
This file contains configuration options for integrating Superset with
|
||||||
|
the Node.js Query Sidecar Service that generates QueryObjects from form_data.
|
||||||
|
"""
|
||||||
|
|
||||||
|
# Query Sidecar Service Configuration
|
||||||
|
# Enable or disable the query sidecar service integration
|
||||||
|
QUERY_SIDECAR_ENABLED = True
|
||||||
|
|
||||||
|
# Base URL for the Node.js query sidecar service
|
||||||
|
# This service transforms form_data into QueryObjects for Alerts & Reports
|
||||||
|
QUERY_SIDECAR_BASE_URL = "http://localhost:3001"
|
||||||
|
|
||||||
|
# Timeout for sidecar service requests in seconds
|
||||||
|
QUERY_SIDECAR_TIMEOUT = 10
|
||||||
|
|
||||||
|
# Example production configuration:
|
||||||
|
# QUERY_SIDECAR_BASE_URL = "http://superset-query-sidecar:3001"
|
||||||
|
# QUERY_SIDECAR_ENABLED = True
|
||||||
|
# QUERY_SIDECAR_TIMEOUT = 30
|
||||||
|
|
||||||
|
# Example Docker Compose setup:
|
||||||
|
# services:
|
||||||
|
# superset-query-sidecar:
|
||||||
|
# image: superset-query-sidecar:latest
|
||||||
|
# ports:
|
||||||
|
# - "3001:3001"
|
||||||
|
# environment:
|
||||||
|
# - NODE_ENV=production
|
||||||
|
# - SUPERSET_ORIGINS=http://localhost:8088
|
||||||
|
#
|
||||||
|
# superset:
|
||||||
|
# # ... your superset config
|
||||||
|
# environment:
|
||||||
|
# - QUERY_SIDECAR_BASE_URL=http://superset-query-sidecar:3001
|
||||||
259
superset/utils/query_sidecar.py
Normal file
259
superset/utils/query_sidecar.py
Normal file
@@ -0,0 +1,259 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
"""
|
||||||
|
Query Sidecar Client
|
||||||
|
|
||||||
|
Provides functionality to communicate with the Node.js sidecar service
|
||||||
|
that generates QueryObjects from form_data, eliminating staleness issues
|
||||||
|
in Alerts & Reports.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import logging
|
||||||
|
from typing import Any, Dict, Optional
|
||||||
|
from urllib.parse import urljoin
|
||||||
|
|
||||||
|
import requests
|
||||||
|
from flask import current_app
|
||||||
|
|
||||||
|
from superset.common.query_object import QueryObject
|
||||||
|
from superset.exceptions import SupersetException
|
||||||
|
|
||||||
|
logger = logging.getLogger(__name__)
|
||||||
|
|
||||||
|
|
||||||
|
class QuerySidecarException(SupersetException):
|
||||||
|
"""Exception raised when the query sidecar service fails."""
|
||||||
|
|
||||||
|
|
||||||
|
class QuerySidecarClient:
|
||||||
|
"""
|
||||||
|
Client for communicating with the Node.js query sidecar service.
|
||||||
|
|
||||||
|
This client transforms form_data into QueryObject format using the same
|
||||||
|
logic as the frontend, ensuring consistency and eliminating staleness.
|
||||||
|
"""
|
||||||
|
|
||||||
|
def __init__(self, base_url: Optional[str] = None, timeout: int = 10) -> None:
|
||||||
|
"""
|
||||||
|
Initialize the query sidecar client.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
base_url: Base URL of the sidecar service. If None, reads from config.
|
||||||
|
timeout: Request timeout in seconds.
|
||||||
|
"""
|
||||||
|
self._base_url = base_url or current_app.config.get(
|
||||||
|
"QUERY_SIDECAR_BASE_URL", "http://localhost:3001"
|
||||||
|
)
|
||||||
|
self._timeout = timeout
|
||||||
|
self._session = requests.Session()
|
||||||
|
|
||||||
|
# Set default headers
|
||||||
|
self._session.headers.update(
|
||||||
|
{
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
"Accept": "application/json",
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
def build_query_object(
|
||||||
|
self, form_data: Dict[str, Any], query_fields: Optional[Dict[str, str]] = None
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""
|
||||||
|
Build a QueryObject from form_data using the sidecar service.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
form_data: The form data from Superset frontend/chart configuration
|
||||||
|
query_fields: Optional query field aliases for visualization-specific
|
||||||
|
mappings
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict containing the QueryObject data
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
QuerySidecarException: If the service is unavailable or returns an error
|
||||||
|
"""
|
||||||
|
if not form_data:
|
||||||
|
raise QuerySidecarException("form_data is required")
|
||||||
|
|
||||||
|
# Validate required form_data fields
|
||||||
|
if not form_data.get("datasource") or not form_data.get("viz_type"):
|
||||||
|
raise QuerySidecarException(
|
||||||
|
"form_data must include datasource and viz_type"
|
||||||
|
)
|
||||||
|
|
||||||
|
url = urljoin(self._base_url, "/api/v1/query-object")
|
||||||
|
payload = {
|
||||||
|
"form_data": form_data,
|
||||||
|
"query_fields": query_fields,
|
||||||
|
}
|
||||||
|
|
||||||
|
try:
|
||||||
|
logger.debug("Calling sidecar service at %s", url)
|
||||||
|
response = self._session.post(url, json=payload, timeout=self._timeout)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
data = response.json()
|
||||||
|
if "error" in data:
|
||||||
|
raise QuerySidecarException(f"Sidecar service error: {data['error']}")
|
||||||
|
|
||||||
|
return data["query_object"]
|
||||||
|
|
||||||
|
except requests.exceptions.Timeout as ex:
|
||||||
|
logger.error("Timeout calling query sidecar service")
|
||||||
|
raise QuerySidecarException(
|
||||||
|
"Query sidecar service timeout. Please check service availability."
|
||||||
|
) from ex
|
||||||
|
except requests.exceptions.ConnectionError as ex:
|
||||||
|
logger.error("Connection error calling query sidecar service")
|
||||||
|
raise QuerySidecarException(
|
||||||
|
"Unable to connect to query sidecar service. "
|
||||||
|
"Please check service availability."
|
||||||
|
) from ex
|
||||||
|
except requests.exceptions.HTTPError as ex:
|
||||||
|
logger.error("HTTP error calling query sidecar service: %s", ex)
|
||||||
|
if ex.response.status_code >= 500:
|
||||||
|
raise QuerySidecarException(
|
||||||
|
"Query sidecar service is experiencing issues. "
|
||||||
|
"Please try again later."
|
||||||
|
) from ex
|
||||||
|
else:
|
||||||
|
try:
|
||||||
|
error_data = ex.response.json()
|
||||||
|
error_message = error_data.get("error", str(ex))
|
||||||
|
except (ValueError, KeyError):
|
||||||
|
error_message = str(ex)
|
||||||
|
raise QuerySidecarException(
|
||||||
|
f"Query sidecar service error: {error_message}"
|
||||||
|
) from ex
|
||||||
|
except Exception as ex:
|
||||||
|
logger.error("Unexpected error calling query sidecar service: %s", ex)
|
||||||
|
raise QuerySidecarException(
|
||||||
|
f"Unexpected error communicating with query sidecar service: {ex}"
|
||||||
|
) from ex
|
||||||
|
|
||||||
|
def create_query_object_from_form_data(
|
||||||
|
self,
|
||||||
|
form_data: Dict[str, Any],
|
||||||
|
query_fields: Optional[Dict[str, str]] = None,
|
||||||
|
datasource: Optional[Any] = None,
|
||||||
|
) -> QueryObject:
|
||||||
|
"""
|
||||||
|
Create a QueryObject instance from form_data using the sidecar service.
|
||||||
|
|
||||||
|
This is a convenience method that returns a QueryObject instance
|
||||||
|
rather than raw dictionary data.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
form_data: The form data from Superset frontend/chart configuration
|
||||||
|
query_fields: Optional query field aliases for visualization-specific
|
||||||
|
mappings
|
||||||
|
datasource: Optional datasource instance to attach to the QueryObject
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QueryObject instance
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
QuerySidecarException: If the service is unavailable or returns an error
|
||||||
|
"""
|
||||||
|
query_data = self.build_query_object(form_data, query_fields)
|
||||||
|
|
||||||
|
# Create QueryObject instance from the returned data
|
||||||
|
# Convert the dictionary to QueryObject constructor parameters
|
||||||
|
query_object = QueryObject(
|
||||||
|
datasource=datasource,
|
||||||
|
columns=query_data.get("columns"),
|
||||||
|
metrics=query_data.get("metrics"),
|
||||||
|
filters=query_data.get("filters", []),
|
||||||
|
granularity=query_data.get("granularity"),
|
||||||
|
extras=query_data.get("extras", {}),
|
||||||
|
orderby=query_data.get("orderby", []),
|
||||||
|
annotation_layers=query_data.get("annotation_layers", []),
|
||||||
|
row_limit=query_data.get("row_limit"),
|
||||||
|
row_offset=query_data.get("row_offset"),
|
||||||
|
series_columns=query_data.get("series_columns"),
|
||||||
|
series_limit=query_data.get("series_limit", 0),
|
||||||
|
series_limit_metric=query_data.get("series_limit_metric"),
|
||||||
|
order_desc=query_data.get("order_desc", True),
|
||||||
|
time_range=query_data.get("time_range"),
|
||||||
|
)
|
||||||
|
|
||||||
|
return query_object
|
||||||
|
|
||||||
|
def health_check(self) -> bool:
|
||||||
|
"""
|
||||||
|
Check if the sidecar service is healthy.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
True if service is healthy, False otherwise
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
url = urljoin(self._base_url, "/health")
|
||||||
|
response = self._session.get(url, timeout=5)
|
||||||
|
response.raise_for_status()
|
||||||
|
|
||||||
|
data = response.json()
|
||||||
|
return data.get("status") == "healthy"
|
||||||
|
|
||||||
|
except Exception as ex:
|
||||||
|
logger.warning("Query sidecar health check failed: %s", ex)
|
||||||
|
return False
|
||||||
|
|
||||||
|
|
||||||
|
# Global client instance
|
||||||
|
_sidecar_client: Optional[QuerySidecarClient] = None
|
||||||
|
|
||||||
|
|
||||||
|
def get_query_sidecar_client() -> QuerySidecarClient:
|
||||||
|
"""
|
||||||
|
Get the global query sidecar client instance.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QuerySidecarClient instance
|
||||||
|
"""
|
||||||
|
global _sidecar_client
|
||||||
|
|
||||||
|
if _sidecar_client is None:
|
||||||
|
_sidecar_client = QuerySidecarClient()
|
||||||
|
|
||||||
|
return _sidecar_client
|
||||||
|
|
||||||
|
|
||||||
|
def build_query_object_from_form_data(
|
||||||
|
form_data: Dict[str, Any],
|
||||||
|
query_fields: Optional[Dict[str, str]] = None,
|
||||||
|
datasource: Optional[Any] = None,
|
||||||
|
) -> QueryObject:
|
||||||
|
"""
|
||||||
|
Convenience function to build a QueryObject from form_data using the sidecar
|
||||||
|
service.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
form_data: The form data from Superset frontend/chart configuration
|
||||||
|
query_fields: Optional query field aliases for visualization-specific mappings
|
||||||
|
datasource: Optional datasource instance to attach to the QueryObject
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
QueryObject instance
|
||||||
|
|
||||||
|
Raises:
|
||||||
|
QuerySidecarException: If the service is unavailable or returns an error
|
||||||
|
"""
|
||||||
|
client = get_query_sidecar_client()
|
||||||
|
return client.create_query_object_from_form_data(
|
||||||
|
form_data, query_fields, datasource
|
||||||
|
)
|
||||||
195
tests/unit_tests/commands/report/test_execute_sidecar.py
Normal file
195
tests/unit_tests/commands/report/test_execute_sidecar.py
Normal file
@@ -0,0 +1,195 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
"""Tests for sidecar integration in report execution."""
|
||||||
|
|
||||||
|
from datetime import datetime
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
from uuid import uuid4
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
|
||||||
|
from superset.commands.report.execute import BaseReportState
|
||||||
|
from superset.reports.models import ReportSchedule
|
||||||
|
from superset.utils.query_sidecar import QuerySidecarException
|
||||||
|
|
||||||
|
|
||||||
|
class TestReportExecutionSidecar:
|
||||||
|
"""Tests for sidecar service integration in report execution."""
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
@patch("superset.commands.report.execute.get_query_sidecar_client")
|
||||||
|
@patch("superset.commands.report.execute.db")
|
||||||
|
def test_generate_query_context_via_sidecar_success(
|
||||||
|
self, mock_db, mock_get_client, mock_app
|
||||||
|
):
|
||||||
|
"""Test successful query context generation via sidecar."""
|
||||||
|
# Setup mocks
|
||||||
|
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||||
|
mock_sidecar_client = MagicMock()
|
||||||
|
mock_get_client.return_value = mock_sidecar_client
|
||||||
|
mock_sidecar_client.build_query_object.return_value = {
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"filters": [],
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create test report schedule with chart
|
||||||
|
mock_chart = MagicMock()
|
||||||
|
mock_chart.id = 1
|
||||||
|
mock_chart.form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
}
|
||||||
|
mock_chart.query_context = None
|
||||||
|
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = mock_chart
|
||||||
|
|
||||||
|
# Create BaseReportState instance
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
# Test the method
|
||||||
|
report_state._generate_query_context_via_sidecar()
|
||||||
|
|
||||||
|
# Verify sidecar was called with correct form_data
|
||||||
|
mock_sidecar_client.build_query_object.assert_called_once_with(
|
||||||
|
mock_chart.form_data
|
||||||
|
)
|
||||||
|
|
||||||
|
# Verify query_context was updated
|
||||||
|
assert mock_chart.query_context is not None
|
||||||
|
mock_db.session.commit.assert_called_once()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
def test_generate_query_context_via_sidecar_no_chart(self, mock_app):
|
||||||
|
"""Test error when no chart is associated with report schedule."""
|
||||||
|
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||||
|
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = None
|
||||||
|
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="No chart associated with report schedule"
|
||||||
|
):
|
||||||
|
report_state._generate_query_context_via_sidecar()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
@patch("superset.commands.report.execute.get_query_sidecar_client")
|
||||||
|
def test_generate_query_context_via_sidecar_client_error(
|
||||||
|
self, mock_get_client, mock_app
|
||||||
|
):
|
||||||
|
"""Test handling of sidecar client errors."""
|
||||||
|
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||||
|
mock_sidecar_client = MagicMock()
|
||||||
|
mock_get_client.return_value = mock_sidecar_client
|
||||||
|
mock_sidecar_client.build_query_object.side_effect = Exception("Sidecar error")
|
||||||
|
|
||||||
|
mock_chart = MagicMock()
|
||||||
|
mock_chart.form_data = {"datasource": "1__table", "viz_type": "table"}
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = mock_chart
|
||||||
|
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="Failed to generate query context via sidecar"
|
||||||
|
):
|
||||||
|
report_state._generate_query_context_via_sidecar()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
@patch(
|
||||||
|
"superset.commands.report.execute.BaseReportState._generate_query_context_via_sidecar"
|
||||||
|
)
|
||||||
|
def test_ensure_query_context_available_sidecar_enabled(
|
||||||
|
self, mock_generate_sidecar, mock_app
|
||||||
|
):
|
||||||
|
"""Test _ensure_query_context_available when sidecar is enabled."""
|
||||||
|
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||||
|
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
report_state._ensure_query_context_available()
|
||||||
|
|
||||||
|
mock_generate_sidecar.assert_called_once()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
@patch(
|
||||||
|
"superset.commands.report.execute.BaseReportState._generate_query_context_via_sidecar"
|
||||||
|
)
|
||||||
|
@patch(
|
||||||
|
"superset.commands.report.execute.BaseReportState._update_query_context_legacy"
|
||||||
|
)
|
||||||
|
def test_ensure_query_context_available_fallback(
|
||||||
|
self, mock_legacy, mock_generate_sidecar, mock_app
|
||||||
|
):
|
||||||
|
"""Test fallback to legacy method when sidecar fails."""
|
||||||
|
mock_app.config.get.return_value = True # QUERY_SIDECAR_ENABLED
|
||||||
|
mock_generate_sidecar.side_effect = QuerySidecarException("Sidecar failed")
|
||||||
|
|
||||||
|
mock_chart = MagicMock()
|
||||||
|
mock_chart.query_context = None
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = mock_chart
|
||||||
|
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
report_state._ensure_query_context_available()
|
||||||
|
|
||||||
|
mock_generate_sidecar.assert_called_once()
|
||||||
|
mock_legacy.assert_called_once()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
@patch(
|
||||||
|
"superset.commands.report.execute.BaseReportState._update_query_context_legacy"
|
||||||
|
)
|
||||||
|
def test_ensure_query_context_available_sidecar_disabled(
|
||||||
|
self, mock_legacy, mock_app
|
||||||
|
):
|
||||||
|
"""Test _ensure_query_context_available when sidecar is disabled."""
|
||||||
|
mock_app.config.get.return_value = False # QUERY_SIDECAR_ENABLED = False
|
||||||
|
|
||||||
|
mock_chart = MagicMock()
|
||||||
|
mock_chart.query_context = None
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = mock_chart
|
||||||
|
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
report_state._ensure_query_context_available()
|
||||||
|
|
||||||
|
mock_legacy.assert_called_once()
|
||||||
|
|
||||||
|
@patch("superset.commands.report.execute.app")
|
||||||
|
def test_ensure_query_context_available_existing_context(self, mock_app):
|
||||||
|
"""Test _ensure_query_context_available when query_context already exists."""
|
||||||
|
mock_app.config.get.return_value = False # QUERY_SIDECAR_ENABLED = False
|
||||||
|
|
||||||
|
mock_chart = MagicMock()
|
||||||
|
mock_chart.query_context = '{"queries": []}' # Existing context
|
||||||
|
mock_report_schedule = MagicMock(spec=ReportSchedule)
|
||||||
|
mock_report_schedule.chart = mock_chart
|
||||||
|
|
||||||
|
report_state = BaseReportState(mock_report_schedule, datetime.utcnow(), uuid4())
|
||||||
|
|
||||||
|
# Should not raise any errors or call legacy method
|
||||||
|
report_state._ensure_query_context_available()
|
||||||
246
tests/unit_tests/utils/test_query_sidecar.py
Normal file
246
tests/unit_tests/utils/test_query_sidecar.py
Normal file
@@ -0,0 +1,246 @@
|
|||||||
|
# Licensed to the Apache Software Foundation (ASF) under one
|
||||||
|
# or more contributor license agreements. See the NOTICE file
|
||||||
|
# distributed with this work for additional information
|
||||||
|
# regarding copyright ownership. The ASF licenses this file
|
||||||
|
# to you under the Apache License, Version 2.0 (the
|
||||||
|
# "License"); you may not use this file except in compliance
|
||||||
|
# with the License. You may obtain a copy of the License at
|
||||||
|
#
|
||||||
|
# http://www.apache.org/licenses/LICENSE-2.0
|
||||||
|
#
|
||||||
|
# Unless required by applicable law or agreed to in writing,
|
||||||
|
# software distributed under the License is distributed on an
|
||||||
|
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||||
|
# KIND, either express or implied. See the License for the
|
||||||
|
# specific language governing permissions and limitations
|
||||||
|
# under the License.
|
||||||
|
|
||||||
|
from unittest.mock import MagicMock, patch
|
||||||
|
|
||||||
|
import pytest
|
||||||
|
from requests.exceptions import ConnectionError, HTTPError, Timeout
|
||||||
|
|
||||||
|
from superset.utils.query_sidecar import (
|
||||||
|
build_query_object_from_form_data,
|
||||||
|
QuerySidecarClient,
|
||||||
|
QuerySidecarException,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class TestQuerySidecarClient:
|
||||||
|
"""Tests for the QuerySidecarClient class."""
|
||||||
|
|
||||||
|
def test_init_default_config(self):
|
||||||
|
"""Test client initialization with default configuration."""
|
||||||
|
with patch("superset.utils.query_sidecar.current_app") as mock_app:
|
||||||
|
mock_app.config.get.return_value = "http://test:3001"
|
||||||
|
client = QuerySidecarClient()
|
||||||
|
assert client._base_url == "http://test:3001"
|
||||||
|
assert client._timeout == 10
|
||||||
|
|
||||||
|
def test_init_custom_config(self):
|
||||||
|
"""Test client initialization with custom configuration."""
|
||||||
|
client = QuerySidecarClient("http://custom:3000", timeout=30)
|
||||||
|
assert client._base_url == "http://custom:3000"
|
||||||
|
assert client._timeout == 30
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_build_query_object_success(self, mock_session_class):
|
||||||
|
"""Test successful query object building."""
|
||||||
|
# Mock the session and response
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.json.return_value = {
|
||||||
|
"query_object": {
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"filters": [],
|
||||||
|
"extras": {},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mock_session.post.return_value = mock_response
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
}
|
||||||
|
|
||||||
|
result = client.build_query_object(form_data)
|
||||||
|
|
||||||
|
assert result["metrics"] == ["count"]
|
||||||
|
assert result["columns"] == ["name"]
|
||||||
|
mock_session.post.assert_called_once()
|
||||||
|
|
||||||
|
def test_build_query_object_missing_form_data(self):
|
||||||
|
"""Test error when form_data is missing."""
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
|
||||||
|
with pytest.raises(QuerySidecarException, match="form_data is required"):
|
||||||
|
client.build_query_object(None)
|
||||||
|
|
||||||
|
def test_build_query_object_invalid_form_data(self):
|
||||||
|
"""Test error when form_data is missing required fields."""
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {"viz_type": "table"} # Missing datasource
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="must include datasource and viz_type"
|
||||||
|
):
|
||||||
|
client.build_query_object(form_data)
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_build_query_object_service_error(self, mock_session_class):
|
||||||
|
"""Test handling of service errors."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.json.return_value = {"error": "Service error"}
|
||||||
|
mock_session.post.return_value = mock_response
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
}
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="Sidecar service error: Service error"
|
||||||
|
):
|
||||||
|
client.build_query_object(form_data)
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_build_query_object_timeout(self, mock_session_class):
|
||||||
|
"""Test handling of timeout errors."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_session.post.side_effect = Timeout("Request timeout")
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
}
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="Query sidecar service timeout"
|
||||||
|
):
|
||||||
|
client.build_query_object(form_data)
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_build_query_object_connection_error(self, mock_session_class):
|
||||||
|
"""Test handling of connection errors."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_session.post.side_effect = ConnectionError("Connection failed")
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
}
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="Unable to connect to query sidecar service"
|
||||||
|
):
|
||||||
|
client.build_query_object(form_data)
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_build_query_object_http_error(self, mock_session_class):
|
||||||
|
"""Test handling of HTTP errors."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.status_code = 500
|
||||||
|
mock_response.json.return_value = {"error": "Internal server error"}
|
||||||
|
mock_http_error = HTTPError("500 Server Error")
|
||||||
|
mock_http_error.response = mock_response
|
||||||
|
mock_session.post.side_effect = mock_http_error
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
}
|
||||||
|
|
||||||
|
with pytest.raises(
|
||||||
|
QuerySidecarException, match="Query sidecar service is experiencing issues"
|
||||||
|
):
|
||||||
|
client.build_query_object(form_data)
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_create_query_object_from_form_data(self, mock_session_class):
|
||||||
|
"""Test creating QueryObject instance from form_data."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.json.return_value = {
|
||||||
|
"query_object": {
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
"filters": [],
|
||||||
|
"extras": {},
|
||||||
|
"row_limit": 1000,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
mock_session.post.return_value = mock_response
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
form_data = {
|
||||||
|
"datasource": "1__table",
|
||||||
|
"viz_type": "table",
|
||||||
|
"metrics": ["count"],
|
||||||
|
"columns": ["name"],
|
||||||
|
}
|
||||||
|
|
||||||
|
query_object = client.create_query_object_from_form_data(form_data)
|
||||||
|
|
||||||
|
assert query_object.metrics == ["count"]
|
||||||
|
assert query_object.columns == ["name"]
|
||||||
|
assert query_object.row_limit == 1000
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_health_check_healthy(self, mock_session_class):
|
||||||
|
"""Test health check when service is healthy."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_response = MagicMock()
|
||||||
|
mock_response.json.return_value = {"status": "healthy"}
|
||||||
|
mock_session.get.return_value = mock_response
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
assert client.health_check() is True
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.requests.Session")
|
||||||
|
def test_health_check_unhealthy(self, mock_session_class):
|
||||||
|
"""Test health check when service is unhealthy."""
|
||||||
|
mock_session = MagicMock()
|
||||||
|
mock_session_class.return_value = mock_session
|
||||||
|
mock_session.get.side_effect = ConnectionError("Connection failed")
|
||||||
|
|
||||||
|
client = QuerySidecarClient("http://test:3001")
|
||||||
|
assert client.health_check() is False
|
||||||
|
|
||||||
|
|
||||||
|
class TestModuleFunctions:
|
||||||
|
"""Tests for module-level functions."""
|
||||||
|
|
||||||
|
@patch("superset.utils.query_sidecar.get_query_sidecar_client")
|
||||||
|
def test_build_query_object_from_form_data(self, mock_get_client):
|
||||||
|
"""Test the convenience function for building query objects."""
|
||||||
|
mock_client = MagicMock()
|
||||||
|
mock_get_client.return_value = mock_client
|
||||||
|
mock_query_object = MagicMock()
|
||||||
|
mock_client.create_query_object_from_form_data.return_value = mock_query_object
|
||||||
|
|
||||||
|
form_data = {"datasource": "1__table", "viz_type": "table"}
|
||||||
|
|
||||||
|
result = build_query_object_from_form_data(form_data)
|
||||||
|
|
||||||
|
assert result == mock_query_object
|
||||||
|
mock_client.create_query_object_from_form_data.assert_called_once_with(
|
||||||
|
form_data, None, None
|
||||||
|
)
|
||||||
Reference in New Issue
Block a user