mirror of
https://github.com/apache/superset.git
synced 2026-09-06 07:21:35 +00:00
feat(gtf): add task dependencies (DAG) with chain-icon Task List display (#43408)
This commit is contained in:
@@ -333,6 +333,33 @@ assert task.uuid == task2.uuid # True
|
||||
print(task2.status) # "success" (terminal status)
|
||||
```
|
||||
|
||||
## Task Dependencies
|
||||
|
||||
Tasks can declare prerequisite tasks, forming a directed acyclic graph (DAG). Pass the prerequisite `Task` objects (returned by `.schedule()`) via `depends_on`:
|
||||
|
||||
```python
|
||||
from superset_core.tasks.types import TaskOptions
|
||||
|
||||
totals = totals_task.schedule(options=TaskOptions(task_key="totals_123"))
|
||||
|
||||
# `dependent` only runs once `totals` has finished successfully.
|
||||
dependent = dependent_task.schedule(
|
||||
options=TaskOptions(depends_on=[totals])
|
||||
)
|
||||
```
|
||||
|
||||
Passing the `Task` object is the canonical pattern. For convenience, a prerequisite's `UUID` (or UUID string) is also accepted where you don't hold the `Task` itself.
|
||||
|
||||
**Semantics (`all_success`).** A task runs only once **every** direct prerequisite has reached a terminal `SUCCESS`. If **any** prerequisite ends in a non-`SUCCESS` terminal state (`FAILURE`, `ABORTED`, or `TIMED_OUT`), the dependent does **not** run and is transitioned to `FAILURE`. This propagates transitively: because a failed dependent is itself non-`SUCCESS`, its own dependents fail in turn, so a failure anywhere short-circuits everything downstream.
|
||||
|
||||
**Scheduling model (block-and-wait).** All tasks in a DAG are enqueued immediately. Each dependent's worker blocks — holding its worker slot — until its prerequisites finish; while waiting, the task remains `PENDING` (shown as "waiting on N prerequisites" in the Task List). Tasks are enqueued in dependency order, so a dependent is rarely dequeued before its prerequisites.
|
||||
|
||||
:::warning Worker fleet sizing
|
||||
Because dependents hold a worker slot while awaiting their prerequisites, a deep or wide DAG can occupy many workers simultaneously. Deployments that use chained tasks heavily must size their Celery worker fleet large enough to absorb the idle waiting, or a large DAG can exhaust the pool and deadlock.
|
||||
:::
|
||||
|
||||
Cycles (including self-dependencies) are rejected at schedule time. Dependency edges are removed automatically when either endpoint task is pruned.
|
||||
|
||||
## Task Scopes
|
||||
|
||||
```python
|
||||
@@ -409,13 +436,15 @@ By default, abort detection and sync join-and-wait use database polling. Configu
|
||||
TaskOptions(
|
||||
task_key: str | None = None,
|
||||
task_name: str | None = None,
|
||||
timeout: int | None = None
|
||||
timeout: int | None = None,
|
||||
depends_on: list[Task | UUID | str] | None = None
|
||||
)
|
||||
```
|
||||
|
||||
- `task_key`: Deduplication key (also used as display name if `task_name` is not set)
|
||||
- `task_name`: Human-readable display name for the Task List UI
|
||||
- `timeout`: Timeout in seconds (overrides decorator default)
|
||||
- `depends_on`: Prerequisite tasks to wait for before running. Pass the scheduled `Task` objects (canonical); a `UUID` or UUID string is also accepted (see [Task Dependencies](#task-dependencies))
|
||||
|
||||
:::tip
|
||||
Provide a descriptive `task_name` for better readability in the Task List UI. While `task_key` is used for deduplication and may be technical (e.g., `chart_export_123`), `task_name` can be user-friendly (e.g., `"Export Sales Chart 123"`).
|
||||
|
||||
Reference in New Issue
Block a user