QT/superset2
1
0
Fork 0
mirror of https://github.com/apache/superset.git synced 2026-09-07 15:54:55 +00:00
Code Issues Packages Projects Releases Wiki Activity

refactor(coordination): reliable Redis Streams await/notify (replace at-most-once pub/sub) (#43409)

Browse Source
This commit is contained in:
Ville Brofeldt
2026-08-21 20:28:37 -07:00
committed by GitHub
parent f3ae075ccd
commit c7fcebf522
10 changed files with 450 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/developer_docs/extensions/tasks.md
+1 -1
View File
@@ -403,7 +403,7 @@ The prune job only removes tasks in terminal states (`SUCCESS`, `FAILURE`, `ABOR
See `superset/config.py` for a complete example configuration.
:::tip Distributed Coordination for Faster Notifications
By default, abort detection and sync join-and-wait use database polling. Configure `DISTRIBUTED_COORDINATION_CONFIG` to enable Redis pub/sub for real-time notifications. See [Distributed Coordination Backend](/admin-docs/configuration/cache#signal-cache-backend) for configuration details.
By default, abort detection and sync join-and-wait poll the task row in the metadata database. Configure `DISTRIBUTED_COORDINATION_CONFIG` (Redis/Valkey) and these become event-driven: completion and abort are signalled over Redis **Streams**, so a waiter wakes when the signal lands instead of polling the database. Because stream entries are persisted, a waiter that reads slightly late, reconnects, or fails over still receives the signal. Each signal stream keeps only its latest entry and is given a TTL, so streams for tasks that are never awaited do not accumulate; set the retention window with `DISTRIBUTED_COORDINATION_SIGNAL_TTL` (default 24h). See [Distributed Coordination Backend](/admin-docs/configuration/cache#signal-cache-backend) for configuration details.
:::
## API Reference