mirror of
https://github.com/apache/superset.git
synced 2026-09-09 08:44:32 +00:00
Hugh A. Miles IIandClaude Opus 5
6f5fb95215
Datasets on Hadoop-family engines are often partitioned on a technical column -- an epoch integer, a lowercased region key -- that no analyst would filter on. Unless a query carries a predicate on that column the engine scans every partition, and today the only workaround is hand-writing the predicate as custom SQL in a virtual dataset. This is the first of four PRs making that a dataset setting. It adds the storage and the save-time validation; nothing reads the mapping yet. Four columns, following the `always_filter_main_dttm` / `currency_code_column` precedent for "a dataset-level setting that names a column": tables.partition_column the physical partition column tables.partition_mapped_column override; NULL follows main_dttm_col table_columns.partition_value_transform the `:value` expression table_columns.partition_transform_is_monotonic gates range mirroring The monotonic flag is NOT NULL DEFAULT false rather than a nullable tri-state, matching `normalize_columns` -- a nullable boolean invites `if x:` bugs where None and False need distinguishing and don't get it. Validation runs in two tiers. Structural and safety errors block the save: unknown columns, a column mapped onto itself, Jinja in the transform, and non-deterministic functions. Everything else -- an unparseable transform, a transform missing `:value` -- saves and leaves the mapping inactive, so a half-written transform doesn't cost the owner the rest of their edits. Note the self-mapping check validates the *effective* mapped column. Checking only the explicit override misses the case an owner actually hits: pointing `partition_column` at the column that is already `main_dttm_col`. `SQLStatement.get_niladic_functions` is added because the denylist cannot be purely name-based: on Hive and Impala `unix_timestamp()` means "now" while `unix_timestamp(x)` -- the canonical transform for this feature -- is pure. Gated behind the `PARTITION_FILTER_MAPPING` feature flag, off by default. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>