mirror of
https://github.com/apache/superset.git
synced 2026-05-22 08:15:36 +00:00
2db41bb2b2
End-to-end working pipeline replacing the legacy notebook for one
worldview / admin level. Verified locally:
$ ./build.sh
Country Map build — pinned to NE v5.1.2 (f1890d9f)
Loaded 10 name override entries
Building worldview=ukr admin_level=0
Downloading NE ne_10m_admin_0_countries_ukr (worldview=ukr)…
mapshaper: ne_10m_admin_0_countries_ukr.shp → _raw_ukr_admin0.geo.json
loaded 249 features
name_overrides: applied 0 field updates across 10 entries
wrote .../output/ukr_admin0.geo.json (23,639,348 bytes)
Done.
What's wired:
- NE download from pinned tag (v5.1.2 / SHA f1890d9f) with cache
- Shapefile → GeoJSON via mapshaper CLI
- YAML config loading (currently just name_overrides)
- name_overrides transform with {match, set} semantics, including
the {in: [...]} list-membership matcher
- Output writes to scripts/output/ (gitignored)
- build.sh wrapper validates Python + Node + PyYAML are available
What's stubbed for future commits (TODO inline):
- Multiple worldviews (currently UA only)
- Admin 1 build (where name_overrides actually fire — currently no
features in Admin 0 match the FRA/PHL admin1 entries)
- flying_islands, territory_assignments, regional_aggregations,
composite_maps transforms
- Simplification (mapshaper -simplify)
- Procedural escape-hatch orchestration
- Manifest with NE SHA + build metadata
The 0 overrides applied is correct, not a bug: all current entries
target Admin 1 features.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Country Map data pipeline
This directory contains the build pipeline that turns upstream Natural Earth data into the GeoJSON files consumed by @superset-ui/plugin-chart-country-map.
It replaces the legacy scripts/Country Map GeoJSON Generator.ipynb notebook. See SIP_DRAFT.md in the parent directory for the full design rationale.
Layout
scripts/
build.sh # one-shot reproducible build
README.md # this file
config/ # declarative YAML — handles ~95% of fixes
name_overrides.yaml # typos, deprecated ISO codes, admin renames
flying_islands.yaml # repositioning + bbox drops for far-flung territories
territory_assignments.yaml # add features from sibling Admin 0 records
regional_aggregations.yaml # dissolve Admin 1 into administrative regions
composite_maps.yaml # multi-country composites (e.g. France-with-Overseas)
procedural/ # escape hatch — handles the rare 5%
README.md # when to use, when not
NN_<descriptive_name>.py # one focused script per genuine edge case
output/ # gitignored — build artifacts
Operating principles
- Default tool: declarative YAML. Most touchups are renames, repositions, dissolves, or filters — all expressible in YAML. Diffs are small, conflicts localize cleanly to one entry, contributors can submit "fix typo X" as a one-line PR.
- Escape hatch:
procedural/directory of small, named, single-purpose Python scripts for the rare cases YAML can't express cleanly. Each script has a header comment explaining why it's not in YAML. Seeprocedural/README.mdfor the bar. - Build is reproducible from a pinned NE version.
build.shrecords the NE git SHA it consumed; outputs are deterministic given inputs. - CI regenerates on schema change and opens a PR if outputs differ. Maintainers review the cartographic diff in legible GeoJSON, not opaque notebook JSON.
Workflow for adding a fix
- Identify the upstream NE issue (wrong name, missing territory, etc.).
- Try YAML first. Add the smallest possible entry to the appropriate config file with a
descriptionfield explaining the fix. - If YAML can't express it cleanly, add a numbered script in
procedural/with a header comment explaining why YAML didn't fit. - Run
build.shlocally, verify the output GeoJSON looks right. - Open PR. Reviewer sees the YAML diff (or new procedural script) plus the regenerated GeoJSON.
See also
SIP_DRAFT.md(parent dir) — design rationale, notebook audit, obsolescence checkprocedural/README.md— when to use the escape hatch