Previously the build only emitted the ukr (Ukraine) worldview, so the
worldview dropdown had a single option even though it claimed otherwise.
Build now produces Admin 0 GeoJSON for every NE-published editorial:
default, arg, bdg, bra, chn, deu, egy, esp, fra, gbr, grc, idn, ind, iso,
isr, ita, jpn, kor, mar, nep, nld, pak, pol, prt, pse, rus, sau, swe, tur,
twn, ukr, usa, vnm (33 total).
NE does not publish per-worldview Admin 1 variants, so subdivisions within
a country come from a single shared file. The frontend now always points
Admin 1, regional aggregation, and composite URLs at the ukr-prefixed
shared outputs regardless of the selected worldview — the worldview
control only affects the world (Admin 0) map.
- build.py: expand WORLDVIEWS_ADMIN_0 to 33 worldviews; main() builds
Admin 0 for all of them, Admin 1 only for ukr
- transformProps.ts: introduce SHARED_ADMIN1_WORLDVIEW = 'ukr'; pin all
non-Admin-0 URLs to it
- controlPanel.tsx: WORLDVIEW_LABELS now covers all 33 codes; unrecognized
codes still fall back to raw code for forward-compat
- transformProps.test.ts: cover shared-Admin1 contract (admin1+chn still
resolves to ukr_admin1_*)
- pre-commit: exclude .geo.json from check-added-large-files (existing
rule only excluded .geojson and would block these ~2MB worldview files)
- README + SIP: document the worldview model and check off Phase 1 item
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add trailing newline to all 220 geo.json outputs to satisfy
end-of-file-fixer pre-commit hook
- build.py: post-process step ensures future regens emit trailing newlines
- CountryMap.tsx: import t/useTheme from @apache-superset/core
(matches the rest of the codebase) and use antd theme tokens
(colorBgSpotlight, colorTextLightSolid, colorErrorText, etc.)
instead of legacy theme.colors.* paths
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- transformProps: read snake_case via rawFormData (ChartProps.formData
is camelCased), fixing 4 failing jest tests
- CountryMap.tsx: replace literal colors with theme tokens; wrap user
strings with t() for i18n
- build.py: add proper dict[str, Any] type params, drop unused type:ignore,
emit manifest.json with trailing newline for prettier/EOF parity
- test_build.py: top-of-file mypy ignore (unittest test scaffolding)
- pyproject.toml: per-file ruff ignores for the standalone build pipeline
(TID251/S310/S603/S607/E501/C901/PT009 all intentional/inapplicable)
- regen workflow: surface drift via PR comment + step summary instead of
failing — cross-platform mapshaper output reproducibility is still WIP
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coordinated fixes for the 25 CI failures on the initial PR push:
1. **Lockfile sync.** Added @superset-ui/plugin-chart-country-map as
a workspace dep in the previous commit but didn't update
package-lock.json. CI's `npm ci` failed across frontend-build,
cypress (12 jobs), playwright (4 jobs), docker (2 jobs), and
frontend-check-translations. Re-ran `npm install --package-lock-only`
to add the new workspace's 71 lock entries.
2. **License headers added** to 13 new files flagged by License Check:
- 5 markdown READMEs / SIP_DRAFT (HTML-comment headers)
- 5 YAML config files (`# Licensed ...`)
- 2 Python files (`# Licensed ...`)
- 1 shell script (preserves shebang)
3. **Reproducible build outputs.** The regen workflow detected drift
on manifest.json + ukr_admin1_CAN.geo.json. Two root causes:
- `build_timestamp_utc` field made manifest non-deterministic →
dropped from the schema
- Floating mapshaper version (`npx --yes mapshaper`) caused subtle
simplification differences across runners → pinned to
`mapshaper@0.7.15` via `npx --yes mapshaper@<version>`
Verified locally: rebuild from clean cache reproduces every output
byte-identically except the manifest (which now also matches once
the timestamp is gone).
Files changed:
.gitignore — re-include rule for static dir
superset-frontend/package-lock.json — +71 lines for new workspace
13 new files — ASF headers
build.py — pin mapshaper, drop timestamp
manifest.json (× 2) — regenerate w/o timestamp
README.md (in static dir) — header
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Covers the pure-Python helpers and YAML-applied transforms that the
build pipeline relies on. Subprocess calls (mapshaper, NE download)
are not exercised — those are integration concerns covered by the
regen workflow itself.
Test categories:
- _matches (4 tests): scalar equality, AND'd conditions, `in: [...]`
list-membership matcher, missing property
- _bbox_center (2 tests): unit square, offset square
- _translate_and_scale (4 tests): pure translate, scale-around-centroid,
combined transform, multipolygon handling
- _translate_and_scale_with_pivot (1 test): shared pivot preserves
relative positions of grouped features (the Paris-petite-couronne case)
- _drop_parts (2 tests): drops specified indices, polygon unchanged
- _bbox_contains (2 tests): inside-bbox, outside-bbox-west
- apply_name_overrides (1 test): applies only to matching features,
respects match conditions across countries (FRA "Seien" vs GBR "Seien"
don't collide)
- apply_flying_islands (2 tests): repositions matched features,
drop_outside_bbox guarded to Admin 1 only (the bug we fixed earlier)
Wired into the regen workflow as a step that runs BEFORE the build,
so a broken transform fails CI before producing potentially-bad output
files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace hardcoded WORLDVIEW / COUNTRY / REGION_SET / COMPOSITE choice
tables in controlPanel with options derived from the build pipeline's
manifest.json. Adding a new entry to a YAML config + re-running
./build.sh now populates the control automatically; no plugin code
change needed.
Build script: also writes manifest.json to the plugin's
src/data/manifest.json so controlPanel can `import` it synchronously
(no async fetch needed at chart-edit time). Data files (the actual
GeoJSONs) still live only at superset/static/assets/country-maps/ —
we don't want to bundle 17MB of choropleth data into the JS payload.
Lookup helpers preserve human-friendly labels:
- WORLDVIEW_LABELS — maps NE worldview codes (ukr, default, ind, ...)
to friendly names ("Ukraine (default — Crimea as Ukrainian)" etc.);
unmapped codes render as the raw code
- COUNTRY_LABELS — ISO_A3 → English country name (~85 entries);
formatCountry renders as "France (FRA)"; unmapped codes render raw
- REGION_SET_LABELS / COMPOSITE_LABELS — same pattern
Manifest's regional_aggregations array is grouped by country into
the {country: [(set_id, label), ...]} shape the control panel needs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move build pipeline outputs from a sibling `output/` directory into
Superset's Flask-served `superset/static/assets/country-maps/` so the
plugin can fetch them at runtime without webpack involvement (Flask
serves the tree at `/static/...` directly).
Commit the 220 generated GeoJSONs + manifest.json so a fresh
ephemeral environment can render the chart immediately, no build
step required at deploy time. Trade-off: ~17 MB of generated files
in-tree. (For comparison the legacy plugin commits ~34 MB of
GeoJSON; net change is -17 MB once we remove the legacy plugin in a
future major version.)
Files committed:
superset/static/assets/country-maps/
README.md (humans, not data)
manifest.json
ukr_admin0.geo.json 2.1 MB
ukr_admin1_<adm0_a3>.geo.json × 214 ~50 KB - 662 KB each
regional_<country>_<set>_ukr.geo.json × 4 ~30 KB each
composite_france_overseas_ukr.geo.json 322 KB
Build script changes:
- OUTPUT_DIR computed via SCRIPT_DIR.parents[3] / "superset" /
"static" / "assets" / "country-maps"
- mkdir(parents=True) so a fresh checkout works first run
- Stale `output/` entry kept in scripts/.gitignore for safety
(some local checkouts may have it from earlier iterations)
.gitignore: add re-include lines so superset/static/assets/country-maps/**
gets committed despite the broader superset/static/* exclusion.
SIP_DRAFT updated with the hosting-decision rationale.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a manifest.json output the plugin can fetch at runtime to
populate worldview / country / region-set / composite dropdowns
dynamically. Adding a new entry to the YAML configs no longer requires
a plugin code change — re-run build.sh, manifest updates, plugin
controls reflect the new options.
Manifest schema:
ne_pinned_tag, ne_pinned_sha, build_timestamp_utc
worldviews: [<wv>, ...]
admin_levels: [0, 1]
countries_by_worldview: {<wv>: [<adm0_a3>, ...]}
regional_aggregations: [{country, set_id, worldview, size_bytes}, ...]
composites: [{id, worldview, size_bytes}, ...]
Sample current output:
1 worldview (ukr), 211 countries with subdivisions, 4 regional sets,
1 composite (france_overseas). Build pinned to NE v5.1.2.
Follow-up commit will replace the hardcoded choice tables in
controlPanel.tsx with manifest-driven options.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The monolithic ukr_admin1.geo.json (15MB / 4595 features) was a single
file that any chart would have to download in full just to render one
country's subdivisions. Replace with per-country files keyed by
adm0_a3, each individually simplified.
Also drops single-subdivision countries (useless as choropleths) at
this stage, mirroring the notebook's auto-purge.
Output stats from full run:
Files: 220 total
1 × admin0 (world) ............ 2.1 MB
4 × regional aggregations ..... 23-32 KB each
1 × composite (france_overseas) 322 KB
214 × per-country admin1 ........ 17 KB - 662 KB each (GBR largest)
Per-chart payload:
world choropleth → ukr_admin0.geo.json 2.1 MB
France departments → ukr_admin1_FRA.geo.json 308 KB
US states → ukr_admin1_USA.geo.json ~250 KB
Türkiye NUTS-1 → regional_TUR_nuts_1_ukr.geo.json 23 KB
France w/ overseas → composite_france_overseas_ukr.geo.json 322 KB
All well within usable browser payload range. The plugin will lazy-load
only what's needed for the current chart's worldview/admin-level/country.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fifth and final transform from the notebook audit.
A composite combines a base country's Admin 1 features with:
- base_repositions (with optional `group: true` for grouped transforms
like Paris + petite couronne treated as one body)
- additions (features pulled from sibling countries' Admin 1, with
optional dissolve, drop_parts, reposition, and attribute set)
Verified on France-with-Overseas:
france_overseas: 108 features → composite_france_overseas_ukr.geo.json
(322,058 bytes)
108 = 101 FRA admin1 departments + 7 additions (Polynésie française,
Terres australes et antarctiques françaises, Wallis-et-Futuna,
Nouvelle-Calédonie, Saint-Pierre-et-Miquelon, Saint-Martin,
Saint-Barthélémy).
Bug fix during implementation: composites pull additions from Admin 1
of sibling countries (Windward Islands is a PYF Admin 1 subdivision,
not an Admin 0 country), not from Admin 0. Initial implementation got
this wrong and warned 0 features. Fixed by sourcing from base_admin1
(the global Admin 1 dataset, which contains all countries'
subdivisions).
New helpers:
- _drop_parts(geom, indices) — drop sub-polygon indices from MultiPolygon
- _translate_and_scale_with_pivot — explicit pivot (vs feature centroid),
used for `group: true` transforms
==== Build pipeline status ====
All 5 declarative transforms implemented and verified:
✓ name_overrides (19 updates per Admin 1 build)
✓ flying_islands (12 reposition + 5 bbox drop)
✓ territory_assignments (4 features added: TWN/HKG/MAC/ALD)
✓ regional_aggregations (4 region sets: TUR/FRA/ITA/PHL)
✓ composite_maps (1 composite: france_overseas)
Current outputs (UA worldview):
ukr_admin0.geo.json 2.1 MB 249 features
ukr_admin1.geo.json 15 MB 4595 features
regional_TUR_nuts_1_ukr.geo.json 23 KB 12 regions
regional_FRA_regions_ukr.geo.json 32 KB 18 regions
regional_ITA_regions_ukr.geo.json 32 KB 20 regions
regional_PHL_regions_ukr.geo.json 32 KB 17 regions
composite_france_overseas_ukr.geo.json 322 KB 108 features
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fourth transform: dissolve Admin 1 subdivisions into
coarser administrative regions. Supports two mapping styles:
- explicit_mapping: per-region {name, members: [iso_3166_2, ...]}
- grouping_field: dissolve by an existing NE field (e.g. region_cod)
Verified counts match notebook expectations exactly:
TUR/nuts_1: 81 subdivisions → 12 regions → 23 KB
FRA/regions: 101 subdivisions → 18 regions → 32 KB
ITA/regions: 110 subdivisions → 20 regions → 32 KB
PHL/regions: 118 subdivisions → 17 regions → 32 KB
Per-region-set output is its own file (`regional_<country>_<set>_
<worldview>.geo.json`) so the plugin can lazy-load only what's needed
for the current chart.
Implementation:
- Filter base geo to country features
- Tag each with derived `_region_code` and `_region_name` (via reverse
lookup of explicit_mapping, or via grouping_field value)
- mapshaper -dissolve handles the polygon merging in one pass
- Rename derived fields → standard `iso_3166_2` and `name` on output
Output sizes are tiny — each per-chart payload becomes ~30 KB instead
of pulling the full Admin 1 layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the third transform: pull features from sibling Admin 0
records into a destination country's Admin 1 view. Used for:
- China + Taiwan/HK/Macau (NE keeps each as separate Admin 0)
- Finland + Åland (missing from FIN admin 1; NE keeps Åland as ALD
admin 0)
Verified on real data:
Building worldview=ukr admin_level=1
territory_assignments: added 4 features from sibling Admin 0 records
(4 = TWN/HKG/MAC + ALD; ARMM-renamed BARMM region picks up correctly
because name_overrides ran first.)
Two bugs fixed along the way:
1. **Property name casing.** NE Admin 0 ships with uppercase property
names (ADM0_A3, NAME_EN), Admin 1 with lowercase. All transforms
downstream assume lowercase, so we now normalize to lowercase at
shapefile-conversion time. Bonus: fixes a silent flying_islands
bug where `adm0_a3` filters never matched at Admin 0 because the
props were uppercase.
2. **drop_outside_bbox at Admin 0.** A country's multi-polygon often
includes overseas territories (Netherlands → Caribbean), so bbox
filtering at Admin 0 would drop entire countries. Now guarded to
only run at Admin 1 where each feature is a single subdivision.
3. **Åland's NE code.** NE uses ALD, not the ISO 3166-1 ALA. Updated
territory_assignments.yaml with comment noting the divergence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add mapshaper -simplify as a final-stage pass after the Python
transforms. Two-stage so the transforms operate on full-resolution
geometry (no risk of repositioning ghost features that got simplified
away) but the shipped output is browser-sized.
Default 5% with keep-shapes — preserves recognizable country shapes
while dropping the per-vertex bloat. keep-shapes prevents tiny features
(small island chains) from being collapsed away entirely.
Results on real outputs:
Admin 0: 23.6 MB → 2.1 MB (-91%, 249 features intact)
Admin 1: 67.7 MB → 15 MB (-78%, 4591 features, transforms preserved)
Per-feature simplify factors (the notebook had a size-based ladder)
can be added later if specific countries need tuning. 5% is the
mapshaper-recommended "good enough for web" default.
Open thought for future: split Admin 1 output by country
(`<worldview>_admin1_<adm0_a3>.geo.json`) so the per-chart payload is
~50KB-1MB instead of one 15MB global file. Will pair naturally with
the territory_assignments pass (which is per-country anyway).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add Admin 1 build path and the second declarative transform. Exercises
the YAML config layer on real data:
Building worldview=ukr admin_level=1
loaded 4596 features
name_overrides: applied 19 field updates across 10 entries
flying_islands: repositioned 12 features, dropped 5 (outside-bbox)
wrote ukr_admin1.geo.json (67,677,079 bytes, 4591 features)
Counts verified against expectations:
- 19 name_overrides = 2 France typos + 6 France ISO codes
+ 5 PHL Caraga renames + 6 PHL BARMM renames
- 12 repositions = 2 USA + 1 NOR + 2 PRT + 2 ESP + 5 FRA
- 5 drops = NLD Caribbean + GBR overseas territories
New: pure-Python translate/scale geometry transform (no shapely dep);
operates on Polygon/MultiPolygon coordinates. Scale pivot is the bbox
center of each matched feature — good enough for the visual layout
purposes we use it for. Output bbox correctness verified by counts.
Refactor: extract `build_one(worldview, admin_level, ...)` so the
target matrix can grow in subsequent commits.
What's stubbed (TODO inline): territory_assignments, composite_maps,
regional_aggregations, simplification, procedural/. Output is
uncompressed and unsimplified (67MB) — simplification will land with
the mapshaper -simplify pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
End-to-end working pipeline replacing the legacy notebook for one
worldview / admin level. Verified locally:
$ ./build.sh
Country Map build — pinned to NE v5.1.2 (f1890d9f)
Loaded 10 name override entries
Building worldview=ukr admin_level=0
Downloading NE ne_10m_admin_0_countries_ukr (worldview=ukr)…
mapshaper: ne_10m_admin_0_countries_ukr.shp → _raw_ukr_admin0.geo.json
loaded 249 features
name_overrides: applied 0 field updates across 10 entries
wrote .../output/ukr_admin0.geo.json (23,639,348 bytes)
Done.
What's wired:
- NE download from pinned tag (v5.1.2 / SHA f1890d9f) with cache
- Shapefile → GeoJSON via mapshaper CLI
- YAML config loading (currently just name_overrides)
- name_overrides transform with {match, set} semantics, including
the {in: [...]} list-membership matcher
- Output writes to scripts/output/ (gitignored)
- build.sh wrapper validates Python + Node + PyYAML are available
What's stubbed for future commits (TODO inline):
- Multiple worldviews (currently UA only)
- Admin 1 build (where name_overrides actually fire — currently no
features in Admin 0 match the FRA/PHL admin1 entries)
- flying_islands, territory_assignments, regional_aggregations,
composite_maps transforms
- Simplification (mapshaper -simplify)
- Procedural escape-hatch orchestration
- Manifest with NE SHA + build metadata
The 0 overrides applied is correct, not a bug: all current entries
target Admin 1 features.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The original draft was missing entries because notebook cell 63 was
truncated in the audit. Reading the full cell surfaced:
- Saint Martin (MAF) + Saint Barthélémy (BLM) as additional sister
Admin 0 territories (small Caribbean islands, scaled up significantly
for visibility — 5x and 8x respectively)
- Paris + petite couronne (Hauts-de-Seine, Seine-Saint-Denis,
Val-de-Marne) as a metropolitan zoom-in (group + translate + scale 3x)
- Per-territory metadata renames (Polynésie française, Nouvelle-
Calédonie, etc.) + ISO 3166-2 code assignments (FR-PF, FR-NC, etc.)
Schema additions:
- base_repositions[].group: true — when match yields multiple features,
transform them as a single MultiPolygon then split back out
(preserves per-feature attributes). Used for the Paris zoom-in.
- additions[].set: { name, iso_3166_2, ... } — override attributes on
the added/dissolved feature
SPM offset placeholder is gone; composite definition now matches the
notebook's output exactly (modulo the build script implementing the
declarative schema).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First-pass schemas for the build pipeline's declarative config layer.
Each schema is documented inline + populated with concrete entries
ported from the legacy notebook's audited touchups (those that the
obsolescence check determined still need to ship).
scripts/
├── README.md — pipeline overview, layout, workflow
├── config/
│ ├── name_overrides.yaml — France typos, ISO codes; PHL renames
│ ├── flying_islands.yaml — USA/NOR/PRT/ESP/FRA repositions; NLD/GBR drops
│ ├── territory_assignments.yaml — China + SARs; Finland + Åland
│ ├── regional_aggregations.yaml — Turkey NUTS-1; FRA/ITA/PHL regions
│ └── composite_maps.yaml — France-with-Overseas
└── procedural/
└── README.md — escape-hatch rules + skeleton (currently empty)
All five YAML files parse cleanly (validated with PyYAML).
Schema design choices:
- Every entry has a `description:` field. Forces honest documentation
of why each fix exists; reviewers can scan rationale at a glance.
- Match semantics: simple AND-of-conditions; supports `{ in: [...] }`
for value-set matching.
- composite_maps and territory_assignments share the "pull feature
from sibling Admin 0" primitive; build script can implement once.
- composite_maps.yaml has a TODO marker for SPM offsets — notebook
cell 63 was truncated in the audit; will backfill during build
script implementation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
