The monolithic ukr_admin1.geo.json (15MB / 4595 features) was a single
file that any chart would have to download in full just to render one
country's subdivisions. Replace with per-country files keyed by
adm0_a3, each individually simplified.
Also drops single-subdivision countries (useless as choropleths) at
this stage, mirroring the notebook's auto-purge.
Output stats from full run:
Files: 220 total
1 × admin0 (world) ............ 2.1 MB
4 × regional aggregations ..... 23-32 KB each
1 × composite (france_overseas) 322 KB
214 × per-country admin1 ........ 17 KB - 662 KB each (GBR largest)
Per-chart payload:
world choropleth → ukr_admin0.geo.json 2.1 MB
France departments → ukr_admin1_FRA.geo.json 308 KB
US states → ukr_admin1_USA.geo.json ~250 KB
Türkiye NUTS-1 → regional_TUR_nuts_1_ukr.geo.json 23 KB
France w/ overseas → composite_france_overseas_ukr.geo.json 322 KB
All well within usable browser payload range. The plugin will lazy-load
only what's needed for the current chart's worldview/admin-level/country.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fifth and final transform from the notebook audit.
A composite combines a base country's Admin 1 features with:
- base_repositions (with optional `group: true` for grouped transforms
like Paris + petite couronne treated as one body)
- additions (features pulled from sibling countries' Admin 1, with
optional dissolve, drop_parts, reposition, and attribute set)
Verified on France-with-Overseas:
france_overseas: 108 features → composite_france_overseas_ukr.geo.json
(322,058 bytes)
108 = 101 FRA admin1 departments + 7 additions (Polynésie française,
Terres australes et antarctiques françaises, Wallis-et-Futuna,
Nouvelle-Calédonie, Saint-Pierre-et-Miquelon, Saint-Martin,
Saint-Barthélémy).
Bug fix during implementation: composites pull additions from Admin 1
of sibling countries (Windward Islands is a PYF Admin 1 subdivision,
not an Admin 0 country), not from Admin 0. Initial implementation got
this wrong and warned 0 features. Fixed by sourcing from base_admin1
(the global Admin 1 dataset, which contains all countries'
subdivisions).
New helpers:
- _drop_parts(geom, indices) — drop sub-polygon indices from MultiPolygon
- _translate_and_scale_with_pivot — explicit pivot (vs feature centroid),
used for `group: true` transforms
==== Build pipeline status ====
All 5 declarative transforms implemented and verified:
✓ name_overrides (19 updates per Admin 1 build)
✓ flying_islands (12 reposition + 5 bbox drop)
✓ territory_assignments (4 features added: TWN/HKG/MAC/ALD)
✓ regional_aggregations (4 region sets: TUR/FRA/ITA/PHL)
✓ composite_maps (1 composite: france_overseas)
Current outputs (UA worldview):
ukr_admin0.geo.json 2.1 MB 249 features
ukr_admin1.geo.json 15 MB 4595 features
regional_TUR_nuts_1_ukr.geo.json 23 KB 12 regions
regional_FRA_regions_ukr.geo.json 32 KB 18 regions
regional_ITA_regions_ukr.geo.json 32 KB 20 regions
regional_PHL_regions_ukr.geo.json 32 KB 17 regions
composite_france_overseas_ukr.geo.json 322 KB 108 features
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fourth transform: dissolve Admin 1 subdivisions into
coarser administrative regions. Supports two mapping styles:
- explicit_mapping: per-region {name, members: [iso_3166_2, ...]}
- grouping_field: dissolve by an existing NE field (e.g. region_cod)
Verified counts match notebook expectations exactly:
TUR/nuts_1: 81 subdivisions → 12 regions → 23 KB
FRA/regions: 101 subdivisions → 18 regions → 32 KB
ITA/regions: 110 subdivisions → 20 regions → 32 KB
PHL/regions: 118 subdivisions → 17 regions → 32 KB
Per-region-set output is its own file (`regional_<country>_<set>_
<worldview>.geo.json`) so the plugin can lazy-load only what's needed
for the current chart.
Implementation:
- Filter base geo to country features
- Tag each with derived `_region_code` and `_region_name` (via reverse
lookup of explicit_mapping, or via grouping_field value)
- mapshaper -dissolve handles the polygon merging in one pass
- Rename derived fields → standard `iso_3166_2` and `name` on output
Output sizes are tiny — each per-chart payload becomes ~30 KB instead
of pulling the full Admin 1 layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the third transform: pull features from sibling Admin 0
records into a destination country's Admin 1 view. Used for:
- China + Taiwan/HK/Macau (NE keeps each as separate Admin 0)
- Finland + Åland (missing from FIN admin 1; NE keeps Åland as ALD
admin 0)
Verified on real data:
Building worldview=ukr admin_level=1
territory_assignments: added 4 features from sibling Admin 0 records
(4 = TWN/HKG/MAC + ALD; ARMM-renamed BARMM region picks up correctly
because name_overrides ran first.)
Two bugs fixed along the way:
1. **Property name casing.** NE Admin 0 ships with uppercase property
names (ADM0_A3, NAME_EN), Admin 1 with lowercase. All transforms
downstream assume lowercase, so we now normalize to lowercase at
shapefile-conversion time. Bonus: fixes a silent flying_islands
bug where `adm0_a3` filters never matched at Admin 0 because the
props were uppercase.
2. **drop_outside_bbox at Admin 0.** A country's multi-polygon often
includes overseas territories (Netherlands → Caribbean), so bbox
filtering at Admin 0 would drop entire countries. Now guarded to
only run at Admin 1 where each feature is a single subdivision.
3. **Åland's NE code.** NE uses ALD, not the ISO 3166-1 ALA. Updated
territory_assignments.yaml with comment noting the divergence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add mapshaper -simplify as a final-stage pass after the Python
transforms. Two-stage so the transforms operate on full-resolution
geometry (no risk of repositioning ghost features that got simplified
away) but the shipped output is browser-sized.
Default 5% with keep-shapes — preserves recognizable country shapes
while dropping the per-vertex bloat. keep-shapes prevents tiny features
(small island chains) from being collapsed away entirely.
Results on real outputs:
Admin 0: 23.6 MB → 2.1 MB (-91%, 249 features intact)
Admin 1: 67.7 MB → 15 MB (-78%, 4591 features, transforms preserved)
Per-feature simplify factors (the notebook had a size-based ladder)
can be added later if specific countries need tuning. 5% is the
mapshaper-recommended "good enough for web" default.
Open thought for future: split Admin 1 output by country
(`<worldview>_admin1_<adm0_a3>.geo.json`) so the per-chart payload is
~50KB-1MB instead of one 15MB global file. Will pair naturally with
the territory_assignments pass (which is per-country anyway).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add Admin 1 build path and the second declarative transform. Exercises
the YAML config layer on real data:
Building worldview=ukr admin_level=1
loaded 4596 features
name_overrides: applied 19 field updates across 10 entries
flying_islands: repositioned 12 features, dropped 5 (outside-bbox)
wrote ukr_admin1.geo.json (67,677,079 bytes, 4591 features)
Counts verified against expectations:
- 19 name_overrides = 2 France typos + 6 France ISO codes
+ 5 PHL Caraga renames + 6 PHL BARMM renames
- 12 repositions = 2 USA + 1 NOR + 2 PRT + 2 ESP + 5 FRA
- 5 drops = NLD Caribbean + GBR overseas territories
New: pure-Python translate/scale geometry transform (no shapely dep);
operates on Polygon/MultiPolygon coordinates. Scale pivot is the bbox
center of each matched feature — good enough for the visual layout
purposes we use it for. Output bbox correctness verified by counts.
Refactor: extract `build_one(worldview, admin_level, ...)` so the
target matrix can grow in subsequent commits.
What's stubbed (TODO inline): territory_assignments, composite_maps,
regional_aggregations, simplification, procedural/. Output is
uncompressed and unsimplified (67MB) — simplification will land with
the mapshaper -simplify pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
End-to-end working pipeline replacing the legacy notebook for one
worldview / admin level. Verified locally:
$ ./build.sh
Country Map build — pinned to NE v5.1.2 (f1890d9f)
Loaded 10 name override entries
Building worldview=ukr admin_level=0
Downloading NE ne_10m_admin_0_countries_ukr (worldview=ukr)…
mapshaper: ne_10m_admin_0_countries_ukr.shp → _raw_ukr_admin0.geo.json
loaded 249 features
name_overrides: applied 0 field updates across 10 entries
wrote .../output/ukr_admin0.geo.json (23,639,348 bytes)
Done.
What's wired:
- NE download from pinned tag (v5.1.2 / SHA f1890d9f) with cache
- Shapefile → GeoJSON via mapshaper CLI
- YAML config loading (currently just name_overrides)
- name_overrides transform with {match, set} semantics, including
the {in: [...]} list-membership matcher
- Output writes to scripts/output/ (gitignored)
- build.sh wrapper validates Python + Node + PyYAML are available
What's stubbed for future commits (TODO inline):
- Multiple worldviews (currently UA only)
- Admin 1 build (where name_overrides actually fire — currently no
features in Admin 0 match the FRA/PHL admin1 entries)
- flying_islands, territory_assignments, regional_aggregations,
composite_maps transforms
- Simplification (mapshaper -simplify)
- Procedural escape-hatch orchestration
- Manifest with NE SHA + build metadata
The 0 overrides applied is correct, not a bug: all current entries
target Admin 1 features.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The original draft was missing entries because notebook cell 63 was
truncated in the audit. Reading the full cell surfaced:
- Saint Martin (MAF) + Saint Barthélémy (BLM) as additional sister
Admin 0 territories (small Caribbean islands, scaled up significantly
for visibility — 5x and 8x respectively)
- Paris + petite couronne (Hauts-de-Seine, Seine-Saint-Denis,
Val-de-Marne) as a metropolitan zoom-in (group + translate + scale 3x)
- Per-territory metadata renames (Polynésie française, Nouvelle-
Calédonie, etc.) + ISO 3166-2 code assignments (FR-PF, FR-NC, etc.)
Schema additions:
- base_repositions[].group: true — when match yields multiple features,
transform them as a single MultiPolygon then split back out
(preserves per-feature attributes). Used for the Paris zoom-in.
- additions[].set: { name, iso_3166_2, ... } — override attributes on
the added/dissolved feature
SPM offset placeholder is gone; composite definition now matches the
notebook's output exactly (modulo the build script implementing the
declarative schema).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First-pass schemas for the build pipeline's declarative config layer.
Each schema is documented inline + populated with concrete entries
ported from the legacy notebook's audited touchups (those that the
obsolescence check determined still need to ship).
scripts/
├── README.md — pipeline overview, layout, workflow
├── config/
│ ├── name_overrides.yaml — France typos, ISO codes; PHL renames
│ ├── flying_islands.yaml — USA/NOR/PRT/ESP/FRA repositions; NLD/GBR drops
│ ├── territory_assignments.yaml — China + SARs; Finland + Åland
│ ├── regional_aggregations.yaml — Turkey NUTS-1; FRA/ITA/PHL regions
│ └── composite_maps.yaml — France-with-Overseas
└── procedural/
└── README.md — escape-hatch rules + skeleton (currently empty)
All five YAML files parse cleanly (validated with PyYAML).
Schema design choices:
- Every entry has a `description:` field. Forces honest documentation
of why each fix exists; reviewers can scan rationale at a glance.
- Match semantics: simple AND-of-conditions; supports `{ in: [...] }`
for value-set matching.
- composite_maps and territory_assignments share the "pull feature
from sibling Admin 0" primitive; build script can implement once.
- composite_maps.yaml has a TODO marker for SPM offsets — notebook
cell 63 was truncated in the audit; will backfill during build
script implementation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Most touchups fit declarative YAML cleanly (typos, ISO codes,
repositions, dissolves, composites). A few don't — e.g., the
France-with-Overseas notebook drops a specific sub-polygon by index
to avoid visual conflict with Corsica.
Add a `procedural/` directory of small, named, single-purpose Python
scripts as an escape hatch. Build orchestrator runs YAML transforms
first, then iterates procedural/*.py in numeric order.
Why this is much better than the notebook for the same content:
- Each fix is a separate file → conflicts localize to one fix at a
time, not "the whole notebook touched the same cell"
- No kernel state, no output churn, pure functions on data
- Naming forces documentation; reviewers see add/remove at a glance
- Bounded growth: a 50-file procedural/ dir is annoying enough to
signal "push back into YAML or upstream"; the notebook had no
such signal and just bloated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verified each notebook touchup against current Natural Earth Admin 1
data. Headline wins:
OBSOLETE (NE caught up or worldview handles it):
- Vietnam diacritics: NE has correct values in name_vi field; manual
rewrites collapse to zero by exposing name_language=vi
- Crimea/Sevastopol: handled by _ukr worldview (already confirmed)
- India Kashmir/Ladakh geometry: NE has UTs with correct ISO codes;
worldview selection adjusts northern boundary
- Russia Chukchi antimeridian: NE has split geometry into properly-
bounded polygons (x range [-180, 180]); D3 projection should handle
- Latvia third-party file: NE has 119 fine-grained admin1 features
- Philippines third-party file: NE has 118 admin1 features
- external_overrides.yaml DROPPED from the design
STILL NEEDED (NE still wrong / no upstream support):
- France typos (Seien-et-Marne, Haute-Rhin still in NE)
- France ISO codes (FR-75 → FR-75C, FR-GP → FR-971, etc. — 6 features)
- Philippines admin renames (Dinagat→Caraga, ARMM→BARMM, 11 features)
- China + SARs (Taiwan/HK/Macau still separate Admin 0 in NE)
- Finland + Åland (missing from FIN admin 1 in NE)
- Regional aggregations (Turkey NUTS-1, France/Italy/PHL regions)
- France-with-Overseas composite
- Flying-island repositions (USA, Norway, Portugal, Spain, France DOM)
Net: design surface shrinks, no third-party data dependencies in-tree,
config files stable enough that ongoing maintenance is small.
Two follow-up verifications still pending: Russia Chukchi renders
correctly under D3, India J&K geometry matches current Indian
boundary expectations under _ukr worldview.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Read all 96 cells of the legacy notebook; categorized every kind of
work it does into 14 buckets and mapped each to either:
- ✅ already covered by the proposed design
- 🟡 covered but requires a YAML config we'll need to port
- 🟠 needs a NEW config file or pipeline feature beyond what was sketched
- ⚪ might be obsolete in current NE — verify per case
Surfaced four NEW config-driven build features beyond the original
sketch:
- territory_assignments.yaml (China + SARs, Finland + Åland)
- regional_aggregations.yaml (Turkey NUTS-1, France/Italy/Philippines
regions; new "Aggregated regions" admin level)
- composite_maps.yaml (France-with-Overseas multi-country composite)
- external_overrides.yaml (India/Latvia/Philippines third-party
GeoJSON; verify whether still needed against current NE)
Plus extended flying_islands.yaml with drop|reposition action modes
to subsume the bounds-clipping pattern (Netherlands, UK).
Updated open questions, smoke-test fixtures (now 5 cases that
exercise the full surface), and Phase 1 of the implementation plan
with the obsolescence check + external-GeoJSON check as concrete
follow-ups.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Living document tracking the design for the Country Map rework. Will
be filed as a proper SIP issue when the POC matures; this file is
working scratch in the meantime.
Captured so far:
- Motivation: disputed-borders flashpoints, Jupyter notebook pain,
legacy explore_json endpoint
- Goals: configurable worldview, Admin 0 + Admin 1, per-deployment
+ per-chart customization, mapshaper-based reproducible build
pipeline, modern chart/data endpoint
- Spike findings: NE UA worldview vs Default — confirms Crimea-as-
Ukrainian + several other commonly-expected positions (Kosovo,
Western Sahara, Palestine, Cyprus, Kashmir)
- Deprecation plan: in-UI "switch to new Country Map" button + banner
on legacy, modeled on existing deprecated-chart pattern
Open questions tracked inline; implementation phases sketched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
