End-to-end working pipeline replacing the legacy notebook for one
worldview / admin level. Verified locally:
$ ./build.sh
Country Map build — pinned to NE v5.1.2 (f1890d9f)
Loaded 10 name override entries
Building worldview=ukr admin_level=0
Downloading NE ne_10m_admin_0_countries_ukr (worldview=ukr)…
mapshaper: ne_10m_admin_0_countries_ukr.shp → _raw_ukr_admin0.geo.json
loaded 249 features
name_overrides: applied 0 field updates across 10 entries
wrote .../output/ukr_admin0.geo.json (23,639,348 bytes)
Done.
What's wired:
- NE download from pinned tag (v5.1.2 / SHA f1890d9f) with cache
- Shapefile → GeoJSON via mapshaper CLI
- YAML config loading (currently just name_overrides)
- name_overrides transform with {match, set} semantics, including
the {in: [...]} list-membership matcher
- Output writes to scripts/output/ (gitignored)
- build.sh wrapper validates Python + Node + PyYAML are available
What's stubbed for future commits (TODO inline):
- Multiple worldviews (currently UA only)
- Admin 1 build (where name_overrides actually fire — currently no
features in Admin 0 match the FRA/PHL admin1 entries)
- flying_islands, territory_assignments, regional_aggregations,
composite_maps transforms
- Simplification (mapshaper -simplify)
- Procedural escape-hatch orchestration
- Manifest with NE SHA + build metadata
The 0 overrides applied is correct, not a bug: all current entries
target Admin 1 features.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The original draft was missing entries because notebook cell 63 was
truncated in the audit. Reading the full cell surfaced:
- Saint Martin (MAF) + Saint Barthélémy (BLM) as additional sister
Admin 0 territories (small Caribbean islands, scaled up significantly
for visibility — 5x and 8x respectively)
- Paris + petite couronne (Hauts-de-Seine, Seine-Saint-Denis,
Val-de-Marne) as a metropolitan zoom-in (group + translate + scale 3x)
- Per-territory metadata renames (Polynésie française, Nouvelle-
Calédonie, etc.) + ISO 3166-2 code assignments (FR-PF, FR-NC, etc.)
Schema additions:
- base_repositions[].group: true — when match yields multiple features,
transform them as a single MultiPolygon then split back out
(preserves per-feature attributes). Used for the Paris zoom-in.
- additions[].set: { name, iso_3166_2, ... } — override attributes on
the added/dissolved feature
SPM offset placeholder is gone; composite definition now matches the
notebook's output exactly (modulo the build script implementing the
declarative schema).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First-pass schemas for the build pipeline's declarative config layer.
Each schema is documented inline + populated with concrete entries
ported from the legacy notebook's audited touchups (those that the
obsolescence check determined still need to ship).
scripts/
├── README.md — pipeline overview, layout, workflow
├── config/
│ ├── name_overrides.yaml — France typos, ISO codes; PHL renames
│ ├── flying_islands.yaml — USA/NOR/PRT/ESP/FRA repositions; NLD/GBR drops
│ ├── territory_assignments.yaml — China + SARs; Finland + Åland
│ ├── regional_aggregations.yaml — Turkey NUTS-1; FRA/ITA/PHL regions
│ └── composite_maps.yaml — France-with-Overseas
└── procedural/
└── README.md — escape-hatch rules + skeleton (currently empty)
All five YAML files parse cleanly (validated with PyYAML).
Schema design choices:
- Every entry has a `description:` field. Forces honest documentation
of why each fix exists; reviewers can scan rationale at a glance.
- Match semantics: simple AND-of-conditions; supports `{ in: [...] }`
for value-set matching.
- composite_maps and territory_assignments share the "pull feature
from sibling Admin 0" primitive; build script can implement once.
- composite_maps.yaml has a TODO marker for SPM offsets — notebook
cell 63 was truncated in the audit; will backfill during build
script implementation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Most touchups fit declarative YAML cleanly (typos, ISO codes,
repositions, dissolves, composites). A few don't — e.g., the
France-with-Overseas notebook drops a specific sub-polygon by index
to avoid visual conflict with Corsica.
Add a `procedural/` directory of small, named, single-purpose Python
scripts as an escape hatch. Build orchestrator runs YAML transforms
first, then iterates procedural/*.py in numeric order.
Why this is much better than the notebook for the same content:
- Each fix is a separate file → conflicts localize to one fix at a
time, not "the whole notebook touched the same cell"
- No kernel state, no output churn, pure functions on data
- Naming forces documentation; reviewers see add/remove at a glance
- Bounded growth: a 50-file procedural/ dir is annoying enough to
signal "push back into YAML or upstream"; the notebook had no
such signal and just bloated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verified each notebook touchup against current Natural Earth Admin 1
data. Headline wins:
OBSOLETE (NE caught up or worldview handles it):
- Vietnam diacritics: NE has correct values in name_vi field; manual
rewrites collapse to zero by exposing name_language=vi
- Crimea/Sevastopol: handled by _ukr worldview (already confirmed)
- India Kashmir/Ladakh geometry: NE has UTs with correct ISO codes;
worldview selection adjusts northern boundary
- Russia Chukchi antimeridian: NE has split geometry into properly-
bounded polygons (x range [-180, 180]); D3 projection should handle
- Latvia third-party file: NE has 119 fine-grained admin1 features
- Philippines third-party file: NE has 118 admin1 features
- external_overrides.yaml DROPPED from the design
STILL NEEDED (NE still wrong / no upstream support):
- France typos (Seien-et-Marne, Haute-Rhin still in NE)
- France ISO codes (FR-75 → FR-75C, FR-GP → FR-971, etc. — 6 features)
- Philippines admin renames (Dinagat→Caraga, ARMM→BARMM, 11 features)
- China + SARs (Taiwan/HK/Macau still separate Admin 0 in NE)
- Finland + Åland (missing from FIN admin 1 in NE)
- Regional aggregations (Turkey NUTS-1, France/Italy/PHL regions)
- France-with-Overseas composite
- Flying-island repositions (USA, Norway, Portugal, Spain, France DOM)
Net: design surface shrinks, no third-party data dependencies in-tree,
config files stable enough that ongoing maintenance is small.
Two follow-up verifications still pending: Russia Chukchi renders
correctly under D3, India J&K geometry matches current Indian
boundary expectations under _ukr worldview.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Read all 96 cells of the legacy notebook; categorized every kind of
work it does into 14 buckets and mapped each to either:
- ✅ already covered by the proposed design
- 🟡 covered but requires a YAML config we'll need to port
- 🟠 needs a NEW config file or pipeline feature beyond what was sketched
- ⚪ might be obsolete in current NE — verify per case
Surfaced four NEW config-driven build features beyond the original
sketch:
- territory_assignments.yaml (China + SARs, Finland + Åland)
- regional_aggregations.yaml (Turkey NUTS-1, France/Italy/Philippines
regions; new "Aggregated regions" admin level)
- composite_maps.yaml (France-with-Overseas multi-country composite)
- external_overrides.yaml (India/Latvia/Philippines third-party
GeoJSON; verify whether still needed against current NE)
Plus extended flying_islands.yaml with drop|reposition action modes
to subsume the bounds-clipping pattern (Netherlands, UK).
Updated open questions, smoke-test fixtures (now 5 cases that
exercise the full surface), and Phase 1 of the implementation plan
with the obsolescence check + external-GeoJSON check as concrete
follow-ups.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Living document tracking the design for the Country Map rework. Will
be filed as a proper SIP issue when the POC matures; this file is
working scratch in the meantime.
Captured so far:
- Motivation: disputed-borders flashpoints, Jupyter notebook pain,
legacy explore_json endpoint
- Goals: configurable worldview, Admin 0 + Admin 1, per-deployment
+ per-chart customization, mapshaper-based reproducible build
pipeline, modern chart/data endpoint
- Spike findings: NE UA worldview vs Default — confirms Crimea-as-
Ukrainian + several other commonly-expected positions (Kosovo,
Western Sahara, Palestine, Cyprus, Kashmir)
- Deprecation plan: in-UI "switch to new Country Map" button + banner
on legacy, modeled on existing deprecated-chart pattern
Open questions tracked inline; implementation phases sketched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
