Two small UX wins on top of the previous control-panel fixes:
1. Rename admin_level options to "World" / "Country" / "Aggregated
regions" (from the technical "Countries (Admin 0)" / "Subdivisions
(Admin 1)"). The control's own label becomes "Map view" with an
explanatory description. Underlying form_data values stay 0 / 1 /
aggregated so saved charts don't break.
2. Auto-migrate form_data when a user switches a saved legacy
country_map chart's viz type to country_map_v2:
- select_country: 'france' → admin_level=1, country=FRA
- select_country: 'france_overseas' → composite=france_overseas
- select_country: 'turkey_regions' → admin_level=aggregated,
country=TUR, region_set=nuts_1
- …same pattern for france_regions / italy_regions /
philippines_regions and ~180 other legacy country files.
The mapping lives in src/plugin/migrateFromLegacy.ts (covered by
12 unit tests). Migration only fills fields the user hasn't already
set on the new chart, so explicit edits win over inferred values.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Four UX bugs reported on the new country map chart:
1. **"Country: cannot be empty" stuck on the Data tab even at Admin 0.**
The Country control had a hard `validators: [validateNonEmpty]`, but
it's only rendered (and only meaningful) when admin_level != 0 and no
composite is set. The validator was firing for the hidden empty value
and trapped the user — the Update Chart button was permanently
disabled with no visible control to satisfy. Switched to a dynamic
validator via mapStateToProps so it only fires when the field is
actually needed.
2. **"Aggregated regions" threw TypeError: t.map is not a function.**
The region_set control used `choices: ({ controls }) => ...`, but
SelectControl's `choices` must be a literal array. Moved the
country-dependent choice computation into `mapStateToProps` (same
pattern other plugins use).
3. **Composite map always visible and "sticky" overriding admin_level.**
At Admin 0 (world choropleth) the composite override produced a map
that didn't change when you toggled admin_level. Hidden composite
control at Admin 0 entirely, and hidden it whenever the build
pipeline didn't emit any composites — leaves room for future
per-country scoping (e.g. only show france_overseas when country=FRA).
4. **Show flying islands checkbox was a no-op.** The build pipeline
doesn't currently tag features as flying, so the runtime drop logic
had nothing to act on. Removed from the control set rather than ship
a misleading control; a single line brings it back once the build
tags features.
Tests updated to cover all four behaviors.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Previously the build only emitted the ukr (Ukraine) worldview, so the
worldview dropdown had a single option even though it claimed otherwise.
Build now produces Admin 0 GeoJSON for every NE-published editorial:
default, arg, bdg, bra, chn, deu, egy, esp, fra, gbr, grc, idn, ind, iso,
isr, ita, jpn, kor, mar, nep, nld, pak, pol, prt, pse, rus, sau, swe, tur,
twn, ukr, usa, vnm (33 total).
NE does not publish per-worldview Admin 1 variants, so subdivisions within
a country come from a single shared file. The frontend now always points
Admin 1, regional aggregation, and composite URLs at the ukr-prefixed
shared outputs regardless of the selected worldview — the worldview
control only affects the world (Admin 0) map.
- build.py: expand WORLDVIEWS_ADMIN_0 to 33 worldviews; main() builds
Admin 0 for all of them, Admin 1 only for ukr
- transformProps.ts: introduce SHARED_ADMIN1_WORLDVIEW = 'ukr'; pin all
non-Admin-0 URLs to it
- controlPanel.tsx: WORLDVIEW_LABELS now covers all 33 codes; unrecognized
codes still fall back to raw code for forward-compat
- transformProps.test.ts: cover shared-Admin1 contract (admin1+chn still
resolves to ukr_admin1_*)
- pre-commit: exclude .geo.json from check-added-large-files (existing
rule only excluded .geojson and would block these ~2MB worldview files)
- README + SIP: document the worldview model and check off Phase 1 item
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The plugin imports src/data/manifest.json. The plugin's own tsconfig
already had resolveJsonModule + json in include after the previous fix,
but the root superset-frontend/tsconfig.json (which tsc --build walks)
also needs to resolve the manifest.json across the lib/ and esm/ babel
outputs. Add the explicit json glob and register the plugin as a
project reference so tsc finds it in build mode.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Add trailing newline to all 220 geo.json outputs to satisfy
end-of-file-fixer pre-commit hook
- build.py: post-process step ensures future regens emit trailing newlines
- CountryMap.tsx: import t/useTheme from @apache-superset/core
(matches the rest of the codebase) and use antd theme tokens
(colorBgSpotlight, colorTextLightSolid, colorErrorText, etc.)
instead of legacy theme.colors.* paths
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- transformProps: read snake_case via rawFormData (ChartProps.formData
is camelCased), fixing 4 failing jest tests
- CountryMap.tsx: replace literal colors with theme tokens; wrap user
strings with t() for i18n
- build.py: add proper dict[str, Any] type params, drop unused type:ignore,
emit manifest.json with trailing newline for prettier/EOF parity
- test_build.py: top-of-file mypy ignore (unittest test scaffolding)
- pyproject.toml: per-file ruff ignores for the standalone build pipeline
(TID251/S310/S603/S607/E501/C901/PT009 all intentional/inapplicable)
- regen workflow: surface drift via PR comment + step summary instead of
failing — cross-platform mapshaper output reproducibility is still WIP
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
- Remove unused CategoricalColorNamespace import in CountryMap.tsx
- Remove unused isAdminSubdivision helper in controlPanel.tsx
- Add resolveJsonModule + include manifest.json in tsconfig
- Cast chartProps.formData to CountryMapFormData in transformProps
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coordinated fixes for the 25 CI failures on the initial PR push:
1. **Lockfile sync.** Added @superset-ui/plugin-chart-country-map as
a workspace dep in the previous commit but didn't update
package-lock.json. CI's `npm ci` failed across frontend-build,
cypress (12 jobs), playwright (4 jobs), docker (2 jobs), and
frontend-check-translations. Re-ran `npm install --package-lock-only`
to add the new workspace's 71 lock entries.
2. **License headers added** to 13 new files flagged by License Check:
- 5 markdown READMEs / SIP_DRAFT (HTML-comment headers)
- 5 YAML config files (`# Licensed ...`)
- 2 Python files (`# Licensed ...`)
- 1 shell script (preserves shebang)
3. **Reproducible build outputs.** The regen workflow detected drift
on manifest.json + ukr_admin1_CAN.geo.json. Two root causes:
- `build_timestamp_utc` field made manifest non-deterministic →
dropped from the schema
- Floating mapshaper version (`npx --yes mapshaper`) caused subtle
simplification differences across runners → pinned to
`mapshaper@0.7.15` via `npx --yes mapshaper@<version>`
Verified locally: rebuild from clean cache reproduces every output
byte-identically except the manifest (which now also matches once
the timestamp is gone).
Files changed:
.gitignore — re-include rule for static dir
superset-frontend/package-lock.json — +71 lines for new workspace
13 new files — ASF headers
build.py — pin mapshaper, drop timestamp
manifest.json (× 2) — regenerate w/o timestamp
README.md (in static dir) — header
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mark Phase 2 (plugin scaffolding) substantially complete: hosting
path wired, click-to-zoom ported. Phase 3 (controls) fully done with
manifest-driven choices replacing hardcoded tables. Phase 5
substantially done: 41 tests across 4 test files, CI workflow,
UPDATING.md entry.
Remaining items, all non-blocking for SIP filing:
- "Switch to new Country Map" button on legacy plugin renderer
- Cross-filter integration via setDataMask
- Composite projection support (geoAlbersUsa for USA)
- Real example images (need actual rendering capture)
- Update Superset docs site
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Completes the test trio (transformProps + buildQuery + controlPanel).
Verifies:
- All 9 new controls (worldview / admin_level / country / region_set /
composite / region_includes / region_excludes / show_flying_islands /
name_language) are present in the panel
- Worldview defaults to 'ukr' (Superset's editorial choice)
- show_flying_islands defaults to true
- name_language defaults to 'en'
- admin_level offers exactly the 3 expected codes
- Country selector visibility hides on Admin 0 OR when composite set
- Region-set selector only visible when admin_level === 'aggregated'
- Region-set choices key off the selected country (TUR → nuts_1,
FRA → regions, USA → empty)
- Composite selector exposes france_overseas
These tests would fail loudly if anyone refactored the visibility
predicates or accidentally removed/renamed a control.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Covers the pure-Python helpers and YAML-applied transforms that the
build pipeline relies on. Subprocess calls (mapshaper, NE download)
are not exercised — those are integration concerns covered by the
regen workflow itself.
Test categories:
- _matches (4 tests): scalar equality, AND'd conditions, `in: [...]`
list-membership matcher, missing property
- _bbox_center (2 tests): unit square, offset square
- _translate_and_scale (4 tests): pure translate, scale-around-centroid,
combined transform, multipolygon handling
- _translate_and_scale_with_pivot (1 test): shared pivot preserves
relative positions of grouped features (the Paris-petite-couronne case)
- _drop_parts (2 tests): drops specified indices, polygon unchanged
- _bbox_contains (2 tests): inside-bbox, outside-bbox-west
- apply_name_overrides (1 test): applies only to matching features,
respects match conditions across countries (FRA "Seien" vs GBR "Seien"
don't collide)
- apply_flying_islands (2 tests): repositions matched features,
drop_outside_bbox guarded to Admin 1 only (the bug we fixed earlier)
Wired into the regen workflow as a step that runs BEFORE the build,
so a broken transform fails CI before producing potentially-bad output
files.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
New workflow `.github/workflows/country-map-build-regen.yml` that:
1. Triggers on PRs touching the build pipeline configs (or manually
via workflow_dispatch).
2. Sets up Python 3.11 + Node 22 + PyYAML.
3. Runs `./scripts/build.sh` from a clean checkout.
4. Detects output drift (uncommitted changes in
`superset/static/assets/country-maps/` or `src/data/manifest.json`).
5. If drift detected on a PR: leaves a comment telling the contributor
to re-run the build locally and commit, then fails CI.
This forces "configs and outputs stay in sync" — a contributor can't
add a name override / region aggregation / composite without also
updating the committed GeoJSON outputs.
Tagged contents permissions only — does not auto-PR (which would
require deeper repo setup); contributors are expected to regenerate
locally and push.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add a "Country Map plugin redesigned (legacy plugin deprecated)"
section under the Next release covering:
- What changed (modern endpoint, worldview, Admin 0+1, aggregated
regions, composite maps, region include/exclude, flying-islands
toggle, name-language selector, build pipeline)
- Migration behavior (existing dashboards keep working, no DB
migrations, deprecation badge in chart picker)
- For maintainers/power users (the notebook is no longer the source
of truth; local touchups go in YAML configs or procedural/ scripts)
Pointer to SIP_DRAFT.md for full design rationale.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Click a region: viewport zooms in (4x) centered on its centroid.
Click again or click the empty background: zoom back out.
600ms CSS transition on the SVG group.
Implementation note: zoom state lives on the DOM (zoomedFeature
local + group transform attribute) rather than React state. Two
reasons:
- Cleaner unmount (the transition completes naturally; no orphaned
React state)
- Avoids a re-render cycle on every click that would re-fit the
projection and undo the zoom
Background rect added to capture clicks outside any feature.
Cross-filter integration intentionally NOT added in this commit —
that needs the chart props' setDataMask path and a settings UI for
which feature property to use as the filter key. Separate concern,
follow-up commit.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace hardcoded WORLDVIEW / COUNTRY / REGION_SET / COMPOSITE choice
tables in controlPanel with options derived from the build pipeline's
manifest.json. Adding a new entry to a YAML config + re-running
./build.sh now populates the control automatically; no plugin code
change needed.
Build script: also writes manifest.json to the plugin's
src/data/manifest.json so controlPanel can `import` it synchronously
(no async fetch needed at chart-edit time). Data files (the actual
GeoJSONs) still live only at superset/static/assets/country-maps/ —
we don't want to bundle 17MB of choropleth data into the JS payload.
Lookup helpers preserve human-friendly labels:
- WORLDVIEW_LABELS — maps NE worldview codes (ukr, default, ind, ...)
to friendly names ("Ukraine (default — Crimea as Ukrainian)" etc.);
unmapped codes render as the raw code
- COUNTRY_LABELS — ISO_A3 → English country name (~85 entries);
formatCountry renders as "France (FRA)"; unmapped codes render raw
- REGION_SET_LABELS / COMPOSITE_LABELS — same pattern
Manifest's regional_aggregations array is grouped by country into
the {country: [(set_id, label), ...]} shape the control panel needs.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Move build pipeline outputs from a sibling `output/` directory into
Superset's Flask-served `superset/static/assets/country-maps/` so the
plugin can fetch them at runtime without webpack involvement (Flask
serves the tree at `/static/...` directly).
Commit the 220 generated GeoJSONs + manifest.json so a fresh
ephemeral environment can render the chart immediately, no build
step required at deploy time. Trade-off: ~17 MB of generated files
in-tree. (For comparison the legacy plugin commits ~34 MB of
GeoJSON; net change is -17 MB once we remove the legacy plugin in a
future major version.)
Files committed:
superset/static/assets/country-maps/
README.md (humans, not data)
manifest.json
ukr_admin0.geo.json 2.1 MB
ukr_admin1_<adm0_a3>.geo.json × 214 ~50 KB - 662 KB each
regional_<country>_<set>_ukr.geo.json × 4 ~30 KB each
composite_france_overseas_ukr.geo.json 322 KB
Build script changes:
- OUTPUT_DIR computed via SCRIPT_DIR.parents[3] / "superset" /
"static" / "assets" / "country-maps"
- mkdir(parents=True) so a fresh checkout works first run
- Stale `output/` entry kept in scripts/.gitignore for safety
(some local checkouts may have it from earlier iterations)
.gitignore: add re-include lines so superset/static/assets/country-maps/**
gets committed despite the broader superset/static/* exclusion.
SIP_DRAFT updated with the hosting-decision rationale.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Three coordinated changes:
1. **VizType.CountryMapV2 = 'country_map_v2'** — new enum value for the
modern plugin's chart type. Existing dashboards continue to use
`country_map` (legacy plugin); new charts get `country_map_v2`.
Naming follows the established pattern (LegacyBubble='bubble',
Bubble='bubble_v2'). Eventual swap (in a major version) can rename
so 'country_map' points to the new plugin.
2. **Legacy plugin marked deprecated**:
- Display name changed to "Country Map (Legacy)" so the chart picker
shows them side-by-side unambiguously
- label: ChartLabel.Deprecated — surfaces the standard deprecation
badge in the UI without hiding from the picker (existing dashboards
keep working; users see "this is deprecated" when editing)
- labelExplanation pointing users at the new chart
3. **MainPreset registers both**:
- CountryMapChartPlugin (legacy) → VizType.CountryMap
- CountryMapV2ChartPlugin (new) → VizType.CountryMapV2
Plus added @superset-ui/plugin-chart-country-map as a workspace
dependency in superset-frontend/package.json so the symlink resolves
during npm install.
Not yet wired (next commits):
- "Switch to new Country Map" button on the legacy plugin's UI
(the deprecation banner shows; the button to migrate form_data
is a separate change in the explore controls)
- Hosting path for the new plugin's GeoJSON outputs (currently
stubbed in transformProps — no static file serving wired yet)
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a manifest.json output the plugin can fetch at runtime to
populate worldview / country / region-set / composite dropdowns
dynamically. Adding a new entry to the YAML configs no longer requires
a plugin code change — re-run build.sh, manifest updates, plugin
controls reflect the new options.
Manifest schema:
ne_pinned_tag, ne_pinned_sha, build_timestamp_utc
worldviews: [<wv>, ...]
admin_levels: [0, 1]
countries_by_worldview: {<wv>: [<adm0_a3>, ...]}
regional_aggregations: [{country, set_id, worldview, size_bytes}, ...]
composites: [{id, worldview, size_bytes}, ...]
Sample current output:
1 worldview (ukr), 211 countries with subdivisions, 4 regional sets,
1 composite (france_overseas). Build pinned to NE v5.1.2.
Follow-up commit will replace the hardcoded choice tables in
controlPanel.tsx with manifest-driven options.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
transformProps tests cover the URL-derivation logic — the core piece
that maps form_data into the right build-pipeline output:
- Admin 0 + worldview → world choropleth URL
- Admin 1 + country → per-country file
- Region set + country → regional aggregation
- Composite overrides admin level + country
- Worldview defaults to 'ukr' when not specified
- Different worldviews reflected in URL
- Admin 1 without country → null URL (chart UI prompts)
- Pass-through of metricName/numberFormat/linearColorScheme
- Pass-through of query data + width/height
buildQuery tests cover that we're producing a valid chart/data
QueryContext with the expected shape (one query, form_data preserved,
orderby normalized as array).
These are the units most likely to break silently if someone refactors
the form_data → URL mapping or the query layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the placeholder control panel with the real form-data surface.
New controls (none of which exist on the legacy plugin):
- worldview: SelectControl, defaults to 'ukr' per documented editorial
choice; configurable per-deployment + per-chart
- admin_level: SelectControl (Admin 0 / Admin 1 / Aggregated regions)
- country: SelectControl, visible only when admin_level !== 0 and no
composite is selected
- region_set: SelectControl, visible only when admin_level === 'aggregated';
choices depend on selected country (sourced from the regional_aggregations
YAML's snapshot table)
- composite: SelectControl (france_overseas etc.); when set, overrides
admin_level + country
- region_includes / region_excludes: free-form multi-select for client-side
filtering at render time (these drive the projection refit too — the
renderer's `filterFeatures` uses them)
- show_flying_islands: CheckboxControl, default true
- name_language: SelectControl with 20 NE-supported languages
The choice tables (WORLDVIEW_CHOICES, COUNTRY_CHOICES, REGION_SET_CHOICES_
BY_COUNTRY, COMPOSITE_CHOICES) are hardcoded snapshots for the POC.
Follow-up commit will replace these with options pulled from the build
pipeline's manifest.json so adding a new worldview or country only
requires a build script run, not a plugin code change.
Standard query controls (entity, metric, adhoc_filters, row_limit,
linear_color_scheme, number_format) preserved from the legacy plugin.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replace the placeholder renderer with a real D3-based choropleth.
Modernizes the legacy plugin's rendering in three ways:
1. **Modern D3 modules** (d3-geo, d3-array, d3-color) instead of the
d3 v3 monolith. No more @ts-nocheck — fully typed.
2. **Built-in include/exclude filtering** (formData.region_includes /
region_excludes) before render. Projection auto-fits the FILTERED
feature set, so excluding Russia from a Europe view actually zooms
to Europe (the "fit to selection" behavior).
3. **Multi-language display names** via name_language formData field.
Falls back through name_<lang> → name → '' so any worldview/dataset
degrades gracefully.
Other changes:
- Tooltip is React state instead of D3-driven HTML (cleaner unmount,
no XSS surface from .html())
- Color scale uses sequential scheme registry from @superset-ui/core
(same path as legacy)
- Cleanup on unmount via useEffect cleanup function
Stubbed for follow-ups:
- Click-to-zoom (legacy had this; revisit after deprecation banner)
- Cross-filter integration
- Composite projection support (geoAlbersUsa for USA, etc.) — falls
back to plain Mercator + build-time repositioning for now
- Flying-islands toggle: feature-tagging in the build pipeline still
needs to land before the toggle can drop them at render time
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Captures things the maintainer should manually verify before merge,
organized by category:
- Build pipeline (data correctness)
- Visual/cartographic quality (in-chart rendering)
- Controls UX
- Backward compatibility
- Per-deployment customization (config)
- Performance + ops
Items get added/checked off as the implementation progresses.
Includes specific spot-checks the agent can't validate alone:
Crimea-as-Ukrainian visual, Russia Chukchi connectedness, India
J&K under _ukr worldview, French composite layout, etc.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Mark Phase 1 substantially complete (5/5 transforms working,
220 outputs verified, all schemas in place) and Phase 2 partially
complete (modern plugin scaffold compiles end-to-end, real renderer +
controls still TODO).
Add a "Current PR state" section showing the build script's actual
output so reviewers reading the SIP can see at a glance what the POC
produces today.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Bootstrap the new @superset-ui/plugin-chart-country-map package with
the modern ChartPlugin pattern. Mirrors plugin-chart-handlebars'
structure (the leanest existing modern plugin in the repo).
Files:
package.json — workspace pkg with d3-* deps
tsconfig.json — extends frontend root tsconfig,
references core/chart-controls
types/external.d.ts — ambient types for png/jpg/geojson
README.md — what this plugin is + status
src/
index.ts — package entry
types.ts — CountryMapFormData + transform props
CountryMap.tsx — placeholder renderer (real D3 port
in a follow-up commit)
plugin/
index.ts — ChartPlugin class with metadata
buildQuery.ts — modern chart/data query (this is
the core difference from the legacy
plugin's explore_json path)
controlPanel.tsx — minimal placeholder; full control
set (worldview / admin level /
country / region include-exclude /
fly-islands / language) ports next
transformProps.ts — derives geoJsonUrl from form_data
using the build-script's output
naming convention
Renderer scaffold fetches the resolved GeoJSON URL and shows a small
diagnostic block (URL + feature count + data row count + metric name)
so the plugin compiles end-to-end and can be wired to the chart-type
registry without a real D3 render. Real renderer + controls land in
the next commits.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The monolithic ukr_admin1.geo.json (15MB / 4595 features) was a single
file that any chart would have to download in full just to render one
country's subdivisions. Replace with per-country files keyed by
adm0_a3, each individually simplified.
Also drops single-subdivision countries (useless as choropleths) at
this stage, mirroring the notebook's auto-purge.
Output stats from full run:
Files: 220 total
1 × admin0 (world) ............ 2.1 MB
4 × regional aggregations ..... 23-32 KB each
1 × composite (france_overseas) 322 KB
214 × per-country admin1 ........ 17 KB - 662 KB each (GBR largest)
Per-chart payload:
world choropleth → ukr_admin0.geo.json 2.1 MB
France departments → ukr_admin1_FRA.geo.json 308 KB
US states → ukr_admin1_USA.geo.json ~250 KB
Türkiye NUTS-1 → regional_TUR_nuts_1_ukr.geo.json 23 KB
France w/ overseas → composite_france_overseas_ukr.geo.json 322 KB
All well within usable browser payload range. The plugin will lazy-load
only what's needed for the current chart's worldview/admin-level/country.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fifth and final transform from the notebook audit.
A composite combines a base country's Admin 1 features with:
- base_repositions (with optional `group: true` for grouped transforms
like Paris + petite couronne treated as one body)
- additions (features pulled from sibling countries' Admin 1, with
optional dissolve, drop_parts, reposition, and attribute set)
Verified on France-with-Overseas:
france_overseas: 108 features → composite_france_overseas_ukr.geo.json
(322,058 bytes)
108 = 101 FRA admin1 departments + 7 additions (Polynésie française,
Terres australes et antarctiques françaises, Wallis-et-Futuna,
Nouvelle-Calédonie, Saint-Pierre-et-Miquelon, Saint-Martin,
Saint-Barthélémy).
Bug fix during implementation: composites pull additions from Admin 1
of sibling countries (Windward Islands is a PYF Admin 1 subdivision,
not an Admin 0 country), not from Admin 0. Initial implementation got
this wrong and warned 0 features. Fixed by sourcing from base_admin1
(the global Admin 1 dataset, which contains all countries'
subdivisions).
New helpers:
- _drop_parts(geom, indices) — drop sub-polygon indices from MultiPolygon
- _translate_and_scale_with_pivot — explicit pivot (vs feature centroid),
used for `group: true` transforms
==== Build pipeline status ====
All 5 declarative transforms implemented and verified:
✓ name_overrides (19 updates per Admin 1 build)
✓ flying_islands (12 reposition + 5 bbox drop)
✓ territory_assignments (4 features added: TWN/HKG/MAC/ALD)
✓ regional_aggregations (4 region sets: TUR/FRA/ITA/PHL)
✓ composite_maps (1 composite: france_overseas)
Current outputs (UA worldview):
ukr_admin0.geo.json 2.1 MB 249 features
ukr_admin1.geo.json 15 MB 4595 features
regional_TUR_nuts_1_ukr.geo.json 23 KB 12 regions
regional_FRA_regions_ukr.geo.json 32 KB 18 regions
regional_ITA_regions_ukr.geo.json 32 KB 20 regions
regional_PHL_regions_ukr.geo.json 32 KB 17 regions
composite_france_overseas_ukr.geo.json 322 KB 108 features
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the fourth transform: dissolve Admin 1 subdivisions into
coarser administrative regions. Supports two mapping styles:
- explicit_mapping: per-region {name, members: [iso_3166_2, ...]}
- grouping_field: dissolve by an existing NE field (e.g. region_cod)
Verified counts match notebook expectations exactly:
TUR/nuts_1: 81 subdivisions → 12 regions → 23 KB
FRA/regions: 101 subdivisions → 18 regions → 32 KB
ITA/regions: 110 subdivisions → 20 regions → 32 KB
PHL/regions: 118 subdivisions → 17 regions → 32 KB
Per-region-set output is its own file (`regional_<country>_<set>_
<worldview>.geo.json`) so the plugin can lazy-load only what's needed
for the current chart.
Implementation:
- Filter base geo to country features
- Tag each with derived `_region_code` and `_region_name` (via reverse
lookup of explicit_mapping, or via grouping_field value)
- mapshaper -dissolve handles the polygon merging in one pass
- Rename derived fields → standard `iso_3166_2` and `name` on output
Output sizes are tiny — each per-chart payload becomes ~30 KB instead
of pulling the full Admin 1 layer.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Implements the third transform: pull features from sibling Admin 0
records into a destination country's Admin 1 view. Used for:
- China + Taiwan/HK/Macau (NE keeps each as separate Admin 0)
- Finland + Åland (missing from FIN admin 1; NE keeps Åland as ALD
admin 0)
Verified on real data:
Building worldview=ukr admin_level=1
territory_assignments: added 4 features from sibling Admin 0 records
(4 = TWN/HKG/MAC + ALD; ARMM-renamed BARMM region picks up correctly
because name_overrides ran first.)
Two bugs fixed along the way:
1. **Property name casing.** NE Admin 0 ships with uppercase property
names (ADM0_A3, NAME_EN), Admin 1 with lowercase. All transforms
downstream assume lowercase, so we now normalize to lowercase at
shapefile-conversion time. Bonus: fixes a silent flying_islands
bug where `adm0_a3` filters never matched at Admin 0 because the
props were uppercase.
2. **drop_outside_bbox at Admin 0.** A country's multi-polygon often
includes overseas territories (Netherlands → Caribbean), so bbox
filtering at Admin 0 would drop entire countries. Now guarded to
only run at Admin 1 where each feature is a single subdivision.
3. **Åland's NE code.** NE uses ALD, not the ISO 3166-1 ALA. Updated
territory_assignments.yaml with comment noting the divergence.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add mapshaper -simplify as a final-stage pass after the Python
transforms. Two-stage so the transforms operate on full-resolution
geometry (no risk of repositioning ghost features that got simplified
away) but the shipped output is browser-sized.
Default 5% with keep-shapes — preserves recognizable country shapes
while dropping the per-vertex bloat. keep-shapes prevents tiny features
(small island chains) from being collapsed away entirely.
Results on real outputs:
Admin 0: 23.6 MB → 2.1 MB (-91%, 249 features intact)
Admin 1: 67.7 MB → 15 MB (-78%, 4591 features, transforms preserved)
Per-feature simplify factors (the notebook had a size-based ladder)
can be added later if specific countries need tuning. 5% is the
mapshaper-recommended "good enough for web" default.
Open thought for future: split Admin 1 output by country
(`<worldview>_admin1_<adm0_a3>.geo.json`) so the per-chart payload is
~50KB-1MB instead of one 15MB global file. Will pair naturally with
the territory_assignments pass (which is per-country anyway).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Add Admin 1 build path and the second declarative transform. Exercises
the YAML config layer on real data:
Building worldview=ukr admin_level=1
loaded 4596 features
name_overrides: applied 19 field updates across 10 entries
flying_islands: repositioned 12 features, dropped 5 (outside-bbox)
wrote ukr_admin1.geo.json (67,677,079 bytes, 4591 features)
Counts verified against expectations:
- 19 name_overrides = 2 France typos + 6 France ISO codes
+ 5 PHL Caraga renames + 6 PHL BARMM renames
- 12 repositions = 2 USA + 1 NOR + 2 PRT + 2 ESP + 5 FRA
- 5 drops = NLD Caribbean + GBR overseas territories
New: pure-Python translate/scale geometry transform (no shapely dep);
operates on Polygon/MultiPolygon coordinates. Scale pivot is the bbox
center of each matched feature — good enough for the visual layout
purposes we use it for. Output bbox correctness verified by counts.
Refactor: extract `build_one(worldview, admin_level, ...)` so the
target matrix can grow in subsequent commits.
What's stubbed (TODO inline): territory_assignments, composite_maps,
regional_aggregations, simplification, procedural/. Output is
uncompressed and unsimplified (67MB) — simplification will land with
the mapshaper -simplify pass.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
End-to-end working pipeline replacing the legacy notebook for one
worldview / admin level. Verified locally:
$ ./build.sh
Country Map build — pinned to NE v5.1.2 (f1890d9f)
Loaded 10 name override entries
Building worldview=ukr admin_level=0
Downloading NE ne_10m_admin_0_countries_ukr (worldview=ukr)…
mapshaper: ne_10m_admin_0_countries_ukr.shp → _raw_ukr_admin0.geo.json
loaded 249 features
name_overrides: applied 0 field updates across 10 entries
wrote .../output/ukr_admin0.geo.json (23,639,348 bytes)
Done.
What's wired:
- NE download from pinned tag (v5.1.2 / SHA f1890d9f) with cache
- Shapefile → GeoJSON via mapshaper CLI
- YAML config loading (currently just name_overrides)
- name_overrides transform with {match, set} semantics, including
the {in: [...]} list-membership matcher
- Output writes to scripts/output/ (gitignored)
- build.sh wrapper validates Python + Node + PyYAML are available
What's stubbed for future commits (TODO inline):
- Multiple worldviews (currently UA only)
- Admin 1 build (where name_overrides actually fire — currently no
features in Admin 0 match the FRA/PHL admin1 entries)
- flying_islands, territory_assignments, regional_aggregations,
composite_maps transforms
- Simplification (mapshaper -simplify)
- Procedural escape-hatch orchestration
- Manifest with NE SHA + build metadata
The 0 overrides applied is correct, not a bug: all current entries
target Admin 1 features.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
The original draft was missing entries because notebook cell 63 was
truncated in the audit. Reading the full cell surfaced:
- Saint Martin (MAF) + Saint Barthélémy (BLM) as additional sister
Admin 0 territories (small Caribbean islands, scaled up significantly
for visibility — 5x and 8x respectively)
- Paris + petite couronne (Hauts-de-Seine, Seine-Saint-Denis,
Val-de-Marne) as a metropolitan zoom-in (group + translate + scale 3x)
- Per-territory metadata renames (Polynésie française, Nouvelle-
Calédonie, etc.) + ISO 3166-2 code assignments (FR-PF, FR-NC, etc.)
Schema additions:
- base_repositions[].group: true — when match yields multiple features,
transform them as a single MultiPolygon then split back out
(preserves per-feature attributes). Used for the Paris zoom-in.
- additions[].set: { name, iso_3166_2, ... } — override attributes on
the added/dissolved feature
SPM offset placeholder is gone; composite definition now matches the
notebook's output exactly (modulo the build script implementing the
declarative schema).
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
First-pass schemas for the build pipeline's declarative config layer.
Each schema is documented inline + populated with concrete entries
ported from the legacy notebook's audited touchups (those that the
obsolescence check determined still need to ship).
scripts/
├── README.md — pipeline overview, layout, workflow
├── config/
│ ├── name_overrides.yaml — France typos, ISO codes; PHL renames
│ ├── flying_islands.yaml — USA/NOR/PRT/ESP/FRA repositions; NLD/GBR drops
│ ├── territory_assignments.yaml — China + SARs; Finland + Åland
│ ├── regional_aggregations.yaml — Turkey NUTS-1; FRA/ITA/PHL regions
│ └── composite_maps.yaml — France-with-Overseas
└── procedural/
└── README.md — escape-hatch rules + skeleton (currently empty)
All five YAML files parse cleanly (validated with PyYAML).
Schema design choices:
- Every entry has a `description:` field. Forces honest documentation
of why each fix exists; reviewers can scan rationale at a glance.
- Match semantics: simple AND-of-conditions; supports `{ in: [...] }`
for value-set matching.
- composite_maps and territory_assignments share the "pull feature
from sibling Admin 0" primitive; build script can implement once.
- composite_maps.yaml has a TODO marker for SPM offsets — notebook
cell 63 was truncated in the audit; will backfill during build
script implementation.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Most touchups fit declarative YAML cleanly (typos, ISO codes,
repositions, dissolves, composites). A few don't — e.g., the
France-with-Overseas notebook drops a specific sub-polygon by index
to avoid visual conflict with Corsica.
Add a `procedural/` directory of small, named, single-purpose Python
scripts as an escape hatch. Build orchestrator runs YAML transforms
first, then iterates procedural/*.py in numeric order.
Why this is much better than the notebook for the same content:
- Each fix is a separate file → conflicts localize to one fix at a
time, not "the whole notebook touched the same cell"
- No kernel state, no output churn, pure functions on data
- Naming forces documentation; reviewers see add/remove at a glance
- Bounded growth: a 50-file procedural/ dir is annoying enough to
signal "push back into YAML or upstream"; the notebook had no
such signal and just bloated
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Verified each notebook touchup against current Natural Earth Admin 1
data. Headline wins:
OBSOLETE (NE caught up or worldview handles it):
- Vietnam diacritics: NE has correct values in name_vi field; manual
rewrites collapse to zero by exposing name_language=vi
- Crimea/Sevastopol: handled by _ukr worldview (already confirmed)
- India Kashmir/Ladakh geometry: NE has UTs with correct ISO codes;
worldview selection adjusts northern boundary
- Russia Chukchi antimeridian: NE has split geometry into properly-
bounded polygons (x range [-180, 180]); D3 projection should handle
- Latvia third-party file: NE has 119 fine-grained admin1 features
- Philippines third-party file: NE has 118 admin1 features
- external_overrides.yaml DROPPED from the design
STILL NEEDED (NE still wrong / no upstream support):
- France typos (Seien-et-Marne, Haute-Rhin still in NE)
- France ISO codes (FR-75 → FR-75C, FR-GP → FR-971, etc. — 6 features)
- Philippines admin renames (Dinagat→Caraga, ARMM→BARMM, 11 features)
- China + SARs (Taiwan/HK/Macau still separate Admin 0 in NE)
- Finland + Åland (missing from FIN admin 1 in NE)
- Regional aggregations (Turkey NUTS-1, France/Italy/PHL regions)
- France-with-Overseas composite
- Flying-island repositions (USA, Norway, Portugal, Spain, France DOM)
Net: design surface shrinks, no third-party data dependencies in-tree,
config files stable enough that ongoing maintenance is small.
Two follow-up verifications still pending: Russia Chukchi renders
correctly under D3, India J&K geometry matches current Indian
boundary expectations under _ukr worldview.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Read all 96 cells of the legacy notebook; categorized every kind of
work it does into 14 buckets and mapped each to either:
- ✅ already covered by the proposed design
- 🟡 covered but requires a YAML config we'll need to port
- 🟠 needs a NEW config file or pipeline feature beyond what was sketched
- ⚪ might be obsolete in current NE — verify per case
Surfaced four NEW config-driven build features beyond the original
sketch:
- territory_assignments.yaml (China + SARs, Finland + Åland)
- regional_aggregations.yaml (Turkey NUTS-1, France/Italy/Philippines
regions; new "Aggregated regions" admin level)
- composite_maps.yaml (France-with-Overseas multi-country composite)
- external_overrides.yaml (India/Latvia/Philippines third-party
GeoJSON; verify whether still needed against current NE)
Plus extended flying_islands.yaml with drop|reposition action modes
to subsume the bounds-clipping pattern (Netherlands, UK).
Updated open questions, smoke-test fixtures (now 5 cases that
exercise the full surface), and Phase 1 of the implementation plan
with the obsolescence check + external-GeoJSON check as concrete
follow-ups.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Living document tracking the design for the Country Map rework. Will
be filed as a proper SIP issue when the POC matures; this file is
working scratch in the meantime.
Captured so far:
- Motivation: disputed-borders flashpoints, Jupyter notebook pain,
legacy explore_json endpoint
- Goals: configurable worldview, Admin 0 + Admin 1, per-deployment
+ per-chart customization, mapshaper-based reproducible build
pipeline, modern chart/data endpoint
- Spike findings: NE UA worldview vs Default — confirms Crimea-as-
Ukrainian + several other commonly-expected positions (Kosovo,
Western Sahara, Palestine, Cyprus, Kashmir)
- Deprecation plan: in-UI "switch to new Country Map" button + banner
on legacy, modeled on existing deprecated-chart pattern
Open questions tracked inline; implementation phases sketched.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
