viz: a region NAME is not a unique identity — two same-named regions with equal values silently merge

Split out of #4526, where this was handed over from local review of #4534 (PR #4536). It is a distinct question from that issue's coarse-geography denominator hole and blocks on a different decision, so it gets its own number.

Problem

viz smart keys its region-level dedupe on the owning-region cell, and is_region_concept deliberately accepts city and county NAMES (geo.city, geo.county, town, municipality) alongside the code concepts — #4529 widened it on purpose, since an event-level dataset keyed by CITY repeats its regional values exactly like one keyed by county FIPS.

A name is not unique. Two Springfield rows can be different cities in different states:

values for the duplicated name	outcome	safe?
differ (Aurora CO 386,261 vs Aurora IL 180,542)	registers as a conflict; the KPI tile is omitted	yes — closed by PR #4536
exactly equal	the two regions merge into one key and the total silently undercounts	no — measured 4,260,700 against a true 4,560,700

Not introduced by #4534: measure_by_dim_panel keys its own dedupe the same way, so this is a property of the merged #4529 design. #4534 makes it numerically visible, because that path now prints a figure where it previously printed nothing.

Why this is blocked, and on what

The obvious fix — a composite identity qualifying the name by an enclosing column — requires choosing that enclosing column. Cardinality cannot make that choice: two cardinality-only ownership rules have already shipped and both were reviewed as defects, in opposite directions (see the record on #4526). Building a third at the consumption site would re-open that oscillation.

Note also what does not work: qualifying by any other region column. A city + zip_code dataset has many zips per city, so a legitimate per-city population would be flagged ambiguous and suppressed. The qualifier has to be an enclosing column, and picking it is the open problem.

Prior art already in the tree

The choropleth path solved a closely related problem for region matching in #4481 — qualifier_spec / canonicalize_qualified / qual_raws_by_cand key aliases by (name, qualifier) because "a region spelling alone no longer identifies one feature". That mechanism is a plausible starting point, and it shows the project has already accepted that a name needs qualifying; what it does not settle is which column qualifies when nothing declares it.

Current state

Recorded as a KNOWN LIMITATION on region_deduped_totals in src/cmd/viz.rs, which currently points at #4526 — that pointer should be re-aimed here now that the halves are split. The unsafe-but-detectable half (differing values) is closed; only the silent-merge half remains.

Anchors

region_deduped_totals, RegionDedupe, is_region_concept, and the dedupe in measure_by_dim_panel — src/cmd/viz.rs
Sibling issues from the same split: viz: reject a dictionary x-qsv.denominator from a COARSER geography than the region key #4526 (coarse-geography denominator), viz: skip the rate panel when an x-qsv.denominator takes ONE distinct value across >=2 regions #4547 (degenerate single-value denominator)

describegpt: propose a per-field UCUM `x-qsv.unit` for numeric measur…

Investigation

The claim checks out, but three things change the shape of this issue.

1. The exposed surface is narrower than the issue implies

is_region_concept accepts REGION_CODE_LEAVES ∪ CITY_NAME_LEAVES. Of those, only these can carry a non-unique value:

geo.city / geo.town / geo.municipality — always names, ambiguous nationally.
geo.county — ambiguous only when it holds names. COUNTY_NAME_LEAVES' own doc notes a county-tagged column "may hold either county FIPS CODES or county NAMES", told apart by VALUE SHAPE at resolution time.

Not exposed: zip_code / zip / postal_code / zcta / county_fips / state_fips / place_fips / fips (nationally unique codes), and state / country (unique as code or name). Minor caveat: a bare census_tract number is not nationally unique — a full 11-digit GEOID is. Low priority, since a tract column is an unlikely KPI owning region.

2. The blocking premise is already half-answered in the tree

The issue says the fix requires choosing the enclosing column, and that cardinality cannot choose. The second half is right — two cardinality rules shipped and both were reviewed as defects, in opposite directions. But the tree already chose, and not from cardinality:

/// The first state column a data dictionary names, if any. This is `viz smart`'s stand-in for
/// `--region-state`: 422 county names are carried by more than one county, and on the smart
/// path there is no flag to resolve them with, so the dictionary is the only place a state can
/// come from.
///
/// Deliberately NOT filtered through `candidates_by_leaves`, whose `cardinality >= 2` rule is
/// right for a column that must KEY a map and wrong for one that merely qualifies another...
fn smart_state_column(col_sems: &[ColSemantics]) -> Option<usize>   // viz.rs:6569

STATE_LEAVES = ["state", "state_fips", "state_code"] — the choice is made by concept. The function is unconditional (no #[cfg(feature = "geocode")]), takes only col_sems, and col_sems is already in scope at both owning_region_idx call sites (viz.rs:17877 and 35250). QualifierSpec::StateFips then normalizes via state_fips_for_any, so PA, 42 and Pennsylvania all yield "42".

So a (region cell, state FIPS) composite identity needs zero new selection rules — it reuses the qualifier #4481 already picked for this same ambiguity, at a second consumption site. That is precisely what keeps it out of the cardinality oscillation this issue is worried about.

Useful property: qualifying is a no-op when the owning column holds codes (42125 keys one region whatever the state cell says), so there is no need to shape-detect names vs FIPS — it could apply unconditionally.

3. Two traps that would make a clean-looking fix spawn a second review round

(a) Do not reuse RegionDedupe::conflict for an "identity may be ambiguous" signal. measure_by_dim_panel treats conflict as abandon the collapse → fall back to the ordinary row-wise verb. Setting it on an ambiguity signal converts a silent undercount into an inflated total — the same shape roborev 4542 killed on the KPI side. A detection-only variant needs its own field that the KPI caller reads and measure_by_dim_panel does not.

(b) The blank qualifier cell is the real open question. A blank state cell yields qualifier "" while its siblings yield "42", splitting one region into two keys and double-counting it — the opposite error from the one being fixed. This has to be decided explicitly rather than left to fall out. RegionDedupe::unattributed_rows is the natural home, noting the settled asymmetry that blanks are reported beside the tile, not carried at full weight (that rule belongs to the grouped bar only).

Residual hole after a state-qualifier fix

A name-keyed dataset with no enclosing region column at all, holding two same-named regions with exactly equal values. Undetectable from the data — nothing in the file distinguishes the rows. That is a genuine permanent limitation, and a more accurate subject for the KNOWN LIMITATION comment than "the whole question is open".

On frequency

The headline figure (4,260,700 vs a true 4,560,700) is synthetic. The real Aurora values differ (CO 386,261 vs IL 180,542), which is the already-closed safe case. Two same-named places with exactly equal measure.population is rare; rounded measure.area is the more plausible collision, REGION_LEVEL_CONCEPTS being exactly those two. The merge is real and silent, so this is not a phantom — but it is not a live data-corruption bug either. Worth fixing on merit, not urgency.

Two corrections to the issue body

The "Current state" paragraph is stale. The KNOWN LIMITATION pointer was already re-aimed from viz: reject a dictionary x-qsv.denominator from a COARSER geography than the region key #4526 to viz: a region NAME is not a unique identity — two same-named regions with equal values silently merge #4548 in 3aa7ad678; no edit is needed there.
The comment has now been updated to record the smart_state_column avenue, the blank-qualifier open question, and the "must not read an ambiguity flag" constraint on measure_by_dim_panel, so the next reader does not re-derive them.

If a test is written: geo.county_name fails is_region_concept — assert on geo.county, and mutation-test the assertion.

Recommendation: defer — no code change, keep this open

Following on from the investigation above. The merge is real, but every available fix costs more than it buys on realistic inputs.

The asymmetry that decides it

There are two viable fix shapes and neither is free:

shape	blank-qualifier policy	new failure it introduces
Qualify unconditionally	blank cell gets its own key	Inflation on FIPS-keyed data — one region splits into two keys and is double-counted. Same failure family roborev 4542 already killed on the KPI side.
Qualify name-valued owning columns only	blank → `unattributed_rows`	Needs value-shape detection, so the fix becomes three coupled decisions (shape test, qualifier, blank policy) in an area that has already produced three review oscillations.

Blank state cells are far more common in real data than two same-named regions carrying exactly equal values. So the unconditional version probably makes aggregate accuracy worse, and the careful version is a three-part change guarding a narrow conjunction (name-keyed owning region and two same-named regions in one file and exactly equal values).

Meanwhile the dangerous half is already closed: when the values DIFFER, conflict fires and the tile is omitted (#4536). What remains is a silent undercount in the rare exactly-equal case.

Exit condition

Deferring is only defensible with a trigger attached, so: reopen the moment a real dataset or a gallery fixture publishes a materially wrong number from this path. That evidence would also settle which of the two shapes to build, which is currently the thing no one can decide from first principles.

If closing is preferred over deferring

The smallest safe change is detection-only, not composite identity: if the owning column is name-valued and an enclosing-concept column varies (as distinct from being blank) within a name key, mark the identity ambiguous and omit the tile. It produces no new arithmetic and reuses the settled omit disposition, so it cannot introduce an inflation path.

Caveats if that route is taken:

It needs its own RegionDedupe field, never conflict — measure_by_dim_panel reads conflict as abandon-the-collapse and would fall back to an inflated row-wise total.
It buys a missing tile in place of a rare wrong one, which is an improvement but not free.

One correction to the investigation comment above

looks_like_county_names (viz.rs:5688) covers county names only. The exposed surface is geo.city/town/municipality plus name-valued geo.county, so that helper is not a drop-in shape test for a name-valued-only variant.

viz(gallery): add a real-geography denominator-aware rate map

Navigation Menu

Repository navigation

viz: a region NAME is not a unique identity — two same-named regions with equal values silently merge #4548

Problem

Why this is blocked, and on what

Prior art already in the tree

Current state

Anchors

Investigation

1. The exposed surface is narrower than the issue implies

2. The blocking premise is already half-answered in the tree

3. Two traps that would make a clean-looking fix spawn a second review round

Residual hole after a state-qualifier fix

On frequency

Two corrections to the issue body

Recommendation: defer — no code change, keep this open

The asymmetry that decides it

Exit condition

If closing is preferred over deferring

One correction to the investigation comment above

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

viz: a region NAME is not a unique identity — two same-named regions with equal values silently merge #4548

Description

Problem

Why this is blocked, and on what

Prior art already in the tree

Current state

Anchors

Activity

jqnatividad commented on Sep 9, 2026

Investigation

1. The exposed surface is narrower than the issue implies

2. The blocking premise is already half-answered in the tree

3. Two traps that would make a clean-looking fix spawn a second review round

Residual hole after a state-qualifier fix

On frequency

Two corrections to the issue body

jqnatividad commented on Sep 9, 2026

Recommendation: defer — no code change, keep this open

The asymmetry that decides it

Exit condition

If closing is preferred over deferring

One correction to the investigation comment above

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions