Replace Java DUKE with a re-tuned pure-Python rapidfuzz matching engine

[FabianHofmann]

powerplantmatching uses fuzzy record comparison for two jobs: linking records of the same plant across sources (GEO, GPD, ENTSO-E, OPSD and the rest), and deduplicating records within one source before those links are drawn. Both were done by DUKE, a Java record-linkage engine shipped as seven .jar files in the package data, invoked as a subprocess and configured through two XML files. Every user needed a working JVM to build a dataset, and the matching configuration lived outside Python where nothing could test it.

This PR replaces DUKE with powerplantmatching.linkage, a vectorised pure-Python backend built on rapidfuzz and numpy, and makes it the sole engine. The jars, the XMLs, powerplantmatching/duke.py and the Java setup step in CI are gone, rapidfuzz becomes a hard dependency, and a full build takes 378 s instead of 950 s on identical inputs.

Note

AI generated draft. Still experimental — see Known limitations.

The engine

linkage keeps DUKE's overall shape. Two records are compared field by field; each field's similarity is mapped onto a probability between a configured low and high bound; the fields are combined by a Fellegi-Sunter belief update over a 0.5 prior; pairs above a threshold are accepted. LINKAGE_FIELDS, DEDUP_FIELDS and the two thresholds are the direct successors of the deleted XMLs and now live in Python where tests can pin them.

It is not a reimplementation, though. DUKE maps similarity onto probability with a quadratic curve and a hard floor below 0.5, treats geographic distance as always-positive evidence in [0.5, 1], and uses the overlap coefficient for q-grams; linkage uses a linear ramp, a [0, 1] distance score and Dice. Around 10 % of accepted GEO×GPD pairs differ as a result, and constants fitted to one curve carry no justification under another, so the thresholds were re-tuned rather than inherited. The linkage threshold moves from 0.965 to 0.85; the field bounds and the dedup threshold ship unchanged.

Names are compared token by token, following DUKE's JaroWinklerTokenized: the similarity is the mean over one record's tokens of their best Jaro-Winkler match in the other, normalised by the longer token count. This is the one field where a character-level ratio is not usable, because plant records distinguish units of a station by a trailing designator — Kozloduy 1 through Kozloduy 6, Doel 1 and Doel 4, Neurath and Neurath F — and a one-character difference in six reads as near-identical to any character kernel. Sibling units score 0.50 and below and stay separate, while a genuine typo stays high: Gravelines 1 against Gravelins 1 scores 0.99. Reducing over the shared token vocabulary rather than over record pairs keeps this vectorised, at 6.5 s for a 6000-row dedup.

Scoring runs in row blocks capped at BLOCK_CELLS, so peak memory is flat in n rather than quadratic: the MASTR dedup group of 8662 rows needs 337 MB instead of 4.33 GB, and takes 12.3 s instead of 17.8 s.

Benchmarks

Two benchmarks live in analysis/, answering different questions.

analysis/benchmark_linkage.py is new and is the tuning instrument. It recovers 567 GEO↔GPD pairs from the projectID column of the production powerplants.csv as a cross-source ground truth, imports the production comparator, and evaluates a config in about 25 ms by caching the per-field similarity matrices. It is bit-exact with the production path — identical pairs to compare_two_datasets, symmetric difference 0. Thresholds were fitted with a country-held-out protocol, and at the shipped 0.85 the engine reaches precision 0.920 / recall 0.847 / F1 0.882, against 0.934 / 0.771 / 0.844 at the old 0.965. This ground truth is DUKE-era pipeline output, so agreement with it partly measures agreement with DUKE.

analysis/compare-with-entsoe-stats.py is the independent check: the built dataset against ENTSO-E installed capacity for 2025, summed over the 162 country × fueltype cells ENTSO-E reports, excluding Wind and Solar (extended separately) and Other (an incomparable residual). The DUKE baseline is a dataset built from a worktree at the merge-base on identical inputs.

	signed gap	Σ|gap|	RMSE
DUKE	−17.6 GW	104.2 GW	1.234
linkage	−5.5 GW	101.9 GW	1.227

The new engine is marginally better on absolute error and considerably better on bias. Nuclear lands 2.9 GW over the statistics against DUKE's 4.8 GW over, and Polish hard coal 18.6 GW against 19.0 reported. The largest residuals are data coverage rather than matching — IT Hard Coal −6.1, FR Natural Gas −4.4, NL Natural Gas −3.2, FR Hydro −3.2 — while slightly less merging than DUKE adds modest overestimates at DE Lignite +6.3 (DUKE +4.3), CH Hydro +4.2 and AT Hydro +3.8.

That script needed repair to produce any of these numbers. entsoe-py 0.7.0 silently returns all-NaN installed capacity for every CET/EET country, because yearly P1Y documents start at 23:00 UTC and date_range(freq='12MS') snaps forward past the only data point; the statistics are now parsed from the raw XML and cached under analysis/data/. Two crashes and two taxonomy errors are fixed alongside, and the reference year moves from 2022 to 2025.

Other fixes

projectID and EIC were aggregated with set, whose iteration order follows the interpreter's hash seed, so two runs over identical inputs serialised different identifier strings. They are sorted lists now, and two full builds under PYTHONHASHSEED=0 and =12345 come out byte-identical across all 167,632 rows.

match() no longer ends in a **_ catch-all, so unknown keywords raise instead of being swallowed — which is how threads came to be dead end-to-end while workers=-1 was hardcoded. It is now wired to process.cdist(workers=), taking a 6000-row dedup from 20.1 s to 6.3 s. A parallel_duke_processes key in an existing user config was silently ignored, dropping parallelism to 1; it now warns and aliases through to parallel_processes.

aggregate_units fills string columns with "" before dedup, so the presence mask never fired: two records both missing Technology scored a perfect match while missing-versus-populated scored a penalty. Zero-versus-zero capacity now scores identical rather than maximally different, results are dtype-stable, and the comparator moved onto FieldSpec, deleting the stringly-typed dispatch.

Tests

6 → 35 linkage tests, offline and deterministic, plus test/test_linkage_integration.py covering the aggregate_units and compare_two_datasets paths that had no non-network coverage. Field bounds and thresholds are pinned by a test, since the deleted XMLs were their only record. New cases assert that sibling units of one station do not merge, that the 1:1 reduction rather than the threshold resolves siblings across sources, that aggregated identifiers come out ordered, and that results are invariant across BLOCK_CELLS from 1 to 1e12.

One existing case flipped: a 50× capacity difference at identical coordinates with the same fueltype, technology and country now links. It previously passed on the name score rather than on capacity, and strengthening the capacity veto to restore it is not supported by the benchmark, so the test id now says what it actually asserts.

Compatibility

Breaking changes:

• Java is no longer required or used.
• powerplantmatching.duke is gone and add_geoposition_for_duke is removed.
• parallel_duke_processes is renamed to parallel_processes, with a deprecating alias.
• projectID and EIC hold sorted lists instead of sets, and a missing EIC is [] rather than {nan}, so set operations on those values need an explicit set(...).
• Matching results differ from previous versions, since both the scoring curve and the threshold changed.

Known limitations

DateOut: "max" is a latent hazard. It skips NaN, so any aggregate mixing a retired unit with an operating one inherits the shutdown year. Returning NaN whenever a unit lacks a shutdown year was tried and measured worse — sources omit DateOut for plants that are in fact closed, so it resurrects them, taking Σ|gap| from 101.9 to 121.1 GW — and was reverted. "max" is the better estimator, but the ambiguity between "missing" and "still running" is unresolved in the data model.

Deduplication has no pairwise ground truth. The intra-source truth in powerplants.csv is circular, since production used DUKE for dedup. ENTSO-E validates it end to end instead, which is much coarser: it would not catch a merge error that conserves capacity within a country and fueltype. UK, Ukraine, Moldova and Kosovo return nothing from the transparency platform and sit outside that check entirely.

Duplicate index labels silently collapse singlematch. That is pre-existing, but nothing validates index uniqueness now that this is the only matcher.

Background on the engine choice, the tuning protocol and the alternatives that were searched and rejected is in analysis/linkage_findings.md.