Duplicate check — 2026-10-04

Superseded in part (2026-10-04, iteration 6). The AI Fairness 360 duplicate described below has been merged (2019 journal version kept), and a title-similarity dedup rule was added. Current state: duplicate-check-v2.md. The text below records the state when written; its source data is data/duplicate_check.json as of commit c1b8872.

Free; no API calls; no corpus was changed. Produced by python -m backend.app.corpus.duplicate_check (writes data/duplicate_check.json); the table is generated by python -m backend.app.corpus.doc_numbers.

The diet audit's three "duplicate" verdicts

All three are the same two papers counted more than once through different claims, not two copies of one work in the corpus:

Pair 7 is the same paper pair as pair 6 (W2109129984 vs W2311763102, alcohol and all-cause mortality): a second claim from the 2016 paper restating the abstainer-bias adjustment.
Pairs 9 and 10 are the same paper pair as pair 8 (W1976428272 vs W2127427274, triglycerides and coronary heart disease): the EPIC-Norfolk and Reykjavik component estimates of the 2006 pooled result.

The audit already keeps them out of the headline. The shortlist can produce this because it pairs claims, not papers: two papers with several similar claims yield several pairs.

Duplicate papers in the corpora

library	papers	after replaying the 4-pass dedup	candidates for hand review
LLM calibration	113	113	1
Diet & mortality	100	100	0
ML fairness	100	100	2

Each candidate, reviewed by hand against titles, DOIs, years and abstracts:

library	candidate	verdict
ML fairness	`W4289438483` (arXiv 2018, "AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias") and `W2974817986` (IBM J. Res. Dev. 2019, "AI Fairness 360: An extensible toolkit for detecting and mitigating algorithmic bias")	Duplicate: preprint and published version of one work. Same authors, near-identical abstract.
ML fairness	`W2100960835` ("Fairness through awareness", 2012) and `W2892038960` ("Fairness through Causal Awareness", 2019)	Distinct papers (different years, different contribution; two shared authors).
LLM calibration	`W7131427313` (2025) and `W7147133480` (2026), near-identical titles, same authors, two IEEE conferences	Distinct works: the 2025 paper is a systematic literature review of 34 studies; the 2026 paper is an empirical study of RAG and MCP pipelines on MedQuAD. Abstracts differ.

Count of duplicate papers: llm-calibration 0, diet-and-mortality 0, ml-fairness 1 pair (2 records of one work).

What the ML-fairness duplicate affected

The two AI Fairness 360 records produced 4 of the 95 shortlisted claim pairs in ML fairness, all classified supports: the paper agreeing with its own preprint. They are not contradictions and feed no gap card (ML fairness has none), but any count of supporting pairs for that library is inflated by 4. They were left in place; the corpus is not changed in this run.

Should the 4-pass dedup have collapsed it? Why it did not

Two reasons, both checked in code:

The 4-pass dedup was never run on the two new libraries. normalizer.deduplicate() runs in build_expanded_manifest.py (LLM calibration only). The multi-domain pipeline (multi_domain.py) keeps papers unique by OpenAlex work ID only (seen_ids), and its prelabel output's n_pre_dedup field is the count before trimming, not before any dedup.
Even replayed, the 4-pass dedup does not merge this pair (the table above shows no collapses in any library). Pass 1 needs the same DOI (they differ: 10.48550/arxiv.1810.01943 vs 10.1147/jrd.2019.2942287). Passes 2–4 need the same normalized title, and the journal version dropped "Understanding" and "Unwanted". Pass 4 is the preprint/published pass, but it too requires identical titles.

So by its own rules the dedup is behaving as designed; by its purpose (no work counted as two independent papers) it should catch this case. A fix would need a title-similarity rule for arXiv/non-arXiv DOI pairs with shared authors. Changing the dedup policy is a decision for docs/merge-policy.md and is not made here.