Does it catch real bugs — without crying wolf?

Eight hundred eighty pull requests that introduced a bug a maintainer later fixed — replayed against the introducing PR, exactly as it stood. The world chose these bugs; the projects' own maintainers adjudicated them. Strictly out‑of‑distribution: real code, real defects, ground truth we didn't write.

Finder model is disclosed per row: batches 1–7 Opus 4.8 n=1–70: 44/70 (63%); batches 8–90 Grok 4.5 n=71–890: 537/817 (66%); batches 91–92 Opus 5 n=891–941: 47/51 (92%). Campaign total 628/938 (67%). Treat mixed-model aggregates carefully — and treat that 92% as n=51, not as a rate: it is one model on 51 pairs, the newest and smallest stratum here, and it is not comparable to the Grok window without controlling for which pairs were mined. Same-pair re-run of those 51 on Grok 4.5: 44/51 (86%) — same briefs, same fixtures, blind finders-only; 3 HIT behind Opus 5 on this set (evidence).

Detection by specialty

Campaign 628/938 (67%) after batches 91–92, the first pairs scored on Opus 5 (47/51 HIT). A same-pair Grok 4.5 re-run of that cohort scored 44/51 (86%) — not a campaign re-bank; a confound-controlled model comparison. The four banked misses are the informative part. In three of them the reviewer read the changed code and affirmatively cleared it — solr-529 as “a faithful port”, and in pulsar-23062 all three reviewers cleared the very executor change that arms the deadlock, reading “it still runs on the same executor” as reassurance. In springframework-6e97587 a reviewer named the exact defect in its own coverage attestation and triaged it away as low severity. pulsar-23901 is additionally a weak test — the flawed pattern predates the change under review — and is discounted rather than counted against capability. Batch 91 was a clean 10/10, which fires the campaign’s perfect-decade gate: a blind re-score of 5 of those 10 — ground truth and raw finder output only, verdict and rationale stripped — agreed 5/5. One proposed pair was discarded as a duplicate of an already-mined PR and is not counted. Prior: batch 90 post-restructure Grok smoke 5/10, 85–89 was 23/50. Strict quote-backed target recall; scoring per pair. Recent hard-gate window (batches 80–89): 42/100 (42%) — residual public gold under Layer B; do not read the drop from a high decade (e.g. 69–73 at 82%) as a silent product regression without the miss postmortem (adjacent-target 39/58 MISSes, empty 12/58).

Recall by surface-directness

Grade is how pointable the introduced bug is on the diff: A a single wrong line, B visible after reading one helper, C cross‑file or subtle. Recall tracks it — the honest read on the headline: the aggregate drifts down as we deliberately mine harder, subtler pairs, not because detection got worse. The residual hard case is a grade‑A bug whose cost is real but emergent — a fold, a downstream fast‑path, an uncached branch — visible only when you model the caller.

Recall by attribution stratum

A mined pair is only a fair test if the defect the fixing PR repairs is visible from the introducing diff. A mechanical screen (no model call) compares the two diffs line by line across every pair and sorts each by how the fix relates to what the intro actually wrote. 28.2% of the misses are not clean in‑diff tests, against 16.2% of the hits — so part of the headline is test validity rather than capability, and the honest read is the split below, not one number. Only 3 pairs were provably invalid (one fix branched before its intro merged; one bank named the same PR as both intro and fix; one fix merged seven weeks before its intro) — those were removed from both sides, taking the corpus from 890 to 887 and the headline from 65.4% to 65.5%. That is the point: the audit buys a clearer picture, not a better percentage. Nothing else was deleted — arms‑sibling and unattributed pairs are harder tests, not broken ones, so they stay and are reported as strata. Method, per‑lane composition, and the wrong‑ref defect it caught: pair‑validity audit.