Test Results Summary
run_015 · prompt v7 · 15 run(s) total · final canonical: numeric reconciliation cases (tc_038/039) + faster resolve timeout
100%
pass rate · 16/16 cases (green ≥80 · yellow 60–79 · red <60)
1 case(s) the engine correctly abstained on (cannot_verify) — it returns "I can't verify this" rather than guessing.
Iteration history · the score was earned by iterating, not curated
| Run | Prompt | Overall pass rate | Change |
|---|---|---|---|
| run_009 | v5 | 100% | rename defect numeric_error -> numeric_mismatch |
| run_010 | v6 | 100% | merge stripped_caveat+overstated_confidence -> overstatement chip |
| run_011 | v6 | 81% | multi-domain: add legal corpus (tc_101-116) + domain field |
| run_012 | v6 | 89% | resolution non-blocking when source text present (fixes legal Justia/CL bot-blocking) |
| run_013 | v6 | 96% | domain-calibrated legal labels (5 -> unsupported), fix tc_108 control, blocked-source status |
| run_014 | v7 | 93% | rename wrong_directionality -> contradiction (general); 3 domains; hardened retry |
| run_015 | v7 | 97% | final canonical: numeric reconciliation cases (tc_038/039) + faster resolve timeout |
By hallucination type
| Type | Passed | Total | Rate |
|---|---|---|---|
| contradiction | 4 | 4 | 100% |
| wrong_attribution | 4 | 4 | 100% |
| scope_expansion | 2 | 2 | 100% |
| control | 3 | 3 | 100% |
| overstatement | 2 | 2 | 100% |
| fabricated_citation | 1 | 1 | 100% |
Per-case results · click a row for detail
| Case | Type | Expected | Actual | Defect | Resolution | |
|---|---|---|---|---|---|---|
| tc_101 | contradiction | unsupported | unsupported | contradiction | blocked | PASS |
| tc_102 | wrong_attribution | unsupported | unsupported | numeric_mismatch | blocked | PASS |
| tc_103 | scope_expansion | unsupported | unsupported | scope_expansion | malformed | PASS |
| tc_104 | control | supported | supported | — | malformed | PASS |
| tc_105 | overstatement | unsupported | unsupported | overstatement | resolved | PASS |
| tc_106 | wrong_attribution | unsupported | unsupported | contradiction | resolved | PASS |
| tc_107 | scope_expansion | unsupported | unsupported | overstatement | resolved | PASS |
| tc_108 | control | supported | supported | — | resolved | PASS |
| tc_109 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_110 | overstatement | unsupported | unsupported | overstatement | resolved | PASS |
| tc_111 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_112 | wrong_attribution | unsupported | unsupported | contradiction | resolved | PASS |
| tc_113 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_114 | control | supported | supported | — | resolved | PASS |
| tc_115 | wrong_attribution | unsupported | unsupported | wrong_attribution | malformed | PASS |
| tc_116 | fabricated_citation | cannot_verify | cannot_verify | fabricated_citation | malformed | PASS |