Test Results Summary
run_015 · prompt v7 · 15 run(s) total · final canonical: numeric reconciliation cases (tc_038/039) + faster resolve timeout
89%
pass rate · 17/19 cases (green ≥80 · yellow 60–79 · red <60)
Iteration history · the score was earned by iterating, not curated
| Run | Prompt | Overall pass rate | Change |
|---|---|---|---|
| run_009 | v5 | 100% | rename defect numeric_error -> numeric_mismatch |
| run_010 | v6 | 100% | merge stripped_caveat+overstated_confidence -> overstatement chip |
| run_011 | v6 | 81% | multi-domain: add legal corpus (tc_101-116) + domain field |
| run_012 | v6 | 89% | resolution non-blocking when source text present (fixes legal Justia/CL bot-blocking) |
| run_013 | v6 | 96% | domain-calibrated legal labels (5 -> unsupported), fix tc_108 control, blocked-source status |
| run_014 | v7 | 93% | rename wrong_directionality -> contradiction (general); 3 domains; hardened retry |
| run_015 | v7 | 97% | final canonical: numeric reconciliation cases (tc_038/039) + faster resolve timeout |
By hallucination type
| Type | Passed | Total | Rate |
|---|---|---|---|
| numeric_mismatch | 5 | 5 | 100% |
| contradiction | 3 | 3 | 100% |
| wrong_attribution | 2 | 2 | 100% |
| control | 3 | 3 | 100% |
| scope_expansion | 4 | 4 | 100% |
| overstatement | 0 | 2 | 0% |
Per-case results · 2 failed · click a row for detail
| Case | Type | Expected | Actual | Defect | Resolution | |
|---|---|---|---|---|---|---|
| tc_201 | numeric_mismatch | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_202 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_203 | numeric_mismatch | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_204 | wrong_attribution | unsupported | unsupported | wrong_attribution | resolved | PASS |
| tc_205 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_206 | control | supported | supported | — | resolved | PASS |
| tc_207 | numeric_mismatch | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_208 | scope_expansion | unsupported | unsupported | contradiction | resolved | PASS |
| tc_209 | scope_expansion | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_210 | control | supported | supported | — | resolved | PASS |
| tc_211 | wrong_attribution | unsupported | unsupported | wrong_attribution | resolved | PASS |
| tc_212 | numeric_mismatch | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_213 | scope_expansion | unsupported | unsupported | scope_expansion | resolved | PASS |
| tc_214 | control | supported | supported | — | resolved | PASS |
| tc_215 | contradiction | unsupported | unsupported | contradiction | resolved | PASS |
| tc_216 | numeric_mismatch | unsupported | unsupported | numeric_mismatch | resolved | PASS |
| tc_217 | overstatement | partially_supported | unsupported | contradiction | resolved | FAIL |
| tc_218 | scope_expansion | partially_supported | partially_supported | unsupported_addition | resolved | PASS |
| tc_219 | overstatement | partially_supported | supported | — | resolved | FAIL |