PolicyLens: Retrieval Ablation
Agentic RAG over insurance & regulatory documents — source on GitHub
Every number below is read directly from committed
eval_results/*.json, regenerated from scratch by make eval
(retrieval) and make eval-generation (generation + judge). Bars show the
mean; the vertical tick marks show the 95% bootstrap confidence interval (n=30
answerable questions, 10,000 resamples).
Retrieval ablation
Dense retrieval alone (S1) underperforms the BM25 baseline (S0) on this corpus — this golden set skews toward precise-terminology questions where exact lexical match beats a small, non-domain-tuned embedding model. Hybrid fusion (S2) recovers the gap and leads on every metric, though a paired bootstrap on the S0→S2 delta shows none of these differences clear 95% significance at n=30 — read the bars as a plausible signal, not a settled result.
Show as table
Generation quality
Refusal accuracy is identical across all three retrieval stages (11/12 unanswerable questions correctly refused). Every false refusal under S2 was traced by hand: none had the correct source chunk in the retrieved top-5 — false refusals here are a retrieval-recall ceiling, not the generator being over-cautious with the right context in hand.