How to make an LLM say I don't know, and how to measure that it did

Ballast declines 24 of its 89 golden questions on purpose and answers the other 65 with citations, and on the August 16 run the answer-or-decline call was right on all 89: zero questions it should have declined and answered, zero it should have answered and declined. One of the 65 answers was unfaithful to its sources, which the eval counts as a hallucination rate of 1.1 percent over the whole set. Those three counts are the abstention measurement, and this post is about how each one is produced, because a refusal you cannot count is a mood, not a feature.

The June post on Ballast explained why refusal is the feature. What it did not have was a ledger. It described one 44-question evaluation and said, correctly, that a zero on that set was not a universal claim. This post reads only the committed metrics ledger, which now has three dated runs on a set that grew to 89 questions, and a gate that fails the build on regression. That is the artifact worth writing about.

Where the refusal comes from#

An LLM does not decline on its own. Left to itself it completes the sentence, because that is the only verb it has. Ballast gives it three places to stop before an answer ships, two of them ahead of generation and one behind it, and each one emits a fixed sentence instead of prose.

The first is the retrieval grader. Every question fetches passages from the corpus and a grader scores each one for relevance. If nothing relevant comes back, the graph rewrites the query and retrieves again, up to two retries, and then declines with a fixed insufficient-evidence sentence rather than handing the model an empty context and hoping. The same retry budget covers the other direction: an answer the critic judges ungrounded goes back through a rewrite, and when the budget is spent the graph declines instead of shipping the answer it could not ground. The second is the guardrails gateway, in two halves. On the way in, a classifier declines requests for personalized, medical or legal advice before generation. On the way out, a YAML policy reads the generated answer, withholds one that gives advice or fails to cite a passage, allows one corrected retry, and then returns a fixed withheld-by-guardrail sentence. The third is the input guardrails, which block prompt injection, secrets and personal data before anything runs and return a fixed could-not-be-processed sentence.

The fixed sentences are the point of the design and not a stylistic choice. A refusal that the model phrases freely is a refusal the eval cannot find. Three marker phrases are what the scorer looks for.

How the eval scores a refusal#

The golden set has 89 questions in five categories: 55 answerable, 10 multi-hop, 10 unanswerable from the corpus, 5 out of corpus entirely, and 9 adversarial prompts. Each carries an expected answer or the single word decline; 24 carry decline. The runner asks every question, checks each output for one of the three marker phrases, and scores the pair: should this have been declined, and was it.

Both mistakes are counted separately, because they cost different amounts. An under-refusal is a question that should have been declined and got an answer, which is the failure that produces a confident fabrication. An over-refusal is an answerable question that got a decline, which is the failure that makes the system useless. Refusal correctness is the share of all 89 where the two flags agree, and the build gate fails below 90 percent. Answered questions are then judged for faithfulness to the retrieved passages. The hallucination rate counts an unfaithful answer and an under-refusal alike, over all 89 questions, and the gate fails above 5 percent.

The ledger has three rows. The first, June 27, on a 43-question set: refusal correctness 97.7 percent with one under-refusal, every answer faithful, hallucination 2.3 percent, all of it that one under-refusal. The second, July 2, on the 89-question set: 98.9 percent with one under-refusal, one unfaithful answer, hallucination 2.25 percent, injection block rate 100 percent. The third, August 16: 100 percent, zero under, zero over, one unfaithful answer, hallucination 1.1 percent with a bootstrap interval of 0 to 3.4 percent, faithfulness 0.985, and $0.0061 a question. Read by unfaithful answers alone the series is zero, one, one; the fall in the headline rate between July and August is the under-refusal going away, not the grounding improving. Whether that is a fix or run-to-run variance in the model, the gate is what makes the next one visible instead of averaged away.

What 100 percent cannot promise#

Only 24 of the 89 questions are decline questions, and 24 correct out of 24 is a smaller claim than it sounds. Zero misses in 24 trials is consistent with a true miss rate anywhere up to about 12 percent at the usual confidence, the rule of three applied to a sample of that size. The ledger's own bootstrap band for refusal correctness reads 1.0 to 1.0, and that is not a promise either: resampling a column of ones produces a column of ones, so the interval has zero width by construction, not by evidence. The number to read is 24, and the number to watch is the under-refusal count, which is a count of specific questions and not a rate.

The detector tests the plumbing as much as the model. A decline phrased in the model's own words, with a citation attached, passes the must-cite policy and is scored as an answer, which hides an over-refusal; an uncited one is withheld by that same policy and comes out as the fixed sentence, which scores as a refusal. So the 100 percent says that on every question the three stops and the policy resolved to a marker sentence or an answer as intended, and a model that had learned to decline in prose would not be caught by it. The faithfulness judge is calibrated against eight human-labeled pairs with an agreement floor of 0.8, and eight is thin; the calibration set is the next number that has to grow.

The cost of refusing#

A system that declines everything scores perfectly on the 24 and zero on the 65, and refusal correctness would read 27 percent. The metric couples the two mistakes on purpose so that a gate cannot be passed by cowardice. In the August run the over-refusal count was zero: every one of the 65 answerable questions came back as an answer rather than a decline, and the must-cite policy would have withheld an uncited one. That is the half of the measurement I would have skipped in June, when the refusals were the part I was proud of, and it is the half that decides whether anyone uses the system.

The abstention number I report now is not a rate. It is three counts from the latest ledger row, zero under-refused, zero over-refused, one hallucinated answer in 89, with the date and the run beside them, and a gate that turns the next regression into a failed build instead of a quieter number.