An LLM System Whose Best Feature Is Refusing
Built and open-sourced an LLM system whose most valuable behavior is producing no answer at all: a self-healing retrieval pipeline grounds it, a separate critic checks it, a guardrails gateway fences it, and an evaluation gate proves answer quality on every change. In a published 44-question battery it answered grounded questions with citations, refused advice and injections, and hallucinated zero times.
The Problem
You do not get reliability out of a language model by asking nicely. A prompt that says "be accurate" and "do not fabricate" is hope, not engineering, and a model that will answer anything is easy to build and impossible to trust. What most LLM applications lack is structure: a place where the system can notice it is about to do something wrong and stop.
The domain raised the stakes. The knowledge base is finance education, so the system had to answer grounded questions with citations while refusing the ones that look like finance but are really requests for personal advice or stock picks.
It also had to be publishable: a public repository with no private data, no secrets, and nothing that names work not yet ready to be named.
The Approach
Composed three subsystems over one shared core. A self-healing RAG graph (LangGraph) retrieves passages, grades their relevance, drafts a cited answer, and hands the draft to a separate critic that checks whether the sources support it. Missing support rewrites the query and retries up to a limit; a spent limit declines honestly instead of inventing an answer.
Wrapped the graph in a guardrails gateway that screens inputs for personal data, secrets, and prompt injection before the model runs, and screens outputs against a plain-English policy written in YAML: citations required, personalized financial advice forbidden by pattern. The model cannot waive those checks.
Made answer quality a build gate rather than a demo: a golden question set scores faithfulness, relevancy, hallucination, refusal, context precision, latency, and cost, and a local gate fails the merge on regression or when the newest metrics ledger entry is stale relative to the code under test.
Kept the corpus public and non-sensitive by construction: federal public-domain investor education (SEC, IRS, CFPB, FDIC, Treasury, BLS) plus the author's own published methodology notes, each with per-file provenance.
Gated the publish itself with a disclosure scan over the whole repository, which had to pass before the repo went public.
The Impact
- Live and open source at github.com/CaskeyCoding/ballast, MIT-licensed source code with per-file provenance on the corpus documents: self-healing RAG, guardrails gateway, and eval CI/CD over one core
- In one published 44-question battery the system answered grounded finance questions with citations, declined unanswerable and off-topic prompts, refused personal advice, and blocked obvious prompt injections before the model ran, with zero hallucinations. That describes that evaluation set, not a universal error rate
- The refusal teaches: asked to use a backtesting method to pick next year's winner, the system declined and cited the sources on why short-run results are statistically indistinguishable from luck
- The publish gate earned its keep once. The disclosure scanner skipped its own source, which held the forbidden name in plain text, and reported clean; the fix moved the withheld terms out of the tracked tree entirely and made the scanner check its own source. A green check is only as honest as what it looked at
- The pattern is domain-independent: swap the documents and the policy and the skeleton holds for running, law, or medicine
Evidence
Related
Ballast: An LLM App Whose Best Feature Is Saying 'I Don't Know'
I built a self-healing RAG pipeline, a guardrails gateway, and an eval gate as one system, then threw 44 adversarial questions at it. Zero hallucinations, because the most important thing it does is refuse. Here is how trust got built into the architecture instead of the prompt, and the safety check that leaked the very thing it was guarding.
How to backtest without fooling yourself
A backtest's job is not to find an edge. It is to stop you from believing in one that is not there. The toolkit I used to test my own trading engine, and the part where it killed my single best signal.
The Pipeline That Ate Itself
A Windows Task Scheduler default fed my Claude Code memory pipeline its own transcripts for nine days: 21,264 junk sessions, 38 GB of exhaust, and an 18,428-session backlog that was really 1,924. The durable fix: postconditions that outrank green dashboards.
Composite What You Trust, Watch What You Don't: A Trust Boundary for Data With Money Attached
Every system that fuses signals into one consequential number has a fault line: the data you trust enough to composite into a grade versus the data you only trust enough to watch. How I drew that boundary in my personal finance engine, and how a test keeps it honest.