The Pipeline That Ate Itself

For nine days in July, my memory pipeline fed on its own output, and every status surface stayed green the whole time.

Some context. Claude Code deletes session transcripts after 30 days, and I lost a month of history to a machine migration before I understood what that meant. So I built a nightly pipeline: snapshot the transcripts, load them into SQLite, have an LLM distill each session into durable facts. Today it holds about 47,000 facts from roughly 24,500 sessions, and it injects a small digest into every new session so my agents stop re-deriving decisions they already made.

A scheduler default did the damage#

Windows Task Scheduler runs a job from C:\Windows\system32 unless you set a working directory. My distiller shells out to the Claude CLI, and the CLI writes its own transcript into a folder derived from the current working directory. Under the scheduler, that folder landed inside the exact tree my snapshot mirrors.

Follow the loop around once. The distiller processes a batch of sessions and, in doing so, writes new transcripts. The snapshot archives them. Ingest loads them as sessions. The queue grows by roughly what the run just processed. The next run distills the distiller.

Nine days of that produced 21,264 junk sessions and about 38 GB of exhaust. No stage failed. The snapshot copied files, ingest parsed them, the distiller distilled them. Every stage reported success because every stage was succeeding.

The 18,428-session backlog was really 1,924#

I found the loop while chasing a different number. The queue query said 18,428 sessions were waiting for distillation, which read like a runaway cost problem. The real queue was 1,924. The gap was tombstones: emptied sessions kept on purpose, because deleting them would orphan the dates on thousands of real facts that cite them.

The wrong number nearly bought the wrong fix. A backlog of 18,000 argues for a bigger nightly cap and more spend. A backlog of 1,900 argues for patience. The honest count only appeared after I added the distiller's own has-content filter to the query, which is a polite way of saying the number on the dashboard and the number the code used had quietly diverged.

The working directory was the trigger, not the cause. Each stage checked its own work, and nothing checked the seams. Ingest growth tracking distill output at a ratio near one is an invariant of the whole pipeline, and for nine days it was violated by a factor of twenty with nobody assigned to notice.

Prose is advice; a check is a guarantee#

The repairs took an afternoon: pin the distiller's working directory, exclude it from the snapshot, and add two independent ingest guards, one keyed on the folder and one on the opening bytes of the distiller's own prompt. The nightly went from 3 hours 11 minutes back to 14 minutes.

Then I wrote the checks that outrank the stages. After every ingest, the pipeline now proves that zero of its own transcripts reached the database, and it fails the night loudly if ingest outpaces distillation past the incident's signature ratio. The guard that prevented the last incident is audited by a postcondition that will catch the next variant, whatever its trigger turns out to be.

The same discipline caught bugs I would never have found by rereading code. A 30-question retrieval eval scored the search layer at 46.7 percent on its first run, because multi-word queries were silently ANDed together and a natural question almost never has all of its words in one stored fact. Last week the gate flagged its own decay: an abstention probe expected "what is Eric's phone number" to return nothing, and it started matching because the corpus had legitimately grown a fact about a phone-number decision. No number ever existed in the store. The check had aged out of truth, and I only know because it failed loudly instead of rotting quietly.

The tool has a kill date#

A memory system invites a comfortable lie: it feels useful every time it answers, and nobody measures what its absence would have cost. So this one carries a standing bet. I built a counterfactual eval: 19 questions whose answers exist nowhere else, each verified absent from my notes, my specs, and every repo's git history before it was admitted to the set. Retrieval currently answers 19 of 19. On September 16 the system faces a review with teeth. Real usage plus that baseline decide whether it keeps running, and losing means deletion, not a backlog item.

Nine days of green taught me what dashboards are for. The goal was never a memory I trust. It is a memory I can check.