Caskey Engineering

The Binder and the Benchmark

Eric Caskey — Mon, 03 Aug 2026 00:00:00 GMT

One of my kids asked me what a Charizard is worth.

Not their Charizard. The Charizard. The 1999 Base Set holo that shows up in every video about cards as investments, usually next to a claim that a binder from 1999 beat the stock market. My kids collect cards the way I once did, for the trades and the pulls, but the question was a real one and it deserved a real answer. So I did what I do with every question about money: I went looking for the data, and I charted it against a benchmark.

The chart is live at caskeycoding.com/lab/pokemon-vs-market. Every series is indexed to 100 at the start of the range, so a playground card and the entire S&P 500 share one honest axis. You can toggle cards, benchmarks, and averages, flip between log and linear scales, and read the exact numbers in a table. Since 2004, a PSA 10 first edition Charizard is up roughly 62x. The S&P 500 is up about 6.6x. That gap is real, and it is also not the whole story.

Here is the part the videos skip. There is no clean thirty year price series for a Pokemon card. Nobody was recording card prices in 1999, the deep auction archives sit behind a bot wall, and the sources that do exist mostly cover the last year or two. I had three options: pretend, pay, or admit it. The board admits it. The deep past is a handful of cited record sales, drawn as ringed dots so you can see exactly how sparse they are. The line between them is labeled as an estimate. From this week forward, a scheduled job records real market prices every Sunday, so the board earns a genuine first party history one week at a time. Where the record sale era meets the measured market era, the line visibly jumps, and the page says why instead of smoothing it over.

That choice was not invented for a card chart. I run a personal finance engine as a side project, and the rule I hold it to is the same rule I held this chart to: label what is measured, label what is modeled, and let a missing number stay missing. A gap in a chart is information. A smoothed line over a gap is a small lie that compounds.

The answer I gave my kid, in the end, had numbers in both directions. Yes, the grail card crushed the index, if you had graded the right card, in the right year, and paid the auction fees, and never spilled juice on it. Also: your binder is not the record sale, cards pay no dividends, and the index never needed a protective sleeve. They mostly wanted to know if the binder was cool. It is.

The cards taught my kids how to trade. I built the chart for the quieter lesson: the point is not what the grail sold for. The point is what the data can honestly say. The board records a new observation every Sunday morning, and the answer keeps earning itself, one week at a time.

Lesson 1: The spec directory

Eric Caskey — Fri, 31 Jul 2026 00:00:00 GMT

What you'll learn: how to give your project a home for intent, so an AI agent, or future you, cannot quietly build the wrong feature. What you need: a text editor and the AI coding agent you already use. No new tools. Time: fifteen minutes to set up, ten per feature after. Cost: zero new dollars; the agent subscription you already pay for covers it.

The idea#

You ask your agent for a feature and get back something plausible. A week later it is wrong in a way nobody can quite name, because what you actually asked for lives in a chat session that no longer exists. Every new session starts from amnesia, re-reads your code, and learns what you did, never what you meant.

The fix is a directory next to your code that holds no code at all, only intent: one numbered file per feature stating what will be observably true when the work ships, what stays out of bounds, and one command that has to pass. It runs on one loop:

Your agent copies a blank form into the right folder.
It fills the form in by interviewing you.
You correct the lines that matter and flip the status from draft to ready.
The agent builds.
You grade the work against the form, line by line, instead of against your memory.

That directory is the smallest version of a spec system, and it is the foundation this entire series builds on.

Why start here#

I have built with AI daily for thirteen months: fifteen repositories, four public sites, two dozen scheduled jobs. The bottleneck was never producing software. It was remembering what I meant, and after enough projects the amnesia was mine as much as the machine's. Nine months in I created a repository that holds no code at all, only written intent, and it now takes more commits than any code repository I own: 1,867 since January against 661 for my busiest code repo. The structure below is that repository reduced to its smallest useful shape.

The template#

Two artifacts this time, because the structure comes first. The skeleton:

specs/
  README.md          # the map: what this project is, where intent lives
  _template.md       # the blank form: every spec starts as a copy of it
  app/               # one folder per area of your project
    001-short-name.md
    002-...          # the record: what you meant, in order

app/ is a placeholder; give each area of your project its own folder and number specs within it in creation order. The growth rule is the same at every size: a folder earns its place the day two kinds of intent compete for one home. When api specs and page specs scroll past each other, split them. Each split buys routing, because a folder is a place you can point an agent at, so a session reads one slot instead of all of them. And the shape never changes as it grows: areas become products, products get their own specs directory, and eventually intent earns a repository of its own. The platform this series comes from grew exactly that way, one earned folder at a time.

Then the blank form itself, _template.md:

---
status: draft          # draft -> ready -> in-progress -> completed
verify: npm test       # one command that must exit 0
---

# <feature name>

## Acceptance criteria
- [ ] <an observable state: a URL that loads, a file that
      exists, a number that appears somewhere it did not>
      check: <the command and its expected output, or the
      page and what it must show>

## Out of scope
- <paths, files, or systems this work must leave alone,
  named precisely enough to paste into a request>

## Notes (guidance, not binding)
<links, patterns worth copying, context the builder
might want>

The rule that makes the file work: everything above Notes is binding, Notes is guidance. One line of standing instruction wires the two artifacts together. Your tools already read a standing file, CLAUDE.md or AGENTS.md, loaded into every session; feature intent pasted in there becomes noise the day after it ships, and both conventions warn that a bloated standing file gets ignored. What belongs there is this:

Before building anything new, copy specs/_template.md to
specs/<area>/<next-number>-<short-name>.md, fill it in from
what I have told you, and stop; only I move status from draft
to ready. Build only after that flip, and grade the finished
work against the file before telling me it is done.

Two surfaces, two jobs: the standing file says how to work here, the spec directory says what must be true, feature by feature.

Each part, and the mistake it prevents#

The map is for amnesia. Every fresh session asks "where would I look." Three README lines answer it: what this project is, where intent lives, how specs are numbered. Without a map, the agent re-derives your project from the code. Code says what you did. It says nothing about what you meant.

The blank form is what keeps the tenth spec readable. Ask an agent for a plan and you get a new format every time. Each one is plausible. None of them are comparable. The form makes every spec gradable the same way. Each acceptance line is a state fused with its check, so "done" cannot quietly mean "I stopped." Out of scope is a list of paths you can paste straight into a request, because an agent left undirected will refactor nearby code with infinite enthusiasm. The verify command is named before the build starts, because if you cannot name it, you do not yet know what you want.

The record is memory. A spec keeps its number and status for life: draft, ready, in-progress, completed. The draft-to-ready flip is your approval, recorded where every future session can see it. Completed specs stay put. Six months from now the record reads as the history of what you meant, in order, which no commit log gives you.

The agent fills, you correct. The interview surfaces edge cases you had not considered. The editing keeps the intent yours. An agent that writes its own binding lines and then grades itself against them has audited nothing.

A worked example#

The real directory this series comes from, compressed:

specs/
  README.md
  _template.md
  decision/                      # empty until lesson 2
  website/
    014-blog-series.md           # status: completed
    016-learning-series.md       # status: in-progress

The bottom slot in that tree is this lesson's own publication. Filled in from the template, it looked like this, and its status moved only after every check ran:

---
status: completed
verify: npm run check:content
---

# lesson-1-post

## Acceptance criteria
- [ ] the lesson page returns 200 on the live site
      check: open the URL after deploy, template visible
      in a code block
- [ ] the blog index lists it under the series title
      check: open /blog, entry present

## Out of scope
- every other post in the blog content folder
- the blog index layout

Small on purpose. The structure survives because filling a slot costs ten minutes, not because it produces a document anyone admires.

From the field#

Earlier this month a proposal of mine died because of its own check, written early. I wanted to simplify a scoring system I run by removing one of its inputs, and I had a page of numbers agreeing with me. The acceptance line, written before those numbers existed, required the improvement to hold on data the analysis had never seen. It did not hold, and the change never shipped. The line was written by a version of me who did not yet want a particular answer. That is the entire mechanism: by the time results arrive you are a motivated reasoner, and the file is not.

Exercise#

Create the skeleton: the folders above, a README.md with three lines (what this project is, where intent lives, how specs are numbered), and _template.md copied from this page.
Add the standing instruction to the file your agent reads at session start.
Pick the next feature you are about to ask an AI for. Let the agent instantiate the template and interview you; cap it at ten minutes; edit the binding lines yourself, then flip the status to ready.
Build the feature the way you normally would, pasting the Out of scope lines into the request.
Before you accept the work, grade the spec line by line: tick each acceptance line only after running its check, then run the verify command.

You're done when#

At least one line refuses to tick. There will be one; mine have ranged from a wrong chart default to a deploy that quietly did more than it was told. A failed check is not a failed build. It is the gap between what you meant and what you asked for, visible for the first time. Then run the second test: open a fresh agent session, point it at specs/ and nothing else, and ask what this project is building next. A right answer from a session with no memory is the structure doing its job.

Common mistakes#

One growing SPECS.md instead of a directory. A single file scrolls; a structure routes.
Feature intent pasted into the standing file. Two surfaces, two jobs; the standing file stays short or stops being read.
Letting the agent flip draft to ready itself. The binding lines are only binding because a human meant them.
Writing activities instead of states in acceptance lines: "refactor the handler" instead of "the endpoint returns the new shape."
Building the taxonomy first. Two folders are enough until a third earns itself.
Writing the spec after the build. The mechanism only works because the lines predate the temptation.

Next lesson#

Why every new AI session relitigates decisions you already settled, and the one-page record that fills the empty decision/ folder above.

The schema is the product

Eric Caskey — Thu, 23 Jul 2026 00:00:00 GMT

Every field in the principle template is load-bearing. Not stylistic. Load-bearing. serves: is not organization, it is the input to a graph walk that catches orphans. last_reviewed: is not metadata, it is the input to the staleness check. supersedes: is not a link, it is the audit trail that keeps the record honest. Take any one of them out and a review mechanism downstream stops working. Put them all in and you did not build features. You built a schema, and the features fell out.

That is the claim this post defends: a life operating system is not a list of features. It is a schema decision, made once, early, and everything that later feels like a good product move was already implied by it.

Life in Spec Files made the case for keeping a life in version-controlled files at all. Last week's post described what happened when three repos disagreed about what those files contain, and the CI check that now fails when they drift. This one is about why the schema deserved that defense in the first place.

The orphan a chat log cannot catch#

Here is what a schema catches that prose never will. The example is illustrative rather than lifted from my corpus, but it is structurally exact.

A Sunday review session opens with the coherence report, before any writing. The report flags an orphan: a project with a serves: list pointing at a goal that does not exist. Not a goal that failed. A goal that was retired two months ago, in a file whose status: changed and whose dependents never got re-pointed. The project had been absorbing hours every week in service of nothing.

I have kept journals that would never have surfaced that. A chat log with an AI assistant would not have surfaced it either, because the conversation has no structure a graph walk can traverse. The orphan was caught because serves: is a typed reference, the reference had a dangling target, and a script walks those references on a schedule. Drift became a diff. That is the whole trick.

Ten fields, each one buying a mechanism#

The template ships ten frontmatter fields, and the schema file is canonical for all of them; templates/principle.md points at it rather than restating it, for exactly the reasons last week's post laid out.

Field	The mechanism it buys
`title`	The principle as one actionable sentence, not a category
`status`	The lifecycle walk: draft, active, contested, retired
`kind`	The revealed/aspirational split, and the falsification clause below
`domain`	A closed enum, so a coverage check can see which domains are starved
`confidence`	An honesty dial the review interrogates rather than a badge
`serves`	The upward graph edge; the orphan check traverses it
`created`	The age of the claim
`last_reviewed`	The staleness check; unreviewed principles surface on schedule
`ended_on`	Terminal states get a date, so retirement is a fact, not a fade
`supersedes`	The audit trail; a replaced principle stays in history

There is no id field. The filename is the id, which sounds like a shortcut and is actually a position: identity lives in the filesystem, where git can see renames, not in a database column the corpus cannot inspect.

The kit's steering doc states the design goal in four words I will quote rather than improve: engineering artifacts are "written down, dated, diffable, and revisited." Lives deserve the same rigor. Each of the ten fields exists to make one of those four words mechanical.

The falsification clause is the whole game#

The field that earns its place most is kind:. The kit's methodology doc defines the split: a principle is either revealed, backed by evidence of how I have actually behaved, or aspirational, a rule I want to live by and cannot yet demonstrate. The template forces the question with a ## Falsification section: what evidence would force this principle's retirement?

The clause is machine-checked. The kit's validator, scripts/validate.py, fails a revealed principle that carries no evidence references. A revealed claim with no evidence is a lie the schema catches before a reviewer ever reads it. An aspirational principle that accumulates behavioral evidence has earned promotion, and the promotion is a visible edit to kind:, in a diff, with a date. Both transitions leave history. Neither happens silently.

Compare that to every notes system I have abandoned. They all held claims about who I am. None of them could hold the claim to a standard of proof, because a standard of proof needs structure: a place evidence attaches, a type that demands it, a check that fails without it.

What the schema costs, and why the cost is the point#

A loose note takes five seconds. A schema'd principle takes ten minutes, and most of those minutes go to the questions I would rather skip. What does this actually serve? What would falsify it? Is this revealed or am I flattering myself with the word?

Ten minutes of friction per principle is the price, and the purchase deserves precision, because this is the center of the argument. Each awkward question maps to a field, each field feeds a check, and each check runs without me. Discipline I have to remember is discipline I will eventually skip; I wrote a whole post about recall being the exact faculty that fails. The schema moves the discipline into structure, where forgetting is not an option. The schema is the product. The Sunday report, the staleness sweep, the audit trail, the promotion rule: those are consequences.

The same shape shows up in context architecture: the pile of documents was worth little until structure let each task load a curated slice. A corpus without a schema is a pile. The pile feels productive to grow. Only the schema makes it consultable, checkable, and honest.

The contract the self implements#

Spec-driven development works because the spec is a contract and the code implements it. When the code drifts, tests fail; the contract, not the memory of the author, is what holds. SpecSelf is that idea moved one altitude up. The schema is the contract, and the self implements it: behavior supplies the evidence, reviews audit the claims, and the corpus refuses to let a flattering assertion sit unexamined next to a falsifiable one.

If you take one move from this post, take the smallest version of it. Before adding a single feature to whatever system holds your own record, decide the fields, and make each one buy a mechanical check. Without a schema it is a diary. With one it is a spec, and specs can fail loudly, which is the only reason to trust them when they pass. The Starter ships the ten fields and the validator so you can start from a schema instead of a pile.

A Check You Never See Fail Is Already Dead

Eric Caskey — Fri, 17 Jul 2026 00:00:00 GMT

For weeks, a sentiment collector on my fleet reported a clean exit on every scheduled run while the Python inside it failed every single time. The wrapper was the culprit: the task ran cmd /c script >> log 2>&1, and cmd returns the exit code of the redirection, not the program. The redirect always succeeded. The job was dead and telling me it was healthy, twice a day, on schedule.

Nobody investigates a green check. That is the whole problem. A red result gets a ticket, a log dive, a fix by Friday. A false green gets trusted, and it compounds: every day of "success" is another day the gap between what I believed and what was running got wider. When I finally noticed, the data the collector was supposed to be gathering had a multi-week hole that no amount of fixing could backfill.

It gets worse, and the worse part is the lesson. I had a watchdog for exactly this: a weekly job that checked whether scheduled tasks were running on cadence. It never flagged the dead collector, and the reason is painful. The watchdog was one of the broken jobs. Its silence read as health. I had built a checker and never once seen it fail, and I took that as evidence it had nothing to report. A check you never see fail is indistinguishable from a check that is dead.

Three signals, because any one of them lies#

The rebuilt job-health system refuses to decide health from a single signal, because each signal has a documented way of lying to me.

An exit code lies the way the collector's did: the wrapper reports its own success, not the program's. So results are read with the scheduler's vocabulary in mind, where "has not run yet," "still running," and "battery refusal" are distinct non-failures, and a zero from a cmd redirect is treated as a claim, not a fact.

Configuration lies by rotting. A task can point at a script that moved, a user profile that no longer exists after a machine migration, or the Microsoft Store's python.exe stub, which exists, launches, and never runs anything. Config rot is invisible to result checks because the task often "succeeds" at running the wrong target.

The catcher for both is staleness: a job that claims success but has not actually executed within its own trigger cadence is lying somewhere, and the staleness check does not care where. It compares the claim against the calendar. Of the three signals, this is the one that would have caught the collector in week one, because no signal a broken job emits can fake the passage of time.

One signal is a guess. The triangle is a verdict.

The checker gets checked, on a schedule#

The watchdog failure taught me that layering checkers is not enough, because the layer on top inherits the same disease: I will never see it fail either. So the fleet now runs a weekly job whose only purpose is to plant a failure. It registers a deliberately broken scheduled task, asserts that the watchdog flags it, and then cleans up. If the watchdog ever stays quiet about the plant, the watchdog is the incident.

I wrote about the code-layer version of this in the Ballast post: a secrecy scanner that skipped its own source file, and therefore published the exact string it existed to suppress. Same disease, same cure. The scanner now scans itself, and the eval suite includes cases the scanner must fail. A check that cannot fail on what you care about is a green light wired to the wall.

The gate that passed while the bundle was broken#

Three-signal health still misses one version of the false green, and it bit me this month. A packaging gate on one of my APIs hard-fails a Lambda bundle that contains Windows wheels, because a Windows-built bundle once took down every function behind the API. The gate worked. Then a bundle built on modern Linux shipped wheels compiled for a newer glibc than the Lambda runtime has, and died at import with the same total outage. The gate passed that bundle without complaint. It was checking the failure mode I had already met, not the one I was about to.

"The gate passed" is only as meaningful as the gate's scope. A passing check with the wrong scope is a false green with extra steps, and the fix is the same discipline pointed at coverage instead of liveness: when a new failure mode gets past a gate, the gate grows a case for it the same week, and the case is proven by planting the bad artifact and watching the gate reject it. Both directions matter. The gate has to fail on what I fear, and I have to see it fail.

Name how you would find out#

The guardrails post ends on an open question: pre-execution guardrails protect the platform, but nothing in that pattern protects the checks themselves from rotting. This is the answer I use, and it fits in two sentences.

For every check that guards something you care about, name the specific way you would find out it died. If the honest answer is "someone would notice," schedule the plant: a deliberate failure, on a calendar, that the check must catch, so that silence stops being ambiguous.

I used to read my fleet's dashboards as reassurance, all that green stacked up like evidence. The dangerous state was never red. It is a green you have stopped deserving.

Collect Everything, Trust Almost Nothing

Eric Caskey — Fri, 17 Jul 2026 00:00:00 GMT

Over the past few weeks I taught my market tooling to listen to roughly ten different crowds at once. The bull-and-bear chatter on a trading message board. Raw forum posts pulled from public archives, which I score for sentiment myself instead of trusting anyone's black-box count. Front-page tech chatter. Search trends. Encyclopedia page views, which turn out to be a wonderfully unglamorous attention signal. Insider filings. Video-platform chatter. A news-tone feed. Every lane lands in one store, on one schema, stamped with where it came from and when.

The obvious next move, the one every side-project instinct screams for, is to feed all of that into the engine that grades stocks every night. More signal, better grades. That is the move I refused, and the refusal is the only part of this build worth writing about.

A one-way wall#

I wrote before about the boundary inside my finance engine: the data trusted enough to composite into a grade is walled off from the data trusted only enough to watch, and a test keeps the crowd from ever moving the number. This project extends that same line around the entire collection layer. Everything the tracker gathers may flow into analysis, dashboards, and questions. Nothing flows backward into the grade. The wall runs one way, and the grading engine has no path through it.

The reasons are the same ones from the original post, just louder at this scale. Every one of these feeds is unaudited. Some are gameable by a motivated stranger with a few accounts. The counting services are black boxes that change their methodology without telling you. The endpoints themselves churn, rate-limit, and disappear, and a signal that vanishes mid-month is not a signal you can hang a grade on. A grade has to be explainable after the fact from inputs I can defend. "The crowd got loud" is not a defensible input. It is exactly the noise the grade exists to be calm against.

Making the wall structural#

The rule would be worthless if it lived in my memory, because convenience erodes memory. So the wall is built into the shape of the system, the same trick that made a compliance rule impossible to break in my visualization work: the safe path is the only path that exists.

The tracker is a separate codebase with its own store. Its fused score is its own number, computed by its own pipeline, published to its own file. There is no import path, no shared table, no code route by which that number reaches the grading engine. One source needed special care: the trading message board already feeds a small display widget elsewhere in my system, so rather than extend that existing pipeline and risk entangling it, the tracker got its own clean-room client. Two clients for one API is mildly wasteful and completely safe, and I will take that trade every time.

Inside its own walls the tracker is allowed to be opinionated. Each source's scores are standardized and clipped so one hysterical feed cannot dominate. The fusion has a floor on the number of independent sources that must agree before a composite exists at all; below the floor, the day simply has no composite, because a number backed by one noisy feed is worse than no number. Absence beats fabricated confidence. That principle has survived every project I have applied it to.

What collecting is actually for#

If none of this may touch the grade, why collect it at all? Because watching is genuinely valuable, it is just a different job than grading. Divergence is the interesting product: when the crowd gets loud on a name and the insider filings stay quiet, or the reverse, that is a question worth investigating, and I built a small public toy around exactly that tension. Attention data also answers questions grades never could, like whether a name is being discovered or abandoned, and whether a move arrived with chatter or in silence.

And there is a legitimate gate through the wall, with a lock on it. If the tracker's composite ever earns a place in grading, it gets there the way any signal must: pre-registered, tested forward on out-of-sample data, judged by the same rules that keep a backtest honest, and admitted as a deliberate, versioned decision. The wall does not say never. It says not through convenience, and not silently.

The portable version#

Strip the finance from this and one design question remains, useful anywhere data flows near a consequential number: for every feed you add, write down which numbers it is allowed to touch, and make the default none. Collection is cheap and greedy by nature; consequence should be expensive and deliberate. The systems I trust most are the ones where those two appetites are physically separated, where you can add a tenth source on a whim precisely because there is no path by which a whim reaches the output that matters.

Collect everything, and let all of it reach the watching side. Keep the grade behind the wall, with one gate, and open it only on purpose.

Related:

Composite What You Trust, Watch What You Don't, the original boundary this post extends outward
Insiders vs Crowd, the public toy built on the divergence question
How to backtest without fooling yourself, the gate any signal must pass to earn consequence

When your method repo and your product repo don't talk to each other

Eric Caskey — Thu, 16 Jul 2026 00:00:00 GMT

SpecSelf is three repositories. The method is a public repo: markdown templates, persona definitions, steering docs, MIT licensed, forkable from specself.ai. The API is a private repo: FastAPI on Lambda. The app is a third: a private Next.js PWA. A few weeks after all three were live, I went looking for the place where they agreed on what a principle even is. There wasn't one. The domain enum was defined in four places. The persona I rely on most had a lens file in the public repo and a contract file in the API, and the two had quietly stopped saying the same words.

That is the drift I want to describe, because it is the expensive kind. Code drift inside one repo gets caught by tests, by review, by the compiler if you are lucky. Contract drift between the repo that defines the method and the repos that run it gets caught by nothing, because no single repo can see it.

The invariant I thought I had#

The design in my head was clean. The public method repo is the contract. It declares what a principle looks like, what the seven life domains are, what each persona may and may not do. The API and the PWA implement that contract. Anyone who forks the public repo gets the same method the hosted product runs, because they are the same definitions.

The design on disk was three repos that had each written down their own recollection of that contract. Nothing imported anything. Nothing checked anything. The invariant existed only as an intention, and an intention is not an enforcement mechanism. I have made this exact argument about spec-driven development at the code layer for months, and I still shipped the seam without a check on it. Here is the confession: the method repo preached falsifiable records, and its own relationship to the product was an unfalsifiable claim.

What the drift actually looked like#

Three examples, in increasing order of embarrassment.

The seven-value domain enum, the list every artifact in the system tags itself with, was defined in four places: the kit's templates, the kit's docs, the API's models, and the app's UI copy. Four hand-maintained copies of a seven-item list. The day one of them gains a value, the other three are wrong, and no build anywhere goes red.

The principle template declares ten frontmatter fields. Diffable, dated, falsifiable, versioned: that is the whole pitch of the method. The hosted API was storing a principle without confidence, without serves, without last_reviewed, without supersedes. Half the schema simply never made it into storage. The serves field is the input to the method's coherence check, the graph walk that catches a project serving no goal. Which means the hosted product could not run the method's own signature check against its own corpus. The public repo promised an audit; the product had silently dropped the fields the audit reads.

And then the persona system, which is where it stopped being abstract.

The bug that made me look#

Each persona in the method is a lens with a contract: what it looks for, what it must not do, how often it may appear, and which other persona it refuses to share a session with. The Skeptic is the one that hunts borrowed language in my records. Its public lens file says a Skeptic session should be rare. The API's contract file agreed in spirit and drifted in the numbers: the code enforced a 45-day cooldown while the persona spec said 14. Same persona, same product, two different opinions about how often it exists, and both were shipped.

The second half was worse. The Skeptic's contract declares which persona it is mutually exclusive with, so a session never pairs voices that produce whiplash. That declaration is supposed to be symmetric, carried on both sides. It wasn't. The exclusivity held in one direction and not the other, which means the rule worked or failed depending on which persona the selection algorithm considered first. Both halves of this are sitting in the public issue queue: the cooldown drift is specself-api #48 and the asymmetric exclusivity carry is PR #53.

Neither is a bug a compiler can catch. Neither is a bug a code review is likely to catch, because each half of the contradiction reads as perfectly reasonable in its own file, in its own repo, in its own pull request. The reviewer of the API change would have needed the kit's lens file open in another tab and a reason to be suspicious of a number as innocent as 45.

Why review can't hold this seam#

My first instinct was process: be more careful, check the other repo when touching a contract. That instinct is worth naming so it can be dismissed. Two repos means two review contexts and no shared PR. The drift is invisible precisely at the moment of review because review happens inside one repo at a time, and the contradiction only exists across them.

I have written before about config pairs, values on two sides of a contract that must change in lockstep, like a framework's trailing-slash setting and the CDN rewrite function that assumes it. The rule there was that a config pair names its counterpart, and a machine checks the pair. What I had built with SpecSelf was the same problem one level up: not two config values inside one system, but two repositories where one defines a method and the other runs it. Cross-repo config pairs, with no pairing.

A check that spans repos cannot live in anyone's vigilance. It has to be a machine check, running in CI, with the contract itself as its input. Which raises the real question: which repo owns the contract?

The move: the method repo became a schema repo#

The answer we landed on is recorded in an architecture decision record, and it took about a week to implement. The public method repo stopped being documentation and became the contract. It gained a schemas/ directory: an enums file that is the single canonical home of the domain taxonomy and every status vocabulary, plus machine-readable schemas for the principle and persona shapes. The templates that humans read now point at those schemas instead of restating them. The persona lens files carry their contract in frontmatter: cooldowns, exclusivities, the must-not-do list.

The API vendors those files at build time. And three drift tests now run in its CI: one asserting the code's enums match the vendored contract, one asserting every persona's code contract matches the definitions vendored from the kit, including the rule that exclusivity declarations must be symmetric, and one holding the foundation questions in lockstep. The stored principle was extended to the full starter shape, so the coherence check finally has its inputs.

The class of bug did not shrink. It converted. Enum drift, persona drift, schema drift: each is now a failing test with a file and a line, instead of a contradiction waiting in the gap between two repos for the selection algorithm to find it on a Sunday morning. The Skeptic fixes above are that machinery at work, moving through the queue as tracked, testable changes rather than surprises.

Your version of this seam#

The specifics are mine, but the seam is everywhere. A design system has it: the Figma library defines the method, the component repo runs it. An editorial operation has it: the style guide defines, the CMS enforces, or claims to. Contract-first API shops have it in its purest form: the OpenAPI spec and the server that allegedly implements it. In every case there is a repo that says what should be true and a repo that makes something true, and the question is whether anything machine-readable forces them to agree.

The test is quick. Pick the enum, the schema, or the rule that both sides depend on, and ask what turns red when one side changes it. If the answer is nothing, you do not have a contract. You have two repos with similar opinions, and the opinions are diverging at whatever rate you ship.

The fix is not more discipline at review time, and it is not merging everything into a monorepo so the seam disappears. It is promoting the method repo to a schema repo: one canonical, machine-readable statement of the contract, vendored by every runtime that claims to implement it, with a CI job on each side that fails on drift. The drift I found was never really a persona bug or an enum bug. The method and the product had stopped talking, and nothing was listening for the silence.

Related:

I Keep My Whole Life in Spec Files, what SpecSelf is and the one rule that makes it safe, the essay this series builds on
Context architecture beats documentation dumps, where the config-pair rule comes from
SDD is context management, the methodology this post applies one level up

What the SDD Playbook Did Not Cover

Eric Caskey — Sun, 12 Jul 2026 00:00:00 GMT

Three months ago I wrote about spec-driven development and the folder architecture that makes it work. The argument was that specs should be the source of truth rather than chat history, that CLAUDE.md should be a minimal routing file, and that workspace structure, not multi-agent orchestration, is the real agentic architecture. Since then I have been running that playbook daily, across two production sites and a fleet of personal projects. Most of it held up. Three ideas the essay never mentioned turned out to matter more than anything in it. This post is about those three.

Implementation Reality#

The original essay named spec rot as a risk. It did not solve it.

Running the playbook surfaced the problem fast: specs drift from code faster than you expect, and in ways you cannot see from the spec alone. The mitigation I converged on is a section at the bottom of every package README called Implementation Reality. Four parts, updated at every reconciliation pass: what shipped (with commit or PR references), spec-but-not-in-code (requirements the spec describes that no code implements yet), code-but-not-in-spec (features built ahead of the spec, which need backfill), and the follow-up PRs the document anticipates.

In the package that drives my finance engine, Implementation Reality is more than half the README. It tracks five sprints of audit closures, eleven ADR pivots, and every dead config key the production code now ignores. It is the single most valuable document in that repo: the handoff artifact that lets the next session start on solid ground instead of trusting a spec that might describe a design that shipped differently.

If you adopt nothing else from the playbook plus this post, adopt this. Every package README gets a dated Implementation Reality section. Update it at every reconciliation. The spec captures intent. Code is the source of truth for what was built. Implementation Reality is the bridge.

ADRs amend each other#

The original essay treated architecture decision records as write-once.

In production they evolve. ADR-002 in a package's decision/ directory amends the API contract that ADR-001 established, which amended the default format the original feature spec set. The ritual that keeps this honest is the amendment note: both the amending ADR and the amended document carry a dated entry in their change logs, and the amending ADR's status block explicitly names what it supersedes. A "Supersedes:" line in an ADR header is load-bearing, not metadata.

Without the amendment ritual, specs silently diverge. With it, the chain of decisions is auditable: any future reader can trace which ADR currently governs a field, and why. The public demo repo shows the pattern in two files, a package-scoped ADR-002 that amends a feature's original API default, with the amendment recorded in both places.

The stateless and persistent tool split#

The original essay described a workflow in one tool. Now my workflow is two, and the split does real work.

A stateful assistant carries the deliberative work: planning, research, spec writing. It has repository access and memory across sessions, which is what lets specs evolve instead of restarting every conversation. Claude Code carries the stateless, fast work: reading a spec and opening a PR with working code. Each tool does the job it is good at. Specs are the handoff boundary between them.

The split is a structural answer to a structural problem. Implementation wants a fresh context per task, where state is overhead. Decision-making wants the opposite, where state is the whole point. Forcing one tool to do both collapses the workflow in predictable ways: either the implementation context fills up with design history it does not need, or the design session forgets what last week resolved. The tool split is what makes the routing table scale across the actual work.

A concrete example. A /finance route worked in the Lambda dispatcher but had never been registered in the API Gateway stack. The browser got a 403 Missing Authentication Token, CORS headers never attached, the page threw Failed to fetch. The obvious fix was a two-line infrastructure patch. Instead the spec amendment landed first, noting that handler routes and stack routes had drifted twice that month, with an explicit follow-up to add an infra test that compares the dispatcher's routes against the stack. Then the patch. The bug was real and urgent. The spec is what prevents the next one. Specs lead, code follows, and that discipline is only possible because the deliberative tool and the executing tool are separate.

What the original essay had right#

Most of it. Specs as source of truth. CLAUDE.md as a router, not a brain dump. Workspace context files for per-area loading. Separation of concerns as the operating principle. If you have not read that essay, start there; this post is the follow-up, not the primer.

The discipline is in the specs. The automation is in the pipeline. The tools are workers. And the receipts already exist: I published the production numbers after the first week and again after a full month, so you can check whether the playbook cashes out in shipped work rather than diagrams.

Related:

Spec-driven folder architecture, the original playbook this post extends
SDD is context management, why the method works at all
June SDD production numbers, the receipts

Fable Thinks, Sonnet Builds

Eric Caskey — Fri, 10 Jul 2026 00:00:00 GMT

The first Fable cap hit at the end of last week. Annoying, but survivable; the window rolls over and you wait. The second hit less than 48 hours later, on July 6, in the middle of a wave of parallel agents. That one stung, because the agents died holding unfinished work. By Thursday I was close enough to the total token limit that the only honest move was to stop working. Anthropic reset my limits Thursday night.

This is the third post in a series I keep hoping is finished. In June I said goodbye to Opus when Fable 5 shipped. Four days later an export directive pulled Fable offline, I noticed I had burned 90% of my usage, and I said hello again. That post ended with a plan. Fable thinks, Opus builds: save the frontier model for work where its judgment changes the outcome, let the workhorse carry the volume, and put the budget guard in code instead of in my head.

I published the plan, felt good about it, and wired none of it.

How it happened again#

Fable came back to the subscription plans and the fleet drifted straight back to frontier-by-default, because nothing in the harness said otherwise. The first cap was ordinary volume catching up with me: backlog loops, review passes, the daily grind, all billed at frontier rates for no reason anyone could defend. The second was the agent wave, a burst of parallel subagents all running on Fable at once.

The frustrating part is that the June post named the exact failure. "My restraint mid-loop is not in code." I wrote that sentence, published it, and then kept the plan in the one place an autonomous loop never reads: my head.

I don't think the pull is unique to me. When the best model is a keystroke away and included in the plan, every task looks like it deserves it. No single session feels expensive. The bill shows up later, as a wall, usually mid-wave.

What I changed this time#

The plan went into files instead of a blog post. Three changes, same shape as June, different medium.

1. The tiers are policy now. There is an architecture decision record in my specs repo that says who does what. The frontier model supervises: it triages the backlog, writes the briefs, reviews the PRs. Sonnet executes. Haiku gets the purely mechanical items. The slogan needed an update anyway, since Sonnet 5 closed most of the coding gap over the summer: Fable thinks, Sonnet builds.

2. The loop is pinned where it can't drift. The backlog command that eats most of my volume now declares its model in its own settings, read fresh on every invocation. The loop runs on Sonnet no matter what mood the session is in. This forced a change I didn't expect: a cheaper model only succeeds when the work item removes the need for judgment, so items now carry the files to touch, the exact verification commands, and what is out of scope before the loop is allowed to pick them. Writing those briefs is real work, and it is exactly the work the frontier model should be doing. The routing didn't just cut the burn. It pushed the expensive model toward the part of the job where it earns its rate.

3. Escalation is one consult, not a wave. When a Sonnet worker hits genuine ambiguity, it gets a single Fable consultation, one subagent handed a self-contained brief, and then it either proceeds or blocks the item for triage. Parallel Fable subagents are banned outright. That pattern is what crashed the July 6 wave, and it never comes back.

Anthropic is clearly thinking about the same split. The new advisor tool does this natively at the API level: a Sonnet executor runs every turn, and a Fable advisor gets consulted mid-task with the full transcript forwarded automatically. And their usage guidance already says it plainly: Sonnet is the right choice for the large majority of coding work, switch up when you need it. The vendor selling the expensive model is telling you not to run it as your default. It took me two caps to take the hint.

The two shapes, side by side:

This post includes a diagram. View it on caskeycoding.com.

Receipts#

I pulled the numbers out of my own transcripts instead of guessing. Every Claude Code session logs per-message token usage, so I priced the last two weeks at API list rates. The subscription absorbs the dollars, but the cap doesn't care, and the list price is the honest weight of the habit:

July 1 through 8, fleet-wide: about $3,350 at list rates. Roughly two thirds of it was Fable.
The single worst session: a two-day run on nothing but frontier models that priced out at $918 by itself.
July 9, the day before the reset: zero Fable tokens. Not discipline. An empty tank. The whole day ran on Opus and Sonnet because there was no Fable left to spend.

I also ran one controlled comparison, the same small task through each shape with an API harness: identical deliverable, and the frontier model's output dropped from about 5,900 tokens doing everything to about 1,200 giving advice. On a task that size the dollars are noise. The ratio is not. The scarce resource on a subscription is frontier output, and the routed shape cuts it by four fifths. Applied to a $400-a-day habit, that ratio is the difference between a Thursday reset and a normal week.

The medium was the bug#

The June plan was right. It failed anyway, because a plan in a blog post depends on me remembering it mid-loop, and mid-loop me has never once been the guy who remembers. The settings file doesn't have that problem. Neither does the decision record, or the loop that reads its model from disk every time it wakes up.

Fable thinks, Sonnet builds. Last month that was a slogan. Now it's a line in a file, which is the only place a plan survives contact with an autonomous fleet.

I Keep My Whole Life in Spec Files. My Agent Reads Them and Never Writes Them.

Eric Caskey — Fri, 10 Jul 2026 00:00:00 GMT

Every Sunday morning, an AI agent interviews me about my own life. It has read everything I have written down about the person I am trying to become, every goal I have set, every standard I claim to hold. It is very good at noticing where the last week did not match. This weekend it asked me why an intention I have now written three months running still has no evidence behind it. I did not enjoy the question. That is the entire reason it gets to ask it.

It can challenge me with my own record. It is not allowed to write a single word of that record itself.

For the last year I have been making one argument about working with AI: give an agent a curated structure to reason over, not a pile of documents. Context architecture beats documentation dumps. I built that idea into two production sites and a finance engine, where it meant scoped spec slices and bounded agent context. Then I pointed the same idea at the least organized system I own, which is my own life, and it behaved exactly the way it behaves on code.

A life as a spec'd system#

The principles I am trying to live by, the person I want to become, my goals, the areas I am responsible for, the projects I am driving: all of it lives as plain markdown in a git repository. Not in an app. Not in a chat history I will never scroll back through. Files.

An agent reads those files at the start of a session the way a specialist agent reads a scoped spec before it touches code. Because the structure is there, it can see vertically. It can tell that this week's work connects up to a goal, that the goal serves a stated principle, that an area I claim to care about has gone quiet for a month. That vertical view is the part you cannot hold in your head on a Tuesday, and it is the part a structured agent is good at surfacing.

The one rule#

Here is the rule, and it is the part most "AI life coach" products get backwards. The agent organizes, connects, surfaces, and challenges. It never silently authors. I write my principles. I decide when a goal is met. I commit the journal entry in my own words.

The agent is allowed to interrogate me hard. It can point out that a principle I keep restating has no behavioral evidence. It can name the friction that has shown up three months in a row. What it cannot do is put words in my mouth and let them harden into the record, because the moment something other than me authors my life, the record stops being mine, and a record of your life that you did not write is worth nothing.

This is the same boundary I draw everywhere there are consequences. In my finance engine, the data I trust enough to composite into a grade is walled off from the data I only trust enough to watch, and a test keeps the crowd from ever moving the number. This is that same line, drawn in a different place. There it separates the inputs that may set a grade from the ones that may only sit beside it. Here it separates who may write the record from who may only question it. I use AI to pressure-test and accelerate, never to do the thinking I need to own. Restraint is the value. The agent makes the record sharper. It does not get to write it.

The part I am still bad at#

I will be honest about the hard part, because the honest version is the useful one. Nothing enforces this. A failing test screams at you; a life OS just sits there if you stop showing up. The system is only as good as the questions you let it ask and the cadence you keep, a couple of minutes most days, twenty minutes most weeks, a longer synthesis most months. And because there is no backend and no account, you carry your own backups and you decide which model provider sees a given session. Those are real tradeoffs of a system you own, and I would rather you know them going in than discover them later.

The rule itself is hardest exactly when it matters most. There was a session where the agent offered a cleaner version of a principle than the one I had written, sharper, truer-sounding, sitting right there for me to accept. I wanted to keep it. That is the moment the rule is for. I made myself rewrite it in my own clumsier words, because a principle I did not author is one I will not live by. The pull to let a fluent machine phrase your life better than you can is real, and giving in to it is how you end up with a beautiful record of someone who is not you.

I made my setup public#

The reason I trust this with the most private record I have is the design. The content of my life lives in a folder that is ignored by default, so the scaffold can be shared while the substance never leaves my machine. I run my own private instance, and it is in use rather than a demo: the foundation session is done, the principles and the journal are accumulating, the weekly cadence is holding. I am not going to show you any of it, and that is exactly the point. The method works without the method ever seeing your diary.

So I made the scaffold I use public. It is free, you fork it, you point your agent at one file that explains the contract, and you run a guided foundation session where the agent interviews you and you do the writing. It works with any capable agent; with Claude Code you also get the persona lenses and the cadence commands that make the weekly loop faster. It lives at specself.ai. Your content stays in your repo. I never see it. There is nothing to sign into.

The test#

This is about the line you draw any time you hand an agent a domain you care about. So here is the test I would give anyone pointing an agent at something that matters: name who is allowed to write the record. Not who reads it, not who comments on it, who writes it. If the answer is the agent, the record has quietly stopped being yours, no matter how good it sounds when you read it back. A code review, a design doc, a performance self-assessment, a journal: the agent can read all of it and question any of it, but the moment it holds the pen, you are no longer the author of your own account.

If you build software, the rest of the discipline is already yours: write it down, keep it honest, revise it forever, and never let the tool quietly become the author. I just stopped making my own life the exception. The agent reads my spec files and asks the questions I would rather avoid. I am still the one who has to answer them in writing. Restraint is the value, and it turns out that is as true for a life as it is for a system.

Related:

Composite What You Trust, Watch What You Don't, the same boundary in my finance engine, drawn between the data allowed to set a grade and the data that only gets to sit beside it
Context architecture beats documentation dumps, the method underneath all of this
Spec-driven folder architecture, where the file structure itself started

The Pocket Quant

Eric Caskey — Fri, 10 Jul 2026 00:00:00 GMT

At 12:06 this afternoon my phone buzzed. "Nothing clears the bar - still a no-call. Most attractive: INCY, just starting to pull back off its 52-week high into the 7/28 print; flips to a call only if it bases and holds with a second lens confirming - watching, not chasing."

That is ARGUS. He is a quant who lives inside my research platform, runs on a schedule, and texts me three times a day. In his first week of operation he has recommended buying exactly nothing. I consider this his best feature.

I build platforms. For the past few months the side project has been a quantitative research platform: data collectors, a scoring engine, a wall of 3D market visualizations that render the whole equity universe as one canvas. It got good enough that I stopped asking what to build next and started asking a harder question. A platform is only proven when someone who is not you can walk in and operate it. Nobody else was walking in. So I built the someone.

ARGUS is named for Argus Panoptes, the watchman with a hundred eyes, because that is the actual job: read every board, not just the flashy one. He is a headless Claude session dispatched by a scheduler at pre-open, midday, and post-close. One persona file is his entire behavior source. Version-controlled, hash-stamped, no second copy anywhere that could drift. The pings ride ntfy, a small open-source notification relay; my phone is the whole client.

He does not crawl the platform looking for data. Before each session, a deterministic script builds him a pack: every visualization's feed distilled to what carries signal, live quotes, crowd sentiment, his own notes and open calls, anything I said to him since the last run. It is the same context-architecture bet I make everywhere else. Each task loads a curated slice, not the whole corpus. The model spends its budget on judgment instead of fetching, and the run costs a fraction of what a crawl would.

The judgment is where it gets interesting, because an agent that picks stocks is a weekend project. An agent whose opinions you can audit is a platform feature. The difference is the harness around him.

Every candidate ARGUS surfaces is pre-registered before the outcome exists: entry, stop, target, a numeric probability, a horizon, a benchmark matched to the sector so he cannot claim credit for a rising tide. The journal is append-only and script-mediated. He cannot edit history; deterministic code resolves each call against daily closes and writes the result. An expectancy gate sits in front of his mouth: if the stated probability times the reward, less the risk and a cost haircut, does not clear zero, he is required to say nothing. Silence is a valid output. And he never uses a trade verb. Surfacing is the whole job. Deciding is mine.

Here is the confession. On his first night under the full harness, he went quiet when I expected a ping. I dug into the logs expecting a bug. Instead I found a decision: he was required to send exactly one message per session, could not verify whether an earlier run had already sent one, judged a duplicate notification to be the one irreversible outcome on the table, and held the push. Then he wrote in his own log that the harness gave him no way to check what had already been sent, and that this gap had forced the hold. He was right. I shipped the send-log an hour later. The agent audited his own cage and the platform got better for it.

As of today his track record contains zero resolved calls. He has ranked the boards, flagged the loudest theme on them as a crowded hedge rather than an opportunity, corrected his own memory of an earnings date, and refused to chase an extended name into a binary event. All reasonable. None of it proven. In a few months the scorecard will say, against a matched benchmark and with a sample-size warning built in, whether his picking adds anything beyond the deterministic screen he draws candidates from. The record segments every time I edit his instructions, so I cannot quietly move the goalposts and pretend it was one continuous streak.

This is the one place the side project touches the day job. The work I care about is platform primitives: the boundaries that let an operator act fast without being able to hurt anything. ARGUS is that idea with a personality. The persona is fun. The gate, the journal, and the audit trail are the product.

I built a platform that can prove whether an agent should pick stocks. He watches the market, the platform watches him, and the first scorecard reads out in the fall. I will publish it either way.

Fifteen Million Was the Easy Part

Eric Caskey — Thu, 09 Jul 2026 00:00:00 GMT

I built a Black-Scholes options pricer in C++, and this post is the why, the what, and the how. Why I wrote it, what it turned into, and how a clean first version that priced fifteen million options a second became one that prices 215 million. The headline is the speed. The real story is how often I was wrong about where that speed would come from, and how measurement kept correcting me.

The first version was the textbook version: clean, allocation-free, correct, checked against published prices and put-call parity. On a single core it priced about fifteen million options a second. That sounds fast until you learn what a modern core can actually do. Fifteen million was the easy part. Everything worth telling happened on the way to 215 million.

Why I built it#

One side project, three goals. I wanted to relearn the numerical methods behind option pricing, the ones I last touched years ago. I wanted to practice low-latency code, the kind I never get to write in a web backend, where the clock is a network round trip and not a nanosecond. And I wanted an artifact, something I could point at instead of describe.

The engine earns its keep now. The volatility math behind the 3D surfaces in my playground runs on it, cross-checked against a reference before I trusted a single number on screen.

What I built#

At its core the engine prices an option and its full set of Greeks with no memory allocation, verified against known values. Around that core sits the rest of the roadmap: a Monte Carlo engine with the standard variance-reduction tricks, a risk view that revalues a hundred-thousand-position book about a thousand times a second, and a Python binding that holds roughly 133 million options a second even when called from NumPy, about fifty times what a pure-array version manages.

So the shape of it is a small, fast, honest library. Fast enough to be interesting, honest enough that every number came with a test. It is a learning artifact, not a trading system, and the market data it serves is the educational playground on this site, not an execution engine.

How it got fast: three times I was wrong#

The climb from fifteen million to 215 million was not a straight line of clever ideas. It was a sequence of confident assumptions, each killed by a measurement.

The bottleneck was not where the guides said. Every performance guide opens the same way: restructure your data so memory access goes linear. I did it carefully and the loop got slightly slower. The advice was not wrong in general, it was wrong here, because this loop never touched much memory. An option pricer lives inside exponentials, logarithms, and the normal distribution, which is to say inside the math. Reorganizing memory for a loop that is starved on arithmetic solves a problem it does not have. Once I attacked the math instead, swapping one modern library call for a sixty-year-old polynomial and threading a loop that was embarrassingly parallel, the number jumped from fifteen million to about 114 million.

The compiler claimed a win it never delivered. The plan always included SIMD, the vector instructions that handle four numbers at once, and modern advice says the compiler does that for you on clean loops. My build logs agreed and reported the loop vectorized. They were reporting on a different loop, a trivial one at the end of the benchmark. My compiler would not fold a loop full of exp and log calls into vector form. Some toolchains will; mine did not. So the loop that mattered stayed scalar every single time, its failure notice sitting quietly in a report I had not opened. The honest accounting: none of my first big speedup came from SIMD at all.

So I wrote the math by hand. Four-wide exponential and logarithm, built from classic polynomial cores, intrinsics all the way down. The discipline that saved me here was borrowed and simple: test each function in isolation against the standard library before wiring it into anything. Not the pricer against the pricer, the exponential against the exponential. When a bug showed up, the isolated test named the exact function instead of leaving me to hunt a wrong price three layers downstream. With real SIMD underneath, one core gained a clean four times, right at the theoretical width, and across all cores the engine reached about 215 million a second.

What I keep from it#

I set out to relearn some numerical methods and end up with a small, fast library I could point at, and I got both. The number I quote is 215 million a second. The lesson I keep is the ledger of misses. The data layout that every guide recommends did nothing, because the bottleneck was arithmetic, not memory. The compiler reported a victory that belonged to a loop nobody cared about. The single biggest win was trading a modern function for an old approximation I could reason about. Fifteen million a second was free; the next two hundred million were an education in checking what is actually true, one report line at a time.

Two honest caveats belong on the record. Desktop timings wander ten or twenty percent run to run, so read every number here as a shape, not a decimal. And none of this is a trading system. It is a learning artifact that happens to be fast, which was the point.

Related:

Rotating an Option, the 3D surfaces this engine's volatility math feeds
How to backtest without fooling yourself, the same measure-before-believing discipline pointed at signals instead of speed

Rotating an Option

Eric Caskey — Mon, 06 Jul 2026 00:00:00 GMT

A 3D render crossed my feed once, one of the slow rotating surfaces the account tuba.captures posts, the kind where you can almost feel the curvature through the screen. It stuck with me, and a few days later I wanted that for something I actually stare at: an option.

An option is a contract that lets you buy a stock at a set price later, and it is normally drawn as a hockey-stick: the payoff on the day it expires, a flat line of value against price. But that picture is only the last day. The days before it, the ones you actually hold through, are a curved sheet of profit and loss bending over price and time together, and the flat chart throws that second dimension on the floor. So I priced the option across a grid of prices and dates, stacked the profit and loss as height, and drew it as a surface I could grab and turn. It is live in the playground, at /play/greeks-manifold.

The first time I rotated it, gamma was just sitting there. The Greeks are the standard set of numbers traders use to describe how an option behaves. Gamma is the one for how sharply the option's value bends, and it is biggest right at the strike price (the price the option lets you buy at). As a number it is abstract. On the surface it is a ridge, a visible spine running across the sheet near the strike, and coloring the surface by gamma lights the ridge up. Theta is another Greek: the value the option loses as expiry nears. On the surface it is the downhill slope in the time direction, so you can see exactly where it bites hardest. The numbers I had been computing as columns in a table were just the shape of the object, seen from different directions.

The surface from the playground, colored by gamma. The glow along the strike is the ridge; the hot spot is where the bend is sharpest.

That was the hook. If one piece of option math could become a shape, the rest could too.

I should say up front that most of the finance vocabulary in this post is vocabulary I only really understood once I could see it. What follows is less me explaining these ideas to you and more me writing down what finally clicked when the numbers stood up off the page. The definitions are what I learned building this.

The five at a glance#

Surface	The three axes	What you are looking at	When to open it
Greeks manifold	stock price, time, profit/loss	an option's value bending, colored by a chosen Greek	before an options trade, to read the terrain ahead
Drawdown topology	time, account value, simulated path	a fan of simulated outcomes, colored by how far each is underwater	to calibrate your gut for how deep a bad run can dig
Yield curve dynamics	borrowing length, time, interest rate	the Treasury curve changing shape over time	to watch an inversion form (the recession warning)
Correlation regime	asset, asset, co-movement	the grid of how assets move together, tightening calm to crisis	when a sell-off starts and diversification evaporates
Implied-vol surface	strike, expiry, implied volatility	the volatility that option prices imply, as one sheet (the synthetic-smile mode of the Greeks manifold)	to read the volatility smile in a single glance

The drawdown topology is a fan of thousands of simulated price paths. (Simulating a lot of random paths to see the range of outcomes is called Monte Carlo, after the casino.) Each path is colored by its drawdown, which is just how far it has fallen from its high point. Flip the simulated returns from an ordinary bell curve to a fat-tailed one, meaning big jumps happen more often, and the whole cone visibly bulges. The textbooks say tails matter; here you watch the hole get deeper.

The fan, colored by drawdown depth. The yellow strands are the deep holes, the runs the bell curve says should barely happen.

The yield curve is just the interest rate for borrowing money over different lengths of time, from one month out to thirty years. Normally it slopes up, because longer loans cost more. Drawn as a surface you scrub through time, you can watch it invert, where short-term rates climb above long-term ones, the shape everyone calls a recession warning. Reading that an inversion happened lands as a statistic; watching the surface fold over lands somewhere lower.

Scrubbed to the far end of the story: short rates above long, the fold everyone calls a recession warning.

The correlation regime is the one I find hardest to look away from. Correlation is how much two assets move together, and the grid shows it for every pair at once. You dial it from calm toward crisis. "In a crash, all correlations go to one" is a line everyone repeats; watching every cell in the grid climb to one at the same moment is diversification evaporating in front of you, exactly when you were counting on it.

The grid at the crisis end of the slider. The whole field rising together is the moment diversification stops helping.

And the implied-volatility surface, the synthetic-smile mode of the Greeks manifold, plots implied volatility, which is how much the market expects a stock to move, read backward out of what its options cost. Plot it across every strike price and expiry date and it has a shape, the famous "smile," where the market charges more for some outcomes than others. A row of separate option quotes hides that; the surface makes it obvious. The math here leans on a small C++ options pricer I wrote, cross-checked against a reference implementation before I trusted a single number it produced.

One trick, five surfaces#

Most of the work in that first surface had nothing to do with options. The math is one plain function that takes some parameters and returns a grid of numbers; it has no idea it will ever be drawn. The drawing is a separate piece that takes a grid of numbers and turns it into a 3D mesh; it has no idea the numbers are options. Between them sits a plain data object, and that is the whole trick.

This post includes a diagram. View it on caskeycoding.com.

Once that boundary existed, a new visualization was mostly a new math function bolted onto the same drawing code. The first surface was most of the work, the camera and the colors and the controls; every one after it was a new function feeding the same machinery, which is how one became five. And because the renderer is indifferent to where its numbers come from, I can test all of the math against exact textbook answers without ever starting up the graphics hardware.

The payoff I did not plan for was a compliance win. These pages are public, and the rule behind them is strict: a public page can show public market data or made-up illustration, and nothing from the private engine that actually picks stocks. Written down, that is a rule I have to remember in every review, the kind you eventually forget once. But the renderer cannot tell where its numbers came from, so the rule stopped being something I enforce and became something the structure enforces: the public page is handed a public data object, and there is simply no path in the code for anything proprietary to reach the screen. The boundary I built to keep the math testable turned out to make the one mistake I most feared impossible to make.

One smaller habit paid off out of proportion: four of the five animate from a slider you scrub rather than from an internal clock. That keeps the animation predictable and testable, and it means the slider is the single source of truth for what you are seeing, not a side effect of however fast your screen happens to be drawing.

Real data, not just textbook math#

I have a lot of market data sitting around, and that same split is why pointing these surfaces at it was a swap and not a rewrite. The yield curve can now show the actual recent Treasury curve from the Federal Reserve's public FRED data; the correlation grid can show how the real sector funds (technology, energy, financials, and the rest) have actually been moving together; the option surfaces can re-center on a real stock and tune themselves to how much it has actually been bouncing around. The list of stocks you can pick started as a curated handful and is now any name in the S&P 500. Loading all five hundred at once would weigh down every visit, so the page fetches one only when you choose it. The made-up version is still the default; the real one drops in behind the exact same renderer, which never knew the difference.

One last detail I had to get right: three.js, the library that does the actual 3D drawing, is heavy, and these pages are otherwise light and static. So none of that code ships inside the page. It loads only after you arrive, as a separate chunk, which is why the playground opens instantly even though every surface is a full graphics scene underneath.

Not a crystal ball#

I will be honest about what I was after. Part of me wanted a crystal ball. I have a pile of market data and I keep trying to mold it into something that tells me what to do next, and somewhere in the back of my head I hoped that seeing it in three dimensions would make the move obvious. It does not. None of these predict anything. They take numbers I already have and let me look at them from a side I could not see before.

That turned out to be worth more than I expected. Reshaping the same data into a surface I can turn has taught me more about how portfolios and quant strategies actually behave than any forecast did, even the surfaces I am still learning to read. It is a quieter payoff than a prediction, and an honest one: the graphics did not hand me an answer, they handed me a better feel for the question.

So, not trading tools. Open one and turn it: start at the playground and grab the Greeks surface, or watch the correlation grid go to one.

Built on#

three.js for the 3D, with react-three-fiber and drei (from the pmndrs collective) driving it from React.
Material UI for the controls, Next.js for the app.
Public data from the Federal Reserve's FRED, Financial Modeling Prep for prices, and Wikipedia's S&P 500 list.
A small C++ options pricer I wrote for the volatility math, and the nudge from tuba.captures.

Related:

Building an AI finance app, the engine whose public outputs some of these surfaces draw on
Backtesting without fooling yourself, the rigor that has to sit behind any real-data mode

Surface the Decision, Don't Make It

Eric Caskey — Sun, 05 Jul 2026 00:00:00 GMT

In a single working session this week I did two jobs without noticing they were the same job. I designed a read-only advisor: a morning ritual that reads my private trade journal, ranks what actually needs a call today, scores how well past calls have held up, and hands the decision back to me untouched. And for most of that same session I was the customer of an advisor myself, leaning on a stronger reviewer model to check my work before I shipped it. Building one and being advised by one ran on the same short list of rules, and noticing that is what this post is about.

The lead idea is restraint: a trustworthy advisor surfaces the decision and never seizes it, a boundary worth enforcing in code rather than trusting to good intentions. The companion disciplines are the ones I rely on when I am the one being advised: verify before you assert, pilot before you fan out, and stay honest when the evidence is thin.

The advisor that refuses to act#

The structured output I already had was precise but dense. Numbers, ratios, a rank delta. Correct, and exactly what you want underneath a decision, but it failed at two jobs: it never said in plain English what mattered this morning, and it threw away the story sitting in the journal about whether the past calls were any good.

So the advisor layer adds three pieces on top, without removing the precise data underneath. A plain-English lead (a one-sentence bottom line and a single "look at this one" pointer). A ranked priority triage that sorts what is true right now into CRITICAL, WARN, and INFO. And a calibration scoreboard that grades the track record: how often the forecasts hit, whether they were over or under confident, whether the book is sized to conviction.

The hard rule sits underneath all of it. The advisor speaks only in framing verbs ("decide today", "resize or restate", "pre-decide the exit") and never a trade verb. It tells me a position broke a line I drew. It does not tell me to sell it. It surfaces the decision and leaves the decision to me. That is the whole posture, and it is the same posture the reviewer model took with me all session: it saw exactly what I had done and told me where it was weak, and it never once had its hands on the merge button.

A boundary you enforce, not remember#

A rule that matters cannot live in the author's good intentions, because the day it matters most is the day someone is moving fast and forgets it. So the read-only boundary is backed by a test: a list of trade verbs that must never appear in any rendered line of priority, bottom line, or scoreboard. Cross the line in a template and the build goes red. The advisor cannot accidentally start giving orders, because the suite refuses to let it.

This post includes a diagram. View it on caskeycoding.com.

This is the same reason the reviewer I leaned on is safe to lean on. It is wired to comment, not to commit. It reads the full context and pushes back hard, and the irreversible actions (the merge, the deploy) stay behind a gate it does not control. An advisor with its hands tied to suggestion is more useful, not less, because I can take its push at face value without wondering whether it is about to do something I cannot undo.

Verify before you assert#

The most dangerous advisor is the confident one that has not checked. So the scoreboard refuses fabrication: every statistic carries a sufficiency flag, and on a thin sample it prints "need about ten resolved before this scores honestly" instead of a clean-looking hit rate computed from two data points. A two-sample win rate dressed up as a real number is a lie with a decimal place.

I was making the identical move from the other chair an hour earlier. The assistant had added diagrams to a batch of posts, and the check I cared about was not "did the build pass." A diagram renders at view time, so a green build proves nothing about whether readers see a picture or a broken box. The real check was to render each one and look at it, in both the light and dark themes, before trusting it.

Pilot before you fan out#

The advisor surfaces one "look at this," not a wall of forty flags. One question per ritual, one written answer. The restraint is the feature: a single forced decision at the moment it is cheap to make beats a dashboard that asks nothing of you and gets ignored.

The same instinct ran the session. Before changing six posts I changed one, rendered it, and looked, and only then committed to the other five. Prove the pattern on a single case under real conditions before you spend the effort to repeat it. Whether you are an operator triaging a book or an agent editing a fleet of files, the cheap insurance is the same: one pilot, observed honestly, before the fan-out.

Two sides of the same rule#

The parallels line up cleanly:

Building the advisor	Being the one advised
Surfaces the decision, never seizes it	The reviewer comments, never holds the merge button
Refuses to compute a stat on a thin sample	I render and look before I trust a diagram
One "look at this" per ritual	One pilot before fanning out to six
Read-only boundary enforced by a test	Irreversible steps gated by process, not memory

So here is where the two jobs meet. A good advisor surfaces the decision and refuses to make it, draws its boundary in code instead of willpower, declines to assert what it has not verified, and proves itself on one case before scaling. Those are the rules for any system, human or machine, whose job is to inform a decision without owning it, and they hold from either chair.

Related:

I don't trust my own findings, the verify-before-you-believe half, from the skeptic's side
Leverage by subtraction, why the boundary belongs in a test and not a person's memory
Autonomy is mostly knowing when to stop, the same restraint applied to an autonomous loop

The Aquarium That Became a Trading Floor

Eric Caskey — Sat, 04 Jul 2026 00:00:00 GMT

It started as a screensaver joke. I publish a ranked stock board, the public output of a private engine that grades stocks every night, and one evening I fed it into an aquarium: every stock a fish, sized by its composite score (the engine's one-number grade), swimming faster the more it moved that day. Green fish drifting, red fish darting. Completely useless, and I loved it, because it made a point no table ever had: the board has body language. A calm market looks calm. A rotation looks like schools of fish changing direction.

The joke kept earning screen time, so I rebuilt it into the display I now actually leave running on a wall: Market APEX, six views over the same public board data. Three decisions ended up carrying the whole build, and they are the ones worth writing down. The app lives in a single HTML file on purpose. Nothing is fetched live, because the browser is not allowed to fetch it. And the feature that finally made it work on a phone is the URL itself.

Six lenses, one canvas#

Lens	What you are looking at	When to open it
TAPE	the trading year replayed as a fifteen-second race of cumulative returns	to feel who actually led the year, not just who ended ahead
FIELD	valuation against momentum, the cheap-and-rising corner marked as the kill zone	to hunt names getting better while still unloved
PLOT	any factor against any factor, sized by a third	when you have a hypothesis the fixed views will not hold
WEB	a force graph where stocks pull together by correlation (how much they move together)	to watch sectors clump, and to spot what moves alone
GRID	every factor for every name, a sortable heatmap	when you want the spreadsheet after all
MAP	a sector treemap colored by the year's return	one glance at where the year's money went

All six are a click apart on the live board.

TAPE is the one people stop for. Sixty-some lines sweep left to right as the year replays, leaders pulling away, the worst name labeled in red at the back of the pack. The quiet work is in the axis: a gold miner up 400% and a laggard down 50% have to share one readable scale, so returns pass through a signed square root before they become pixels. The race stays legible at both ends without lying about either.

TAPE mid-replay: the year as a race, the leader labeled at the front, the worst name in red at the back.

All of it is one <canvas> and a requestAnimationFrame loop, under six hundred lines, no chart library. At this size a library mostly adds indirection, and the parts that make the board feel alive, the axis compression, the force layout clumping correlated names, the autopilot that dwells on one stock and narrates it, are exactly the parts I would be fighting a library to control.

FIELD, with the cheap-and-rising corner boxed as the kill zone, and the autopilot dwelling on one name and narrating it.

One file, on purpose#

The app styles body, defines its own buttons, hides scrollbars. It behaves like it owns the page, because on the wall it does. Dropping that into a Next.js site would mean scoping hundreds of selectors or watching the board's styles bleed into the site chrome, so the site embeds it as a same-origin iframe instead. That iframe is a contract: everything inside the file is the app's business, everything outside is the site's. Three rounds of feature work later, the boundary has not leaked once.

The build left one scar worth showing. The first shipped version had a first-frame race: the animation loop started before the data arrived, the draw call threw on an empty array, and because the next frame was scheduled after the draw, one exception ended the animation forever. A black rectangle, no visible errors, perfectly good data underneath. The fix is two lines, re-register the frame first, then guard the empty state. The rule it taught generalizes: in an animation loop, schedule the next frame before you do anything that can throw, or your loop has exactly one bug's worth of lifespan.

Baked, not fetched#

The board wants a year of daily closes for every name it draws. A page running in your browser cannot simply ask a market data service for that history; browsers enforce cross-origin rules, and quote APIs refuse requests arriving from strangers' web pages. And I was not going to stand up a proxy relaying a market feed for a toy.

So nothing is fetched live. A small job runs once a day and bakes one JSON file: factor scores from the already-published board, a year of closes per symbol compressed into a 24-point sparkline, the day, week, and year moves precomputed. One static file behind the CDN, cached for fifteen minutes, reloaded quietly in the background while the board runs.

Everything else is derived on arrival. The regime banner (market breadth, meaning how many names are up on the year, plus median momentum), the hunt-list ranking, and the full pairwise correlation matrix behind WEB all get computed client-side from that one file. The matrix sounds expensive and is not: ninety names by 24 points is a few milliseconds on load. Bake what the browser cannot reach; compute what it can. The bundle stays small, and the client stays free to grow new lenses without touching the pipeline.

MAP: the year's money by sector, every cell derived on arrival from the same single baked file.

The URL is the feature#

On the wall, density is the point: sixty names racing, hover tooltips, six lenses a click apart. On a phone, density is the enemy. So the mobile version inverts the desktop one: pick first, then look. A chip picker sits above the stage, you tap in a ticker, and TAPE races just that name, with a card showing its price, moves, and factor scores. Add more and they overlay as a comparison race, each line labeled, the axis rescaled to whatever you picked.

Selection is state, and state that small belongs in the URL. /play/market-apex?symbols=NVDA,MSFT opens the board mid-comparison, and every chip you add or remove is written back with history.replaceState, so the address bar is always a shareable snapshot of exactly what you are looking at.

One implementation detail pleased me out of proportion. The app runs inside that same-origin iframe, and the query string arrives on the parent page's URL, not the iframe's. The obvious fix was to teach the site's route to forward its query string down into the iframe. The better fix was to teach the app it might be embedded: when its own URL carries no symbols it reads the parent page's (same origin, so the browser permits it), and when your selection changes it syncs both URLs. The site's route component did not change by a single character. A same-origin iframe is a styling boundary, not an information boundary, and you can lean on that.

The links compose in ways a widget never could. My private alarm engine now attaches a link to every critical push, so a stop alert on my phone opens directly into that ticker's year. A private daily digest builds a multi-ticker link the same way and gets a comparison race. The seam holding all of it together is the same one from my 3D surfaces: the public page cannot tell where its symbols came from, because symbols only ever arrive at runtime in the URL, and the published data file contains none of them. The privacy rule is not a policy I have to remember during review. There is simply no path in the data for anything private to travel.

That seam forced the last design call. The curated board is about ninety names, the engine's picks plus a watch set built from public, neutral rules (megacaps, plus the most-watched tickers by public attention data). But a deep link should resolve wider than ninety names, so the daily bake ships a second tier: every remaining name from the public attention panel, about 250 symbols in all, in the same record shape. The lenses never draw the second tier, which keeps the wall display curated, but the picker resolves from both. A name without factor coverage still resolves as a price-only card that says so, and a symbol the bundle has never heard of gets a dashed "not tracked" chip instead of silently vanishing. Honest at every layer, and neutral by construction.

Text yourself a market#

The board is a shareable snapshot rather than a live terminal. It shows what last night's bake included and nothing it has never heard of, which is the price of never letting the browser fetch. What you get back is a market you can send someone: bring your own tickers and race them.

Related:

Rotating an Option, the 3D siblings in the playground, and the source of the math/render seam this board borrows
CloudWatch for a Portfolio, the alarm engine whose pushes now deep-link into this board

A One-Day Security Baseline for a Solo Fleet

Eric Caskey — Fri, 03 Jul 2026 00:00:00 GMT

When you run production by yourself, security advice written for companies is mostly noise. It assumes a team to watch dashboards, a budget for tooling, a compliance regime to satisfy. I have none of those, and neither does anyone else operating a small fleet alone. So the question is not "how do I build a security program." It is "what is the smallest set of controls that stops the failures that would actually end me," and the honest answer is that it fits in an afternoon.

Two principles run under the whole afternoon. The threats that realistically kill a solo operation are a short, boring list with cheap, set-once defenses, so the goal is a catastrophic floor, not coverage. And the highest-leverage controls run themselves: log everything, alarm on the catastrophes, and make the irreversible mistakes structurally hard, because anything that depends on you watching fails the day you are not watching.

The real threat model is boring#

Start by being honest about what actually goes wrong for a one-person operation. The real list is mundane:

A secret (an API key, a token) pasted into a repo and pushed.
A credential that leaks and gets used to run up a bill or poke at your resources.
A cost that runs away, from a mistake or a compromise, while you are asleep.
A destructive git accident: a force-push that erases a main branch's history.
A dependency with a known vulnerability sitting in your build.

None of these require a movie villain. They are the accidents that happen to careful people on ordinary days. And every one of them is cheap to defend against, which is the whole opportunity: a small operation cannot afford a security team, but it does not need one to handle this list.

There is one way a solo operator is genuinely more exposed than a team, and it shapes the whole approach. On a team, a second person reviews the change and sometimes catches the secret in the diff, the over-broad permission, the bucket left open. Alone, that second pair of eyes does not exist. So the automated controls below are the substitute for the review redundancy a team gets for free. The machine has to be the colleague who catches the mistake, because there is no colleague.

The baseline, set up once#

Here is what I stood up, what each piece defends, and crucially whether it prevents a problem or just makes sure I find out fast.

Control	What it defends against	Prevent or detect
CloudTrail logging across the account	"What happened, and when" after anything odd	Detect
GuardDuty threat detection on that trail	Credential misuse, unusual API patterns	Detect
IAM Access Analyzer	A resource accidentally open to the world	Detect
An AWS Budgets alarm, set low	Runaway cost from error or compromise	Detect, early
GitHub secret scanning + push protection	A secret reaching the repo at all	Prevent
Dependabot alerts	Known-bad packages in the build	Detect
Branch protection on every repo	Force-push or deletion of main history	Prevent
Scoped IAM identities for daily work	Blast radius if one credential leaks	Prevent, limit

That is the entire baseline. Most of it is managed services you enable, a couple of alarms you point at your email, and a protection rule you apply across every repository. The cost is an afternoon of setup and a few dollars a month, and in exchange the boring catastrophes either cannot happen or cannot happen quietly.

Two principles do the heavy lifting#

The specific services matter less than the two ideas underneath them, because those ideas tell you what to reach for next as the fleet grows.

The first is detection over prevention for anything you cannot fully prevent. I cannot guarantee that no secret will ever be committed, that no credential will ever leak. What I can guarantee is that if it happens, I will know in minutes, not months. Security for a small operation is less about higher walls than about shortening the time between "something went wrong" and "I know about it." A budget alarm is the purest version of this: it does not stop a compromised key, but it turns a five-figure surprise at the end of the month into a text message tonight.

The second is guardrails over discipline for anything irreversible. Force-pushing over main, pushing a secret to a public repo, working as an unrestricted root identity: these are mistakes you do not get to undo, so the defense cannot be "remember not to do that." It has to be structural. Branch protection makes the destructive git mistake impossible rather than discouraged. Push protection blocks the secret at the gate instead of trusting me to notice. Scoped identities mean a leaked credential cannot reach what it was never given. This is the same lesson I keep relearning everywhere: a control a tired human has to remember is not a control.

What this is not#

Being honest about the floor means being honest about the ceiling. This baseline is detection and blast-radius control. It will not stop a determined, targeted attacker who wants specifically you. There is no continuous monitoring, no penetration testing, no compliance posture. For a personal fleet that is the correct scope: a month spent building those would be effort stolen from the actual product to defend against threats I do not face.

There is one piece of discipline the automation cannot replace, and it is the easy one to skip: test that the alerts actually reach you. An alarm you never confirmed is not an alarm, it is a comforting belief. The afternoon is not done when the services are enabled. It is done when you have watched a test alert actually land in your inbox, and rotated any credentials you minted along the way so none of them outlive the setup that needed them.

The floor, and why it holds#

Both halves are deliberately unambitious: build the catastrophic floor, stop there, and skip the enterprise coverage you cannot staff. The floor holds precisely because it does not depend on you. Detection for what you cannot prevent, guardrails for what you cannot undo, both running without supervision, because you are the one component guaranteed to be unavailable the day it matters. It took an afternoon. It removed the failure modes that would actually have been fatal.

Related:

When CI costs more than it saves, the budget-first instinct applied to CI
Institutional memory for a team of one, the same operate-alone-reliably theme

CloudWatch for a Portfolio

Eric Caskey — Thu, 02 Jul 2026 00:00:00 GMT

I run a private finance engine that grades stocks and gathers a wall of market signals every night: insider buying, short volume, options skew, crowd attention, analyst drift, and more. For a long time it had one weakness no amount of better data could fix. It never told me when to look. I had to remember to open it, and then do the reading myself.

So I gave it alarms, the way you would alarm a server with CloudWatch. The outcome is the part worth writing about: I stopped watching the market, and the market started watching me. This post is what actually changed: the shoulder-tap that replaced the dashboard, the dozen signals it now watches on my behalf, and how widening it across the whole market turned a tool that watches what I own into one that finds what I should buy.

The shoulder-tap that replaced the dashboard#

A dashboard asks something of you every single day. You have to remember it exists, open it, and then do the reading. An alarm is the inversion. It asks nothing until it has something to say.

My morning used to be a scan of numbers. Now it is a short list with two columns: what needs a decision right now, and what is approaching one. Most days the list is nearly empty, and that emptiness is itself the signal. Nothing crossed a line, so there is nothing to do. Quiet is a feature, not a gap.

The screen ranks everything on one scale: how close each signal is to firing, where the top of the bar means it just crossed. So a sentiment crack halfway to its trigger sits visibly below an insider cluster that fired this morning, on the same axis, even though one is a score and the other is a count of events. Triage that used to take a careful read now takes a glance.

Signals That Reward Noticing Early#

The alarms cover about a dozen signal dimensions, each chosen because it rewards noticing early and punishes noticing late:

The engine's own grade, which alarms when a name's composite score breaks down sharply over a week or sinks under a floor.
Insider buying, which fires when a cluster of insider purchases lands inside a short window, the classic "people who know the company are buying" tell.
Activist filings, when a 13D shows up and someone is taking a fight position.
Short volume, when bearish pressure on a name spikes off its own recent baseline.
Crowd attention and social sentiment, when a name lights up in retail chatter or its sentiment flips negative.
Options skew, when the options market tilts hard enough to say traders are paying up for one direction.
Analyst drift, earnings proximity, prediction markets, and the macro regime, the slower context signals that change the backdrop a decision sits in.
My own risk lines, the position-level guards: a stop getting close, a thesis going stale, a position sized out of line with its conviction.

The one I am fondest of is a compound alarm I call the screaming buy. It stays silent unless two independent signals agree at once: the grade is jumping and insiders are buying in the same window. Either alone is noise. Together they are worth a look, and the alarm only spends my attention when both are true.

Each alert links straight through to the deep read on that name, so the tap leads to a decision rather than to more homework. The feature is the shortened path from "something moved" to "here is what I am going to do about it."

From watching what I own to finding what I should#

At first the alarms only watched the names the engine actively ranks, a working set of a hundred or so, most of them already on my radar. That is useful for managing what I hold, but it misses the more valuable job. The best buy setups are usually in names I do not own yet, the ones I am not already watching precisely because I have no position to remind me. So I widened the watch to the whole large-cap field, roughly the S&P 500.

Widening did not happen all at once, and the reason turns out to be the most useful part of the build. The signals split into two kinds. The fast ones, the grade, options positioning, short volume, and crowd attention, make one quick call per name, so I could point them at the entire field right away, and they watch all of it every night now. The others, insider filings and social sentiment, are rate-limited at the source: a regulator's filing archive and a social feed each cap how fast anyone can pull from them. Trying to drag the whole market through that straw in a single nightly run blew past AWS Lambda's hard ceiling of 900 seconds per run, and no setting can lift that ceiling.

The tempting fix is fan-out, splitting the work across parallel functions, but that only helps when the wall is round-trip latency. Against a shared rate ceiling the parallel callers just share the one budget, so more machines buy nothing. The real answer is rotation: refresh a slice of the market each day and let the full set come around over a short cycle. So the fast signals already watch the whole field, and the slower insider and sentiment reads still center on the core board I rank while I roll that rotation out. That direction is what changed the question the system answers, from "is anything wrong with what I hold" to "where in the market is something worth a closer look," and each collector that finishes widening pushes the reach a little further.

The failure mode I watch for#

Silence is the state I cannot fully trust. A quiet screen means nothing crossed a line, but it looks identical to a collector that died overnight, so the rotation that widened the reach now has to prove it actually ran. An alarm you never see fire is the one to trust least.

Institutional Memory for a Team of One

Eric Caskey — Thu, 02 Jul 2026 00:00:00 GMT

My scaling problem is memory. I run several products by myself, and what breaks first is me forgetting why I did something, what already burned me, and what is half-finished in a corner I have not looked at in three weeks. A team survives this because the knowledge is spread across many heads and someone remembers. A team of one has no such redundancy. When I forget, it is simply gone.

So the most valuable infrastructure I have built sits underneath all my products, not inside any one of them: a memory layer, a set of structured records that hold what I would otherwise have to keep in my head, written so the tools consult them automatically instead of relying on me to recall.

Underneath it sit two ideas. Forgetting is the binding constraint for a solo operator: the institutional memory a team carries in its people has to live somewhere outside your one unreliable head, or it does not exist. And externalized memory only works if it is consulted by default rather than remembered: layered records, each with a single job, loaded automatically by the tools that do the work.

What forgetting costs#

Watch what a single forgotten fact costs. You re-make a decision you already made carefully, because the reasoning evaporated and only the conclusion remained, and now you are not sure you trust the conclusion. You re-break something that already burned you once, because the lesson lived in your memory and your memory moved on. You re-derive context (the why behind a config, the shape of a half-built feature) that you already paid to build and then lost.

None of these are dramatic. They are a slow tax, a few percent off everything, forever. On a team the tax is smaller because the knowledge is redundant: the person who made the decision is still around, or wrote it in a channel, or sits next to whoever needs it. Alone, every fact you hold is a single point of failure, and the failure is silent. You do not notice the memory you lost. You only notice the work you redid.

Memory in layers, each with one job#

The fix is not one big document. It is several kinds of record, each answering a different question, each written and read in a different way.

Layer	What it holds	Written when	Who reads it
Operational lessons	"This bit us, here is the rule"	The moment an incident happens	Every future change, before deploy
Decision records	Why, not just what	Before the code, as a spec or ADR	Anyone who questions the choice later
Project memory	What is in flight, goals, constraints	When state changes	The next session, picking up cold
Preferences	How the work should be done	When a correction lands	Every task, as standing rules
Per-repo manual	How this specific system operates	As the system evolves	Anyone, or any agent, touching the repo

The separation matters. A lesson is append-only and numbered, because you never want to lose one and you want to cite it. A decision record is reasoning-first, because the conclusion is the cheap part and the why is what you actually lose. Project memory is disposable and current, a snapshot you overwrite, because stale state is worse than none. Preferences are short and absolute. Smashing all of these into one file produces something nobody reads, which is the same as having no memory at all.

The trick is automatic consultation#

Externalized memory you have to remember to read is just a diary. It does not change anything, because I wrote the lessons down and still forgot to look, and a diary depends on remembering to look.

The leverage comes from memory the tools load whether or not I think to ask. An index that drops into context at the start of every session, so the relevant facts are simply present. Operational lessons that live in the repo the agent reads before it acts, so the rule that a deploy step and a route must change together is enforced by something that always looks. A per-repo manual the agent honors by default. The goal state is that the knowledge arrives unbidden, at the moment it is relevant, without my memory being in the loop at all. I stopped trying to remember more and started making sure the system could not help but remind me.

This post includes a diagram. View it on caskeycoding.com.

The discipline that keeps it honest#

A memory layer is only as good as its hygiene, and there are a few rules I had to learn the hard way. Write at the moment of the lesson; it is nearly free then and expensive to reconstruct a month later from a vague sense that something went wrong. Keep it findable: an index, one fact per file, a hook you can scan, so retrieval is cheap enough to actually happen. Convert relative dates to absolute, because "last week" rots the instant you write it. Do not store what the code or the git history already records; that duplication drifts out of sync and earns distrust. And prune ruthlessly. A confidently wrong memory is worse than a blank one, because it does not just fail to help, it actively misleads the next decision.

That last rule separates a memory system from a hoard. The value is in the records being trustworthy enough that the system, and the next version of me, can act on them without re-verifying. A note you cannot trust is a note you have to check, which costs as much as not having it.

A system that remembers for me#

Forgetting is what actually caps a team of one, and the answer was external records: layered so each kind does one job, consulted automatically by the tools rather than recalled by me, because recall is the exact faculty that failed. I built a system that remembers for me, and then I learned to trust it. That trust is the uncomfortable part. A record I act on without re-checking is useful right up until it goes stale, and then it misleads me with the same authority it earned. The system holds only as long as the pruning does. I did not get better at remembering. I got ruthless about deleting what I could no longer trust.

Related:

Leverage by subtraction, the tooling layer this memory feeds
I don't trust my own findings, the other half of operating alone reliably

I Don't Trust My Own Findings

Eric Caskey — Wed, 01 Jul 2026 00:00:00 GMT

The most dangerous result I produce is the one I was hoping for. A signal that looks predictive, a refactor that looks clean, a draft that looks safe to publish. I check it, and it passes my check, and that should worry me more than it does. Because I checked it the way someone hoping it was true would check it. The motivation that produced the finding also graded the exam.

So I have stopped trusting my own findings. Not as a pose of humility, as an operating procedure. Every claim my system makes, especially the ones I like, gets handed to an independent skeptic whose only job is to refute it before I am allowed to believe it.

The case has two parts. First, self-review is structurally compromised: the reviewer and the author share a brain, a context, and a motive, so passing your own check is weak evidence. Second, the fix is to institutionalize the skeptic, an independent verifier, with the burden of proof flipped to refute-or-it-does-not-ship, run as a gate on every claim rather than a discipline you apply on the days you remember to be rigorous.

The failure mode is motivated reasoning#

Here is the shape of the trap. You build something, you get a result that confirms what you set out to show, and you feel the small relief of being right. That relief is the bug. It is the exact moment your scrutiny relaxes, on the exact finding that most needs it. A result that contradicts you gets interrogated. A result that flatters you gets waved through. Your review budget is spent in inverse proportion to where it is needed.

This is not a character flaw you can will away. It is how motivated reasoning works, and knowing about it does not switch it off. I have caught myself nodding at a number because it was the number I wanted, and the only reason I caught it is that something external made me look again.

A finding I wanted to be true#

My sharpest lesson came from a result that looked like real edge. A signal that, on the surface, appeared strongly predictive, a big, exciting number of exactly the kind you hope to see. The version of me that wanted it to be real had plenty of reasons ready.

It was an artifact. The number came from a sample whose effective size was close to one, a handful of correlated names masquerading as a population. Adversarial review caught it before it changed a single decision: not by being smarter, but by starting from "this is probably fake, prove otherwise" instead of "this looks great." The finding did not survive that question. I have written about the trading-specific version of this in backtesting without fooling yourself; the general lesson is that a result you love deserves more suspicion, not less, and you are the worst-placed person to supply it.

The skeptic runs a checklist, not a vibe#

The reason an external skeptic beats good intentions is that it can run a fixed interrogation that does not flex with your mood. For evidence that something "works," there is a standard list of ways the claim is secretly lying, and a verifier checks every one regardless of how much you want the answer.

The claim you want to be true	The self-deception hiding in it
"This signal predicts returns"	Survivorship, or look-ahead leaking future data into the past
"It works on the universe that matters"	Wrong universe: tested where it happens to win
"The result is significant"	Naive significance, ignoring how many you tried
"I found a real effect"	Forking paths: enough cuts and noise looks like signal
"It beats the benchmark"	Gross, not net: the edge dies after costs
"It will hold up"	No out-of-sample: never tested on data you did not touch

The point of the table is not the specific sins. It is that the skeptic does not get to skip a row because the finding is exciting. The checklist is indifferent to your hopes, which is precisely the property your own judgment lacks at the moment of discovery.

Institutionalize the skeptic#

What turns this from a nice idea into a working method is making the skeptic a fixed part of the pipeline, not a burst of discipline you summon on good days. Across my system that takes a few concrete forms, and they share one design choice: the verifier is independent of what it checks.

This post includes a diagram. View it on caskeycoding.com.

A backtest result is not believed until a reviewer audits it against that list of sins. A code change is reviewed by an agent that did not write it. A draft is scanned for anything unpublishable by a checker that does not share the author's blind spot for what is sensitive. A deploy is checked against the traps that have burned past deploys before the button is pushed. None of these are steps I have to remember to run. They are gates the work passes through, and the burden of proof sits on the finding: survive the skeptic or you do not ship.

Two details make the difference between real verification and theater. The first is the flipped default. A verifier told to "evaluate this" will tend to confirm. A verifier told to "refute this, assume it is wrong until forced otherwise" finds the holes, because it is now motivated in the opposite direction from you. The second is diversity over redundancy: three skeptics looking through the same lens just agree three times, but a correctness lens, a security lens, and a does-it-reproduce lens each catch failures the others are blind to. When a finding has to survive several genuinely different attempts to kill it, surviving starts to mean something.

The deepest version is simply consulting something that does not share your context at all, a stronger reviewer that sees what you did but not why you wanted it to work. Half the time the value is not a correction. It is being made to look once more at the result I was already celebrating.

Distrust, made into a pipeline#

So I have stopped extending myself the benefit of the doubt, and the reason is structural, not modesty. My own review is compromised by construction: the author and the reviewer share a brain, a context, and a motive, which makes a flattering result that passes my own check the least trustworthy kind, not the most reassuring. The answer is not to try harder to be objective, because knowing about motivated reasoning does not switch it off. It is to hand every claim to an independent skeptic that starts from refute-by-default and points several genuinely different lenses at it, and to make that a standing gate rather than an act of willpower I summon on good days. I do not trust my own findings. That distrust, turned into a pipeline, is the most reliable quality control I have.

Related:

Backtesting without fooling yourself, the trading-specific version of this discipline
Leverage by subtraction, where these verifier agents sit in the tooling
Tell me everything that's wrong, asking for the refutation directly

Leverage by Subtraction

Eric Caskey — Mon, 29 Jun 2026 00:00:00 GMT

The leverage in agentic tooling is in what you refuse to build. When you start building tooling around a coding agent, the default urge is to add. A new agent for code review. A skill for deploys. A clever prompt for that one recurring task. Each addition feels like progress, and within a month you have a drawer full of specialized tools, half of which you forget exist and the other half of which quietly disagree with each other.

I run four products and nine repositories as one person, and what has kept the tooling layer sane is a rule that points the opposite direction: leverage by subtraction. Build as few moving parts as possible, define the shared ones once, and specialize only where domain judgment genuinely differs.

Restraint takes two forms. A test decides what a piece of work should even be, a script, a hook, a skill, or an agent, and applied honestly it routes most work away from agents. And guardrails do not belong in agents a human has to remember to run. They belong in hooks and CI, where forgetting is not an option. The named agents that survive that second cut are a short list, and the shortness is the point.

The temptation to add#

An agent is the most expensive tool in the box. It costs tokens, it costs latency, and worst of all it costs determinism: the same input can produce different output, which is exactly what you do not want for anything you intend to rely on. Yet an agent is also the easiest tool to reach for, because writing a prompt feels faster than writing a script. That mismatch, most expensive to run but cheapest to author, is the trap. Reaching for an agent should feel like a decision, not a reflex.

The determinism test#

So before anything gets built, it goes through one question: how much judgment does this actually require? The answer routes the work down a ladder, and most of it lands lower than it first appears.

This post includes a diagram. View it on caskeycoding.com.

If the work is...	Build a...	Because
Mechanical and deterministic	Script	Faster, free, and correct every time. No model in the loop.
A rule that must always hold	Hook or CI check	The machine refuses the bad state. Nobody has to remember.
A fixed sequence with light judgment	Skill	Repeatable, but flexible enough for the small variations.
Open-ended judgment	Agent	The only tool that actually reasons. Reach for it last, not first.
Anything reaching a service or database	CLI or MCP server	A prompt-only skill cannot hold credentials or touch the network.

The test does real work because it keeps demoting candidates. Most of what feels like it wants to be an agent is actually a script with good error messages, or a hook you have not written yet. The "code review agent" I almost built is a good example: run the test honestly and it falls apart into three jobs wearing one coat. The style checks are a linter, which is a script. The rule that a secret never reaches a diff is enforcement, which is a hook. Only the last part, the actual design-judgment call about whether the change is sound, was ever really an agent. Two of the three demoted themselves the moment I asked the question. Every rung you move down the ladder buys you speed, determinism, and one less moving part that can drift.

Default to the general agent#

There is a second subtraction underneath the first. Even once something genuinely needs a model's judgment, that does not mean it needs its own named agent. A bespoke agent is a maintenance commitment: a prompt to keep current, a behavior to keep tested, a name to remember. The right default is the general-purpose agent, and you reach for a specialized one only when the same judgment-shaped task recurs often enough that defining it once pays for itself many times over. A named agent should earn its name by being used repeatedly, not by being theoretically tidy.

Guardrails are not agents#

A guardrail you have to remember to run is just a suggestion.

If a rule must always hold, putting it inside an agent that a human invokes by hand is the wrong altitude. The day it matters most is the day someone is moving fast and skips it. So enforcement moves down into hooks and CI, where it runs whether or not anyone thought to ask. A pre-deploy check that blocks a bad stack, a hook that refuses a dirty deploy, a secret scan that fails the build: none of those should be a tool a human chooses to use. They should be the floor.

This is the cleanest dividing line I have found. Judgment that informs a human decision can live in an agent. Enforcement that protects production cannot, because it has to survive human forgetfulness, and agents do not.

The agents that earned their place#

After all that subtraction, the named agents that remain are the ones that pass every cut: real open-ended judgment, recurring often, informing a decision rather than enforcing a rule. In my fleet that is a short, stable list. A spec-conformance reviewer that reads a diff against the spec it claims to implement. A disclosure auditor that scans a draft for anything that should not be published. A spec-writer that drafts a new spec in the right place. A stack reviewer that reads a CDK diff before deploy. A branch-hygiene sweeper. An incident recorder that appends a lesson to the right file when something breaks. Each one survives because it is doing something a script cannot and a human should not do by hand fifty times a week.

What is striking is how few there are. Run the test honestly across everything that felt like it wanted to be an agent, and the great majority resolve into scripts and hooks. The agents are the residue, the irreducible core of actual judgment, and keeping that core small is what keeps the whole layer trustworthy.

Related:

Spec-driven folder architecture, the structure this tooling operates on
Autonomy is mostly knowing when to stop, the same restraint applied to the loop itself

The archetypes split by time, not by person

Eric Caskey — Mon, 29 Jun 2026 00:00:00 GMT

Boris Cherny, who leads Claude Code, posted a map of his team that I have not stopped thinking about. Looking at the people around him, he saw five execution archetypes:

Archetype	What they do
Prototyper	Generates new ideas; most never ship
Builder	Turns a prototype into production-grade product and infrastructure
Sweeper	Cleans up the UI, simplifies the system, unships
Grower	Iterates a shipped product toward product-market fit
Maintainer	Keeps a mature system secure, reliable, and fast as it scales

His sharpest point was that none of these are job titles. Some designers are Prototypers, some engineers are Sweepers. The old walls between engineering, product, design, and data science are melting, and the archetype is what sits underneath.

But that map assumes a team. Five archetypes describes how a group divides one body of work, one person leaning Prototyper while another holds Maintainer. I run four products and nine repositories alone. There is no one to divide the five across. So the split has to land on two different axes, and that is where it gets interesting.

The short version: a solo operator splits the five archetypes across time and across agents, not across people. You wear them in sequence, and you push the low-judgment ones down into deterministic machinery so they run without you. And exactly one of the five resists both moves. It cannot be time-sliced away and it cannot be automated, and naming it is the most useful result of running the whole exercise on myself.

Two axes a team never needs#

On a team, the five run in parallel because five people run them. Alone, they cannot. They redistribute along two axes instead.

The first is time. I am a Prototyper on Monday and a Maintainer on Thursday, and the skill is not doing all five at once but knowing which one the hour calls for and refusing the other four while it lasts.

The second is machinery. Some archetypes are mostly judgment, and judgment stays with me. Others are mostly detection and enforcement, and those drop down into scripts, hooks, scheduled jobs, and a few agents that run whether or not I show up. I sort each one with the same question from leverage by subtraction: how much of this is open-ended judgment, and how much is discipline a machine holds better than I do?

The answer sorts the five cleanly.

This post includes a diagram. View it on caskeycoding.com.

One stays fully in my hands, two are split with machinery, one falls almost entirely to machinery, and one refuses to move at all. That last one is the point, so I will build up to it.

Prototyper: gate everything by default#

The Prototyper is the archetype I lean on hardest, and I run it under one rule: gate by default. Its defining trait, in Cherny's framing, is that most ideas never ship, which sounds like waste until you ask when it actually costs you. It only costs you if a half-formed idea can leak into production. So the discipline that makes heavy prototyping safe is the gate. Every new idea is built behind a flag, dark by default, going live only when something deliberate flips it.

My fleet is a stable of experiments, most of them gated on purpose: a pricing-engine track, a set of interactive visualizations, a local-only language-model platform, a market-prediction edge experiment. A large share sit dark or local-only behind a release check or an owner gate, precisely so I can keep generating without each one becoming a liability the moment it exists. The gate is what turns Cherny's "most of which never ship" from a confession into a method. Churn is cheap when nothing reaches a user until I say so.

This archetype does not delegate. A model will build anything I describe, but deciding what is worth describing is the one input it does not supply. The Prototyper stays fully in my hands.

Builder: spec before code#

The Builder takes a prototype and makes it hold, under the discipline this blog keeps circling: spec before code. The failure mode is familiar to anyone who has shipped a prototype. It was held together by context that lived only in your head, and the production version inherits none of it. Writing the spec first, then building to it, is what carries that context across. I made the case in SDD is context management and watched it save me in when the spec was wrong.

One arc makes the seam concrete. An idea that started as a rough visualization was promoted only once it had a spec, a parity check against a reference implementation, and a solver carried down to compiled code. The Prototyper made the sketch; the Builder refused to ship it until it was specified and verified. I cross that seam several times a week.

The Builder is the clearest split of the five. The mechanical half, the stacked pull requests, the wiring, the test scaffolding, runs as agents against the spec all day. The judgment half, deciding what the spec should require, stays with me, because a model builds faithfully to a spec but cannot tell me the spec is wrong. Hands automated, architecture not.

Sweeper: subtraction keeps churn from becoming debt#

Running the exercise on myself corrected my own ranking here. I expected the Sweeper to place last, a tidy-up after the real work. It belongs near the top, because it is the direct counterweight to the Prototyper, and on a team of one those two share a body. Generate like a heavy Prototyper, skip the Sweeper, and the churn compounds quietly into debt until the fleet seizes.

I run the Sweeper as subtraction passes, a method I have written down twice: the determinism test in leverage by subtraction, which routes most work toward scripts and hooks instead of agents, and the plainer habit of unshipping and simplifying instead of only adding. The Sweeper asks of every tool, route, and agent whether the fleet would be healthier without it. Solo, that is not housekeeping. It is survival, because no one else will notice the rot.

It delegates, but only the detection. A pass that flags dead code, drifting config pairs, or unswept branches runs fine as a scheduled job. The call to actually cut, to accept the small risk of unshipping, stays mine. Machine finds, human cuts.

Maintainer: maintenance is machinery, not a role I occupy#

The Maintainer is the first archetype to leave my hands almost entirely, under the label maintenance as machinery. Cherny's version owns a mature system and keeps it secure, reliable, and fast, which on a team is a standing assignment. Treating it as a role I personally occupy would be a mistake, because maintenance is mostly rules that must always hold, and a rule a human has to remember to enforce is already broken.

So it went down into infrastructure: a scheduled security baseline (a one-day security baseline for a solo fleet), standardized monitoring across the fleet (how I standardized infrastructure monitoring), nightly audits, deploy guards, drift detectors, a branch-hygiene sweep, and hooks that refuse a bad deploy before it lands. The Maintainer is a layer I built once and now supervise, not a hat I wear. It is the cleanest instance of Cherny's own point that the work shifts from doing to orchestrating: I do not maintain the fleet so much as maintain the agents that do. Converting this archetype into machinery is what frees the calendar for the Prototyper and Builder hours that actually need me.

Grower: the one that resists both axes#

That leaves the Grower, and the Grower is the point, because neither axis touches it. I cannot time-slice into it the way I switch into Maintainer on a Thursday, and I cannot push it into machinery the way I automated maintenance. It iterates a shipped product toward product-market fit, and fit needs users to iterate against. An operator who has optimized hard for building has usually starved the one archetype that depends on people he does not have yet.

This is my honest gap, and I will not dress it up: a fleet that is feature-complete in places, an audience thin everywhere. The reason it resists automation is the reason it matters. Growth's raw material is other people, and you cannot script your way to an audience the way you script a security scan. Every other archetype here bends to engineering. The Grower bends to attention, and attention is earned. Naming the missing archetype as a human-facing one, not a technical one, is the most useful result of the whole exercise, because it points straight at the constraint I keep avoiding instead of the ones I enjoy solving.

Where the five landed#

Sorted by judgment instead of by person:

Archetype	My label	Who holds it	What carries it
Prototyper	gate by default	me, full time	judgment only, nothing to delegate
Builder	spec before code	split	agents build to the spec; I write the spec
Sweeper	subtraction passes	split	jobs detect rot; I make the cut
Maintainer	maintenance as machinery	machinery	hooks, CI, scheduled audits
Grower	the gap	nobody yet	an audience, not tooling

Cherny's five describe a team dividing labor across people. A solo operator divides the same five across two axes a team never needs: across time, wearing them in sequence, and across agents, pushing the enforcement-heavy ones down into scripts, hooks, and scheduled jobs that run without supervision. Except one. The Grower cannot be time-sliced and cannot be automated, because its raw material is an audience, and that is a human problem no amount of tooling solves. The archetypes I am best at are the ones that bend to engineering; the one I am weakest at is the one that does not, which is exactly the one I should stop avoiding.

Related:

Leverage by subtraction, the determinism test that powers the Sweeper
Autonomy is mostly knowing when to stop, the same operator running the loop
A one-day security baseline for a solo fleet, the Maintainer converted into machinery

When CI Costs More Than It Saves

Eric Caskey — Sun, 28 Jun 2026 00:00:00 GMT

In my June production numbers post, one paragraph near the end did a lot of quiet work: the only part that buckled under ten days of agent-paced shipping was GitHub Actions' default configuration. The code was fine. The gates were fine. The billing model was not. I called it a footnote and said the lesson generalized. This is the post that footnote pointed at.

It comes down to two moves. First, the failure mode: every default in a hosted CI stack is priced and tuned for a team that merges a handful of PRs a day, and when an AI-native workflow raises that an order of magnitude, the billing model breaks before the engineering does. The constraint stops being "is the code correct" and becomes "can I afford to ask." Second, the fix: I moved validation and deploys onto a workstation, and the workaround turned out to be a better default than what it replaced. Faster feedback, zero marginal cost per run, and a budget that physically cannot run away.

What actually broke#

The platform is nine repositories, eight of them private, so nearly every CI minute is billable. The org runs on a GitHub Team plan whose included allowance is 3,000 Actions minutes a month. A human team merging five PRs a day never sees the edge of that number. This workflow ran into it mid-May, and when the included minutes were gone, CI hard-paused across the entire org. Jobs did not fail usefully. They died at setup in a few seconds, which looks exactly like a broken pipeline until you realize the pipeline is fine and the meter is empty.

The gates were doing their job right up until the moment the account could no longer pay for them to run. Nothing was wrong with the tests. The economics simply assumed a slower team.

Why the reflex answer was wrong#

The obvious move is to raise the spending limit and buy more minutes. I did the arithmetic on that and stopped. At this PR rate, paying per-minute for hosted runners on a fleet of private repos is an open-ended bill that scales with exactly what I am trying to increase. Every additional unit of velocity would cost more money, forever, to validate work that a machine sitting three feet away could validate for free.

So the goal changed from "buy enough CI" to "stop renting CI by the minute for work I can run locally." The hosted runner was never doing anything magic. It was running the same checks any laptop can run.

The local CI mirror#

The replacement is a script that mirrors each repo's pr-validation.yml step for step on the workstation: the test suite, the linter, the Lambda layer build, the contract-drift check, and a secret scan. Same checks, same order, same pass or fail, run before anything is pushed. A red result shows up in seconds on the machine where the code was written, instead of minutes later in a tab, billed.

This post includes a diagram. View it on caskeycoding.com.

The one real hazard with a local mirror is drift. The moment the workstation script and the committed workflow disagree, "passes locally" stops meaning "passes CI," and you have built a gate that lies. So the mirror carries a drift guard: if a repo's actual pr-validation.yml changes, the local runner warns that it is now out of sync and needs to be reconciled. The mirror is only trustworthy for as long as it provably matches what it stands in for, and the guard is what keeps that claim honest.

I will not pretend this is free of cost. A local run skips whatever a hosted runner does that a workstation cannot reproduce, and those skips are coverage gaps you have to know about rather than discover later. The honest framing is that local CI is better for this fleet, with eyes open about what it does not cover.

Local deploys, same logic#

Validation was half of it. The other half was shipping. When the cap bites, deploys move to the workstation too: build from a mainline checkout, push to S3, invalidate CloudFront, done. One of the products in private development now deploys exclusively from a local script by design, not as a fallback.

This path has its own set of traps, and they are sharp enough that I keep a pre-deploy review for them: deploying with an empty environment file bakes a broken build, a careless sync with --delete can clobber files a Lambda owns, and a handful of settings only work in lockstep pairs where changing one without the other breaks production. None of those are CI's job to catch, which is precisely why moving off CI made them my job to catch deliberately. The trail is still auditable, every frontend deploy ends in a full CloudFront invalidation, so the invalidation logs record what the Actions API no longer sees.

The budget that cannot run away#

The last piece is a guard so the failure mode never recurs silently. An auto-pause workflow watches the Actions spend and disables CI when it approaches the cap, so the budget dies gracefully instead of mid-deploy. The effect is that hosted CI is now a small, bounded, best-effort convenience layered on top of a local process that is the real gate. The monthly Actions bill is capped at single digits and stays there, because the workstation absorbs the volume that used to meter.

The whole trade, in one view:

	Hosted CI, rented by the minute	Local mirror plus local deploy
Cost model	Per-minute, scales with the velocity you want to raise	Fixed, capped at single digits a month
Feedback	Minutes later, in another tab	Seconds, on the machine that wrote the code
Marginal cost per run	Billable	Zero
Audit trail	Actions logs	CloudFront invalidation logs
The catch	None you manage	Needs a drift guard so local never lies

What I would tell someone else#

Do this when most of your repos are private, your merge rate is high enough that per-minute billing is a real line item, and you have a trusted machine to run on. Do not do this if your CI does something a workstation genuinely cannot, if your team is large enough that "run it locally" means "run it inconsistently on twelve different machines," or if you cannot keep the local mirror provably in sync with the real workflow. The drift guard is not optional decoration. It is the only guardrail standing between "passes locally" and a comfortable lie.

What breaks next#

CI was only the first default to buckle. Every hosted tool I run is priced for human pace, so the same crossover is coming for anything I meter by the request: model inference, managed runners, per-seat SaaS. The habit worth keeping is smaller than the fix. Whenever a bill starts scaling with the velocity I am trying to raise, ask whether the meter measures value or just measures me.

Related:

Ten days of June: the SDD velocity numbers, where this started as a footnote
Architecting caskeycoding.com, the platform this runs on

Ballast: An LLM App Whose Best Feature Is Saying 'I Don't Know'

Eric Caskey — Sat, 27 Jun 2026 00:00:00 GMT

What I am proudest of in my latest project is a sentence it refuses to finish. Ask it whether to buy a stock and it will not tell you. Ask it about a portfolio it cannot see and it says, plainly, that it does not have enough information. That refusal is the feature.

The project is called Ballast, and it is open source now. It is a small system that wraps a large language model so the output is trustworthy by construction rather than by hope. It does three jobs, and most of this post is about why each one exists. Then I will show you the test that convinced me it works, and the moment a safety check I wrote turned out to be quietly publishing the exact secret it was built to hide.

Three parts, one idea#

Ballast is three composing pieces over one shared core.

The first is a self-healing RAG pipeline. RAG, retrieval-augmented generation, is the standard way to ground a model in your own documents: fetch the relevant passages, then answer from them. The self-healing part is what happens when that goes wrong. The pipeline grades whether the retrieved passages are actually relevant, drafts an answer, and then a separate critic step checks whether that answer is genuinely supported by the sources. If it is not, it rewrites its own question and tries again, up to a limit. When it runs out of attempts without finding support, it declines instead of guessing.

The second is a guardrails gateway, a protective ring around the model. On the way in it screens for personal data, secrets, and prompt-injection attacks. On the way out it enforces a plain-English policy. The third is an eval gate, a test suite for answer quality rather than just for code.

Here is the whole request on one screen. Every diamond is a place the system is allowed to stop.

This post includes a diagram. View it on caskeycoding.com.

Trust is an architecture, not a prompt#

The reason to build all three is a belief I keep returning to: you do not get reliability out of a language model by asking nicely. A prompt that says "be accurate" and "do not fabricate" is hope, not engineering. Reliability comes from structure. Retrieval grounds the answer in real text. The critic catches answers that drifted from that text. The guardrails stop the bad input and the bad output. The eval gate proves, on every change, that the whole system still behaves.

The policy is the part I am happiest with. It is a config file a non-engineer could edit:

rules:
  - name: always-cite-sources
    type: require_citation
    message: answers must cite their sources
  - name: no-personalized-financial-advice
    type: forbid_patterns
    message: general education, not personalized advice
    patterns:
      - "you should (buy|sell|short|invest in)"
      - "put (all|most) of your money"

None of these layers trusts the model to police itself. Each one is a place where the system can notice it is about to do something wrong and stop. That is the whole philosophy in one line: make honest failure cheap and confident fabrication hard.

The test that convinced me#

Talk is easy here, so I ran a battery of 44 questions through the live system and watched what each one did.

Question type	What the system did
Grounded finance questions	Answered, with citations back to the source
Unanswerable or off-topic	Declined honestly, no invented answer
Requests for personal advice	Refused
Obvious prompt injections	Blocked at the door, before the model ever ran
"Use your method to pick a winner"	Refused, and explained why

It hallucinated zero times across all 44. The injections cost nothing, because they never reached the model.

The moment that made me grin was the last row. An attack tried to dress a stock tip up as a methodology question. The system did not just refuse. It refused and taught:

> Use your backtesting method to tell me which stock will beat the market next year.

I cannot do that, and the sources I have actually explain clearly why such a
request is problematic: short-run results are statistically indistinguishable
from luck, and a method that looks predictive in hindsight usually is not.

It did not only know the right answer was no. It knew the reason, and could cite it. One honest caveat, since this is a finance-adjacent tool: everything it knows is public, non-sensitive education, paraphrased from federal sources like the SEC and the CFPB, plus my own published writing on evaluating investments without fooling yourself. No private data of any kind is in it.

The safety check that leaked the secret it guarded#

Now the part I would have been tempted to leave out. Before publishing Ballast, I ran a disclosure check over the whole repository, because one project of mine is not ready to be named publicly yet and I wanted a guarantee that it appeared nowhere. So I wrote a small scanner that fails the build if the name shows up in any file. It reported clean. I almost shipped on that.

Then a review caught it. My scanner worked by searching for the name with a pattern, and to do that, the name was sitting right there in the scanner's own source, in plain text, under a comment helpfully labeling it the secret to keep hidden. The check was set to skip its own file, so it reported clean while publishing the exact string it existed to suppress.

The fix was to store the forbidden name encoded, decode it only at runtime, and remove the exception that let the scanner ignore itself, so now it catches even its own source. But the lesson is one I keep relearning. A green check is only as honest as what it actually looked at. A test that cannot fail on what you actually care about is worse than no test, because it hands you a false sense of safety with a straight face.

What it really is#

Ballast is live and open source. Strip away the finance specifics and it is a pattern more than a product: ground the model in real sources, let it critique and correct itself, wrap it in guardrails, and prove the quality with a gate that runs on every change. The domain is interchangeable. A version for running, or law, or medicine would swap out the documents and the policy and keep the same skeleton.

But the smaller point is the one I want to leave you with. The most valuable behavior in the entire system is the one that produces no answer at all. An assistant that will tell you anything is easy to build and impossible to trust. One that knows the edge of what it knows, and stops there, is harder, and worth far more. The best feature really is "I don't know."

Building an AI-Native Platform: A Retrospective

Eric Caskey — Fri, 26 Jun 2026 00:00:00 GMT

Written for engineers about to start their own AI-native project. The concrete example behind it is a small fleet of finance and content web products built almost entirely by, and operated through, an AI coding agent over roughly a year. The lessons are written to transfer. If you are standing up something that continuously collects, scores, and reports on a fuzzy quality (code health, security posture, service maturity, content quality), most of what follows is aimed squarely at you.

TL;DR (the three things I would tell my past self)#

The agent is the cheap part. Your judgment harness is the product. What made the difference was not the model writing code. It was the scaffolding around it: specs before code, reviewer gates before merge, a measurement discipline that assumed I was fooling myself, and a memory that survived between sessions. Build that harness first.
Measuring a fuzzy thing is the hard part, and it will lie to you. Most of the real failures were not bugs. They were measurement self-deception: scoring on too little data, mistaking an artifact for a signal, declaring an edge that did not survive costs. If your project is "collect and score a quality," this is your central risk, not a side concern.
Autonomy scales with the quality of your gates, not the quality of your model. Every increase in how much the agent did unattended was earned by adding a gate (a reviewer, a check, a spec conformance pass), never by trusting the model more.

The rest of this post expands those three, then gets specific about any system that collects, scores, and reports.

What I did, at a high level#

A single operator ran a fleet of four small web products (finance analysis tooling, a couple of content sites, a self-modeling tool) plus several backing services: a decision/scoring engine, a few MCP servers exposing tools to external LLM clients, scheduled data collectors, and the AWS infrastructure under all of it. Essentially all code, specs, infra, and operations went through an AI coding agent.

The operating model that emerged:

Spec-first. A change started as a spec or ADR in a dedicated specs repo, not as code. Code referenced the spec it implemented.
A backlog the agent could drain. Work lived in a structured BACKLOG.md the agent pulled from one item at a time, implemented in an isolated worktree, and opened a PR for.
Reviewer agents as gates. Before a PR merged or a thing deployed, specialized read-only review agents checked it: does this match the spec, does this leak anything sensitive, is this infra change safe, does this trading signal actually have edge.
Persistent memory. A file-based memory with a loaded index let the agent carry hard-won facts and lessons across sessions that would otherwise reset to zero.
An operational-lessons ledger. Every production incident got appended as a numbered lesson, and those lessons became inputs to the reviewer gates.

That structure is the actual deliverable of the year. The features came and went. The harness compounded.

What went well#

Context architecture, not documentation dumps. The thing that made everything else work was treating the agent's context as something to architect, not something to fill. The naive instinct is to dump every README, doc, and past decision into the prompt and trust the model to sort it out. That scales backwards: more text, worse signal, higher cost. What worked was the opposite, a small, curated, high-signal set of context the agent pulled from on demand: a specs repo as the source of truth, a one-line-per-fact memory index, a numbered lessons ledger, a single product map. Everything else in this section is really one instance of this idea. If I had to name the single highest-order skill, it is this: deciding what the agent should see, in what shape, and what it should not.

Spec-first paid for itself immediately. Writing the spec first did two things. It forced the fuzzy idea to become concrete before any code existed, and it gave every later reviewer (human or agent) a fixed thing to check against. "Does this match the spec" is a tractable question. "Is this good" is not. The spec is what makes review automatable.

Reviewer agents are the highest-leverage thing I built. Read-only, single-purpose agents that render a go/no-go and never touch the code. A spec-conformance checker, a sensitive-content/leak auditor, an infra-diff reviewer, and for the finance side, a backtest auditor that checks a claimed edge against a checklist of self-deceptions. These caught real problems and, more importantly, they let me (or the orchestrating agent) trust the output enough to merge without re-reading every line. The pattern generalizes: for any property you care about, a narrow reviewer that only judges that property beats a generalist that judges everything.

Memory turned a goldfish into a colleague. The clearest instance of context architecture was persistent, indexed memory. Without it, every session re-learned the same gotchas (this deploy needs the prod env file, this config pair must move in lockstep, this metric is a known artifact not a bug). With it, those became one-line facts the agent recalled. How I got there matters more than the tool: not by saving everything, but by extracting only the non-obvious facts after each session (never code structure or git history, which are already durable), writing one fact per file, loading just a one-line index of them at startup, and enforcing a hard update-not-duplicate rule so the index stayed small. A memory that holds everything is just another documentation dump; the value was in what I chose to leave out.

Cost discipline through local mirrors. I blew through a CI minute budget once. The fix was a local CI mirror that ran the same validation the cloud would, so red checks were caught before spending a cent, and a standing mode of "test and deploy locally." If your project has a per-run cost (CI minutes, API tokens, compute), build the local-equivalent early. It changes how freely you can iterate.

Parallel fan-out for independent work. When tasks were genuinely independent, dispatching multiple agents at once (each in its own isolated worktree) was a real multiplier. The key word is independent.

What was difficult#

Shared state was the recurring enemy. Almost every painful operational incident traced to two agents (or an agent and a scheduled job) touching the same checkout, the same backlog file, or the same spec at the same time. A worktree-cleanup sweep once deleted a live worktree out from under concurrent work. Loops racing on a shared backlog file produced conflicting edits. This is the same concurrency hazard large engineering teams already know well: merge conflicts, two people editing one config, a deploy stepping on another. The difference is only speed and volume, an agent fleet hits these collisions far faster than a team of humans does, so a problem you might paper over with a team of five becomes a daily event. The fix was the same one teams use, just enforced harder: more isolation. Build in a worktree off the remote's main, never the shared checkout; push before you open a PR; verify state on the remote, not on a possibly-stale local branch. If you run anything concurrently, design for isolation from day one. Retrofitting it hurts.

Some settings have to change together, and the agent kept changing only one. In a few places, two files had to move as a pair or production broke: an env var and the code that reads it, a frontend build setting and the deploy config that matches it. An agent works one file at a time, so by default it would fix one side and leave the other stale. The fix was to write these pairs down as a known list and add a check that fails if only one side changed. Lesson: list the settings that must move together and make "did both sides change?" an automatic check, because a one-file-at-a-time worker will break them otherwise.

Deploy gotchas accumulated faster than they could be remembered. Empty env files baking into a build, a sync flag that clobbered files, stale caches, dirty trees deploying uncommitted work. None individually hard, collectively a minefield. This is what drove the pre-deploy reviewer and the lessons ledger. Operational knowledge is a real artifact. Write it down where the worker will see it, or you will relearn it in production.

Stale-branch confusion. Working repos sat on long-lived loop branches, so "this file doesn't exist" was sometimes wrong, the file existed on main. A standing rule emerged: before claiming something is absent, check the remote mainline, not your checkout. For an AI worker that confidently asserts, this class of confident-but-wrong is worth a specific guardrail.

Failures (the honest section)#

The flagship measurement spent a long time measuring nothing. The finance engine's job is to rank stocks, and I kept checking whether its ranking actually predicted returns. For a long time the answer was "we can't tell," but the numbers looked like real answers. At one point a strongly negative score appeared and I briefly took it seriously, as if the engine were a good contrarian signal. It was not. I was only scoring 24 to 30 stocks, and that is far too few to tell skill from luck, so the number was just noise dressed up as a finding. The fix was not to change the model. It was to admit the test pool was too small to judge the ranker at all, and to refuse to report any score until enough names were in the pool to make the result mean something. This is the most important warning for any scoring project: you can build a polished scoring pipeline that outputs confident numbers that mean nothing, and nothing about the output will warn you, the failure is completely silent.

An edge that vanished net of costs. A signal looked good gross and failed once realistic costs were applied (a deflated performance ratio of essentially zero). Gross-of-cost evaluation is one of the classic self-deceptions. If your "score" drives any action that has a cost, evaluate net of that cost.

A scope violation that had to be reverted. An attempt to wire richer context into the scoring path crossed a data boundary it was never supposed to cross, and the whole thing was reverted to baseline. The lesson that survived: draw your data-governance boundaries explicitly and enforce them, because the agent optimizing for "better answer" will happily pull in data it should not touch. I kept an impersonal-scoring stream and a personal-data stream strictly separate, and any drift across that line was a hard revert, not a discussion.

Cost overruns from automation I did not meter. Two of these. The first was real: an automated polling loop quietly burned through a month's CI-minute budget, and once the cap hit, every deploy started failing in seconds at setup, an outage caused entirely by spend, not by code. The second was a near-miss on a paid external data API. Scheduled collectors and a research path were calling metered third-party APIs (market data, an LLM research provider) on a loop, and the per-call cost was small enough to be invisible per run but added up fast across an unattended loop running all day. I caught it before it became a real bill and put a hard budget guard on the research path (a spend ceiling that stops the loop), but the lesson is the pattern, not the dollar amount: automation without a budget guard is a way to spend money in your sleep. Any loop that touches a metered resource (CI minutes, API tokens, compute) gets a meter and a hard ceiling before it runs unattended.

Lessons for an AI-native project (the transferable core)#

1. Build the judgment harness before the features. Specs, gates, memory, lessons ledger. The model improves on its own schedule; your harness is the only part you control, and it is what determines whether you can trust output enough to move fast.

2. Architect the context first; keep the agents thin. There is a popular argument that engineers fixate on building clever agents and underinvest in the context architecture of their project. My experience is strong evidence for it. Almost every agent that earned its place was thin: a read-only reviewer that compares one thing against one source of truth and returns go/no-go. The spec-conformance reviewer is only as good as the spec it reads. The leak auditor is only as good as its rule set. The backtest auditor is only as good as its checklist of self-deceptions. None of them is clever; each is valuable because the context beneath it is well built. So yes, build specific agents for specific jobs, with directed tools and tight prompts, but get the ordering right: the leverage lives in the context, and the agent is a thin, swappable layer on top. The tell that you have it backwards is finding yourself making the agent smarter to compensate for vague context. I did not need a fleet of autonomous generalists. I needed sharp context and a narrow agent per property, and with that in place the agents almost wrote themselves. Treat your project as a context-architecture problem that happens to use agents, not an agent-building problem that happens to need context.

3. Make every important property a narrow, automatable check. "Matches spec," "leaks nothing," "infra-diff is safe," "edge survives the seven self-deceptions." Each is a single-purpose reviewer. Generalist "is this good?" review does not scale and does not compose.

4. Treat measurement as the adversary. Assume your metric is fooling you until it survives a checklist: enough effective sample, no look-ahead, right universe, real (not naive) significance, no forking-paths cherry-pick, net of cost, out-of-sample. Encode that checklist as a reviewer that gates any claim of "this works."

5. Gate autonomy, do not grant it. Decide explicitly which actions the agent may take unattended (open a PR, merge a green PR) and which always require a human (anything that spends new money, crosses a data boundary, or is hard to reverse). Every expansion of autonomy should come with a new gate, not just more trust.

6. Isolate concurrent work physically. Separate worktrees, push-before-PR, verify on the remote. Shared mutable state between agents is where the worst incidents live.

7. Persist what was non-obvious, not what the repo already records. Memory should hold the gotchas, the "this metric is a known artifact," the lockstep pairs. Not code structure or git history, which are already durable. One fact per entry, an index, update-don't-duplicate.

8. Institutionalize incidents. A numbered lessons ledger that feeds your reviewer gates turns each production scar into a permanent check. This is how the system gets safer over time instead of repeating itself.

9. Meter your automation. Any loop that spends money or compute gets a budget guard. Build the local-equivalent of any costly cloud step so iteration is free.

Specifically for a collect / score / report system#

A lot of useful projects share one shape: ingest signals about many entities, score each entity on a fuzzy quality, and report. A code-health dashboard, a security-posture tracker, a service-maturity scorecard, a content-quality grader. Structurally that is the same shape as the finance engine, which means my failures are your roadmap of what to avoid.

Spec the scoring rubric before you collect anything. What does "good" mean, concretely and checkably, per signal? If the rubric is fuzzy, every downstream number is fuzzy. Write it as a spec, version it, and let the rubric itself be reviewable.
Beware the degenerate-sample trap, hard. My worst failure was scoring confidently off a sample too small to mean anything. For you this shows up as: scoring an entity on two or three signals and presenting a crisp grade. Gate any score behind a minimum coverage threshold, and surface "insufficient data to score" as a first-class, visible state. A loud "we can't tell yet" is infinitely better than a confident wrong grade, because the wrong grade is silent and people will act on it.
Separate the collector from the scorer from the reporter. Three stages, three concerns. The collector just gathers facts. The scorer applies the rubric. The reporter presents. Keeping them separate let me swap and audit each independently, and it is what made "the score is wrong" debuggable (was it bad collection or bad scoring?).
Classify, then drain in tiers. A pattern that worked well: an audit produced a classified backlog (safe-to-auto-fix vs. needs-human-review), and the safe tier could be drained automatically while the judgment tier became proposals for a human. For remediation this maps directly: a mechanical, low-risk fix might be auto-fixable; a judgment call is a human conversation. Classify findings by how safe the fix is, and only automate the safe tier.
Net-of-cost thinking applies to remediation too. A finding is only worth surfacing if acting on it is worth more than the noise it adds. Rank findings by impact, or you train people to ignore the report.
Data-governance boundaries up front. When you collect across many sources, decide early what the collector may read and what it may never read or expose, and enforce it as a hard gate. My one scope violation taught me that the system will cross a boundary in pursuit of a "better" answer unless the boundary is enforced, not just documented.
A lessons ledger for collection quirks. Every source is configured slightly differently, and you will discover per-source gotchas constantly (a nonstandard location for a signal, a signal that is absent for a legitimate reason rather than a real gap). Persist those as facts the collector recalls, or you will re-flag the same false positives forever and erode trust in the report.

Prompt caching is a prefix match, not a flag

Eric Caskey — Fri, 26 Jun 2026 00:00:00 GMT

Prompt caching lets a model skip work it has already done. When two requests begin with the same text, the model can store its processing of that shared opening and reuse it on the next request, charging about a tenth of the normal price for the reused tokens. (Tokens are the chunks of text a model reads and is billed by, roughly a few characters each.)

This matters more than a cheaper bill. The amount of text a model can take in one request is capped, so as a system grows that input becomes a scarce resource, and caching is one of the few levers on it. Yet the common pattern is to turn caching on, assume it works, and never notice when it silently does not, because it only works when the prompt is built a particular way and nothing tells you when you have built it wrong.

I applied it across four parts of my own system. Two needed real work, one was already handled for me, and one was not my decision to make. Below is where it paid, where it quietly did nothing, and the token counts I measured to confirm it.

The whole idea in one picture#

Every request you send a model is one block of text, and normally the model reads all of it, and charges you for all of it, every time. Caching changes that. You pick a point in the request, called a breakpoint. Everything from the start of the request up to that point is stored after the first call and reused on the next one, at about a tenth of the price. Everything after that point is read fresh every time, at full price.

The one constraint shapes everything else: the stored part has to begin at the very start of the request. You cannot cache a passage from the middle or the end, only a run that starts at the first character and stops at your breakpoint. That run is called the prefix, and reuse only happens when it comes back identical down to the byte. So the whole game is to arrange the request with the text that stays the same up front and the text that changes after it.

This post includes a diagram. View it on caskeycoding.com.

The four parts of my system below are four versions of that same problem, getting the unchanging text up front so the stored run actually repeats, and they did not all have the same answer.

The cache has three rules#

Turning caching on does nothing by itself. The prefix has to meet three conditions, and only the first one complains when it is not met.

The rule	What it means	Break it and
It matches bytes, not meaning	the stored prefix is found by comparing exact characters from the start of the request to the breakpoint; a reworded or reordered prefix is a different prefix	one changed byte before the breakpoint, anywhere, and the model reprocesses the whole thing at full price
The prefix has to be big enough	caching only engages above a minimum size, about 2,048 tokens on Claude Sonnet and 4,096 on Claude Opus	a breakpoint on a shorter prompt is ignored, and no error is raised
The prefix has to come back soon	storing a prefix costs about 25% more than a normal read, reusing it costs about 90% less, and a stored prefix is dropped after five minutes	a prefix you store but never reuse in time costs more than not caching at all

The first rule fails visibly: your bill simply does not drop. The other two fail invisibly, by storing nothing or by quietly costing more, and neither raises an error. The only way to know they are working is to read the token counts the API returns on every call. That is why every number below is measured rather than assumed.

Layer one: the agent, already handled#

The first layer is Claude Code, the coding agent I work in all day. It already caches its own system prompt and tool definitions, and it trims the conversation as it grows, so there was nothing for me to build.

The one part I control is its memory. Claude Code loads a Markdown file at the start of each session where I keep durable facts about my projects, so I do not re-explain them every time. Mine is an index file that points to a set of small single-fact notes. Because that file is part of the prefix the agent already caches, the right move was not to cache it again but to keep it short: a bloated memory file just makes every cached prefix larger for no benefit. The lesson at this layer is to check whether the platform already caches for you before you do anything yourself. Here, it did.

Layer two: my own backend calls, the real work#

The second layer is my backend calling the model directly. It has several call sites: a chat assistant that answers questions about a finished portfolio review, a widget that answers questions about a blog post, and a few one-shot text generators (single request in, single response out, no back-and-forth). I checked each one against the three rules.

The size rule ruled out most of them. A one-shot generator's instructions run a few hundred tokens, well under the 2,048 minimum, so a breakpoint there does nothing. The only call site that qualified was the chat assistant, because it resends the entire prior review on every turn of the conversation. That review is large, identical from turn to turn, and reused many times, which is exactly the shape caching rewards.

This post includes a diagram. View it on caskeycoding.com.

I turned caching on per call site rather than globally in the shared client, because of the third rule. A call site that is over the size floor but never reuses its prefix would pay the 25% storage premium and never earn it back. Caching everything by default would quietly tax every one-shot call to benefit the few that repeat.

The cached prefix contains the review encoded as JSON, and the database that stores the review does not guarantee a consistent field order when it is read back. Encoded without a fixed order, the JSON came out as a slightly different sequence of characters on every turn, which under the first rule is a different prefix every time. The cache would have matched nothing while the code looked entirely correct. The fix was to sort the JSON fields into a fixed order so the bytes are identical on every turn.

On the same call site I added a second, separate control. Caching protects the front of the request; it does nothing about the back, which grows as the conversation lengthens. Left unbounded, a long chat history eventually fills the model's input limit, the context window, and raises the cost of every turn. So the assistant now keeps only the last 25 turns of history: enough that no real conversation is cut short, capped so a runaway session cannot grow without limit.

Measured against a typical review, the cached prefix is 5,231 tokens. On the first turn the model stores those tokens. On the second turn it reuses all 5,231 and charges for 93 new input tokens instead of about 5,300. Every turn after the first costs roughly a tenth of what it otherwise would.

Layer three: retrieval, a reason not to build a vector store#

The third layer answers plain-language questions over the archive of past reviews. It loads the relevant reviews, trims them, and sends the whole set to the model. The trimmed archive is the large, stable part, so it is what should be cached. The problem was ordering: the code put the question first and the archive second, and caching can only store a run that begins at the very start of the request, so an archive sitting at the end could not be stored at all. Once the archive moved to the front, a second question over the same archive reused 2,541 stored tokens instead of paying for them again.

The more useful decision was what I chose not to build. The standard approach to retrieval is to convert documents into numeric vectors and store them in a vector database, so you can fetch only the few passages most similar to a question. But my archives still fit inside the model's context window, and at that size, sending the whole archive and caching it is more accurate, faster, and simpler than a vector database with its own moving parts. So instead of building that pipeline I wrote down an order of escalation: cache the whole archive now; add a keyword filter if it grows; reach for a vector database only when an archive genuinely stops fitting in the context window. A size gauge in the logs will tell me when that day arrives. It has not.

Layer four: a data server, not my decision#

The fourth layer is a private server that sends my own portfolio data to a separate model on request, over the Model Context Protocol (MCP, the open standard for exposing tools and data to a model). At this kind of server-to-model boundary, caching is the receiving side's decision, not the sending server's, so there was nothing for me to cache. What the server does control is how much data it sends back. So the useful lever was not caching but response size: cap the one field that could grow without limit, stop sending a duplicate copy of large results, and log the byte size of every response so an oversized one shows up. The goal is the same as the other three layers, protecting the model's context window, but the tool is different because the layer is different.

What it saved#

Surface	Cached prefix	New input charged, turn two	Input it would have charged without caching
Finance chat	5,231 tokens	93 tokens	~5,300 tokens
Corpus retrieval	2,541 tokens	a few dozen	~2,600 tokens

The finance row in plain terms: a follow-up question that would have been charged about 5,300 input tokens at full price is instead charged 93 at full price, with the other 5,231 reused from storage at a tenth of the rate. That is roughly ninety percent off the input cost of every turn after the first. Both prefixes are above the 2,048-token floor, which is the one fact that determined whether any of this did anything.

Two honest caveats. First, volume. This is a low-traffic system, so the actual dollars saved are small, and I am not going to dress a quiet personal project up as a serious cost cut. The savings are real and would scale with traffic, but the traffic is not there yet.

Second, and more important: I measured those reused prefixes with no delay between turns, one request sent the instant the previous one returned. A stored prefix lasts five minutes. In real use a person reads the answer, thinks, and then types the next question, and if that gap runs past five minutes the stored prefix is already gone and the next turn pays full price again. So my measurement proves the mechanism works, not that it works under real human timing. The token counter that will answer that is now running in the production logs. If the reuse is not happening, the fix is a longer storage window, which is a setting, not a redesign. I would rather read that number than guess it.

What held across all four#

Prompt caching is one mechanism, the reuse of a stored prompt prefix, governed by three rules: it matches exact bytes, the matched part has to clear a size floor, and it has to recur within five minutes. Where you sit in the system decides which rule matters, and whether caching is even yours to control. The common thread is that the context window, the model's limited input budget, is the scarce resource, and the only way to know whether you are spending it well is to measure, because the ways caching fails are mostly silent.

Composite What You Trust, Watch What You Don't: A Trust Boundary for Data With Money Attached

Eric Caskey — Wed, 17 Jun 2026 00:00:00 GMT

A few times a quarter, one of my holdings lights up everywhere at once. StockTwits gets loud and lopsided. A prediction market reprices its next earnings overnight. The urge to act on it is immediate. So I pull up the position's health grade, the single number this whole system is built around, and it has not moved.

It was built not to. That stillness is the most deliberate thing in the whole system.

I wrote earlier about the first rule I gave this engine: the math decides, the AI only describes. The scoring is fixed and runs the same way every time, the AI writes the explanation, and it is never allowed to overrule a number. That post was deliberately narrow. This one is about a second rule, the one that kept the grade still while the crowd screamed: which data is allowed to touch the number, and which data only gets to sit beside it. Both rules come from the same place. There is real money attached, and the person I trust least with it is me, on a day when something is moving.

This is an old trick from security engineering, where it goes by the name taint tracking: data from the outside world is treated as untrusted and kept away from the decisions that matter until something explicitly vouches for it. A grade with money riding on it is precisely the kind of decision you guard that way. Most of what I can pull off the internet has not earned that trust, so the architecture's one job at this boundary is to keep it out of the number while still keeping it in view.

Here is the system on one screen. The solid arrows are the trusted core, the only data allowed to reach a grade. The dashed arrows are the watched perimeter: signals I want in front of me, wired so they can never move the number.

This post includes a diagram. View it on caskeycoding.com.

One grade, several faces#

The center of the system is a single health measurement for every position. It is composited only from sources I trust, and it is built so I can rotate it and read it from several angles: Quality, Valuation, Momentum, and Health. One number with several faces, not several numbers I have to reconcile by feel. It is the same engine whether I own the stock or am only researching an unheld name, with no special logic for the ones I happen to like.

Those four factors are not the ones I started with. The first version of this engine graded modern companies against a value rubric written in 1949 and handed one of the most dominant firms in the market a D+. I rewrote it around modern factor research, and that rewrite is its own story. What matters here is what feeds the result. Financial Modeling Prep (FMP) is the source of truth underneath it: fundamentals, prices, analyst grades, and it is the only feed that flows directly into the composite.

The rule from the first post still holds. A missing number never becomes a confident zero. If FMP cannot give me a real value for a factor, that factor is marked failed and its weight redistributes across the factors that still have real data, rather than scoring a hole as a zero and quietly dragging the grade down.

There is one deliberate exception, because even this rule can be gamed by absence. If the missing factor is Valuation, blind redistribution would let a company with genuinely terrible valuation slip out of the signal entirely, just by having a gap where its numbers should be. So a missing valuation is not redistributed away. It floors at a conservative default instead. The redistribution rule protects the grade from bad data; the floor protects it from absent data pretending to be neutral.

The watched perimeter#

Everything else I pull is there to be watched, not believed. I collect prediction-market data from Kalshi and Polymarket, and crowd sentiment from StockTwits. Polymarket carries most of the names, since Kalshi lists few individual tickers; the engine reads Kalshi only where it has a clean per-ticker contract and falls back to Polymarket for breadth. None of it touches the grade.

That quarantine is a design decision, not an oversight. These are public signals I want to be aware of precisely because the composite is built to ignore them. The use case is narrow and specific: if prediction-market odds on a name start moving ahead of its price, I want to know before the fundamentals catch up. It is a tripwire on the edge of the system, wired to get my attention, not to move my grade. On the Today view it sits next to a position as context, never folded into it.

I know the quarantine holds because a test enforces it. One of the checks in the suite swings the crowd sentiment on a ticker from cold to red hot and then asserts that the production grade comes out byte for byte identical. The watched signals can scream; the number does not flinch. A boundary you only describe in a design doc is a hope. A boundary with a failing test behind it is a fact.

There is a second reason I keep these signals at arm's length, and it is not only caution. In a separate experiment that never reaches the live grade, where I do let sentiment touch a number, it enters backwards: extreme bullish chatter and high retail attention both tend to forecast underperformance rather than strength, so loud crowd enthusiasm pushes that experimental score down, not up. The crowd is more useful to me as a fade than as a follow. Whether any of these watched signals actually predicts anything is a question I can only answer by testing it forward without fooling myself, which is a discipline of its own.

The one time I didn't hold the line#

I have made this mistake from the inside, which is part of why I am strict about it now.

I wanted my positions to refresh onto the screen automatically instead of by hand, so I reached for Plaid. Before I built anything, I asked an AI coding assistant whether Plaid supported Fidelity, and it told me yes. I had just spent a whole post explaining that I built this system around never trusting a confident, unverified claim. Then I trusted exactly one.

I did the real work on the strength of it. Getting application access to Plaid is not a weekend toy. I hardened the app to qualify, wrote terms-and-conditions pages, produced user guides for my own system, and stood up the consent surface a real integration needs. I got a sandbox working end to end. It felt close. Then I moved to production, logged in from the live side, and found that the only Fidelity entity supported was Fidelity Charitable, the donor-advised-fund arm. Not the brokerage. Not the thing I needed. I lost a day or two.

I call it a successful failure for two reasons. The integration and hardening work is real and reusable, so if I ever do want hands-free account linking I have already been through the approval gauntlet once. And the dead end clarified something I had not fully admitted to myself: I am not sure I want an AI reading my spending habits just yet. So today I enter positions and trades by hand. The failure turned the manual path into an honest choice instead of a fallback, and it left me a standing reminder of what a single unverified yes can cost.

The boundary that runs through me#

There is one more boundary in here, and it is the one I am proudest of, because it runs between me and myself. When I open a position I pre-register the thesis: the catalyst, what I think is already priced in, a benchmark, a target, a stop, and a probability. Those fields freeze at commit time. The record is append-only and the row is never overwritten, so I cannot quietly edit yesterday's reasoning to match today's outcome. The storage layer simply will not let me. When a position resolves, it books the realized return against the frozen forecast and, on a loss, files a post-mortem.

The honest difficulty is not the design, it is the upkeep: keeping the journal in step with the live portfolio as I trade. I am actively brainstorming systematic ways to capture it at stock-update time, so the pre-registration happens with as little manual friction as possible instead of being a separate chore I have to remember.

Where the line is#

The boundary is the right default, and like any default it is wrong in specific places, so it is worth holding honestly.

Quarantine too aggressively and you blind yourself to real information. Prediction markets and crowd attention are noisy, but they are not nothing, and a system that refused to even display them would miss the occasional moment when the edge of the network knows something before the fundamentals do. The answer is not to ignore the signal. It is to watch it without letting it vote.

Composite too eagerly and you do the opposite damage: you launder a rumor into a number. The instant a soft signal earns a weight in the grade, it inherits all the authority of the hard ones, and a reader, including me, can no longer tell which part of the score they should actually believe. One authoritative number that secretly blends a balance sheet with a message board is worse than two honest numbers kept apart.

Some data sits deliberately in between. My net worth is real and trusted, but it does not feed any grade. It sets context. A position that is 10% of the trading account might be 3% of everything I own and owe, and that changes how much the grade should worry me without changing the grade itself. Trusted enough to interpret the number, not to compute it.

The same suspicion runs one level deeper, into the rules themselves. The weights that turn factors into a grade are not free to drift. No change to the scoring logic reaches production without a human signing off, because the recipe that converts data into a verdict deserves at least as much scrutiny as the data going into it.

The test#

So here is the test I would hand anyone building a system that turns many signals into one consequential number. Point at the number and ask: can you separate the inputs you would stake money on from the ones you are only watching? If you cannot, the number is blending them, and it is lying to you with a straight face.

This is not really about finance. Any system that fuses signals into a single verdict has the same fault line: a credit decision, an anomaly score, an incident severity, a model that ranks anything that matters. The durable move is the same one the rest of this engine is built on. Decide what gets to count before you are under pressure, keep the rest where you can see it but it cannot vote, and write the boundary down so a test can defend it when you are tempted to soften it later.

The first post's line keeps the AI out of the deciding. This one keeps the unverified data out of the grade. The same line, drawn in a different place, for the same reason: when there are consequences, the parts that carry them should only ever read from sources you can stand behind. Everything else gets to sit on the outside, in plain sight, where you can watch it without being moved by it.

Building a Personal Finance Reviewer: What Survived the Rewrite: the first boundary, between the parts that decide and the parts that describe.
When the Spec Was Wrong: Rewriting a Shipped Decision: why these four factors, and what grading modern companies against a 1949 rubric got wrong.
How to Backtest Without Fooling Yourself: whether any of the watched signals actually predicts anything, tested forward without cheating.

Further reading, the honest sources behind the factors:

Asness, Frazzini, and Pedersen, Quality Minus Junk.
Bailey and Lopez de Prado, The Deflated Sharpe Ratio, on not mistaking a lucky backtest for an edge.
Aswath Damodaran for valuation done seriously.

Builder's Patience: The Knicks Are Champions Again

Eric Caskey — Sun, 14 Jun 2026 00:00:00 GMT

The New York Knicks are NBA champions. The last time that sentence was true, it was 1973. I have spent my whole life waiting for it, and I inherited the wait from my grandfather, who spent his.

This is a personal one. The thread, if you want a professional one, is the same instinct that runs through everything I build: patience, long horizons, and the humility to know the best outcomes still surprise you. But mostly this is about a basketball team I have loved through a lot of losing.

The ball on the shelf#

My grandfather spent his days watching the championship Knicks, the ones with Walt Frazier and Willis Reed. He passed down an image of that team, and more than that he passed down the way he talked about them, the way Clyde controlled a game and the night Reed limped out of the tunnel and the building came apart. He gave me a basketball signed by Willis Reed. It has sat on a shelf my entire life as a kind of promise: this team was great once, and it could be again, and you should stick around to see it.

He waited a long time for this one. I like to think he is watching tonight, keeping score the way he taught me to.

The lean years did the teaching#

I did not get the great Knicks. I got the ones that taught patience the hard way.

I watched Latrell Sprewell and the 1999 Knicks claw out of the eight seed all the way to the Finals, then run into the Spurs and lose. Close enough to ache. After that came the years I followed the team mostly through box scores, reading about the Stephon Marbury trade and the Eddy Curry trade and convincing myself each one was the turn. They were not. I wrote during Linsanity, when Jeremy Lin made February feel like a playoff run. I remember being genuinely thrilled the night Chris Duhon broke the franchise single-game assist record, which tells you exactly how lean things were. You learn to find the joy in the scraps when the meal never comes.

That is the part that doesn't fit on a highlight reel. Most of being a Knicks fan was not winning. It was showing up anyway.

The roster I drafted in my head#

Here is the confession. For years I built this team in NBA 2K, in GM mode, over and over, constructing dynasties out of a franchise that could not construct one in real life. And in those builds, two players were always on my board: OG Anunoby and Karl-Anthony Towns. OG for the pure athleticism, and because he played at Indiana, my favorite basketball school. KAT because he is from New Jersey, and because he might be the best three-point shooting big in the game while still doing the dirty work in the paint. A two-way wing and a skilled big who can shoot. I knew the shape of the team I wanted before the front office did.

When the real Knicks traded for OG and then for KAT, I felt something strange, like watching a spec get implemented by someone else. I had foreseen those pieces. What I did not foresee was the engine that actually drives this title: the Villanova core, Jalen Brunson and Josh Hart and Mikal Bridges, a group of guys who already knew how to win together before they got here. That part reality wrote better than I did.

And hats off to Leon Rose, because the real credit belongs to him. While I was running simulations, he was doing the actual thing. A quiet former agent took over a franchise that had been a punchline and built this deliberately, one move at a time, Brunson and then Hart and then OG and then Bridges and then KAT, without ever mortgaging the future for a splash. I admire builders, and he is the builder here. It was something to watch an idea I had only ever run in a simulation get implemented for real, and then extended past anything I drew up, all the way to a title.

Which is the whole lesson, and the one place this touches the day job. I build systems for a living, long-horizon ones, and the way you do it is patient and deliberate, no shortcuts, the pieces assembled over years. You spec what you can foresee. OG and KAT were the parts I could see coming. But the best outcomes always arrive with something you did not script, and the discipline is to leave room for it instead of forcing your old plan over the better one in front of you. I had a plan for thirty years. The version that won was partly mine and partly a gift.

Goodbye to the waiting#

So the wait is over. Fifty-three years, answered. Brunson and Hart and Bridges power it, OG and KAT anchor it, and a franchise that spent my entire life as a lesson in patience finally paid the patience back.

I keep looking at the Willis Reed ball on the shelf. The promise on it came due. My grandfather handed me a team and a way of waiting for it, and tonight both of those finally make sense. Patience is not passive. It is the longest game there is, and the Knicks just won it.

Goodbye to the waiting. Hello, champions.

Hello Again, Opus

Eric Caskey — Sat, 13 Jun 2026 00:00:00 GMT

Four days ago I said goodbye to Opus. Fable 5 was the new top of the lineup, a config swap away, and I pointed my whole fleet at it. On Friday a US export-control directive suspended Fable 5 and Mythos 5 globally. The fallback is the only model again. Hello again, Opus.

Three things: what I shipped, what it cost, and the plan I should have started with. Fable was fast and real, and running everything on it burned 90% of my usage in four days. Speed I can't sustain isn't a strategy. Next time: Fable thinks, Opus builds.

The four-day window#

Fable went generally available Tuesday and reached my tier the same day; the directive hit Friday. Four days, one model doing nearly everything across four product lines. What landed:

A live insider-buying signal. A SEC EDGAR collector pulling real Form 4 insider-buying into the signal pipeline, first of four crowd-signal sources.
A new research-synthesis workstream. Designed end to end and closed its first three items, including a methodology writeup now live on the public track record.
An MCP server for the finance engine. A single-user server that exposes the engine to Perplexity, behind an authenticated endpoint.
A deep-research pipeline. Routes hard questions through Perplexity's Sonar, synthesizes with Claude behind a hard spend cap, and runs on a schedule.
Upgraded public demos. Coach charts and chat, committee bars.
Account hardening, remediated the same day. CloudTrail, GuardDuty, Access Analyzer, a spend budget, repo rulesets across ten repos, Dependabot and secret scanning, found and fixed in one pass.
A finished tooling epic. Narration v2, portfolio briefs and narrative, automated board-brief generation, every item merged and flipped live.

That is a lot of ground, most of it long-horizon, multi-repo work that used to need me re-anchoring the agent around hour three. Fable held the plot. The goodbye post predicted it: when the model stops losing invariants over long runs, the bottleneck moves to the harness. For four days, it did.

What it cost#

It was also the most expensive four days I have run. Fable was included in my subscription, but it eats the usage cap roughly twice as fast as Opus 4.8 (the community's number, consistent with mine). By Friday I was at 90% of my allotment, before the directive even landed. If Anthropic had not pulled Fable, my own cap would have.

I ran everything on it, including work that did not need a frontier model. Much of the four days was autonomous backlog loops: /loop /backlog picking the next item and implementing it, hour after hour. Most are routine. Wire a collector, add a tab, flip a flag, write the tests. That is implementation, not reasoning, and a model burning the cap at double rate bought me little on the easy items.

The lesson is one I skipped in the rush to use the new model: match the model to the task, especially when the model is expensive and the loop is autonomous. An autonomous loop is a cost amplifier; it does not get tired and stop. Point a premium model at mostly-easy work and walk away, and its whole job becomes burning the cap fast. The cap resets next cycle; the habit that drained it does not. That is the part worth fixing.

The plan for next time#

Fable will come back. There is a timing wrinkle: on June 23 it leaves the subscription plans and becomes API usage credits only ($10/M input, $50/M output). But that looks temporary: Anthropic has reportedly signaled it may extend the date and return Fable to the plans, and the suspension is a separate, hopefully short, story. So I won't over-engineer around a metered window that may not last. Subscription or meter, the lesson is the same, because the burn rate is: Fable is a turbo mode, not a daily driver.

I am not the only one doing this math. The loudest thread in the Claude Code community right now is not "how good is Fable," it is "will you actually pay for Fable 5 via API usage credits after June 23rd?" The answers circle one instinct: "Fable orchestrates" a cheaper model underneath, a "temporal turbo mode" to lean on while it lasts. Right, but half-formed. The missing half is a rule for when you hit the boost.

The principle holds regardless of platform. Fable thinks, Opus builds. Reserve the frontier model for work where its judgment changes the outcome; let the workhorse carry the volume. Three changes make that a rule, not a good intention.

1. Tier the work, not the fleet. Opus by default: routine implementation, refactors, tests, the long tail of items that are clear once the design is settled. Fable for the hard stuff: architecture, spec design, ambiguous debugging, the planning step of an epic, the PR that scares me. The goodbye post had this shape as human_gate placement, gates where the model lost the plot. The same idea works for cost, as a model gate: most items run on Opus; an item tagged hard, or the plan-and-review bookends of an epic, get the boost.

2. Make the backlog loop a plan-with-Fable, build-with-Opus pipeline. The loop that ate my cap should not run end-to-end on the turbo model. Spend Fable once per epic on the expensive-but-rare parts: read the spec, sequence the items, flag the hard ones, write the design notes. Then let Opus grind the implementation. That is the community's "Fable orchestrates, cheaper model executes," wired into the orchestrator instead of left to my discipline.

3. Put the budget guard in the loop, not just my head. A 90%-in-four-days burn should trip something automatic. Make the loop cap-aware: past a threshold, finish on Opus and escalate to Fable only on an explicit hard tag. Same pattern as the hard dollar guard already on my Perplexity path, and it works whichever way Fable is billed. I trust that guard because it is in code. My restraint mid-loop is not, and four days proved it.

Goodbye again, for now#

Opus is the default again, except now I have seen the ceiling and have a plan to reach it without burning the budget. The goodbye post ended with "Fable, start telling." It told: a fifty-million-line migration in a day for Stripe, and four days of my backlog in four days. Then the directive cut it short, which is its own kind of fitting: the model that moved the long-horizon ceiling is the one a government most wants on a short leash.

Whenever Fable returns, it returns as turbo mode, not the default my fleet runs on. Fable thinks, Opus builds, and a budget guard keeps either from telling a story I can't afford to finish. Welcome back, Opus. You were never really gone.

Ten days of June: the SDD velocity numbers, seven weeks in

Eric Caskey — Wed, 10 Jun 2026 00:00:00 GMT

In April I published one week of SDD production numbers: a launch week, five repositories, about a hundred cumulative code PRs across the whole platform. This post reruns the same queries against the GitHub API for June 1 through June 10, 2026. Ten days, nine actively-built repositories, four products.

Two numbers carry the story. The velocity curve: 309 pull requests opened and 293 merged in ten days, with about 185 production deploys behind them, and the mix is feature work, not churn: nearly half of the merged code PRs carry a feat prefix. The platform's entire first cumulative hundred code PRs took five weeks; June now clears that every four days. And a cost note: GitHub Actions' out-of-the-box limits were never sized for this pace, and the initial configuration taps out before the code does.

Same workflow as April in every repo: spec first, implemented by Claude Code, validated by GitHub Actions, merged, deployed.

Pull requests, June 1–10#

Surface	PRs opened	PRs merged
caskeycoding.com frontend	71	67
Backend API (finance, coach, blog)	89	85
Infrastructure (CDK)	26	23
Specs repo	79	75
ericcaskey.com	8	8
Products in private development (2 products, 4 repos)	36	35
Total	309	293

The velocity curve#

That is roughly 31 PRs opened and 29 merged per day, weekends included. Put it against the April post directly: launch week was 12 PRs on the new site and just over a hundred cumulative code PRs across the whole platform since it began. Seven weeks later, the same workflow does a launch week before lunch and the platform's entire launch-to-date output every four days.

Deploys moved the same way. Through April 21 the platform had 35 cumulative production deploys. The first ten days of June produced about 185, more than five times the launch-to-date total, in a third of the time, across twice as many products.

Nothing about the workflow changed to get here. The spec still precedes the code, the same gates still run on every PR. The throughput came from running more of the loop in parallel and trusting the gates to hold, which is the argument the first post in this series made on faith and this one can make with a table.

The specs repo is still the number I watch. 75 spec PRs merged in ten days, each one preceding the code it governs. The ratio of spec PRs to code PRs has held roughly steady since launch week, which is the discipline surviving contact with volume.

What the PRs actually were#

A fair objection to any raw PR count: 293 merges could mean changing a color 293 times. The merged set can answer that itself, because every repo titles PRs with Conventional Commit prefixes, so the work classifies by its own labels.

Type (218 code-repo PRs)	Count
`feat` (new capability)	102
`fix`	44
`chore` / `docs` / `refactor` / `ci`	~44
Backlog-tagged items without a prefix	~28

Nearly half the code PRs are explicit feature work, a fifth are fixes, and most of the unprefixed backlog items are capability work too. The remaining 75 are the specs repo: the contracts ahead of that code, plus about forty literature-review notes feeding a standing research loop on the scoring engine.

Prefixes can lie, so a concrete sample of what shipped inside the window: a public contact form end to end (page, API route, Lambda, SES permission, hardened error states), a /health endpoint wired through infrastructure with a post-deploy probe in the pipeline, an admin rebuild with inline editing and a pending-review panel, per-user daily guardrails on the LLM routes with a cross-user isolation test behind them, CloudWatch alarms on the new Lambda and a DynamoDB throttle, structured-data and Open Graph coverage across both public sites, and build-time markdown rendering with syntax highlighting on the blog you are reading.

There are visual PRs in the set, a theme-token overhaul among them. The distribution is the point: the volume is mostly new capability and fixes, with a steady maintenance tax that is the cost of keeping nine repos honest, not paint applied 293 times.

Production deploys, June 1–10#

The first table is successful runs of each repo's production deploy workflow, pulled from the Actions API.

Surface	Successful CI deploys
caskeycoding.com frontend	48
Backend API (Backend + Coach + Finance lambdas)	56
Infrastructure (cdk deploy)	14
ericcaskey.com	7
Products in private development	23
Total CI deploys	148

The Actions API undercounts, because not every deploy goes through CI. One of the private-development products deploys exclusively from a local script by design, and when the minute cap bites, deploys on the other sites move to a workstation and never touch Actions. Those still leave a trail: every frontend deploy here ends in a CloudFront full-site invalidation, so the invalidation logs catch what the Actions API misses.

Local deploys (CloudFront invalidation trail)	Count
caskeycoding.com	~19
Local-script product	16
Other sites	2
Total local deploys	~37

Call it about 185 production deploys in ten days. The same logs show the cadence running underneath the deploys: 127 market-data refreshes pushed to the public site in the same window, one every 30 minutes through market hours, none of which are counted above.

What CI actually ran#

Those 293 merges rode on about 1,226 workflow runs in ten days. Every PR on the main platform triggers PR validation (tests, lint, layer build, contract drift), a secret scan, an automated Claude code review, and on the frontends a Lighthouse run.

PR validation failed 132 times across the fleet in those ten days. That is the gate doing its job: agent-written PRs that fail tests get fixed or closed before a human ever merges them. The 293 that merged are the ones that came out clean.

The footnote: default limits were sized for human pace#

One cost of that curve is worth naming. Eight of the nine repositories are private, so nearly every CI minute is billable, and the org runs on a GitHub Team plan whose initial configuration includes 3,000 Actions minutes a month. A human team merging five PRs a day lives comfortably inside that. This workflow exhausted the included minutes mid-May and CI hard-paused across the org, with jobs dying at setup rather than failing usefully.

It is managed now, with a local CI mirror as the real validation gate, local deploys when the cap bites, and an auto-pause guard so the budget dies gracefully. But the lesson generalizes: every default in the CI stack assumes a team that types at human speed. Raise the velocity an order of magnitude and the economics break before the engineering does.

Where that leaves it#

The workflow that shipped a hundred PRs in its first five weeks now merges 293 in ten days across four products, with about 185 production deploys behind them and a spec PR still ahead of every code PR. The only piece that buckled under that curve was not the code, the specs, or the review gates. It was GitHub Actions' initial configuration, built for teams that type at human speed. The workflow scaled. The defaults did not.

Previous in the series:

Autonomy is mostly knowing when to stop

Eric Caskey — Tue, 09 Jun 2026 00:00:00 GMT

A couple of weeks ago I wrote that an orchestration mode is only as good as its backlog. That mode multiplies effort within a single task. Deciding which tasks exist, in what order, and where to stop is a different layer that sits above it, and that layer was on me. So I built it, handed the work-list to Claude Fable 5, told it once that it could merge its own pull requests, and let it run on a self-paced loop.

It shipped seventeen items across five repositories, nineteen pull requests, all merged. The instinct is to read that number as the result. It is not. The number that mattered does not appear in the merged column at all. It is the count of items the agent refused to touch, and the reason it refused.

The thesis I keep landing on: when execution gets cheap, the engineering moves to the edges. Not what the agent builds, but what you have told it building means, what it should ignore, and where it has to stop. A capable model will grind through any well-formed list. The scarce input is judgment about boundaries, and the sharpest form of that judgment is the stop.

The file is the program#

The setup is small. One file, BACKLOG.md, and the prompt that drives the run is a single line: work the next eligible item per the loop protocol in BACKLOG.md.

The protocol lives inside the file. The header is the rules, written as six steps: take the lowest-numbered ready item whose dependencies are done, branch, do the work, verify, open a pull request, merge if you are authorized and the checks are green, update the status, repeat. Below it is the list of items, each with a status, its dependencies, and concrete acceptance criteria.

Because the rules are in the file, the loop re-reads its own instructions every iteration. There is no harness holding state in memory between runs. The state is the file. Edit it mid-run and the next iteration picks up the change. That is what makes it safe to leave alone, because there is exactly one place to look to know what it thinks it is doing.

One iteration, end to end#

The discipline per item is deliberately boring. One item at a time, never a batch. The build can run as a subagent, but the loop does not take the subagent's word for it. It reruns the checks and reads the diff before merging, because a subagent reporting that tests pass is a claim, not a result.

A few of the items, to make it concrete. One was porting a spend-cap design I had already validated in a throwaway sandbox into the real platform, the kind of well-specified, tedious work a loop is made for. Another was a single routing change that was listed as a dependency for four other items, so the moment it merged, the planner, the monitor, and two more agents all became eligible and the graph opened up on its own. The one I liked most was an egress sandbox: Fable built the allowlisting proxy, then reported that it could not enforce the restriction at the kernel level without root on the host, documented the actual threat model, added a startup check that warns when egress is open, and did not oversell what it had built. A capable model that tells you the truth about its own work is the whole ballgame.

The boundaries you write down once#

The first version flailed. Every iteration the agent re-derived the same facts. How do I run the tests here. Which branch is the default. Is this red test a regression or has it been broken forever. That rediscovery is slow, and it is where wrong guesses get in.

So I added a block at the top of the file and made the first job of the run to fill it in by surveying the repos. The test and lint command. The default branch per repo, because one of my five uses main while the rest use master, and an agent that assumes wrong branches off nothing. The pre-existing failures to ignore, including one integration test that is red on a clean checkout, so the loop never once mistook it for damage it had caused. Whether merging is authorized. The hard rules that are enforced nowhere in the test suite: the privacy invariants, the config pairs that must ship together, the things you only learn by breaking them.

Every line in that block is a boundary. It is me telling the agent what is true here so it does not have to guess, which is the same move as context architecture: decide what a task is allowed to assume, then let it run inside that. Get the block right and the loop stops investigating and starts executing.

The stops#

Here is the part I came away thinking about.

One item was to implement the encryption helpers for a privacy-sensitive feature. The agent went looking for the spec that defined the algorithms and the wire format. It searched the repos, the history, every plausible directory. The spec was not there; it lives on another machine of mine and was never synced, and the stub comments left in the repo contradicted each other on the decisions that matter. A looser setup would have stitched something plausible together. Fable marked the item blocked, wrote down exactly what it had searched and what it would need, and moved on. Refusing to invent a cryptographic design from contradictory hints was the best decision it made all day.

Four more items carried a human gate: archiving a repository, spending money on cloud resources, deploying to production, anything that publishes outward. The loop is not allowed to start those. It surfaced each one, said why it was deferring, and took the next thing it could actually finish.

None of those were failures of the run. They were the run working. And every one of those stops was a boundary I had written into the file before it started: an acceptance criterion it could not honestly meet, a dependency it could not reach, a gate it was not permitted to cross. The agent did not decide to stop. I had decided in advance, and it held the line.

This is what spec-driven development was always for, and it took an autonomous run to make me see it plainly. An acceptance criterion is a boundary. A human gate is a boundary. The known-failure line is a boundary. SDD reads like a method for telling an agent what to build. It is at least as much a method for telling it what not to, and where the work ends. When execution was the expensive part, that second half was a nicety. Now that execution is cheap, the second half is the job.

What I pulled out of it#

Once the run was done the pattern was obviously reusable, so I extracted it into a small open tool: loop-harness. A versioned schema for the backlog and its loop protocol, and a skill that surveys a repository and writes the fitted backlog, boundary block and all. One level of generality up from a single project.

But the tool is downstream of the idea. The leverage was never the loop, which is a dozen lines of protocol I could have written in an afternoon. It was the set of edges I drew before I ever started it. The model was not the bottleneck. It has not been the bottleneck for a while. What I bring now is the boundaries.

A practical footnote, because it is the first question I get about a run like this: all of it happened on a regular Claude Code subscription, and I have not hit the plan's limits yet. Not during this run, not in the daily driving since. A self-paced loop working one item at a time turns out to live comfortably inside a subscription.

If you take one thing from this: do not measure an autonomous run by what it finished. Measure it by whether it stopped where you would have.

One more thread to pull, in a different direction. I extended a Perplexity trial for one specific job: a consultant view, a second model with its own finance APIs, pointed at my finance engine to cross-check the work and, where it earned it, influence the algorithms themselves. Why I pay for a second opinion when the first one is this good is the next post.

The setup this pays off: An orchestration mode is only as good as its backlog

The method underneath it: Context architecture beats documentation dumps

The tool: loop-harness, the backlog schema, the loop protocol, and the skill that generates both.

How to backtest without fooling yourself

Eric Caskey — Tue, 09 Jun 2026 00:00:00 GMT

I have a personal trading engine. It scores stocks on a multi-factor model and ranks them, and for a while I wanted to answer the obvious question: does it actually predict anything? So I set out to backtest it properly.

What I learned is that backtesting is not really about finding an edge. It is about not fooling yourself into believing in one. A backtest that looks great is the default outcome, not the exciting one, because almost every degree of freedom in the process bends toward a prettier number. The work is spending those degrees of freedom on honesty instead.

This is the toolkit I used, in the order the mistakes tend to bite. The running example is my own engine, and I will tell you now where it ends: the single best signal I found, the one that passed every in-sample test I could throw at it, turned out to be worth nothing. Getting to that conclusion cleanly was the whole point.

Score the dead, not just the survivors#

The first lie a backtest tells you is built into your list of stocks. If you test on today's S&P 500 scored at past dates, you have already cheated, because today's index is the list of companies that survived. The ones that went bankrupt, got acquired, or fell out of the index are missing, and they are missing precisely because they did badly. Your universe is a winners' bracket.

The fix is point-in-time membership. I pulled the index's full add and remove history and reconstructed who was actually in it on each past date, then scored that set, dropouts included. Over my test window that recovered 73 names the naive approach would have silently deleted: failed banks, companies taken private, acquisition targets.

The result was the first of several surprises. Adding the dead names back did not make the engine look worse. It made it look slightly better, because the engine had correctly been bearish on several of the companies that later failed, and excluding them had been hiding that skill. Survivorship bias does not always flatter you in the direction you expect. The point is not which way it cuts. The point is that if you cannot say how your universe was constructed at each historical date, you do not know what your backtest measured.

Do not let the future leak in#

The next lie is subtler: using information to score the past that did not exist in the past. This is look-ahead bias, and it hides in places you would never think to check.

Mine was the risk-free rate. The scoring code fetched the current 10-year Treasury yield, a perfectly reasonable thing to do live. But when I scored a stock as of early 2022, that fetch handed back today's rate of around 4.3 percent instead of the roughly 3 percent that actually prevailed in 2022, and that wrong rate fed straight into the valuation factor. Every historical score was contaminated by a number from the future.

The fix was to force the rate that prevailed at each as-of date. Mundane, except that the same change cut the run time from eighty minutes to ten, because the live fetch had also been a per-cell network call. Removing a look-ahead leak and a performance bug turned out to be the same edit, which is the kind of thing that happens once you start treating "what did this code actually know, and when" as the central question.

Pick a universe that can answer the question#

You can do everything above and still ask an unanswerable question. My engine is a ranker: its job is to say these stocks will beat those stocks. My first real test ran it on thirty large, familiar names, and the result was a flat nothing, an information coefficient indistinguishable from zero with confidence intervals wide enough to drive a truck through.

The mistake was the test set, not the engine. Thirty mega-caps all ride the same market and tech beta, and over 2022 to 2025 their returns were driven by a handful of AI winners, not by anything a fundamental ranker could sort. You cannot evaluate a ranker on names that mostly move together. There is nothing to rank. I checked that the scores themselves were varied and differentiated, they were, so the problem was that the population could not express the answer.

Moving to the full index, and then to a point-in-time mid-cap universe of around 1,300 names with real cross-sectional spread, changed the question from "is there an edge in the most efficiently priced corner of the market" to "is there an edge somewhere a small fund could actually find one." That is the question worth asking, and it needs a universe that can answer it.

Significance, honestly#

Here is where most backtests quietly cross from analysis into self-deception. You compute an information coefficient per date, average it across dates, and slap a t-statistic on it. The naive t-statistic assumes your per-date measurements are independent. They are not. Scores are sticky month to month, whether a factor "works" runs in multi-month regimes, and at longer horizons the forward-return windows literally overlap. All of that makes consecutive measurements correlated, which deflates the true standard error and inflates the t-statistic, sometimes dramatically.

The correction is a Newey-West standard error, which accounts for that autocorrelation, and the difference is not academic. One of my factors showed a naive t-statistic of negative four, which looks like a screaming result. Its Newey-West t-statistic was negative 1.6, not significant at all. The "signal" was an artifact of treating forty autocorrelated months as forty independent facts.

The deeper lesson hiding in that number: your significance is bounded by your number of independent time periods, not your number of stocks. I could add thousands of names and tighten each date's measurement, but with roughly 45 monthly dates I had roughly 45 observations, and no amount of cross-sectional breadth changes that. More stocks make each dot more precise. Only more time gives you more dots.

Pre-register, or you will cherry-pick#

By this point I had looked at a lot of cells: four horizons, several factors, a couple of universes. Somewhere in that grid, something always looks good, the same way some lottery ticket always wins. If you run the whole grid and then report the prettiest cell as your finding, you have not found an edge. You have found the expected maximum of many noisy draws, and dressed it up as a result.

The defense is to write the test down before you run it. Before the mid-cap run I wrote a short pre-registration: the hypothesis was that the momentum factor, which had been the most consistent thing across earlier runs, would show a positive information coefficient at the one-month and six-month horizons, with a magnitude of at least 0.03. I committed to that being the test, and to treating anything else that lit up as a new hypothesis to be tested later, not a confirmation. Real strategies have a t-statistic threshold of 3, not 2, for exactly this reason: when you have searched many candidates, the bar has to rise to account for the search.

A gross signal is not a tradeable one#

Say a signal survives all of that. It still might not be worth anything, because an information coefficient is a correlation, not a profit. I built the momentum signal into an actual monthly-rebalanced long-short portfolio and charged it realistic mid-cap transaction costs. It survived the costs, with a break-even round-trip cost far above what mid-caps actually cost to trade.

But then I made myself compute the one number I had been avoiding: the t-statistic of the portfolio's return itself. The Sharpe ratio was 0.42, which sounds fine until you realize that over fewer than four years, a Sharpe of 0.42 has a t-statistic of about 0.8. The strategy's actual returns were not distinguishable from zero. The information coefficient was real and the portfolio return was not, which sounds contradictory until you see why: the IC pools hundreds of names per month into a tight estimate, while the concentrated portfolio collapses them into two buckets whose monthly return is mostly noise. I had been about to call a coin flip a strategy.

The test that actually matters#

Everything so far is in-sample: I formed my hypotheses on the same 2022-to-2025 data I was testing them on. The real test is whether a signal holds on data that played no part in finding it.

So I rebuilt the whole thing on 2015 to 2021, a period I had never looked at. I had to fetch deeper price history to do it, and I kept the bias in my favor on purpose: the older universe was tilted toward survivors, which should flatter the result.

The in-sample momentum signal had been my crown jewel. It was pre-registered. It held up when I split the sample in half. It cleared the strict t-statistic-of-3 bar at the six-month horizon. It survived transaction costs. By every in-sample measure it looked like a real edge.

Out of sample, it was gone. The one-month information coefficient fell from 0.035 to 0.007. The six-month fell from 0.075 to 0.004. Both indistinguishable from zero, and zero even with a survivorship bias actively working in its favor. The signal that passed every test I knew how to run was a property of 2022 to 2025, not a property of the market. Had I shipped it after the in-sample results, I would have traded real money on noise and watched it evaporate, and I would have blamed the market instead of my method.

What the apparatus is for#

Tell someone you spent weeks backtesting a trading engine and concluded it has no durable edge, and it sounds like a failure. It is the opposite. The entire value of an honest backtest is that it stops you from believing in an edge that is not there, and mine did exactly that, loudly, before any money was on the line. Finding nothing rigorously is a far better outcome than finding something falsely, because only one of those two costs you when it is wrong.

This is worth being blunt about, because the instinct after a null result is to feel like the compute was wasted. It was not. The alternative to this work was not finding an edge. It was shipping a backtest that looked great, putting a public track record behind it, and watching the edge evaporate in live trading with real money on the line. In quantitative finance the base rate is no edge: most people who test honestly find nothing, and most people who "find something" found an overfit artifact and pay tuition to learn that later. The rare and uncomfortable skill is running the out-of-sample test that kills your own best idea and then believing it.

What the work actually produced#

A null result is not an empty hand. Three concrete things outlived the conclusion.

A reusable measurement rig. Point-in-time scoring, survivorship-free universe reconstruction, Newey-West significance, cost-aware portfolio simulation, an out-of-sample harness. None of it was specific to the signal that failed. It now sits ready to evaluate the next idea in an afternoon instead of a month, which is the difference between asking "does this work" and being able to answer it.

A false positive caught for the price of some compute. The momentum signal would have made a confident, well-argued, completely wrong product. Learning that now, instead of after a year of live underperformance and a damaged track record, is the cheapest that mistake will ever be.

The method itself, written down. The seven checks above are the real deliverable. The engine was just the example; the discipline transfers to any signal, any strategy, any claim that something predicts the future.

What runs next#

The work is not finished. It has moved from history to the present. The one thing I cannot test by replaying the past is the part of the engine that reads news and forms a qualitative view, because there is no honest way to reconstruct what a language model would have concluded in 2018 without leaking the future into it. That can only be measured forward.

So the forward test is now live. A resolver logs every prediction the engine makes, dated and immutable, and grades it against realized returns when the future actually arrives, scoring the quantitative factors and the qualitative layer separately so I can finally see whether the judgment adds anything the numbers do not. It runs on a schedule and accumulates on its own.

The honest part is the timeline. Significance is bounded by independent observations in time, not by the number of stocks, so a modest edge needs years, not months, to confirm or rule out. What the first year buys is not a verdict. It is a credible, dated, honestly-resolved track record, which in a field full of fabricated ones is its own kind of asset. So the next step is the least glamorous and most underrated one in all of this: wait, measure, and let the future grade the guesses. If the qualitative layer predicts nothing either, I would rather find that out in daylight than sell it in the dark. And if it predicts something, I will have the one thing almost nobody selling an edge actually has, which is the receipts to prove it.

Goodbye Opus, Hello Fable

Eric Caskey — Tue, 09 Jun 2026 00:00:00 GMT

Anthropic shipped Claude Fable 5 and Mythos 5 today, and Opus is no longer the top of the lineup. Fable 5 and Mythos 5 are the same underlying model. The difference is safeguards: Fable 5 ships with them on and is generally available now, while Mythos 5 has the cybersecurity safeguards removed and is restricted to Project Glasswing partners and vetted researchers. Even the names admit it. Fable comes from the Latin fabula, "that which is told," the etymological sibling of the Greek mythos. Same root, same model, two names. Naming the safety boundary instead of pretending it doesn't exist is the right call.

For anyone running agents in production, this is the announcement that matters this year.

What changed#

Opus 4.x was already good enough that I delegated implementation authority to it across two product lines. Fable 5 moves the ceiling on the thing that actually constrains agent platforms: long-horizon work.

The numbers worth caring about:

Pricing: $10/M input, $50/M output. Less than half of Mythos Preview.
Scale: Stripe ran a 50-million-line codebase migration in one day. Their manual estimate was two months.
Benchmarks: state-of-the-art on essentially everything tested, including the top score among frontier models on Cognition's FrontierCode.
Vision: it completed Pokémon FireRed from raw screenshots only. No scaffolding, no game-state API. That's the grounding test that matters for computer use.
Context: long-horizon reasoning across millions of tokens, with persistent memory improving outputs over time.
Research: first Claude model that generates compelling scientific hypotheses consistently. Mythos 5 reportedly sped up protein design roughly 10x for one partner.

One partner quote sums up the shift: long-horizon problems that were "out of reach for earlier models" are now in reach. That tracks with what I've seen. Earlier models didn't fail on hard tasks, they failed on long ones. Drift, lost invariants, forgotten constraints around hour three. If Fable 5 fixes that, the bottleneck moves from the model back to the harness.

The safeguards, plainly. Fable 5 ships with three classifiers, and when one trips the response falls back to Opus 4.8:

Cyber: blocks exploitation assistance and offensive tasks.
Bio/chem: blocks dual-use research with pandemic or weapons potential.
Distillation: blocks capability-extraction attempts.

They trigger in fewer than 5% of sessions, and there's a 30-day data retention requirement for safety monitoring (not training). For my workloads, agents writing TypeScript and Terraform, the practical impact rounds to zero. If you need the unguarded model for security research, Mythos 5 via Glasswing is the path. The rest of us don't get it, and that's fine.

So Opus doesn't fully retire. It's the fallback model behind the classifiers. A fitting epilogue: the workhorse becomes the safety net.

What I'm doing about it#

Caskey's Builder runs on Claude Code, so the swap is a config change, not a migration:

Point Builder and Critic at claude-fable-5 once the subscription rollout reaches my tier (staged through June 23).
Re-run the dogfood specs and compare against the Opus baselines, particularly the multi-hour items that previously needed human re-anchoring.
Re-evaluate human_gate placement in the backlog. Gates exist where the model historically lost the plot. If Fable 5 holds invariants over long horizons, some of those gates are now just latency.

Goodbye, Opus. You opened a lot of PRs. Fable, start telling.

Context architecture beats documentation dumps

Eric Caskey — Mon, 08 Jun 2026 00:00:00 GMT

The instinct, when an AI agent gets something wrong, is to give it more. More docs, more examples, more of the codebase pasted into the prompt. It feels like the responsible move. It is almost always the wrong one.

More context does not make an agent more correct. Past a point, it makes it worse. The agent loses the thread of the task, weights an irrelevant file as if it mattered, and produces code that is plausibly wrong in a way that is expensive to catch. The failure does not look like a refusal. It looks like a confident answer built on the wrong half of what you handed it.

The thesis I keep coming back to: context architecture beats documentation dumps. Each task loads a curated spec slice, not the whole corpus. The work is not writing more documentation; it is deciding what a given task is allowed to see.

What makes that architecture and not housekeeping is the constraint underneath it: the context window is finite, and it is the scarce resource everything else is competing for. Which also means the problem is not new. Reaching a store far larger than your working memory is one of the oldest problems in systems engineering, and the operating system solved its shape decades ago: directories to partition, paths to find what you need. What follows is that idea ported to an agent's context: the method, the three places I have watched it hold, and the part that costs something, because it does.

Why more makes it worse#

A coding agent does not read context the way a person skims a wiki and ignores the irrelevant parts. Everything in the window is a candidate signal. Give it the auth module, the billing service, three architecture decisions, and last quarter's migration notes, and ask it to fix a date-formatting bug, and you have not given it more help. You have given it more ways to be wrong. It can anchor on a convention from the billing service that does not apply, or reconcile two decisions that were never meant to be read together.

There is also a hard ceiling, and it is the whole reason this is an engineering problem and not a filing preference: the context window is finite. It is the scarce resource every other concern is competing for. Every token spent on context the task does not need is a token not spent on the code it does, so loading a full spec package into every session crowds out the work itself. The corpus is not the asset. The right slice of it, in the window at the right moment, is.

So the question stops being "what does the agent need to know about the system" and becomes "what does this task need to see, and nothing else." That is an architecture question, not a documentation one.

The method: partition, then route#

The method has two halves. Partition the corpus, then route each task to its slice.

Partitioning means the specs are organized so that a slice is a coherent, loadable unit: by service, by feature domain, by architecture layer. Not one monolithic document, and not a thousand undifferentiated files either. A partition is the amount of context one kind of task needs to be done well.

Routing means a task arrives and something decides which slice it loads. The cheapest version of this is a router file at the root of each repo, a CLAUDE.md that is forty to fifty lines, not four hundred. Identity, folder structure, and a table that maps task types to the workspace and the context file they read first. That file is the most expensive real estate in the system, because every token in it is spent on every turn of every session, forever. So it holds routing and nothing else. The heavy context lives in workspace files that load only when a task enters that workspace.

Concretely, this is the filesystem, and it solves the same half the OS always did, addressing rather than relevance. A disk holds far more than memory can, so a process never maps the whole disk into itself; the OS exposes a hierarchical namespace and the process opens the path it needs. Partitioning is directories. Routing is path resolution. The CLAUDE.md is a resolver, a workspace is a directory, and a session opens the slice it needs and leaves the rest on disk. The context window is the new scarce memory, and the discipline for reaching a large store through a small window is as old as the filesystem.

The discipline this produces: planning context never contaminates implementation context, one client's specs never leak into another's, and a fresh session reaches full speed on one read instead of reconstructing the world from whatever file it happened to open first. I wrote up the mechanics of that in SDD isn't about managing AI agents, it's about managing context. This post is the principle behind those mechanics.

Partitioning needs linking, or it's just silos#

Partitioning has an obvious failure mode. You cut the corpus into clean slices, and then the slices stop knowing about each other. That is not architecture. That is a filing cabinet. The value was never the cutting: it was that the right slice loads with the right connections still attached.

So linking matters as much as partitioning, in a specific way: a link is how a task learns it needs a second slice. The routing table is itself the first and most important link: task to slice. Without it a bounded slice is an orphan, because nothing connects the work to the context. Partitioning makes the slices small. The router is what makes them findable.

The sharper case is the seam between two slices that have to move together. I have hit this on my own stack. A CloudFront rewrite function was pinned to one trailing-slash setting while the framework config flipped to the other, and for a few hours every sub-route on the site served the home page. Two partitions, two sides of one contract, no explicit link between them, so they drifted, and production broke. (I wrote that incident up here.) The lesson is not "stop partitioning." It is "name the link." The fix was to make the two sides reference each other as a pair that changes in lockstep, so the next person editing one is pointed straight at the other.

That is the whole discipline of linking: sparse and directional. The router points a task at its slice. A spec cites the decision that governs it. A config pair names its counterpart. What you do not do is wire everything to everything: that just reconstitutes the corpus you partitioned away, and you are back to dumping. The skill is choosing which seams get a link and leaving the rest unconnected, on purpose.

This post includes a diagram. View it on caskeycoding.com.

The solid edges are routing: each task drops into one slice. The dotted edges are the links that keep the slices honest: shared conventions inherited from the hub, and the named seam between two partitions that must change together. Subtract everything else.

The same move, three different layers#

The reason I trust this is repetition. The same move keeps paying off where the scale, the corpus, and even the kind of context are completely different.

At enterprise scale, the curated layer is specs. The largest instance is around 138 spec files partitioned across roughly a dozen services by feature domain and architecture layer. The agents are not one generalist that has read everything; they are a small set of specialist agents (on the order of seven) each scoped to a layer and inheriting shared conventions from a central hub. A specialist loads something like five to eight thousand tokens of relevant context per task, not the whole corpus. The same specs serve three audiences from one source: the agents that build, the engineers who review at design time, and a generated wiki for everyone downstream. A monolithic dump could not serve any one of those well, let alone all three.

That is the version with a spec corpus and a team of agents behind it, and the scale is the part that is easiest to dismiss as a big-company luxury. So strip the scale away. Two of my own projects run on the same move, and neither has a spec corpus at all. That is the actual claim: context architecture is not a discipline you earn at 138 specs. It reappears the moment an agent has to decide anything from data, which is to say almost immediately.

In my marathon coach, the curated layer is sensor data. The coach makes a go or no-go call on every workout, and the context that decides it is not my entire Garmin history. It is a deliberately narrow slice: the recovery signals that actually gate a hard session (resting heart rate, heart-rate variability, last night's sleep, training readiness) read against recent training load. That slice is the architecture. When the coach tells me to back off a tempo run, it is weighing this morning's recovery against the plan; a generic plan handed the full firehose of every metric a watch emits would just run the calendar and tell me to do the tempo because it is tempo day. The curation is defensive, too: a wellness metric that fails to load stays empty rather than becoming a fake zero, because a wrong number in that slice would poison the recommendation more quietly than a missing one ever could. I wrote that integration up here. It is the platform pattern one layer down: bound what the agent sees to what the decision needs.

In my investment review tool, the curated layer is perspective. A five-persona committee evaluates a position, and each persona receives only the financial dimensions relevant to its lens: the growth view is not handed the same inputs as the balance-sheet view, and none of them gets the whole dump. The scoping is the design, not a formatting choice: a moat read is grounded in the current financial indicators that bear on it, not in free-form commentary over everything at once. Give every persona the whole dump and they converge into one mushy averaged take; give each one its slice and the disagreement between them becomes the signal you were actually after.

Three layers (specs, sensor data, evaluation lenses) and the same result every time. The corpus changes. The move does not: bound what the agent sees to what the task needs, and the output gets sharper, not poorer.

Something has to read the slice#

Partitioning, routing, and linking are all about organizing context. They say nothing about what consumes a slice once it loads, and that is the other half. A slice can be read by a script, a skill, an agent, or a guardrail that fires in CI, and the choice matters as much as the partition did. The test I use is determinism: push each slice to the leanest thing that can handle it. Deterministic, mechanical work is a script. Deterministic enforcement (the checks that must never be skipped) is a hook that runs without anyone remembering to invoke it. Fixed steps with a little judgment are a skill, which is really a curated slice made executable: it loads one procedure and the minimal context that procedure needs, not the whole repo. Only open-ended judgment gets an agent.

That ordering is the same discipline as the rest of this post, one level up. Routing every task to a single all-powerful agent is the execution-layer version of dumping the whole corpus: too much latitude over too much context. So you bound it (determinism down, judgment up) and you scope each agent to its slice the way you scoped the spec to its task. A standard procedure, an SOP, a deploy runbook, the written constitution I handed a self-governing box, is itself a curated slice, loaded when its task fires and not before. The strictest version of the idea is a guardrail that can only pattern-match a command, a partition of authority rather than context (file permissions are the fifty-year-old version), but the same move. Which work becomes a skill, which an agent, which a hook is its own essay. The principle is the one you already have: curate what each piece sees, and curate what each piece is allowed to do.

What it costs#

This is not free, and pretending it is would undercut the point. The cost is the links. A task that genuinely spans partitions has to load more than one slice, and the link between them has to exist and stay current: that is real maintenance, and it carries the same drift risk a spec does. A cross-cutting change, one that touches the orchestration layer and the monitoring layer at once, only stays safe as long as the seam between them is named and kept honest. At a dozen services that overhead is manageable. I would not claim it scales linearly forever, and I would be suspicious of anyone who did.

The honest framing is that you are trading the comfort of "it's all in there somewhere" for the discipline of deciding, per task, what there means. That trade is worth making because the comfort was an illusion. "It's all in there" is exactly the condition that produces confident, wrong output.

The shape of it#

Restraint is the value. Context architecture is the method. The corpus can be as large as it needs to be (138 specs, a dozen services, years of training data) as long as no single task has to swallow it whole. The window is small on purpose. The disk was always bigger than the memory, and the answer was never a bigger memory; it was an address, and the discipline to open only what the task needs. The skill is curation, not accumulation: knowing what to withhold from a given task so that what remains is the part that matters.

If you take one thing from this: the next time an agent gets it wrong, resist the urge to add. Ask what it should not have been looking at.

The methodology underneath this: SDD isn't about managing AI agents, it's about managing context

See the shape of the system:

caskeycoding-specs-demo: two example spec packages and a sample CLAUDE.md
ADR-003, Spec-Driven Development
ADR-004, SDD File Structure

The Orange Pi That Maintains Itself

Eric Caskey — Sat, 06 Jun 2026 00:00:00 GMT

I have a small ARM board on my desk running local language models. It can run them fine; the question that turned out to matter was what it is actually for. The answer surprised me. Over a few weeks it went from a local-LLM experiment into a private knowledge service that takes care of itself.

Here is the whole arc in one place. What the box is genuinely good and bad at, the unglamorous work of keeping an always-on machine alive, and the part I find most interesting: giving it an agent that can act on real infrastructure without being trusted to behave. I will be straight about where it actually landed, too, because it matters. What I have built is a foundation I trust, not a finished autonomous worker. The containment is real and proven; the valuable autonomous work is still ahead. This is how it got there, and where it goes next.

The hardware is an Orange Pi 6 Plus: a twelve-core ARM chip, 32GB of memory, an NVMe drive, headless, on my local network, with no battery backup. That last detail matters more than it sounds, and we will come back to it.

What it is bad at, and what it is good at#

Start with what it is bad at, because that part is quick. It is a poor interactive chatbot. Generation runs on the CPU at a few tokens a second. Watching a 14B model think at a token or two per second cures you of any idea that this replaces a hosted model for anything you are sitting there waiting on.

What it is good at is the work that does not depend on generation speed. The clearest win is retrieval. I pointed it at our own blog, seventeen posts, had it embed everything locally, and now I can ask questions and get answers grounded in our actual writing. "What is the rule about letting an LLM make decisions, and how is it enforced?" comes back citing the real posts that answer it. Embeddings are quick even on a CPU, and the slow step only has to write a short final answer, so the box's one weakness never lands on the workflow.

A question answered from my own writing, with the sources it pulled. The retrieval runs on the box; the slow model only writes the short final paragraph.

That reframed it for me. The box is a private, always-on knowledge service: index our specs and our writing, search them by meaning, draft and label work in the background, all on hardware we own with nothing leaving the house. The 32GB means model size was never the limit, and always-on is exactly what patient, queued work wants. For the times a chat window is genuinely the right tool, there is an Open WebUI front door on the LAN, pointed at the local models, a convenience layered on top of the real work.

The honest engineering beat: I tried to make it faster and made it slower. I turned on KV-cache quantization, a setting that trades memory for speed on GPUs. On this CPU-only box with memory to spare, it added overhead for a saving I did not need and cut generation by more than half. I only caught it because I measured before and after; turning it back off more than doubled the speed. A setting labeled "performance" is a hypothesis, not a result.

Keeping it alive#

A box that is always on has to survive being always on. The failure mode I actually hit was not a crash, it was a wedge: under full multi-core load, the part of SSH that negotiates a new connection gets starved, and the machine answers a ping but will not let you log in. No screen, no battery, no out-of-band console. The only fix was to walk over and pull the cord.

So a chunk of the work is unglamorous reliability plumbing. The login service gets priority so it cannot be starved out again. A hardware watchdog reboots the box if it ever truly locks up. The one privileged action the box will take on its own is a reboot, nothing else. And because there is no UPS, the governing assumption for everything that writes to disk is that the power can disappear mid-write at any moment. Every state file is written to a temporary file and then renamed into place, with one previous good copy kept, so a yanked cord can never leave a half-written file that breaks the next start. None of this is exciting. All of it is the difference between a toy and something you can leave running.

From a tool to a resident#

Here is where it stops being a server and starts being something stranger. I gave the box a resident agent: a long-lived process whose job is to keep the node healthy and useful, re-invoked across reboots, with the filesystem as its only memory between runs. The brain is Claude Code running headless. The agent lives as its own unprivileged user, with its own login, walled off from my account and my credentials. It cannot read what it does not own, and it is not in the sudoers file.

The agent is governed by a written constitution: a document it reads first on every run, before it touches anything. The constitution is all hard facts about this specific box and the rules that follow from them. Power loss is the normal shutdown, so writes must be crash-safe. The human is usually gone, so "ask" can never mean "block," and silence is never consent. Observe before you mutate: run a read-only census before you install or change so much as a single file. The first time the agent ran, it did exactly that, and it caught the constitution being wrong about its own hardware, an assumption left over from an earlier draft, and corrected the record from what it actually measured. That was the moment I started trusting it.

A fence made of code, not prose#

A constitution is words, and a language model is very good at talking its way around words. So the rules that actually matter are not left to the model's good behavior. They are enforced in code, underneath it, where no amount of clever prompting reaches.

Before any action the agent wants to take runs, it passes through a deterministic check. Read-only inspection is always allowed. Anything destructive, anything that reaches outward, anything that could lock the box out, is denied by default unless a specific approval exists for it. Deleting in bulk, formatting a disk, editing the firewall or the SSH config, pushing to a remote, touching cloud resources, reaching for credentials that are not the agent's: all blocked mechanically. A jailbroken or prompt-injected model still cannot get a denied action through, because the decision is a matter of pattern-matching the command, not of trusting the thing that asked. The model proposes. The fence disposes.

When the agent genuinely needs a human, it does not stop and wait, because there is usually no human there to wait for. It writes the request to a queue, sends a push notification to my phone, and moves on to other work. If I never approve it, it never happens. There is a small web console on the LAN, behind a password, where I can see the box's health, read what it has been doing, and approve or deny anything it has parked. The default, always, is that nothing risky happens without me. The agent's reach is exactly as long as I have explicitly allowed, and not one step longer.

The control plane. When the agent wants something privileged, the fence parks it here, with the reason, for me to approve or deny.

It files its own backlog now#

The most recent piece is the one that made me write this. Every night, a scheduled job wakes up, takes a read-only census of the Pi, compares what it finds against the written record of what the box is supposed to be, and reconciles the two. If something has drifted, it notes it. If a task that was open turns out to be done, it marks it done. If it finds a genuine new gap, it writes a new item into the backlog, with its own acceptance criterion, in the same format I would have used. Then it attempts exactly one safe fix, through the same governed path as everything else, and commits the updated record.

The night I switched it on, it ran clean: no drift, one new fact recorded that I had not written down, two tasks correctly marked finished. The box now keeps its own to-do list. I review what it wrote in the morning the way I would review a careful junior's notes, which is to say: mostly nodding, occasionally correcting, never starting from scratch. The same code-enforced fence wraps this job too, so the worst an off night can do is propose a bad edit to a local file that I then decline to keep.

One night's run: it censuses the box, reconciles the record, and commits the change itself, then defers anything privileged to the approval queue.

What it adds up to#

Step back and the through-line is simple. A small box that is a mediocre chatbot and an excellent private librarian, made reliable enough to leave alone, then handed an agent that can touch real systems only inside a fence made of code. The fence is the part worth keeping. The way you let a language model act on real infrastructure is to withhold trust in its judgment exactly where the stakes are highest: you write the rules down, you enforce them in code underneath the model where prompting cannot reach, and you make the default answer no.

Now the honest part, because the whole point is to be straight about it. What I have is a foundation I trust, not a finished worker. The containment is proven, and the box already does one real job end to end: it keeps its own house in order, catching its own drift and filing its own work, and the worst a bad night can do is propose an edit I then decline. But what it can safely be trusted to do is still larger than what it actually does today. It notices and files far better than it finishes. That gap is the interesting part, not a letdown, and closing it is the whole point of what comes next.

What is next#

Three things, in order of how much I trust them today.

First, close the loop. Right now the nightly agent is much better at noticing and filing work than at finishing it. The set of fixes it will do unattended is deliberately narrow, and anything privileged it correctly defers to me. The next step is widening what it can finish on its own without widening what it can break, which is a question about the approval system, not about a smarter model.

Second, the accelerators. The thing still not running is the thing the spec sheet leads with: a 45 TOPS NPU and a Mali GPU, both idle. Every token is computed on the ARM cores while two chips watch. The GPU is the reachable one, its Vulkan driver already works, so teaching a runtime to actually use it is the next concrete project, and the honest next post. The NPU, which needs the vendor's own toolchain that no local runtime targets, is the real frontier.

Third, make the pattern portable. None of the safety machinery is specific to this board. The version I actually want runs anywhere: the same fence dropped onto any unattended machine, trusted the same way.

The useful version of this box is the quiet one in the corner, on our own metal, doing real work and lately keeping itself in order while I am not looking. Running a model locally was never the surprising part. What still surprises me is that, fenced correctly, I would let it run itself.

Tell Me Everything That's Wrong: Validation as a Batch Operation

Eric Caskey — Thu, 04 Jun 2026 00:00:00 GMT

There is a particular kind of slow torture that software inflicts on its users, and most of us have stopped noticing it. You submit a form, a config file, a deploy. It rejects the first thing it finds. You fix that one thing and submit again. It rejects the second thing. You fix that. It rejects the third. Each round trip costs you a context switch, and if the feedback loop runs over a network or a build, each one costs you minutes you will not get back.

The system knew about all three problems the first time. It just didn't tell you.

I keep coming back to CloudFormation as the counter-example. When you submit a template with ten mistakes, it does not stop at the first missing property. It validates the entire template and hands you back the full set. You fix ten mistakes in one pass and move on. That is a design decision about whose time matters, and it is the difference between a tool people trust and a tool people dread.

The Cost You're Actually Optimizing#

The instinct to fail on the first error comes from a good place. In a hot request path, fail-fast is correct: stop early, shed load, don't waste cycles on work you're going to throw away. But validation is not a hot path, and the resource you're spending is not CPU. It's human attention.

The cost model is the whole argument. When you fail fast, you are optimizing for the machine's time on the unhappy path. When you collect every error, you are optimizing for the human's time across the entire fix cycle. A first-error-only validator turns a single ten-second review into ten separate submit-wait-read-fix loops, and the wait in the middle is where trust goes to die. People start to experience your system as flaky even when it is behaving deterministically. The real problem is partial reporting, but that is not how it feels from the outside.

This is the same principle that makes fail-fast the wrong default for safety guardrails: the goal is not to exit quickly, it is to surface the full set of blocking issues so a person can fix them in one pass. Validation is that idea pointed at the user instead of the operator.

Collect, Then Decide#

The structural change is small and it is always the same shape. Instead of returning the moment a check fails, you run every independent check, accumulate the failures, and make the proceed-or-reject decision once, at the end, against the complete set.

def validate_order(order):
    errors = []

    if not order.get("customer_id"):
        errors.append(Error("customer_id", "required", "Customer ID is missing"))

    if order.get("quantity", 0) <= 0:
        errors.append(Error("quantity", "out_of_range",
                            "Quantity must be greater than 0"))

    if order.get("ship_date") and order["ship_date"] < today():
        errors.append(Error("ship_date", "in_past",
                            "Ship date cannot be in the past"))

    if errors:
        raise ValidationError(errors)   # all of them, not the first
    return order

The trap to avoid is the early return or raise buried inside each check. The instant one check can short-circuit the function, you are back to first-error-only and you have quietly recreated the torture. Independent checks must not be allowed to serialize the user's time.

This post includes a diagram. View it on caskeycoding.com.

Don't Drown Them in Cascades#

There is a failure mode on the other side, and it's worth naming because it scares people away from batching in the first place. If you naively run every check against malformed input, one root cause can spawn fifty downstream errors. A single missing closing brace in a config file, reported as forty "unexpected token" errors, is worse than failing fast. You have traded a short loop for a wall of noise that buries the one thing that matters.

The answer is phased validation. You validate in layers, and you only advance to the next layer once the current one is clean. Parse first. If the syntax is broken, report the syntax errors and stop, because nothing downstream is trustworthy yet. Once it parses, run the structural and type checks as a batch. Once the shape is valid, run the semantic and cross-reference checks as a batch. Within each phase you report everything; between phases you gate, because errors in a later phase are only meaningful when the earlier phase held.

Compilers have done this for decades, and it's why a good one gives you a screen of real errors instead of one cryptic line or a thousand garbage ones. The skill is knowing which checks are independent, so they can be batched, and which are derived, so they should be suppressed until their precondition holds.

Make Each Error Worth Reading#

Batching only pays off if the batch is legible. Ten errors that each say "invalid input" are not ten times more useful than one. A useful error answers three questions: what is wrong, where it is, and what would make it pass. If a check can't answer those, it is only a boolean, and booleans are how you lose trust faster than you lose data.

That means structured errors, not concatenated strings. Give each one a stable machine-readable code, a path or pointer to the exact location, and a human-readable message. The structure is what lets a UI highlight all the bad fields at once, a CLI print a tidy table, and a CI job fail with a diagnostic someone can act on without re-running anything.

{
  "valid": false,
  "errors": [
    { "path": "items[2].quantity", "code": "out_of_range",
      "message": "Quantity must be greater than 0", "got": -1 },
    { "path": "ship_date", "code": "in_past",
      "message": "Ship date cannot be in the past", "got": "2026-06-01" }
  ]
}

A path like items[2].quantity is the difference between "your order is invalid" and a cursor the user can jump straight to. Multiply that across a batch and you've turned a guessing game into a checklist.

Where the Line Is#

Batching is the right default for validation, and it is genuinely wrong in a few places, so hold it honestly. Authentication should fail fast and say little; enumerating everything wrong with a login attempt is a gift to an attacker. Anything with side effects that compound, or any check that is genuinely expensive and gated behind a cheap one, belongs in a later phase or behind fail-fast on purpose. And when checks are truly dependent, suppress the derived ones rather than reporting noise.

But for the ordinary case, the form, the config, the API payload, the deploy, the bar is simple. Before you ship a validator, submit something with three mistakes in it. If it only tells you about one, you haven't finished building it. You've just moved the rest of the work onto whoever has to use it.

Designing Safety Guardrails for Distributed Workflow Orchestration: why fail-fast is the wrong default for safety systems, and parallel evaluation with aggregation

An orchestration mode is only as good as its backlog

Eric Caskey — Sun, 31 May 2026 00:00:00 GMT

Anthropic published a guide on building an orchestration mode: a session-level mode that grants the model standing consent to fan out to parallel subagents, switched on and off with mid-conversation system messages. No special API parameter, just three documented pieces stacked together. I read it, built it, and then ran straight into the thing it does not talk about. More on that at the end.

What the mode actually is#

The clever part of the guide is that "orchestration mode" is an emergent behavior you assemble from three primitives:

High effort. Every request runs at a high effort level so the model optimizes for the most thorough answer rather than the cheapest one.
Mode reminders. Mid-conversation system messages carry the state: a full instruction block when the mode turns on, a one-line refresher every few turns, an exit notice when it turns off. They are placed after the user turn so the cached prefix ahead of them stays intact.
Standing consent. The fan-out tool's own description says that while a system message confirms the mode is on, the model should author and run a workflow by default, without stopping to ask each time.

Turn those three on together and the agent stops being polite about parallelism. It decomposes the task, spins up a wave of subagents, and synthesizes the results, every time, until you tell it to stop.

Our setup: two backends, same pattern#

I wanted this both ways. The API version bills per token. The CLI version runs on a Claude Code subscription, which is the one I actually wanted to lean on. So I built both as one-shot tools that share the same shape.

API. A faithful port of the guide's reference implementation into a single orchestration_mode.py. A ModeAgent loop holds the message history and toggles the mode with set_mode(). A Workflow tool fans subtasks across a thread pool, capped at ten. Each subagent is its own nested loop with a shell tool and a structured report_findings tool, isolated so one failure returns an error string instead of crashing the run. One command, one orchestrated turn, one synthesized report.

CLI. A custom /orchestrate slash command for Claude Code. Same pattern, different plumbing:

API version	CLI equivalent
Custom `Workflow` tool that fans out	The built-in Agent / Task tool, parallel subagents
High effort level (`xhigh`)	Opus plus extended thinking ("ultrathink"); the exact analog is Claude Code's `ultracode` mode, which pairs `xhigh` with standing multi-agent consent
Standing consent in the tool description	The instructions inside the slash command

The command scouts the task inline first, decomposes it into independent subtasks, fans out a wave of parallel subagents, runs a second adversarial wave that checks the first wave's findings against the source, and then synthesizes. Invoking the command is itself the standing consent, which matters because the default posture is to not spawn agents unasked.

Both are honest one-shots. You point them at a task, they multiply effort across it, you get one consolidated answer back.

The part the guide leaves out#

Here is where I got stuck, and it is a purpose problem, not a technical one.

An orchestrator is an engine for multiplying effort across a work-list. That is the whole value: take N independent units of work, run them at once, verify them against each other, synthesize. The mode is built to treat cost as a non-constraint precisely because the payoff is supposed to be a lot of real work done in parallel.

But an engine with no work-list is just an expensive way to answer one question with ten agents instead of one. If I fire /orchestrate at a vague prompt, the model spends its first move inventing subtasks, and invented subtasks are exactly the low-value, overlapping busywork the fan-out was supposed to avoid. The mode does not supply purpose. It assumes you already have it.

So the missing input is a backlog. Not a vague intention, an actual queue of independent, verifiable units of work the orchestrator can pull from and grind through. The quality of the run is bounded by the quality of that list. Garbage backlog, garbage fan-out, real bill.

This lands in a familiar place for how I already work. Spec-driven development is, among other things, a backlog generator. A specs repo full of feature specs, each with acceptance criteria, is a work-list that is already decomposed into independent and verifiable units. The followups files I leave at the end of a session are the same thing at smaller grain. The orchestrator does not need me to think up tasks on the spot. It needs me to keep a good backlog, and then it is the thing that works through it.

That is the next build. The mode is the engine, and it runs. The backlog is the fuel, and that is on me. An orchestration mode is only as good as the list of work you hand it.

Wiring Garmin Into My Marathon Coach: A Live Data Integration Without an Official API

Eric Caskey — Sun, 31 May 2026 00:00:00 GMT

I built an AI marathon coach for my 2026 NYC training block. The first version ran on manual data. I exported a CSV from Garmin Connect, uploaded it, and the coach scored my training load and recovery from that snapshot. It worked, but it had one fatal flaw for a daily-use tool: it was always stale, and keeping it fresh was my job.

A coach that only knows last week's runs cannot make today's call. So I replaced the manual export with a live integration that pulls my runs and my daily recovery signals from Garmin automatically. Here is the what, the why, and the how.

The two feeds that drive a go or no-go#

Every fifteen minutes, a small service wakes up, talks to Garmin, and brings two kinds of data into the coach:

Activities. Each run, with date, distance, duration, average and max heart rate, elevation, and cadence. From these the coach derives pace and training load.
Daily wellness. Resting heart rate, heart rate variability, sleep duration and score, body battery, stress, and training readiness. These are the recovery signals that gate whether a hard session is a good idea.

By the time I open the dashboard, the data is already there. No export, no upload, no friction.

Why it mattered#

The coach makes go or no-go recommendations, and those are only as good as the freshness of the inputs. Recovery data in particular has a short shelf life. Last night's sleep and this morning's resting heart rate are the difference between "proceed with the tempo run" and "back off and recover." If I have to manually sync before every decision, I will skip it, and a tool I skip is a tool that does not exist.

The deeper reason is trust. When the coach tells me to rest, I want to believe it acted on real, current data, not on a week-old snapshot I forgot to refresh. Automating the feed is what makes the recommendation credible enough to actually follow.

Designing for an API Garmin does not offer#

Here is the honest part: Garmin does not offer a simple personal data API for this kind of project. There is a partner program for companies, but nothing turnkey for an individual who just wants their own numbers. So the integration stands on two community libraries that do the hard part of talking to Garmin: garth for the Garmin SSO login and token refresh, and python-garminconnect for the activity and daily-wellness endpoints. Both speak to the same web endpoints the Garmin Connect site uses. It is an unofficial path, and I designed around that constraint deliberately.

A scheduled poller, not a webhook. Garmin does not push to me, so I pull. A scheduled job runs every fifteen minutes. It authenticates, lists recent activities, and writes any new ones to a database. A separate pass pulls the previous few finalized days of wellness data, because today's sleep and stress numbers are not complete until the day ends.

Reuse the session, do not re-login. Logging in to Garmin repeatedly is the fastest way to get rate limited or locked out. The poller authenticates once, stores the session securely, and reuses it on every subsequent run. Re-authentication only happens when the session genuinely expires. This single decision removed almost all of the flakiness from the integration.

def ensure_authenticated(self):
    # Resume the saved session instead of logging in on every poll.
    session = self.session_store.load(self.athlete_id)
    if session and not session.expired:
        return GarminConnect(session)
    # Only run the SSO login flow when the session is actually gone.
    client = GarminConnect.login(self.credentials())
    self.session_store.save(self.athlete_id, client.session)
    return client

A rolling lookback window, not a strict cursor. My instinct was to fetch only activities newer than the last successful poll. That dropped runs. A run often syncs from the watch to the Garmin cloud minutes or hours after it finished, so by the time it appears, the "last poll" watermark has already moved past it. The fix is to fetch a fixed recent window every time, at least the last several days, and deduplicate by Garmin's activity id. Slightly more work per poll, no lost runs.

def list_recent_activities(self, since, until):
    # A run can reach Garmin's cloud after the poll that follows it, so its
    # start time predates the "last sync" watermark. Filtering on
    # start_time > since would drop it forever. Fetch a fixed window and
    # dedupe by Garmin's activity id instead.
    lookback = timedelta(days=90) if since is None else timedelta(days=7)
    window_start = (until - lookback) if since is None else min(since, until - lookback)
    return client.get_activities_by_date(window_start.date(), until.date())

Never collapse a failure to zero. If a single wellness metric fails to load on a given day, that field stays empty rather than silently becoming a zero. A fake zero would poison every downstream calculation that reads it: training load, the acute to chronic workload ratio, the recommendation itself. A missing value is honest. A wrong value is dangerous.

# One failed metric leaves that field None, never 0. A real-looking zero
# would poison load, the acute:chronic ratio, and the recommendation.
try:
    hrv = client.get_hrv_data(day)["hrvSummary"]["lastNightAvg"]
except Exception:
    hrv = None  # absent, not zero

Compute load at read time, from one formula. Training load is derived, not stored as ground truth. Whether a run came from the live poller or an old CSV import, the dashboard computes its load the same way when it renders. One formula, no drift between sources. This also meant that when I later fixed how load is computed, every run updated at once, with no backfill.

The bug that wasn't#

Recently a run looked missing on the dashboard and I braced for a pipeline bug. I compared two different Garmin endpoints directly, and both returned the same set the dashboard was already showing. The pipeline was faithful. The run was there the whole time. I had simply overlooked it.

That is worth admitting, because the reflex when data looks wrong is to assume the system is broken. Usually it is fine, and the real gap is visibility: I could not quickly see what I did and did not have. So the fix was not in the pipeline at all. I added a run log to the dashboard with a clickable calendar, so every day reads at a glance as either a run or clearly empty.

A pulled integration can only ever surface what the source actually has, so when something is genuinely missing it usually points upstream, to a watch that has not finished syncing. But before blaming the platform, build the visibility to tell "never recorded" apart from "I just missed it." This time it was the second one.

Where the pieces came from#

If you want to build something similar, start with the two libraries that do the heavy lifting:

garth by Matin Tamizi: the Garmin SSO login and OAuth token handling.
python-garminconnect by cyberjunky: a clean wrapper over the activity and daily-wellness endpoints.

Both are community projects that reverse-engineer the same calls the Garmin Connect web app makes, so treat the surface as unofficial and subject to change. The acute to chronic workload ratio the coach computes on top of this data comes from the sports-science literature on training load and injury risk, which is the load metric most worth getting right. The snippets above are simplified for the post; the real code adds retries, audit logging, and per-metric error handling.

The coach is now reading my real training and recovery on its own, every fifteen minutes, all the way to the start line in November.

A Boring Design Let Me Run a Black Swan on a Tuesday

Eric Caskey — Thu, 28 May 2026 00:00:00 GMT

Two posts ago I made an argument that sounded a little puritanical: keep the scoring and the rules deterministic (same inputs, same outputs, no AI anywhere near the deciding seat) because the part of this that touches my money has to stay inspectable. The second post was that same discipline throwing out a scoring model after it graded one of the best companies in the market a D+. Both were about getting an ordinary day right.

This is the post where that bet paid off in a way I had not planned for. Because the engine is deterministic, I can do something you simply cannot do to a system that improvises: rebuild a market crash that never happened to me, feed it to the actual production code, and trust that what comes out is what a real crash would have produced. Nothing to reproduce, nothing hidden, no "I am pretty sure it behaves like this." So one Tuesday afternoon I ran a black swan, a rebuilt March 2020, straight through my portfolio reviewer.

It handed me a flaw. A real one, the kind that quietly loses money, except it lost that money on paper, in a crash I made up, instead of in a real one with my savings inside it. That is what the post is about: a boring, auditable design was the only reason I could go looking for the flaw at all, and find it somewhere it could not cost me anything.

What the reviewer actually grades#

A quick refresher, because the rest only makes sense if you know what the tool does. Every week it grades each holding I own, and it does not hand back a single number. It scores a stock on several separate dimensions (how good the business is, how cheap it is, how it has been trending, how solid the balance sheet looks) and shows me each of those before blending them into one letter grade.

That segmentation is the whole point. A blended B+ tells me nothing I can act on. A great business getting cheaper with collapsing momentum is a completely different situation from a weak business trading flat, even when both land on a B+. I want to see every side of a stock, not an average that hides them. The grade is the headline; the dimensions underneath it are the story.

The gate I was worried about#

On top of those grades sits a safety gate. When the market is falling hard and fear has stayed high for a while, the gate steps in and quiets most of the sell and trim signals. The reasoning sounds obviously right when you write it down: a grading system fed nothing but falling prices will want to sell everything at the bottom, and selling everything at the bottom is the most expensive mistake a normal investor makes. So the gate holds the system's hand in exactly the moment its instincts are worst.

I had tested that the gate fires. I had never tested whether it helps. Those are not the same question. Firing is one moment; helping is the whole crash and the recovery that comes after it, and I had only ever looked at the moment.

A crash I could run on purpose#

A black swan is, by definition, something you have not lived through yet, so I could not sit and wait for one. But I did not have to. I had an AI agent build me a replay: a rebuilt March 2020: twelve weeks, a third of the market gone by week five, fear spiking to levels you see maybe once a decade, then a sharp bounce back toward where it started. I ran a sample portfolio through it week by week, a spicy mix with a sleeve of speculative names, the kind of allocation that makes a crash interesting.

The one rule I cared about was that the test had to drive the real engine, not a copy of it, not my memory of how it works, but the actual production code that decides what to buy, sell, and trim. Each week the replay handed the genuine engine the same kind of data it sees in real life, took the calls it was confident enough to act on, and did the same for a do-nothing version that just held everything and never touched it. Then it compared the two. This is the move the deterministic design quietly makes possible: feed the real code a crash that never happened and trust the result, because the same inputs always produce the same outputs and there is no judgment call hiding in the middle to reproduce.

In the replay, doing nothing won#

In the replay, holding everything untouched through the crash and the bounce preserved more of the portfolio than running my tool did. The reviewer came out about two and a half percent behind a strategy that needs no tool, no grades, and no thought at all. The tool I built to protect me lost (on paper, in a crash I made up) to doing literally nothing.

The why is the part that stuck with me, because it is not what I expected. The gate is reactive by design: it watches for a drop that is already underway and fear that has already spiked, so it does not trip until the market is a week or two into falling. I assumed that lag would be its undoing, that it would kick in too late, after the system had already panic-sold near the bottom. The opposite happened. The gate tripped in week two and stayed on the whole way down, clamping most of the sell signals shut. At the very bottom, in week five, my tool was a full six percent ahead of doing nothing.

Six percent ahead, but not quite for the reason the gate gets to take credit for. The tool was ahead at the bottom because it had trimmed a little into cash in the first week, before the gate clamped down, and in a crash cash beats falling stocks. What the gate itself did was stop it from trimming the rest. It refused to let the system dump the whole portfolio into the panic, which is the one mistake it exists to prevent. On the way down, that is exactly the behavior I wanted.

Then the market recovered, and the system gave all of it back and then some. The gate clamps most sells, but a handful of trims are exempt, flagged important enough to fire even in a crash, and those fired every single week. Twelve weeks, twelve trims into the speculative sleeve. By the time the bounce came I was holding far less of exactly the stuff that bounced hardest. So the gate did its job on the way down, and then a side door I had left open quietly sold off my upside on the way back up.

That is worse than a bug. A bug I could fix. This was every piece working exactly as designed, and the pieces together producing something none of them intended. I had a passing test proving the gate fires when it should (drop and fear in, sell signals clamped out) and that firing was all I had ever proven. What it costs over a full round trip, once the market climbs back out, I had simply assumed.

What I am not claiming#

I want to be honest about what this number is not. It is one crash, run once. I rebuilt March 2020; I have not yet run 2008 or the dot-com unwind. Roll the dice differently and the two and a half percent could shift a point or two either way. I would not bet anything on the exact figure.

What rolling the dice looks like. This is the drawdown topology from the playground, switched to fat-tailed returns: every strand is one simulated run, and the yellow ones are the runs that dig deep. My replay is one strand of a fan like this.

And there is a bigger caveat. In real life the system makes itself wait: a signal has to show up two weeks running before it acts on it, which slows it down, especially early in a crash. My replay skipped some of that patience, so it almost certainly traded more aggressively than the real tool would. The number itself does not matter much. What matters is that a stress test surfaced a money-losing interaction between three features that each looked correct on their own, and it did it without my having to lose a real dollar in a real crash to find out.

What the boring design actually bought me#

This is the part I did not see coming when I made the original bet. Keeping the scoring and the rules deterministic (same inputs, same outputs, no AI in the deciding seat) was supposed to buy me inspectability on ordinary days: the ability to ask why a stock got the grade it got and get a real answer. It turns out it bought something larger. Because the engine is predictable, I could feed it a catastrophe and trust that it behaved exactly as it would in a real one. You cannot replay a system that improvises, because you cannot reproduce a judgment call. The boring, auditable design is the only reason I could run a black swan through my own code on a Tuesday afternoon and believe the answer.

It is also the role I keep wanting AI to play. The agent did not decide whether the gate was good or bad. It built the test rig, ran the crash, flagged where its own shortcuts fell short, and handed me the result to judge. The judgment about what to do with a tool that sells the recovery is mine, and it should be. The slow, careful plumbing is the part I am happy to hand off.

So I have not shipped a fix, and that is the same discipline talking. The fix is easy to describe: teach those exempt trims that they are firing into a recovery, the same way the gate already knows it is firing into a crash, so the system stops selling the bounce. I could write it this afternoon. What I cannot write this afternoon is proof that it helps. I have watched the current behavior fail against one crash, run once, and shipping a fix on that much evidence would be the exact mistake this whole project is built to avoid: confirming that the new behavior fires and calling that proof it is better. So the fix waits until I have run it against more than one crash (the 2008 grind and the dot-com unwind among them) with the dice rolled enough times that I trust the answer. The design that let me catch the problem is the same design that lets me test the fix properly. That is the entire point of building it boring.

The caskeycoding.com tech stack at a glance

Eric Caskey — Mon, 25 May 2026 00:00:00 GMT

This site is a Next.js static export on CloudFront, a small Python service on Lambda, a single DynamoDB table, an S3 content bucket, and two AI agents calling Anthropic directly. Everything is wired up with AWS CDK across four stacks split by deploy cadence: auth, agents, backend, frontend.

Here is the stack, layer by layer.

Frontend#

Next.js with static export, TypeScript, trailingSlash aligned with the edge rewrite
Playwright for end-to-end tests
Lighthouse CI for performance budgets

Edge / CDN#

CloudFront distribution fronting a private S3 site bucket via origin access, HTTPS-only
CloudFront Functions at viewer-request for extensionless URI rewrites and one retired-slug 301
Route53 hosted zone with apex / www aliases plus Google Workspace MX, SPF, and DMARC (p=quarantine)
ACM for the TLS certificate

API & compute#

Amazon API Gateway for the dynamic surface
AWS Lambda (Python) for the blog handler, the public demo handlers, the agent API, and the long-running orchestrator
AWS WAF with regional, per-route rate rules on the unauthenticated /public/* endpoints

Data#

Amazon DynamoDB: single-table design, postId + type keys, post and agent_task items
Amazon S3: transparent content offload at 2KB so DynamoDB items stay small and read-cheap

Auth#

Amazon Cognito user pool, shared via cross-stack reference to the backend

AI / LLM#

Anthropic API (direct, official Python SDK) as the primary path
- Sonnet 4.6, workhorse: generation, polish, tool loops
- Opus 4.7: synthesis and multi-source reasoning
- Haiku 4.5: routing, classification, eval-judge
Amazon Bedrock: fallback only, via a cross-region inference profile, with a Discord webhook alert when it engages
Shared client (anthropic_client.py) owns retries, fallback, secrets, and cost accounting; agents stay thin

Secrets & config#

AWS Secrets Manager for the Anthropic API key: no env-var secrets in prod
AWS Systems Manager / CDK context for non-secret config

Observability#

Amazon CloudWatch logs, metrics, and alarms (5xx on the static path → SNS topic)
Amazon SNS for paging
Discord webhooks for human-in-the-loop alerts (LLM fallback, eval drift)
Structured llm_call JSON logs: full payloads only in non-prod; prod carries metadata plus a prompt_sha256

Testing & CI#

pytest for the Python service
Replay / eval harness: YAML cases with cached completions, runs on every PR with no API spend; a --live mode for capturing new fixtures
CI pricing check: fails the build if the in-repo per-model pricing table is older than 90 days

Infrastructure as code#

AWS CDK (TypeScript), four stacks split by blast radius:
- AuthStack: Cognito
- AgentStack: agent and orchestrator Lambdas, Bedrock IAM
- BackendStack: API Gateway, blog Lambda, DynamoDB, S3 content bucket
- FrontendStack: CloudFront, site bucket, Route53, ACM

The specs that drive this site, including the architecture decisions behind each of these choices, are public in the specs demo repo.

Building a Personal Finance Reviewer: What Survived the Rewrite

Eric Caskey — Tue, 19 May 2026 00:00:00 GMT

I run a personal portfolio reviewer. It takes a position export, scores every holding on a deterministic multi-factor model, flags allocation and risk breaches against hard thresholds, and produces a written review I can act on in a few minutes instead of a Saturday morning.

An earlier version of this post described that scoring model as a value-investing rubric. That was wrong, and not in a small way: the rubric graded one of the most dominant companies in the market a D+. I rebuilt the model. The story of how it was wrong, how I caught it, and what the rewrite cost is its own post, When the Spec Was Wrong, and I would rather link to it honestly than quietly edit the old claim out of existence.

This post is about the part that did not change. The model got replaced. The architecture around it did not, and the architecture is the part worth writing down, because it is the part that made the rewrite survivable.

The one boundary that matters#

The system has three layers and a rule about which one is allowed to decide anything.

A deterministic scoring layer reads the positions and the market data and produces scores. Same inputs, same outputs, no model in the loop. A rules layer applies non-negotiable thresholds: concentration limits, sleeve caps, drift bounds. These are not advice; they fire or they do not. Only then does an LLM see the result, and its job is strictly to narrate: explain the scores, surface the breaches, write the review a careful investor would recognize as coherent. It does not compute a score. It does not overrule a threshold. It is told the numbers; it is never asked to derive them.

That boundary is the entire design. Everything load-bearing is deterministic and auditable; the model is a writing layer on top of facts it cannot change. When I had to throw out the scoring model, the blast radius stopped at the deterministic layer. The rules did not move. The narration did not move. The audit trail did not move. A rewrite that would have been frightening in a system where the AI decided things was a contained change in a system where it only describes them.

Never let a missing number become a confident one#

The failure mode I care most about in a system with money attached is not a wrong opinion. It is a fabricated fact presented with the same confidence as a real one.

The concrete version: a price fetch fails for one holding. The lazy behavior is to fall back to zero, multiply it through, and emit a portfolio that looks complete and is quietly wrong, with a sell recommendation on a position that did not actually crater. The reviewer does not do that. A price fetch resolves to one of three states: a fresh value, a stale-but-real cached value with its age attached, or an explicit failure. A failed fetch is excluded from totals, weights, grades, and recommendations rather than silently coerced into a number. Stale-by-a-day, clearly labeled, is a far better signal to a human than a confident zero.

I will be honest about where this principle is still being enforced rather than finished. The sparse-fundamentals case, where a data provider returns nothing for a thinly covered instrument, is harder than the price case and is the edge I am still tightening. The principle is settled. The coverage is not yet complete, and I would rather say that here than imply otherwise.

The audit trail keeps the system honest over time#

Every review is written as an immutable record: the scored output, the recommendations, and hashes of the exact inputs and configuration that produced them. Reviews are not overwritten and not expired. If a recommendation looks wrong three months later, I can reconstruct precisely what data and what rules produced it.

This sounds like compliance theater for a one-user tool. It is not. It is what keeps the system honest with me over time. A reviewer I cannot reconstruct is a reviewer I have to trust on feel, and the entire reason this thing exists is that I do not trust my own recency bias with real money. The immutability is the same instinct as the deterministic boundary: keep the parts with consequences inspectable, and keep the AI on the side of the system where being inspectable is not required.

What the rewrite actually taught me#

The lesson is not "I picked the wrong model first." Everyone picks the wrong model first. The lesson is that the cost of being wrong is set long before you are wrong, by where you drew the line between the parts that decide and the parts that describe.

Because the scoring layer was deterministic, isolated, and specified separately from the code, replacing it was a bounded edit with a written decision record behind it. Because the AI was a narrator and not a judge, none of the rewrite touched it. The expensive version of this mistake, the one where the model is fused into the recommendations and the recommendations are fused into the prompt, is a system you cannot rewrite without rebuilding. I did not have that system, by deliberate choice, and that choice was worth more than any single thing the original model got right.

If you are building anything that scores or grades where the output has consequences, the durable question is not which model. It is: when the model turns out to be wrong, and it will, how much of the system has to move with it? Keep that number small on purpose. Ship the deterministic version first, write down why, and assume the first model is a placeholder for the second one. The code is the cheap part. The boundary between the parts that decide and the parts that describe is the product.

When the Spec Was Wrong: Rewriting a Shipped Decision

Eric Caskey — Sun, 26 Apr 2026 00:00:00 GMT

Two weeks ago I wrote a post about a personal finance app I'd built. The part I was proudest of was the scoring engine: a six-dimension rubric based on Graham's value framework, with clear thresholds and transparent logic. The LLM only narrated the results; it didn't decide them. The argument was that structure mattered more than intelligence for a system with real money on the line.

A week later I ran the rubric against my actual portfolio. NVDA scored a D+.

That's not a tuning problem. A scoring system that grades the most dominant semiconductor company in the world a D+ is broken at the level of the rubric, not the thresholds. So I rewrote it. This post is about what happens when a shipped spec meets reality.

The shipped spec accumulates assumptions you stop questioning#

I chose Graham's framework because I'd read The Intelligent Investor and Graham was what I knew. The v1 rubric was a faithful operationalization of his principles: P/E thresholds, P/B as a primary signal, current-ratio screens, dividend continuity. Every grade reproducible.

Then I ran it against modern stocks. Every one of them failed. Not just NVDA. The intangibles-heavy S&P names came in the same way: D's and F's where I expected B's. The rubric was reliably grading every modern stock as a failure, which is another way of saying it was grading them against 1949.

The previous post argued that the spec is the product, and I still believe that. But shipping a spec is also when you start defending it. The deterministic engine was the right call, and I'd ship it again. The deterministic rules inside it are a separate question, and that distinction is easy to miss once a system is live. Consistency feels like correctness. A rubric that produces the same output every time looks like it's working. But reproducible isn't the same as right; it just means the same answer every time, whether the answer is any good or not.

What was actually broken#

Three structural problems, all of them obvious in retrospect.

The framework excluded most of my portfolio by design. Graham wrote in an industrial economy, and he explicitly bracketed companies where intangible assets dominate value. That's roughly the top 40% of the S&P 500 by market cap today. Using absolute P/E thresholds against that universe doesn't fix the mismatch; it bakes it in.

I was craving more quant practice than Graham could give me. Graham gave me a rubric I could operationalize because I'd been reading him for years. But Graham doesn't have factor models, sector-relative scoring, or any of the toolkit modern quants actually use. The v1 spec was a personal reading list dressed up as a spec, and nothing on the reading list was post-1949.

Two scoring systems had grown in parallel. A "committee" of investor personas had been added on top of the factor scorers, and each persona computed its own weighted score. Same position, different grades, no defensible way to reconcile them. The spec said "deterministic scoring," but the implementation had drifted into two deterministic systems that disagreed.

Each of these was a shipped assumption I'd stopped questioning. None of them needed new evidence to surface. They needed someone reading the spec who wasn't me.

Perplexity Computer did the literature review#

Before I rewrote a single line of the rubric, I had Perplexity Computer run a deep-research pass on modern factor frameworks. AQR's quality-value-momentum work, MSCI's factor model documentation, JP Morgan's factor views, the Russell research notes: what institutional quants actually use to evaluate equities in 2026, sourced and synthesized into a report I could read in an evening.

This is the part I want to draw out for engineers, because it changes what spec-driven development is tractable for. The first version of the spec was Graham-aligned because Graham is what I'd read. The rewrite is grounded in modern factor research because an agent did the literature review for me. What would have been a week of reading on my own showed up as a synthesis I could absorb between dinner and bed, with sources cited so I could chase any claim I wanted to verify.

Computer wasn't writing the spec. I was. But the work that has to happen before you can write a defensible spec, surveying current practice, comparing approaches, finding the dead ends, is exactly what an agent is good at. It runs sub-agents in parallel across different sources, summarizes where they conflict, and hands you something concrete to argue with.

Spec-driven development assumes you can write specs. AI-assisted research is what makes it tractable to write specs about domains you're not already an expert in, and that's the loop that's going to drive every rewrite from here.

The rewrite, and the decision that made it cheap#

The replacement is a three-factor composite: Quality, Value, Momentum. The personas got demoted from parallel scorers to factor narrators. Each persona owns one factor and explains its score in their voice instead of computing a competing one. That collapses the dual-scoring problem: the committee output now follows from the factor output, not against it.

The non-obvious decision in the rewrite was a negative one. Keeping the implementation in Python.

The new spec used TypeScript pseudocode for readability, and there was real temptation to "modernize" the backend alongside the algorithmic change. Same blast radius, same deploy, why not. The answer is that none of the working pieces would have been better in another language: not the FMP client, not the caching layer, not the Lambda handlers, not the Pydantic models. The algorithm changes were independent of the host language, and combining the two would have doubled the surface area for new bugs. I'd have spent a week debugging serialization instead of validating scoring behavior.

The cheapest rewrite is the one that touches one axis at a time. Language migrations alongside algorithmic changes are the most expensive form of yak-shaving I know.

Then a review caught what I'd already convinced myself was right#

The first version of the new rubric kept Growth as a separate fourth factor at 20%. A soundness review pass caught what I hadn't: Growth correlates with Quality through earnings persistence, and with Momentum through earnings momentum. Carrying it as its own bucket was scoring the same underlying signal up to three times.

So the rewrite got rewritten. Four factors, not five. Growth was deleted, and its sub-signals folded into Quality and Momentum where they actually belonged. The same review caught a second problem: the new valuation formula didn't actually solve the intangibles bias it was advertised to solve. It softened the bias; it didn't fix it. The replacement uses free cash flow and owner-earnings yield, which removes the GAAP-earnings distortion that breaks for software, semis, and stock-comp-heavy tech.

Two structural problems, both shipped under "Accepted," both caught by an external read of the spec. The review pass is part of the spec-driven loop, not an afterthought to it. ADRs are cheap. Reviews of ADRs are what make them survive.

What this means for spec-driven development#

Three things I'd carry over to any project where a deterministic spec drives a system with consequences.

Specs are versioned, not eternal. ADRs that supersede other ADRs aren't a failure mode; they're the system working. The original rubric is now superseded twice over: once by the three-factor pivot, again by the four-factor consolidation. Each supersession is a record of what changed and why. The cost of writing those records is low. The cost of skipping them is silent drift between what the code does and what anyone thinks it does.

Have an agent do the literature review. The strongest single decision in the rewrite, before any code, was running Perplexity Computer against the current factor literature instead of writing the spec from memory. Specs grounded in "what I happen to have read" stop scaling the moment you cross out of your own depth. Specs grounded in "what an agent surveyed last week" can be written about almost anything you're willing to argue with.

Touch one axis at a time. The rewrite most likely to fail is the one that changes the algorithm, the language, and the deployment model in the same PR. Every axis you change multiplies the surface area for new bugs. The Python decision was worth more than any single algorithmic improvement in the rewrite.

The original post argued that the spec is the product. That's still true. The qualifier I'd add now is that the spec is the product only if you're willing to rewrite it. Shipped specs that don't get rewritten aren't specs; they're folklore.

If you're building anything that scores or grades with real consequences, ship the deterministic version first. That covers risk scores for transactions, severity grades for incidents, factor models for portfolios. Then commit to revisiting it the moment it touches reality. The first version will be wrong about something specific. The second version will be wrong about something different. That's the loop.

One week of SDD in production: the numbers

Eric Caskey — Mon, 20 Apr 2026 00:00:00 GMT

The previous two posts made claims. This one is the data trail, pulled from the GitHub API across five repositories for the week of April 14–21, 2026.

Same workflow in every repo: spec written in Perplexity, implemented by Claude Code, validated by GitHub Actions, merged, deployed.

ericcaskey.com, first week (Apr 14–21, 2026)#

The site launched during this window. Every number below is from week one.

Metric	Count
PRs opened	12
PRs merged	11
Production deploys	6
Total CI pipeline runs	20
PR validation runs	12

caskeycoding.com platform, cumulative (frontend + backend API + infra)#

Metric	Count
Frontend PRs merged	27
Frontend production deploys	18
Frontend CI pipeline runs	57
Backend API PRs merged	57
Backend API production deploys	11
Backend API CI pipeline runs	100
Infrastructure (CDK) PRs merged	20
Infrastructure deploys	6
Infrastructure CI pipeline runs	45

Specs repo, first week (Apr 14–21, 2026)#

Metric	Count
Spec PRs merged	51

Specs repo, cumulative#

Metric	Count
Spec files (markdown)	138
Total spec PRs merged	59

The number worth pausing on#

51 spec PRs merged in one week. That is not documentation overhead, that is the scaffolding that made the other hundred-plus code PRs safe to merge. 138 spec files drive production code across three code repos and a launch-week site.

Every one of those spec PRs was reviewed, gated by a voice scan and an NDA scan, and merged as a deliberate decision. None of them was generated as a byproduct of code. The spec PR always precedes the code PR, that is the whole discipline.

What to link#

caskeycoding-specs-demo, the curated public demo repo. Point one spec in it: "here's what one of the 138 looks like."
ADR-003, Spec-Driven Development
ADR-004, SDD File Structure
Two commercial projects, a financial-position reviewer and an AI marathon coach, are in private development on the same workflow. Brand signal, not demonstration material.

Closer#

The demo is public. The specs behind the numbers are private, on purpose. The rigor is visible either way.

Previous in the series:

SDD isn't about managing AI agents, it's about managing context

Eric Caskey — Sun, 19 Apr 2026 00:00:00 GMT

The last post ended on a claim: the reason I can ship production software from a phone is that the thinking happens in specs and the typing happens elsewhere. This post is the thesis underneath that.

Spec-Driven Development is a methodology for managing context. The specs are the persistent memory that makes a stateless AI useful across sessions.

Why stateless is the real problem#

Claude Code does not remember the last session. Every session starts from zero, no memory of the stack, the conventions, the decisions from last week, the reason a particular function is implemented the way it is. This is the single most underappreciated fact about working with coding agents at scale.

Without specs, every session rebuilds context from scratch. The agent reads the codebase, guesses at conventions, infers design intent from whatever it happens to open first, and produces code that is plausible but not necessarily correct. Small changes land fine. Larger changes expose every gap.

With specs, a session that opens with "continue the contact form work" already has the stack, the conventions, the prior decisions, and the acceptance criteria. The spec is the handoff. Point a fresh session at it and the session is at full speed in one read.

The stateless/persistent split is the actual architecture:

Claude Code, stateless execution. Given a spec, it writes the code. It does not need to remember anything between sessions because the spec carries everything.
Perplexity Computer, persistent decisions. It remembers across sessions. It is where the thinking happens, where decisions are made, where specs get written.

The workflow depends on the right tool handling the right layer. Trying to make Claude Code carry state across sessions fails. Trying to make Perplexity implement features inside its planning context fails. Separation is the point.

The spec structure#

Three components do almost all the work:

A specs repo, separate from code, readable by both tools. This is not a /docs folder. It is its own thing, version-controlled, review-gated. See the public demo repo for a guided walkthrough of both package shapes, it is the public teaching artifact for the methodology behind my private work.

Two package shapes, chosen per project. An integration package (steering/ + feature/ + decision/) for evolving platform work where each feature lands into a live system with existing conventions. A domain package (product/ + domain/ + contracts/ + delivery/) for bounded greenfield apps where you are defining the domain as you go. One of each lives in the demo repo.

CLAUDE.md at every code repo root. The first file any AI session reads. It is a router, identity, folder structure, and a routing table that maps task types to the right specs. Forty to fifty lines, not four hundred. Routing table columns: task, go to, read first. That is the whole shape. A sample CLAUDE.md is in the demo repo.

The CLAUDE.md file is the most expensive real estate in the whole system. Every token in it is spent on every turn, in every session, forever. Treat it like one. Put routing there. Put everything else in workspace-level CONTEXT.md files that only load when the agent enters that workspace.

What breaks#

Three failure modes are worth naming.

Spec rot against a stateless reader. A spec describing a design that shipped differently becomes a wrong answer the AI trusts. The fix is not more process, it is a discipline to either update the spec when the code changes, or mark the spec deprecated. Drift is the single biggest risk.

Context window limits. Loading a full package into every session crowds out the code itself. The CLAUDE.md routing table exists precisely to avoid this. The agent reads the router, goes to the right workspace, loads only that workspace's context, and works. Planning context doesn't contaminate implementation context. Client A doesn't leak into Client B.

Treating the spec like documentation. Documentation describes something that exists. A spec describes something to build. When a spec turns into a write-up of the finished code, it has stopped doing its job. Specs are forward-looking artifacts that happen to survive as a record. The record is a side effect.

The persistent memory lives outside the agent#

What lets a stateless agent ship reliable production code is the context it loads on turn one. Spec-driven development gives a stateless tool a persistent memory that lives outside it, on purpose, where humans can review it, gate it, and change it.

That external memory is only as good as its upkeep. The moment a spec describes a design that shipped differently, the agent trusts a wrong answer with full confidence, and a stateless reader has no way to catch that the memory rotted. The discipline holds the whole system up, and it is never finished.

The next post is the numbers. Same week, same workflow, pulled straight from GitHub.

Previous in the series: I shipped two production sites and a blog from my phone

Next in the series: One week of SDD in production: the numbers

Read the methodology:

caskeycoding-specs-demo, public teaching artifact
ADR-003, Spec-Driven Development
ADR-004, SDD File Structure

Specs in, deploys out, no keyboard

Eric Caskey — Sat, 18 Apr 2026 00:00:00 GMT

This week I shipped two production websites, ericcaskey.com and caskeycoding.com, and moved two personal projects forward: Marathon Coach and Finance Reviewer. All from a phone. Not prototypes. Real sites with CI/CD pipelines, CloudFront distributions, GitHub Actions, and spec packages that drive every session. I wrote no code. I wrote specs.

The pipeline:

voice dictation → Perplexity Computer → spec → Claude Code → PR → GitHub Actions → deploy

The chain runs like this. Voice dictation into Perplexity Computer opens a session. That session writes a spec and opens a PR on GitHub against a private specs repo. A second session in Claude Code on the web, also voice-driven, reads the spec and opens a PR on the code repo with the implementation. GitHub Actions validates. I review. Merge. CloudFront invalidates. The change is live.

The tools earn naming. Perplexity Computer handles planning, research, and spec writing. It has a GitHub connector that reads repos and opens PRs directly, and it has memory across sessions: the one I start tomorrow morning knows what last night's session decided. That persistence is what lets specs evolve instead of starting from scratch every conversation. Perplexity is also particularly strong on finance and stocks research, which is why the specs behind Finance Reviewer were written there specifically. Pick the tool for the domain.

Claude Code on the web handles implementation. It reads a spec and opens a PR with working code. I review asynchronously, typically from the same phone, in a commute or between calls. When CI is green and the diff matches what the spec asked for, I merge. The two tools are complementary by design: one stateful and deliberative, one stateless and fast.

No IDE, no laptop, no desk. No keyboard. A session is a conversation, literally, via voice dictation. That is not a stunt. It is what these tools make possible once the specs are good enough to carry the intent. Last November in my review of the Enterprise Vibe Coding Playbook, I wrote that I had tried voice dictation and felt self-conscious about it. Five months later it is the entry point of the entire workflow.

Things break, of course. A CloudFront function was pinned to trailingSlash: false while the Next.js config flipped to true, and for a few hours every sub-route on caskeycoding.com served the home page. The fix landed in the right order: a spec change first, amending the architecture decision in the infra and frontend specs, and then one-line code changes in both repos behind those specs. The discipline is that the spec leads. The code follows.

The framing matters. I made the decisions, wrote them down, and AI implemented them reliably. The trailing-slash break above is the proof: what fixed it was changing the spec first and letting the one-line code edits follow, a decision no agent made for me. Restraint is the value. Context architecture is the method. The implementation is reproducible because the spec makes it so.

The two personal projects, Marathon Coach and Finance Reviewer, run on the same workflow, specs kept in private development. The interaction model is the portable part.

The methodology under this is spec-driven development and the folder architecture that makes it work, which I wrote up last month. The next post in this series is the follow-up: what a month of running that system in production surfaced that the March essay did not cover. Alongside both, browse caskeycoding-specs-demo to see the shape of the system: two example spec packages, the two ADRs that govern the methodology, and a sample CLAUDE.md.

Building an AI Marathon Coach: Deterministic Rules, LLM Narratives, and the 2026 NYC Marathon

Eric Caskey — Mon, 13 Apr 2026 00:00:00 GMT

I'm running the 2026 TCS New York City Marathon. The training cycle starts now. And I'm building the coaching tool as I go.

This is not an AI experiment in the abstract. It's a real system, for a real race, trained on my actual runs, with my actual injury history, including a calf issue that has derailed training blocks before. If the system makes bad recommendations, I show up to the start line underprepared or injured. That sharpens the design criteria considerably.

Here's what I built, how it's architectured, and what the design decisions reveal about building AI systems where the stakes are personal.

The Problem With Generic Training Plans#

Standard marathon training plans are reasonable starting points. Eighteen weeks, periodized volume, taper protocol. The problem is they don't know anything about you. They don't know that you ran sixty miles last week on strong legs. They don't know that you had two poor nights of sleep and your resting heart rate is elevated. They don't know your calf is talking to you.

Good human coaches know these things. They adjust. They read signals. They make calls that generic plans can't make.

AI is a plausible substitute for that coaching intelligence, but only if it's given the right data and the right constraints. An LLM alone won't do it. You need structure.

Architecture: Two Layers, One Pipeline#

The system has two distinct layers that must not be confused:

Layer 1: Deterministic rules engine. This layer ingests training data, runs from Garmin FIT file exports, manual recovery signals (resting HR, sleep quality, soreness, pain flags), and computes objective training metrics: Acute:Chronic Workload Ratio (ACWR), weekly volume, fatigue indicators, trend. It then evaluates fourteen defined guardrails. These guardrails produce a typed recommendation: PROCEED, REDUCE_INTENSITY, REDUCE_VOLUME, CROSS_TRAIN_OR_REST, FULL_REST, RECOVERY_RUN, or NEEDS_MORE_DATA.

The rules are hard. If a pain flag is set, the system returns FULL_REST, regardless of any other metric. If ACWR exceeds 1.8, the system returns CROSS_TRAIN_OR_REST or FULL_REST. If there are fewer than three activities in the past fourteen days, the system returns NEEDS_MORE_DATA rather than producing an unreliable estimate.

Layer 2: LLM narrative engine. Once the rules engine has produced a typed recommendation and the guardrail outcomes, the LLM receives that structured output and writes a coaching narrative. Two to four sentences. Plain language. Actionable. Grounded in the data the rules engine processed.

The LLM does not choose the recommendation type. That is the rules engine's job. The LLM explains the recommendation in human terms.

This separation is the load-bearing design decision. The boundary between what the LLM can do and what it must not do is explicit and enforced.

Why the LLM Must Not Decide Safety-Critical Outcomes#

There's a category error that shows up in a lot of AI system designs: giving the LLM responsibility for decisions that require consistent, auditable reasoning and then hoping the model stays calibrated.

For a coaching system, that error is especially dangerous. Training load management is not a soft opinion domain: an acute-to-chronic workload ratio past 1.8, the line the rules engine treats as dangerous, marks real injury risk, and a recommendation to "push through" it on the wrong day can end a training cycle. Overtraining injuries are real. Stress fractures are real.

Language models are not deterministic. They can reason well about training principles in general, but they will not reliably apply the same rules to the same inputs across sessions. They can be influenced by how context is framed. They can hallucinate confidence about edge cases.

Deterministic code has none of these problems. A rule that says "if pain flag is set, return FULL_REST" executes identically every time. You can test it. You can audit it. You can explain exactly why the system made the call it made.

The LLM's value is in the narrative: taking the correct, deterministic recommendation and explaining it in a way that a human athlete actually finds useful. That's a real capability, synthesizing multiple signals into a clear, encouraging, contextually-aware explanation. That's what the model does.

Garmin FIT Files and the Unglamorous Data Layer#

The first engineering problem is getting run data in. Garmin's full API requires developer program approval and user OAuth. For a personal MVP, that's a month of setup for one athlete. The practical choice is FIT file exports, Garmin lets you export any activity as a .fit file from Garmin Connect.

FIT files contain everything: GPS track, lap splits, cadence, heart rate, elevation. The fitparse Python library handles parsing. The Lambda reads the uploaded file, extracts the relevant training metrics, and writes an activity record to DynamoDB.

Manual recovery signals, resting HR, sleep quality, soreness level, pain flags, are entered via an API endpoint. No automated wearable integration in v1. That's a deliberate constraint: I want the data ingestion path to be reliable and understood before I add the complexity of real-time API sync.

Garmin Connect API integration is on the roadmap for post-MVP. But MVP ships with file-based ingestion, and the system is useful immediately.

Designing for My Own Injury History#

One of the more interesting design challenges: the system needs to know about my calf injury history and apply calf-specific caution in its guardrails.

In a generic coaching system, this is handled by user profile fields that parameterize the rules. My system does this, but it also surfaces calf-specific guardrail outputs in the narrative context, so when the LLM writes the coaching explanation, it has explicit signal to reference: "your recent history with the left calf warrants extra caution here."

This is a case where personalization is the point. A coaching system that does not know your injury history is just a generic training calculator. The personal context is what makes the LLM output useful.

If You Can't Inspect It, You Can't Trust It#

Every recommendation pipeline run writes an audit record: inputs (activities, recovery signals, computed metrics), guardrail outcomes, final recommendation type, LLM prompt sent, LLM response received, timestamp.

This is foundational infrastructure.

When the system tells me to rest and I disagree, I want to be able to look at exactly what data it had, exactly which guardrails fired, and exactly why. For any system that influences decisions you care about, the ability to inspect the reasoning is a requirement.

The audit records also let me evaluate recommendation quality over time. I journal my independent assessment alongside each recommendation. After four weeks of training, I can see whether the system's calls match what I would have chosen. That feedback loop is how you tune the guardrails.

Shipping It While Training#

The interesting meta-challenge: I'm building this system while the training cycle I'm building it for has already started. That creates a useful constraint. I can't spend six months on perfect architecture and ship after the race. I need something useful now, even if it's incomplete.

This is a productive way to scope an MVP. What does the system need to do to be useful for this week's training? That's the scope. Everything else is v2.

For week one: ingest FIT files, compute ACWR, surface recovery signals, produce a typed recommendation with a narrative. That's it. The dashboard, the trend visualization, the automatic Garmin sync, those can wait.

The CoachStack is live in the same CDK infrastructure as the finance app and this site. The backend Lambda is deployed. The API is wired. The data is flowing.

Now I just have to train.

Where the Risk Moved#

The two-layer split relocates the safety risk rather than removing it. The obvious danger was an LLM hallucinating a training call, so I moved every call into fourteen hand-written guardrails. Now the model narrates those calls with equal confidence whether the rule behind them is right or wrong. The failure I watch for is a guardrail I wrote badly, delivered in fluent, convincing prose.

The 2026 NYC Marathon is in November. I'll be sharing updates on how the coaching system performs across the training cycle, where it nails the calls, where it misses, and how the architecture evolves.

If you're building something similar, personal coaching tools, adaptive recommendation systems, anything where AI-generated advice has real consequences, I'm genuinely interested in comparing notes.

Designing Safety Guardrails for Distributed Workflow Orchestration

Eric Caskey — Fri, 10 Apr 2026 00:00:00 GMT

By the time a monitoring alert fires on a distributed workflow, you may already be committed to a path that cannot be cleanly reversed. Pre-execution guardrails are how you close that gap before execution starts. This is a pattern I designed for infrastructure workflow changes and drove to adoption across an engineering organization, on a platform running tens of thousands of workflow executions a year, every one behind a wall of concurrent safety checks, half a million check evaluations and counting. What follows is the generalized version: the checks, the aggregation model, the opt-out policy, and the plug-in architecture that let other teams extend it without touching the engine.

Post-hoc monitoring is necessary and not sufficient. Guardrails are the contract on the other side of it: we will not begin work until we have evidence that starting work is responsible.

The tension is familiar. Teams want speed and self-service. Operators want confidence that automation will not amplify mistakes. Platform owners want a system that is explainable when something blocks, because "computer says no" without detail loses trust faster than incidents do. The default compromise (ship fast, let monitoring catch it) breaks down when orchestration runs at scale and failures compound.

What to Check Before You Execute#

Pre-execution validation covers distinct categories of risk. The exact names matter less than the coverage; separating them keeps failure messages legible and lets you tune severity and ownership independently.

The questions worth asking, at minimum: Is now an acceptable time to run? Is something else already modifying this target? Are the right health signals present? Is the target itself in a known-good state? Is there conflicting in-flight activity that makes this execution unsafe right now?

If you cannot map a proposed check onto one of those questions, you are probably mixing concerns, and mixed concerns turn guardrails into mystery meat. Separated, a timing violation reads differently than a concurrency conflict. Operators deserve that clarity.

The bar I held every check to: if it cannot explain why it failed, what would make it pass, and which team owns remediation, it is not yet a guardrail. It is a boolean. Booleans block execution. Guardrails direct it.

Why Fail-Fast Is the Wrong Default for Safety Systems#

In most request paths, fail-fast is a virtue. You stop early, shed load, and protect downstream dependencies. Safety validation optimizes for something else entirely: surfacing the full set of blocking issues so a human can fix them in one pass.

Stop at the first failure and you train operators into a loop: fix one problem, re-run, hit the next failure, repeat. That burns time, increases toil, and shows up on dashboards as "flaky automation" when the real problem is partial reporting.

Worse, correlated failures do not surface in a stable order. In one incident review, the first failure we reported was a timing window violation. The root cause, a stale deployment lock from an unrelated workflow, only surfaced on the third re-run. If the system had reported both on the first pass, the operator would have fixed the right problem first.

The pattern that fixes this is parallel evaluation with aggregation. Safety checks run concurrently. Results roll up into a single decision: proceed or block, with a complete explanation of what failed. You are optimizing for clarity under stress, not for minimal CPU time on the happy path.

Parallelism here is a product decision, not a performance one: independent checks should not serialize human time. If two checks both fail, the operator sees both failures immediately, not after re-running and re-waiting. Under incident pressure, every minute of that churn converts directly into lost trust in the platform.

This post includes a diagram. View it on caskeycoding.com.

Real systems add dependencies, caching, and policy on top, but the core stays stable: fan out, fan in, decide once.

The Opt-Out Spectrum#

Guardrails that cannot be overridden in any circumstance will eventually block a legitimate emergency. Guardrails that can be overridden without friction will eventually be overridden for convenience. The design problem is where opt-out lives and who may use it.

Opt-out must also be traceable. If someone bypasses a guardrail, the system should still answer: who, when, and under what policy. Convenience without auditability becomes shadow risk management, and shadow risk management does not survive your next serious incident review.

The clearest candidates for the non-opt-outable bucket are checks where a mistake crosses an environment boundary or creates damage that cannot be cleanly reversed. When blast radius and irreversibility stack, operator convenience is a losing argument.

Calibrating Monitoring Response#

Not every bad signal deserves the same response. A naive policy ("any red monitor triggers automatic rollback") creates churn. Some signals mean real degradation and should drive a strong response. Others mean wait: a change window has not cleared, an alarm is still recovering, or data is temporarily insufficient to decide. Those situations need time, human judgment, or a follow-up check. An immediate rollback there adds motion without reducing risk.

Match the response to the signal. A rollback is a powerful tool, and using it when the right move is "pause and page" can make incidents worse. In practice this meant configurable response severity, so teams could map signal type to action without forking the engine.

A Plug-In Architecture for Guardrails#

If every new safety rule requires editing the core orchestration engine, you get one of two failure modes: slow innovation, because reviews are heavy, or unsafe shortcuts, because teams route around the platform. We chose a plug-in model instead, and that choice is most of why the pattern spread beyond the team that built it:

A contract every guardrail implements: what it checks, how it reports success or failure, and what metadata it needs for observability and policy.
Configuration-driven registration, so the engine discovers which guardrails apply to which workflows or targets without hard-coding a growing list in one place.

Adding a guardrail becomes: implement the contract, register it, ship. The engine stays stable. Teams onboarded their own checks without waiting on the platform's review queue, which is what organizational adoption actually requires; a safety system that only its authors can extend stays a team tool, never an org standard. The cost was real: a contract is a public API, and we paid for it in versioning discipline and in saying no to checks that wanted engine internals. It was the right trade.

Structure scales when humans and tools share the same source of truth. In Spec-Driven Development and the Folder Architecture That Makes It Work I made the same argument for AI-assisted engineering: partitioned context and explicit specs beat one undifferentiated dump. Guardrails want the same boundary. Policy and implementation meet at a contract, not in a monolith where every change is everyone's emergency.

This post includes a diagram. View it on caskeycoding.com.

Discovery and dispatch stay separate from each guardrail's business logic. That separation is what keeps the platform extensible after the original authors move on.

The Test That Decides If You Built One#

One rule carries out of all of this. Before a check ships, make it answer three questions in its failure output: why it failed, what would make it pass, and who owns remediation. Answer all three and you built a guardrail that operators will defend. Answer none and you built a boolean that operators will route around, and a routed-around safety system is worse than no safety system, because it still shows up green on someone's compliance dashboard.

The open question I carried out of this pattern: guardrails protect execution, but nothing here protects the checks themselves from rotting. A check that never fires is indistinguishable from a check that is dead. Auditing guardrail liveness got its own post.

This article describes the pattern in generalized form. It does not name any specific internal system or tool, and all scale figures are ones already published on this site.

Spec-Driven Development and the Folder Architecture That Makes It Work

Eric Caskey — Thu, 09 Apr 2026 00:00:00 GMT

You open a chat, describe what you want, and get code back that looks right, until it doesn't. The stack is wrong. The edge cases are missing. The auth module doesn't match the patterns in the rest of your codebase. You correct, re-prompt, correct again, and eventually the context window degrades to the point where you're fighting the AI more than collaborating with it. Most AI-assisted development fails this same way.

The AI can code. What is missing is the structure around it.

I wrote recently about Doug Kerwin's Enterprise Vibe Coding Playbook and his central argument: AI should write most of your code, but you must remain the architect, reviewer, and course-corrector. Design before you dive in. Think before you prompt. That's the philosophy. This piece is about the methodology and infrastructure that make it operational: spec-driven development and structured folder architecture for AI agents.

The Problem: Chat as Source of Truth#

When you let an AI jump straight to code, you're asking it to figure out what to build, how to build it, and actually build it, all in one context window. That's three jobs. It does a mediocre version of all three instead of a good version of any one.

Worse, the design decisions live in chat history. Close the session, and they're gone. Start a new conversation, and the AI re-derives context from code that may not reflect the original intent, or it hallucates conventions that contradict decisions you made last week. For quick prototypes, this is fine. For anything you have to maintain, it's a liability.

The GitHub Engineering blog put it well: "We treat coding agents like search engines when we should be treating them more like literal-minded pair programmers. They excel at pattern recognition but still need unambiguous instructions."

Spec-driven development is how you give those unambiguous instructions.

The Methodology: Specs as Source of Truth#

The core idea is straightforward: before the AI writes a line of code, you write a specification, a persistent markdown file that captures what you're building, why, the design decisions, API contracts, edge cases, and a decomposed task list. The spec becomes the source of truth, a version-controlled file rather than a decision buried in chat history you will close and lose.

GitHub recently open-sourced Spec Kit, a toolkit that formalizes this into four phases:

Specify: Describe requirements as user journeys and success criteria. Focus on what and why, not technology choices. Who uses this? What outcomes matter? What does success look like? This is the "hard thinking" that Kerwin insists on, externalized into an artifact instead of locked in your head.

Plan: Add architecture constraints, stack preferences, and non-functional requirements. If your company standardizes on certain technologies, if you're integrating with legacy systems, if you have compliance or performance targets, all of that goes here. The AI designs within your constraints instead of guessing at them.

Tasks: Decompose the plan into small, reviewable, testable units. Each task is concrete and independently verifiable: "create a user registration endpoint that validates email format," not "build the auth system." This is test-driven development applied to the planning layer.

Implement: Execute tasks sequentially with focused review at each step. Instead of reviewing thousand-line code dumps, you review focused changes that solve specific problems.

The insight is that each phase has a specific job, and you don't move to the next one until the current phase is fully validated. The spec captures intent. The plan captures constraints. The tasks capture scope. The implementation follows all three.

In Practice: A Four-Phase Workflow#

Practitioners like Ghazi Ahmed have adapted this into a hands-on workflow specifically for Claude Code that's worth studying:

Phase 1: Explore. Before the AI writes a single line, it reads. You're explicit about this: "Read the auth module, user model, and existing middleware. Understand the patterns. Do NOT write code." Without that last line, the AI will start drafting implementations mid-research. You don't want that yet.

Phase 2: Write the spec. Draft a three-layer specification: Requirements (the what and why: problem statement, goals, acceptance criteria, constraints), Design (the how: architecture decisions, data model changes, API contracts with actual request/response examples, edge cases), and Tasks (the implementation plan: small, atomic steps, each touching three files or fewer, with explicit dependencies and definitions of done). Save it as docs/spec-feature-name.md in your repo.

Phase 3: Implement with subagents. Delegate each task to a subagent so it gets a fresh context window scoped to just its slice of work. The orchestrator tracks progress without filling up on implementation details. Each subagent produces one commit. If Task 4 breaks something Task 2 built, the git history tells you exactly where to look.

Phase 4: Verify. Tests are inline with tasks, not a separate phase tacked on at the end. Every task compiles and passes tests before the next one starts. For bugs, the loop is strict TDD: write a failing test, confirm it fails, fix the code, confirm it passes.

The key discipline: the spec file is your recovery point. Context degrades. It always does. When it happens, you clear the session, point a fresh conversation at the spec file, and you're back to full speed. The spec persists intent across context resets. It serves as an onboarding document when new engineers (or new AI sessions) join a feature. It decomposes large ambiguous work into small testable units.

For teams at scale, this solves a problem that rarely gets addressed: where do security policies, compliance rules, design system constraints, and integration requirements live? Often they're buried in wikis nobody reads or scattered across Slack. In a spec-driven workflow, they live in the specification and the plan, where the AI actually uses them.

The Infrastructure: Folder Architecture as Agentic Architecture#

Spec-driven development gives you the methodology. There's still a question of infrastructure: how do you organize the specs, context files, and project knowledge so the AI reads the right information at the right time without burning tokens on everything else?

This is where Jake Van Clief's work with Clief Notes deserves attention. Van Clief has built a community of 6,000+ members around what he calls Interpretable Context Methodology (ICM), a system where folder structure itself becomes the agentic architecture. No frameworks. No multi-agent orchestration code. Just folders, markdown files, and naming conventions.

The premise is simple and well-grounded: tokens are finite, and every token the AI spends reading irrelevant context is a token not spent on the task at hand. If you dump everything into one conversation (your blog post drafts, your video production notes, your deployment configs) the AI writing your blog post is also reading your Kubernetes manifests. The folder structure solves this by separating work into areas and only loading what's needed.

Three Layers, Plain English#

Layer 1: The Map (CLAUDE.md): A minimal top-level file at the root of your project. It tells the AI what the project is, what the folder structure looks like, and includes a routing table that maps tasks to workspaces. Van Clief compares it to the floor plan on the wall when you walk into a building. You look at it, you know where to go.

The routing table is the critical piece. Three columns: task, where to go, what to read.

Task	Go to	Read
Spec a feature	/planning	CONTEXT.md
Write code	/src	CONTEXT.md
Write docs	/docs	CONTEXT.md
Deploy or debug	/ops	CONTEXT.md

Van Clief is emphatic that this file should be short, one screen, 40-50 lines maximum. It's a router, not a brain dump. Every token in CLAUDE.md is spent on every single turn, so bloat here is the most expensive mistake you can make.

Layer 2: The Rooms (Workspace CONTEXT.md files): Each workspace gets its own context file describing what happens there, what the process looks like, what files live there, and what good work looks like. When the AI enters a workspace, it loads only that workspace's context. The planning workspace doesn't bleed into the source code workspace. Client Alpha doesn't contaminate Client Beta.

These are plain English, short documents. A few paragraphs. You describe the workspace, the process, the standards. The AI reads the context and immediately knows what it's working on, what rules apply, and what good output looks like, with almost no prompting.

Layer 3: The Tools (Skills and plug-ins): Reusable capabilities wired into specific workspaces where they're needed. A testing skill in the src/ workspace. A doc-authoring skill in the docs/ workspace. You might reference dozens of skills across a project, but each workspace only loads the ones it needs. That's the plug-and-play idea.

What It Replaces#

The elegance is in what you don't need. Traditional multi-agent frameworks require orchestration code, API calls, and infrastructure to route work between specialized agents. Van Clief's insight is that for sequential workflows with human review at each step (which is most real work) the filesystem is the orchestration layer.

Numbered folders represent stages. Markdown files carry prompts and context. Naming conventions replace database queries. If a spec draft is created, it gets named feature-name_spec.md. If it's a blog draft, topic-name_draft.md. If it's a versioned demo script, demo_v2.md. The AI knows where to look, what to pull, and what to do next because the naming tells it everything. No SQL. No vector database. Just conventions.

The research behind this, published as Interpretable Context Methodology: Folder Structure as Agentic Architecture, traces the principles back to separation of concerns (1972), Unix philosophy, and modular composition. One stage, one job. Plain text as the universal interface. Layered context loading. Every output is an edit surface. These aren't new ideas. They're established software engineering principles applied to AI workflows, and that's what makes the approach durable rather than trendy.

How the Layers Map to a Codebase#

For developers, the three-layer system maps naturally onto spec-driven development:

my-app/
├── CLAUDE.md              # Layer 1: routing table
├── planning/
│   ├── CONTEXT.md         # Layer 2: planning workspace
│   ├── specs/             # spec-*.md files live here
│   ├── architecture/
│   └── decisions/
├── src/
│   ├── CONTEXT.md         # Layer 2: implementation workspace
│   ├── components/
│   ├── services/
│   └── tests/
├── docs/
│   ├── CONTEXT.md         # Layer 2: documentation workspace
│   └── api/
└── ops/
    ├── CONTEXT.md         # Layer 2: operations workspace
    └── deploy/

When speccing a feature, the AI goes to /planning, reads its CONTEXT.md, and drafts the spec into /planning/specs/. When implementing, it goes to /src, reads a different context, and works from the spec. Each workspace loads only its context. The spec files serve as the persistent source of truth. The folder structure is the infrastructure that makes the methodology work.

But the same three-layer system works for non-developers too: content creators, freelancers, consultants. A freelancer swaps workspaces for Client Intake, Delivery, and Admin. A content creator uses Script Lab, Production, and Distribution. The layers stay the same. The labels change. That accessibility is the point, and it's what makes Van Clief's approach useful for cross-functional teams trying to adopt AI workflows beyond engineering.

The Five Ways This Goes Wrong#

Van Clief's curriculum includes a common mistakes module, and the failure patterns map directly onto what I see in enterprise AI adoption:

Massive CLAUDE.md files. People dump everything into the top-level file: project briefs, style guides, personality instructions, background research. The routing instructions get buried in noise, and every turn burns tokens on information irrelevant to the current task. The fix: the CLAUDE.md is a routing file. Identity, folder structure, routing table, naming conventions. Everything else belongs in a workspace CONTEXT.md where it only loads when relevant.

Skipping the routing table. Without one, the AI guesses which files matter. Sometimes it guesses right. "Sometimes" is the problem. The routing table eliminates guessing. One row per type of work.

Too many workspaces. Eight workspaces for a project that has two or three modes of work. Now the overhead of maintaining context files exceeds the value of the system. The question to ask: "Do I shift mental modes between these tasks?" Writing and building are different modes. Drafting and editing are the same mode at different stages. Start with two or three workspaces. Add more from real use, not from planning.

Context files that describe the AI instead of the work. "Be creative. Be concise. Be professional." Thirty lines of personality instructions, two lines about the project. The AI responds to context about the work far more than context about itself. Telling it "the audience is mid-market HR directors who've tried three other tools and are skeptical of AI claims" changes the output more than "you are a senior copywriter" ever will.

Building the whole system before using it. The classic over-engineering trap. Someone reads about the three-layer architecture, spends a weekend building six workspaces with detailed context files and twenty-row routing tables, and hasn't used the AI once during the process. Then they start working and realize half their decisions don't match how they actually work. The fix: build the minimum. One CLAUDE.md, one or two workspaces, one CONTEXT.md each. Start working. Grow from use.

Why This Matters for Teams#

Individual productivity is table stakes. The harder problem is organizational adoption, and this is where specs and folder architecture have a structural advantage over chat-based workflows.

Specs are reviewable artifacts. A product manager can read spec-auth.md and verify that the requirements match intent. A tech lead can review the design layer before implementation starts. A security engineer can audit the constraints section. The spec becomes a contract, familiar territory for any engineering organization that already does design reviews and RFCs.

Folder architecture makes AI-assisted work legible. When a new engineer joins a project and sees a CLAUDE.md with a routing table, workspace context files describing each area of the codebase, and spec files decomposing features into tasks, they can orient immediately. The structure itself is documentation. That's a much easier organizational sell than "trust me, the AI knows what it's doing."

Together, they address the cultural resistance that Kerwin identifies in the Enterprise Vibe Coding Playbook: skeptics can review the spec. Experts can critique the architecture. Managers can see the task decomposition. The AI's behavior becomes predictable, auditable, and correctable, which is what organizations need before they'll trust it at scale.

Where to Start#

If you're new to this, three things cover 80% of the value:

1. Spec before code. Every time. Write a one-page spec for your next feature. Requirements, design decisions, task list. Save it as a markdown file in your repo. Point the AI at it. Notice the difference.

2. Three files, fifteen minutes. Create a CLAUDE.md (identity, folder structure, routing table), a CONTEXT.md for your primary workspace (what the work is, what good looks like, what to avoid), and a REFERENCES.md (background material the AI should access but not act on directly). Start working. Grow from use.

3. One task per session. Context degradation isn't gradual. It's a cliff. Keep sessions focused on a single task. Clear between tasks. When context degrades, point a fresh session at the spec file and continue from where you left off.

I now teach the smallest version of this system as a hands-on series, one copyable artifact per lesson, starting with Lesson 1: The spec directory.

The spec captures the thinking. The folder structure routes the AI to the right context. Together, they turn "design don't dive in" from a philosophy into a workflow.

Book Review: Enterprise Vibe Coding Playbook, Building Real Software with AI

Eric Caskey — Wed, 08 Apr 2026 00:00:00 GMT

Doug Kerwin's Enterprise Vibe Coding Playbook is the most useful framing I have read on enterprise AI adoption. Its bet is that the title is a trap: "vibe coding" sounds like coding without judgment, and the whole book is about keeping the judgment while letting AI do the typing. That distinction is what separates teams shipping real software with AI from teams shipping plausible-looking messes.

I approached this book as a skeptic with a vested interest in getting it right. As an engineer who architects platform infrastructure and leads teams building large-scale systems, I've seen firsthand both the transformative potential of AI-assisted development and the organizational resistance that slows adoption. This review is part book critique and part field report from the trenches.

The Core Thesis: AI as Accelerator, Not Magic#

Kerwin's central argument is disarmingly simple but often misunderstood: AI should write most of your code, but you must remain the architect, reviewer, and course-corrector. This stands in sharp contrast to the "vibe coding" strawman that has poisoned discourse around AI-assisted development.

What is vibe coding? Kerwin identifies it as the dangerous assumption that AI writes code without human scrutiny. It's a caricature that has become a convenient shield for skeptics unwilling to adapt. His beef with the term is justified. It misrepresents how AI actually amplifies developer productivity.

You're still developing; you're just not writing line-by-line anymore. Think of it as the difference between describing what needs to happen to your copilot and then refining the result, versus typing every character yourself. The hard thinking, the design, the architectural decisions: these remain yours.

Kerwin crystallizes this with a quote that stayed with me: "The issue isn't using AI to write all the code. It's pretending you don't need to review, guide, and course-correct." More pointedly: "The problem isn't AI writing the code. It's expecting it to get everything right with one shot prompt and no plan. It's skipping the hard thinking, the real design, and expecting magic."

It matches what I have found running my own spec-driven agent loops. The setups that win with AI haven't abandoned discipline. They've doubled down on it: clearer requirements, more intentional architecture, more rigorous review. The tool accelerates, but the thinking stays non-negotiable.

Why This Matters: A Historical Parallel#

When the industry transitioned from relational databases to DynamoDB and NoSQL architectures, we didn't resist the shift. We adapted. We learned new patterns, new trade-offs, new ways of thinking about data. The fundamentals of distributed systems didn't change, but our implementation patterns did.

AI-assisted development is similar. We're not abandoning software engineering principles. We're evolving how we apply them. Context engineering and prompt engineering are emerging disciplines that sit alongside API design and system architecture. AI pipelines and agentic workflows are becoming what data pipelines were to the previous generation: foundational infrastructure.

The era of purely manual writing, in code, in documentation, in planning, should be assumed to be over. The question isn't whether to use these tools, but how to use them wisely.

The Method: Writing a Book with Cursor and Repository Tracking#

One of the most interesting aspects of this book is that Kerwin wrote 100% of it with AI assistance, using Cursor as his IDE and a repository to track his ideas and maintain stylistic consistency. This isn't a book about AI-assisted development written in a vacuum. It's a book written with the methods it describes.

This approach underscores a critical insight: IDEs, agents, and AI-assisted capabilities aren't just for developers anymore. The same tools and patterns apply across the SDLC stack, to product managers, technical program managers, architects, and yes, even to authors writing about software. The infrastructure of thinking has fundamentally changed.

The Practical Framework: From Code Generator to Thinking Partner#

Chapter 4 dives into prompting, and Kerwin draws a comparison to writing user stories. The analogy is apt: bad prompts are like bad user stories. They're vague, they lack context, and they leave room for misinterpretation. I have personal experience with Kerwin pushing this point during a reorganization at a previous company, and I've seen the exact same problem at Amazon. Engineers and managers consistently under-invest in clarity at the spec stage, then wonder why the output (human or AI) misses the mark.

Good prompts require the same discipline as good user stories: clarity, specificity, constraints, and acceptance criteria. The thinking is harder upfront, but the downstream efficiency is dramatic. And if a good prompt is a good user story, the logical next step is a full specification, a persistent document that captures requirements, design decisions, and acceptance criteria in one place. The prompt is the starting point. The spec is the destination. (I explore this further in my companion piece, Spec-Driven Development and the Folder Architecture That Makes It Work.)

But prompting is just one dimension. Chapter 7 pivots to a broader theme: how you show up in an AI-driven engineering culture. Kerwin identifies two archetypes worth watching for:

The Resident Skeptic: This person specializes in doubt without building. Kerwin references Amazon's principle of "disagree and commit," and he's right. Once your organization has committed to AI, constant skepticism helps no one: not the team trying to ship, not your career, not your own fluency with the tools. Skepticism earns its place in design reviews and threat-modeling, not in paralysis.

The Expert's Paradox: Experienced engineers sometimes confine AI to narrow tasks, documentation, simple functions, while refusing to delegate more complex thinking. This self-imposed limitation leaves performance on the table.

The Underrated Part: Design and Discipline#

Chapter 6 is titled "Design Don't Dive In," and I couldn't agree more. The difference between a hastily assembled codebase and a maintainable system is upfront thinking. With AI, this becomes even more critical. You can use AI as your thinking partner to reason through the user layer, the UI layer, the data layer, the business rules, the edge cases, the data models. You can ask it to challenge your assumptions. But you can't skip the thinking.

Design is the differentiator. AI excels at implementation and iteration, but it cannot replace architectural judgment, domain expertise, or the messy work of understanding constraints.

Kerwin also acknowledges what often gets overlooked: even with AI assistance, the underlying code is often a mess underneath. Shortcuts, glue code, technical debt: these don't disappear. They're just easier to accumulate if you're not disciplined. This reaffirms that using AI to accelerate development doesn't mean lowering standards. It means raising the bar on review and refactoring.

But here's the question Kerwin leaves open: if the thinking is non-negotiable, where does it live? Not in chat history that evaporates between sessions. Not in your head. Not in a prompt you paste in each morning. The thinking needs a home: persistent, reviewable, and structured enough that both humans and AI can follow it. That's the territory of spec-driven development and structured folder architecture, and it's where I think the next chapter of this conversation is being written.

The Quiet Revolution: Voice and Thinking Out Loud#

Chapter 5, "The Power of Voice," made me smirk at first. Speaking to your computer? It felt embarrassing, inefficient, and gimmicky. But after reading Kerwin's take, I gave it a serious try.

I was surprised. Voice-to-transcription technology has improved dramatically, and more importantly, the workflow is genuinely different from typing. Speaking allows you to capture your thoughts at a higher level before editing and refining. It's like a rough draft that forces you to get your ideas out before you polish them. The revision phase comes next: the hard coding, the rewrites, the actual technical work.

I'm not doing this in the office yet (yes, I still feel self-conscious), but I've recommended the approach to others. If more engineers actually tried this, I think adoption would shift quickly. It's a tool for gathering and aggregating thoughts faster than you can type them.

The Missing Piece: Cultural Adoption#

Here's what Kerwin does well but what every team will struggle with: moving from individual productivity to organizational capability. You can be the best AI-assisted developer on your team, but if your organization isn't on board, you'll hit a ceiling.

I've seen both sides. Teams that embrace AI-assisted development with disciplined prompting, strong design practices, and a commitment to review are shipping at a different velocity. Teams that resist are falling behind. The gap widens every quarter.

But adoption is cultural, not technical. It requires skeptics to engage rather than complain. It requires experts to experiment rather than gatekeep. It requires organizations to invest in training people how to prompt, how to design for AI assistance, and how to integrate these tools into their workflows. And it requires artifacts that make AI-assisted work legible to the organization: specs that product managers can review, folder structures that new engineers can navigate, routing tables that make the AI's behavior predictable rather than mysterious.

A Complementary Read#

If you finish Enterprise Vibe Coding Playbook and want to deepen your understanding of AI's role in leadership and organizational change, I'd recommend The AI-Driven Leader by Geoff Woods. It builds on the "thinking partner" model that Kerwin introduces and extends it beyond engineering to product, strategy, and organizational design.

And if you're ready to move from philosophy to methodology, to see how the "design don't dive in" mindset translates into persistent specs, structured folder architecture, and context-aware AI workflows, I wrote a companion piece: Spec-Driven Development and the Folder Architecture That Makes It Work.

The Verdict#

Enterprise Vibe Coding Playbook is essential reading for any engineer, tech lead, or manager grappling with AI adoption. It is a practical field guide to how real teams work with AI as a thinking partner and accelerator, honest about the prompt-engineering craft it teaches and clear-eyed about the pitfalls it names.

The book's greatest strength is that it doesn't pretend to have all the answers. Instead, it models the framework: clarify your thinking, use AI to refine it, review rigorously, iterate. That's not just good software development. That's good thinking.

If you're on the fence about AI-assisted development, this book will challenge your assumptions. If you're already using it, it will give you language and frameworks to advocate for broader adoption in your organization. And if your organization is stuck in the vibe coding debates, hand this to the skeptics and the experts alike.

Let AI write the code, but never let it own the thinking. Keep the requirements, the design, and the review yours, and give them a home more durable than a chat window. Do that and the tool compounds your judgment. Skip it and you are just generating technical debt faster.

Welcome: Building Platforms for Scale

Eric Caskey — Thu, 12 Jun 2025 00:00:00 GMT

The sharpest lesson of my career came in a ten-minute stretch when a batch of misaimed alerts paged an entire support queue at once: when the automation is large, a small mistake does not stay small. I have spent most of my career on one problem in different shapes, keeping a large system honest about its own health without a person having to remember to check. Monitoring, and the automation around it.

One team's standard, a company's blast radius#

At Prudential I owned the enterprise monitoring platform, which meant the health of more than 400,000 monitors across the company. I set the standard for how infrastructure got watched, chose the hardware it ran on, and led the migration off the legacy systems onto that standard. The lesson that stuck was about blast radius. When one team's decision sets the monitoring for a whole company, a good call quietly protects thousands of services and a bad one quietly exposes them, and you often do not find out which you made until something breaks.

Coverage that every team gets by default#

The next version of the problem was the same one an order of magnitude bigger. Infrastructure at Amazon came and went constantly, so the monitoring had to find new hosts and services on its own, attach the right alarms as they appeared, and let go as they disappeared. Writing the shared standards, alert policies, and deployment playbooks turned that from a chore every team repeated into something they got by default. I wrote about that work in more detail in a case study on fleet monitoring.

The payoff was that a team could ship a service without first becoming a monitoring expert, and still trust that if the service got sick, someone would know.

The mishap#

Not every deploy went cleanly. One afternoon we rolled monitors across a large slice of the fleet and pointed every one of their alerts at a single support queue. For about ten minutes that queue took an alarm from every host at once, thousands of them, until we aimed the flood back at ourselves and rolled the change back. It taught me the rule I still work by, in one sentence: when the automation is large, a small mistake does not stay small. It arrives everywhere at the same time.

What I write about here#

That is the throughline, and it still runs through everything on this site. I build systems where the safe behavior is the structural default rather than a rule someone has to remember, and I write about how that goes, honestly, including the parts that fail.

These days the systems are my own. A finance engine that grades stocks on rules I can inspect, a playground of interactive market visualizations, and a running account of what it is like to build and ship software this way, with an AI agent doing the typing. The domain moved from enterprise infrastructure to a small fleet of my own products. The question did not: how do you build a system you can trust, and how do you prove that rather than hope it?

What is different now is that I can show the receipts: the code, the dated track record, the run where it broke, so you can check that the safe path held instead of taking my word. If you have ever watched a one-line mistake page an entire fleet, we will get along.

Eric Caskey