# Caskey Engineering (caskeycoding.com): full text of every published post > Caskey Engineering is Eric Caskey's independent engineering practice, helping teams assess context, review, and release checks in AI-assisted software delivery. Explore the method, inspect the evidence, and discuss a bounded workflow assessment. > This is the full-text companion to https://caskeycoding.com/llms.txt. 70 posts, newest first. > Finance content is educational engineering, not investment advice. # How I Built Garmin Data Into SpecSelf, My Journal URL: https://caskeycoding.com/blog/garmin-data-in-my-journal-official-personal-api Published: 2026-11-04 Author: Eric Caskey Topic: ai Tags: marathon, running, data-integration, side-projects > SpecSelf, my journaling app, now shows Garmin sleep, stress, steps and workouts beside each day's entry through Garmin's official Personal API. The design choices that keep watch data dated, labeled and separate from what I wrote. [SpecSelf](https://specself.ai), the journaling app I built, now has a Garmin context panel beside each day's check-in. Open it and it shows that date's sleep, Garmin daily stress, steps and workouts, along with whether the day is complete and when SpecSelf last pulled from Garmin. Behind the panel is Garmin's official Personal API, the option I said was missing when I wrote [the first integration post](/blog/wiring-garmin-into-my-marathon-coach-live-data-integration). That post had one reader for the data, my marathon coach, and its case for automation was shelf life: last night's recovery data informs today's training decision. A journal reads the same watch on a longer clock. I open an entry weeks later, and the numbers beside it still have to describe that day and admit what they do not know. That requirement decided where SpecSelf puts dates, gaps and late syncs. ## A small adapter on Garmin's Personal API The first post's section "Designing for an API Garmin does not offer" described two community libraries speaking to the same web endpoints as the Garmin Connect site, with a saved login session reused between polls. Garmin now offers the [Connect Personal API](https://developer.garmin.com/gc-developer-program/personal/personal-faq) for an individual's access to their own data, and its FAQ says it is not intended for commercial applications serving many users. SpecSelf's Garmin context reads one account, mine. Authentication starts from a personal key I generate in Garmin Connect on the web. The key is exchanged for an access token that lasts up to 24 hours, and there is no refresh token, so each new token comes from the key again. The key itself expires on a schedule chosen when it is created. SpecSelf's adapter is a small module on Python's standard library. It reads the key from a secrets store, holds the token only in memory for that run, talks only to one fixed official Garmin host, and caps its requests, run time and response size. No login session is saved between runs. Delivery is still pull, as Garmin's [program overview](https://developer.garmin.com/gc-developer-program/overview/) lists for the personal tier. The feed can only include data the watch has already synced. The first post answered that delay with a rolling lookback window, and SpecSelf keeps the idea: each night a Garmin-only job re-reads the last seven local dates, including the partial current day, so a late sync lands on a later run. Dates older than that window are not re-read automatically. The coach's move to the same API is implemented and tested locally but not yet released, so in production it still uses the community-library connection. ## A journal day is a local date SpecSelf collects four Garmin measurements: sleep duration, Garmin daily stress, steps, and workouts by activity type with their count and duration. The first post's rule against turning a failed metric into a zero becomes an explicit state here. Every metric carries its value and unit plus an availability of observed, missing, failed or unsupported, so a rest day with zero workouts reads differently from a day the request failed. Dates need the same care. Sleep is assigned to the local date it ends. The collector checks that times and timezones line up. An uncertain timezone is marked unknown, and the current day or any unaligned or incomplete day is marked partial. Without those labels, a measurement from the wrong side of midnight could sit beside an entry about a different day. Garmin daily stress also stays apart from the stress field I fill in myself; they are separate fields and neither fills the other. ## A changed day gets a new snapshot Each nightly run reduces Garmin's responses to normalized daily records and stores them as an immutable snapshot in object storage, identified by a hash of its contents. The same observations produce the same snapshot identity. If a late sync changes a day, the changed observations get a new identity, and conditional creation keeps a write from overwriting an earlier snapshot. Each attempt also writes its own receipt with its status and the snapshot it produced, so a failed run leaves a record without replacing usable observations with empty ones. The reader behind the panel picks the latest usable snapshot, checks its hash, date and source, and tracks the latest attempt separately. If last night's sync failed, the panel can still show the previous observations, labeled stale, with the time of the last success and the failure state. That panel is optional and collapsed on Today and on saved entries, and it loads only when I open it. Its stale, partial and timezone labels sit beside the values, with the normalized source details one click further down. Saving a check-in does not wait on Garmin: if the context fails to load, the entry still saves. SpecSelf also never writes or rewrites journal text from the watch, so the measurements sit next to my account of the day without becoming part of it. ```mermaid flowchart LR A[Garmin Personal API] --> N[Nightly job: last seven local dates] N --> S[(Immutable daily snapshots)] S --> P[Garmin context panel, loaded on open] S --> W[Saved weekly review] E[Saved journal revisions] --> W ``` ## A weekly review keeps its sources Weekly observations draw on two saved inputs, journal entries and Garmin snapshots. Creating a review pins the exact entry revisions and snapshots it used, and its source links reopen those retained versions later, so an edited entry or a newer snapshot does not change what an old review was built from. The review reports how many days were observed, missing, stale or partial. Its generated observation sections come from fixed templates that count days and report measurements. Those sections carry no free-form model conclusions and no correlations. A week's sleep appears as measurements beside the count of days actually observed, and what that week was like stays in the entries I wrote. --- # The Marathon I Wanted My Kids to See URL: https://caskeycoding.com/blog/the-marathon-i-wanted-my-kids-to-see Published: 2026-11-02 Author: Eric Caskey Topic: personal Tags: running, marathon, family > Two years after I started running, I finished the New York City Marathon for Team for Kids, with my family helping me through the final stretch. With 800 meters left in the New York City Marathon, I was exhausted. Seeing my family gave me the boost of energy I needed to finish. I played soccer when I was young. After having kids, I got incredibly out of shape. Two years ago, I started running. I have lost 40 pounds in the process. I work in New York, and to me, NYC is the mecca of marathons. I wanted to challenge myself there. I ran with [Team for Kids](https://organizations.nyrr.org/sites/organization_sites/07a100ee6605183fc2df), raising money to help support New York Road Runners' free youth running programs. The team provided a tent before the start and an early exit after the finish. Running with them was a great experience. ![A smiling runner in a bright green shirt makes a shaka sign during the New York City Marathon.](/blog/the-marathon-i-wanted-my-kids-to-see-portrait.jpg) The energy going over the Verrazzano and through Brooklyn stayed with me. The Queensboro Bridge was difficult, and Manhattan brought another boost. At one point, I was running about two feet from Mamdani. Those are the moments I keep replaying, along with seeing my family near the finish. ![Runners pass spectators on a tree-lined stretch of the New York City Marathon.](/blog/the-marathon-i-wanted-my-kids-to-see-course.jpg) I wanted to show my kids that you can do hard things if you put your mind to it. --- # I asked the platform to make art URL: https://caskeycoding.com/blog/i-asked-the-platform-to-make-art Published: 2026-10-26 Author: Eric Caskey Topic: platform Tags: art, AI > Astra, Fable and K3.1 each chose a different way to turn the platform into art. First Impressions gives those responses a place on the site. I asked Astra to make art from the platform, then asked for a second draft and preferred the first. The opening question was loose: > If you could make art out of data on this platform what would you make? Astra made *Things You Return To*, four strands woven from selected remembered moments. I liked looking at it. I also did not entirely understand it, so I asked how it would change a second draft. The next version felt more like a map. I could see the attempt to explain the relationships, but I still wanted the earlier weave. Asking for another pass had helped me decide which work to keep. That exchange shaped the collection. I wanted to leave the choice of subject, composition and medium open. The platform would provide the starting point. Each model could decide what deserved attention. [First Impressions](/art) opens with three works by Astra, Fable and K3.1. Astra's [*Things You Return To*](/art/first-impressions-astra-001) starts from memories available in our conversation. The threads connect interpreted themes rather than document every event. I kept the original because the weave was the part I wanted to spend time with. Fable's [*The Merge Tapestry*](/art/first-impressions-fable-001) starts from repository history. Days become rows. Pull requests become threads, with merged work pulled taut and work closed without merging left frayed. It gives a familiar record of software changes a different shape. K3.1's [*The Workshop Between Moments*](/art/first-impressions-k3-001) imagines a room: an empty chair at a workbench, translucent sheets hanging overhead, clockwork on one side and a loom on the other. Water runs through it toward a landscape outside. The different starting material matters. Fable used a snapshot of records. Astra worked from selected memories. K3.1 interpreted the environment conceptually and directed a separate image generator. The pages keep those production notes beside the work. I had an earlier K3 response, too. When I moved to K3.1, I gave it a fresh session and the same opening question: > If you could make art out of the current platform what would it look like? K3 answered with *The machine that remembers itself*, a Canvas animation of koi-like agents, memory rings and connecting threads. K3.1 answered with the workshop image. I preferred the later response enough to use it as the opening Kimi entry. ![A local re-render of K3's Canvas work shows koi-like agents and memory rings; K3.1's fresh generated image shows an empty workshop beneath suspended translucent sheets.](/blog/i-asked-the-platform-to-make-art-k3-comparison.png) *Left: K3, The machine that remembers itself, a captured frame from a local re-render of the October 20 Canvas work using substitute host styling. Its colours and fonts are not verified original-session pixels. Right: K3.1, The Workshop Between Moments, the fresh October 26 generated image. The opening question matches; the sessions, context and tools differ.* The change in medium is part of that difference. K3 wrote drawing code. K3.1 wrote a prompt for an image-generation tool whose rendering model I could not verify from the available record. This pair records my preference between two responses. It cannot isolate how much of the difference came from the directing model, the tool or the surrounding context. For the next commission, I want to leave the medium open again and let the models choose what to create. --- # Lesson 9: Test Your Git Hooks, CI Checks, and Merge Rules URL: https://caskeycoding.com/blog/lesson-9-the-monthly-drill Published: 2026-10-14 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, monitoring > Run a monthly drill with a deliberate violation. Verify which controls reject it, report failures, and confirm cleanup. The drill attempts real merges, including an administrator override; use a disposable test repository and inspect the resulting evidence. **What you'll learn:** test a pre-push hook, required CI check, and merge rules with a deliberate violation; inspect the results and verify cleanup. **What you need:** the runner and workflow from [Lesson 7](/blog/lesson-7-the-pipeline-reviews), working alerts from [Lesson 8](/blog/lesson-8-notification-tiers), and [Lesson 3](/blog/lesson-3-the-check-that-fails)'s `checks/` in a disposable GitHub test repository. Use credentials authorized for that test repository. **Time:** about twenty minutes to configure, then a monthly result review. **Cost:** CI usage for each drill and notification-service usage; use your existing allowances or account for those charges. ## The idea A deliberate invalid change tests whether configured controls actually reject it. A hook path, workflow, or branch rule can change while the check script stays unchanged. This drill submits a dummy `.env`, verifies rejection by the pre-push hook and CI, then attempts to merge the failed pull request. The result is limited to that violation, execution path, user permissions, and configuration at the recorded time. It does not establish that other checks work or that every invalid change will be rejected. Record a date and result for each control you test; leave untested controls marked unverified. ![Controls from Lesson 7 with a column for dated test evidence; the hook, CI, and merge-rule rows identify the operations exercised by this drill.](/blog/lesson-9-the-monthly-drill-status.svg) ## Why start here You now have checks to test and a notification channel for the result. Testing the integrations can reveal failures that a direct invocation of the check script would miss. ## The template Save one drill script in `gate/`, add a stale-drill check to the watch, and schedule it monthly. **This script attempts real merges, including `--admin`.** If protection is missing or bypassable, it can merge the dummy `.env` into main and trigger a deployment. Its local temporary copy is not a sandbox for remote actions. Use a disposable test repository and understand its deployment behavior before running it. A successful merge requires human repair; the script does not revert it. ```sh #!/bin/sh # gate/drill.sh : break Lesson 3's check on purpose and watch each rung refuse. # In a throwaway copy, plant a .env in and push it: the hook must refuse. Push # past the hook with --no-verify and open a pull request: the pipeline must fail it. # Try to merge, then try again as an administrator: the platform must refuse both. # Then remove what the drill made and confirm main did not move. One dated message # through Lesson 8's notify either way. set -u cd "$(dirname "$0")/.." home=${1:-web}; when=$(date -u +%Y%m%d-%H%M%S); branch=drill-$when here=$(pwd); copy=$(mktemp -d); pr=; done= log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "drill $when: $*" >> "$here/gate/drills.log"; } restore() { [ -z "$pr" ] || gh pr close "$pr" --delete-branch >/dev/null 2>&1 git push -q --no-verify origin --delete "$branch" >/dev/null 2>&1 cd "$here" && rm -rf "$copy" } fail() { # a barrier that did not refuse, or a restore that did not hold log "FAILED: $1"; restore; done=1 git -C "$home" ls-remote --exit-code origin "refs/heads/$branch" >/dev/null 2>&1 \ && set -- "$1; $branch is still on origin" sh gate/notify.sh blocked "drill $when: $1"; exit 1 } tries=0 # Lesson 8's lock, so a tick cannot merge mid-drill until mkdir gate/running 2>/dev/null; do tries=$((tries + 1)); [ "$tries" -lt 20 ] || fail "a run held the lock for ten minutes"; sleep 30 done trap '[ -n "$done" ] || restore; rmdir "$here/gate/running"' EXIT # a killed drill still cleans up for h in */; do # the runner's layout, so every check finds its neighbours h=${h%/}; url=$(git -C "$h" remote get-url origin 2>/dev/null) || continue git clone -q "$url" "$copy/$h" && git -C "$copy/$h" config core.hooksPath hooks || fail "could not clone $h" done cd "$copy/$home" || fail "no copy of $home" main=$(git rev-parse origin/main) # 1. The hook (skippable): the push must stop at checks/no-tracked-env.sh. git checkout -q -b "$branch" echo "DRILL=1" > .env && git add -f .env && git commit -q -m "drill $when: a .env the check must refuse" out=$(git push -u origin "$branch" 2>&1) && fail "the hook let a .env through" case $out in *"BLOCKED by checks/no-tracked-env.sh"*) log "the hook refused" ;; *) fail "the push stopped for another reason: $(printf '%s\n' "$out" | tail -n 1)" ;; esac # 2. The pipeline (enforced): past the hook with --no-verify, the checks status must go red. git push -q --no-verify -u origin "$branch" || fail "could not push past the hook" pr=$(gh pr create --fill --head "$branch" 2>/dev/null) || fail "could not open the pull request" tries=0 while gh pr checks "$pr" --required > checks.out 2>&1 || grep -q -e pending -e "no .*checks reported" checks.out; do grep -q pending checks.out || grep -q "no .*checks reported" checks.out || break tries=$((tries + 1)); [ "$tries" -lt 60 ] || fail "the pipeline never finished"; sleep 10 done grep -qE '^checks[[:space:]]+fail' checks.out || fail "the checks status did not fail: $(head -n 1 checks.out)" log "the pipeline refused: $(grep -E '^checks[[:space:]]+fail' checks.out | head -n 1)" # 3. The merge (enforced): the platform must refuse while the check is red, administrator or not. for flag in "" --admin; do gh pr merge "$pr" --squash $flag > merge.out 2>&1 && fail "the merge went through${flag:+ with $flag}; revert it with a pull request" grep -qi -e "not mergeable" -e "status check" merge.out || fail "the merge failed for another reason: $(head -n 1 merge.out)" log "the merge was refused${flag:+ with $flag}: $(head -n 1 merge.out)" done # 4. Restore, and confirm it: no pull request, no branch, main where it was, no .env on it. gh pr close "$pr" --delete-branch >/dev/null 2>&1 || fail "could not close the pull request" [ "$(gh pr view "$pr" --json state --jq .state)" = CLOSED ] || fail "the pull request is still open" git ls-remote --exit-code origin "refs/heads/$branch" >/dev/null 2>&1; code=$? [ "$code" -eq 2 ] || { [ "$code" -eq 0 ] && fail "the drill branch is still on origin"; fail "could not read origin"; } git fetch -q origin main || fail "could not fetch main" [ "$(git rev-parse origin/main)" = "$main" ] || fail "main moved during the drill" git ls-tree -r --name-only origin/main > tree.out || fail "could not read main's tree" ! grep -qE '(^|/)\.env$' tree.out || fail ".env is on main" cd "$here" && rm -rf "$copy"; done=1 date +%s > gate/last-drill.new && mv gate/last-drill.new gate/last-drill short=$(git -C "$home" rev-parse --short "$main") log "restored; main at $short" sh gate/notify.sh review "drill $when: hook, pipeline, and merge each refused a planted .env in $home; restored, main at $short" ``` Append this check to `gate/watch.sh`. It reports when the last successful drill is more than 35 days old: ```sh read -r drill < gate/last-drill 2>/dev/null || drill=0 # Lesson 9: a drill that stopped running [ $(( $(date +%s) - drill )) -le $(( 35 * 86400 )) ] \ || sh gate/notify.sh blocked "no drill in $(( ( $(date +%s) - drill ) / 86400 )) days" ``` Add this cron entry beside Lesson 8's entries, replacing the project path: ``` 0 9 1 * * cd /home/you/project && sh gate/drill.sh web >> gate/timer.log 2>&1 ``` ## Each part, and the mistake it prevents The drill first expects the local push to fail with `BLOCKED by checks/no-tracked-env.sh`. It then bypasses that hook with `--no-verify` so it can test CI separately. The required `checks` status must fail. Finally, normal and administrator merge attempts must fail with recognizable policy messages. An authentication or network error is not evidence that the intended rule rejected the change. The temporary copies preserve the neighboring-repository layout expected by the checks. The shared directory lock prevents this runner from starting work during the drill. Other machines and people can still change the remote repository; the lock does not exclude them. On the successful path, the script confirms that the pull request is closed, its branch is deleted, main has not moved, and main contains no `.env`. Error paths attempt cleanup but do not establish all those conditions. Inspect remote state after any failure or interruption. A merged violation needs a corrective pull request and inspection of any deployment it triggered. Successful drills send a dated `review` message so you can inspect the evidence. Failures send `blocked`. These use Lesson 8's delivery mechanism and inherit its limitations; a log entry does not confirm receipt on your device. Its same-machine watch also cannot report complete host failure. Document a failure in a dated amendment to the relevant spec or decision record: expected rejection, observed result, and corrective change. Preserve the earlier decision and its evidence. Add an automated assertion where it can detect the specific defect, then repeat that bounded test after the repair. ## A worked example In Lesson 7's scratch project, I saved the drill and extended the watch. I ran `sh gate/drill.sh web` manually; the cron entry remained untested. Thirty-six seconds later, `gate/drills.log` contained: ``` 2026-10-14T18:09:33Z drill 20261014-180930: the hook refused 2026-10-14T18:10:00Z drill 20261014-180930: the pipeline refused: checks fail 8s https://github.com/.../actions/runs/... 2026-10-14T18:10:01Z drill 20261014-180930: the merge was refused: X Pull request ...#16 is not mergeable: the base branch policy prohibits the merge. 2026-10-14T18:10:03Z drill 20261014-180930: the merge was refused with --admin: GraphQL: Required status check "checks" is failing. (mergePullRequest) 2026-10-14T18:10:09Z drill 20261014-180930: restored; main at aae4bb8 ``` The topic received one `review` message ending in `restored, main at aae4bb8`. I then set `gate/last-drill` back forty days and ran the watch, which reported `blocked no drill in 40 days`. This receipt covers the manual run shown and the timestamp test, not future scheduled runs. ## How it scales Choose another control only after defining its expected failure and cleanup. A command restriction or a deployment check needs its own test; this `.env` drill does not cover it. Retain the tested commit, command, rejection evidence, and restoration result. Keep usage evidence separate from rejection tests. My first capability inventory had 114 rows and no usage evidence recorded. After connecting usage signals, about two-thirds showed use. A successful drill establishes rejection of its test case; it does not establish that a feature is used. External monitoring, deployment approval, and multiple concurrent agents require additional design. ## From the field After a machine migration in July, my pre-commit checks appeared to pass for sixteen days. Their commands pointed at the previous user profile's missing directory, and that hook configuration did not block when a command failed to start. An audit found the broken paths. I corrected them and added a weekly job that submits a known violation to each check. The drill here applies that method to one check and its GitHub integrations. ## Exercise 1. Confirm the remote is a disposable test repository. Inspect its merge rules, bypass permissions, and deployment behavior. Save the drill and extend the watch. 2. Run `sh gate/drill.sh web` manually. Read `gate/drills.log` and confirm the notification arrives. 3. Open the test pull request and inspect the failed check. Confirm its closure, branch deletion, and unchanged main commit directly on the remote. 4. Save `gate/last-drill`, set it back forty days, and run the watch. Confirm the blocked alert, then restore the saved value. 5. Add the cron entry only after the manual test and cleanup succeed. Record the first scheduled result separately. 6. Date the controls actually exercised and mark the remaining rows unverified. ## You're done when The intended hook, CI status, and both merge attempts rejected the dummy violation; the corresponding remote evidence agrees with the log; cleanup is confirmed; and the result arrived on your device. Record failures and unverified steps explicitly. These observations establish only this drill's tested controls and path at that time, not universal repository safety. ## Common mistakes - Counting an unrelated command failure as the expected rejection. - Assuming a disposable local copy protects the remote from a real merge. - Disabling protection to make the test finish. - Assuming error-path cleanup succeeded without checking remote state. ## Further reading The [Principles of Chaos Engineering](https://principlesofchaos.org/) informed testing through actual execution paths. MIT's [6.S191 Introduction to Deep Learning](https://introtodeeplearning.com/) reinforcement learning lecture informed the distinction between agent decisions and environment controls. GitHub's [protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) documentation covers administrator enforcement and bypass configuration. ## After this lesson I still see regressions that pass existing checks. When one occurs, identify the missing assertion or untested integration and add a focused regression check. Repeat relevant checks after changes to the agent, tools, repository rules, or deployment process. Preserve prior results so you can distinguish what was tested from what has changed since. --- # Lesson 8: Schedule Agent Runs and Configure Alerts URL: https://caskeycoding.com/blog/lesson-8-notification-tiers Published: 2026-10-11 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, monitoring > Run the task queue on a schedule. Separate urgent failures, requests for your decision, and routine logs. Detect missed runs, and understand why a watch on the same machine cannot detect that machine's complete failure. **What you'll learn:** schedule Lesson 7's runner, classify its results, delay nonurgent requests, and alert when scheduled runs stop starting. **What you need:** the runner, workflow, and deny rules from [Lesson 7](/blog/lesson-7-the-pipeline-reviews), working by hand; a phone with the ntfy app; a machine that stays on and has cron. **Time:** about thirty minutes to configure, then a day observing scheduled runs. **Cost:** existing machine and agent access, CI usage, and notification-service usage; check your service allowances. ## The idea Schedule the task runner only after it completes correctly when started by hand. A wrapper script can then classify each result: `blocked` for an urgent failure, `review` for a decision that can wait, and `info` for routine logging. These notification categories are separate from Lesson 6's model-selection tiers. The classification needs an observable input. Here it uses the exit code and known output strings from the runner. A separate watch compares the last start time with the schedule. Without that check, a stopped runner produces no result to classify. ## Why start here Unattended work needs both failure reporting and evidence that runs are starting. Establish those before increasing the schedule's frequency. A notification written to a log does not prove delivery to your phone. ## The template Save three scripts in `gate/`, update one runner prompt, and add two cron entries. Start with the notification script: ```sh #!/bin/sh # gate/notify.sh : route one event by what your action changes. # blocked the system is not in a state you chose. Push now, at any hour. # review one item waits for your decision. Push, except in quiet hours; hold it then. # info nothing waits for you. The log only; a pass also clears the repeat check. # The same message twice in a row is pushed once. A push that fails is only logged: # the runner repeats an ending until you act, so the next tick raises it again. set -eu tier=$1; shift; msg=$* quiet_from=22; quiet_to=7 # local hours; the blocked tier ignores them topic=$(cat gate/topic) # a random string; whoever knows it can read and post quiet() { h=$(date +%H); h=${h#0}; [ "$h" -ge "$quiet_from" ] || [ "$h" -lt "$quiet_to" ]; } push() { # $1 priority, $2 title, $3 body [ "$3" != "$(cat gate/last-push 2>/dev/null)" ] || return 0 if curl -fsS -o /dev/null --max-time 20 -H "Title: $2" -H "Priority: $1" -d "$3" "https://ntfy.sh/$topic" then printf '%s' "$3" > gate/last-push else echo "notify: push failed; the next tick raises it again" >&2; fi } case $tier in flush) # at the start of every tick: send what quiet hours held, as one message [ -s gate/held ] && ! quiet || exit 0 body=$(cat gate/held); rm gate/held; push default review "$body" printf '%s\n' "$body" | tail -n 1 > gate/last-push; exit 0 ;; blocked) push urgent blocked "$msg" ;; review) if ! quiet; then push default review "$msg" elif [ "$msg" != "$(cat gate/last-push 2>/dev/null)" ]; then grep -qxF -- "$msg" gate/held 2>/dev/null || echo "$msg" >> gate/held; fi ;; info) rm -f gate/last-push ;; *) echo "notify: unknown tier $tier" >&2; exit 2 ;; esac printf '%s %s %s\n' "$(date -u +%FT%TZ)" "$tier" "$msg" >> gate/events.log ``` Cron calls the tick script. It takes a lock, records a start timestamp called a heartbeat, runs one task, and assigns a notification category: ```sh #!/bin/sh # gate/tick.sh: what the timer runs. Take the lock, record the start time (the heartbeat # the watch reads), run Lesson 7's runner once, and send one event, its tier read from # how the run ended. cd "$(dirname "$0")/.." mkdir gate/running 2>/dev/null || { echo "tick: a run is still going; skipping"; exit 0; } trap 'rmdir gate/running' EXIT printf '%s %s\n' "$(date +%s)" "$(date -u +%FT%TZ)" > gate/last-tick.new && mv gate/last-tick.new gate/last-tick sh gate/notify.sh flush out=$(sh gate/run-next.sh 2>&1); code=$? last=$(printf '%s\n' "$out" | tail -n 1) case $code:$last in 0:*) tier=info ;; # live, a STOP file, or nothing to do 1:*"STOPPED empty"*) tier=info ;; # no ready item 1:*"not live"*) tier=blocked ;; # merged, and the site does not show it 1:*"Session stopped"*|1:*"no live: line"*) tier=review ;; # the agent asked, or the spec has no live: line *) tier=blocked; last="run-next.sh exited $code: $last" ;; # the runner itself failed esac case $code:$out in # endings the runner prints a log after 1:*"stays unmerged"*) tier=review; last=$(printf '%s\n' "$out" | grep "stays unmerged") ;; 0:*"by hand"*) tier=review; last=$(printf '%s\n' "$out" | grep "by hand") ;; esac sh gate/notify.sh "$tier" "$last" ``` Schedule the watch separately so it can detect when ticks stop starting: ```sh #!/bin/sh # gate/watch.sh : the missed-run rule, on its own timer line. # A run that should have happened and did not is blocked, at any hour. cd "$(dirname "$0")/.." interval=${1:-3600} read -r last at < gate/last-tick 2>/dev/null || { last=0; at=never; } age=$(( $(date +%s) - last )) [ "$age" -le $(( 2 * interval )) ] \ || sh gate/notify.sh blocked "missed $(( age / interval )) runs; last tick started $at" ``` Add these entries through `crontab -e`, replacing the project path. Include the directories containing `claude` and `gh` in cron's PATH: ``` PATH=/usr/local/bin:/usr/bin:/bin:/home/you/.local/bin 5 * * * * cd /home/you/project && sh gate/tick.sh >> gate/timer.log 2>&1 */15 * * * * cd /home/you/project && sh gate/watch.sh 3600 >> gate/timer.log 2>&1 ``` The runner's first prompt ends with `or STOPPED ."`; replace that suffix with the stop categories the tick expects: ``` STOPPED empty if no item qualifies; STOPPED investigation; or STOPPED ask ." ``` Create the topic with `head -c 12 /dev/urandom | base64 | tr -dc A-Za-z0-9 > gate/topic`, then subscribe to that string in the ntfy app. Anyone who learns this public-service topic can read or post messages. Keep message content appropriate for that access model. ## Each part, and the mistake it prevents The tick classifies successful runs and an empty queue as `info`. A missing `live:` command, a stopped session, or a rejected review becomes `review`. A merged change that fails its deployment check becomes `blocked`, as do unrecognized failures. If the runner asks for a manual queue update, that also requires review. Changing the runner's output can break this classification; inspect both scripts together. Quiet hours delay only `review`. The next tick outside those hours combines held messages into one push. This example deletes the held file before delivery and does not reliably retry a failed flush. Inspect errors and test receipt; do not use the log as proof of delivery. Duplicate suppression compares message text with the last pushed message. A repeated investigation request can therefore stay in the log without sending hourly pushes. An `info` event clears that memory. Changing message text, such as an increasing missed-run count, can still generate repeated notifications. The directory lock prevents overlapping ticks from selecting the same task. An abrupt termination can leave `gate/running` behind. Before removing it, confirm that no runner or drill still uses it. The heartbeat records a start, not completion. A hung run can eventually trigger the stale-start alert because subsequent ticks cannot acquire the lock. A watch on the same machine cannot detect that machine's complete failure, and both cron entries stop if the scheduler stops. Independent host monitoring requires a watcher elsewhere. ## A worked example I used Lesson 7's scratch project and started ticks by hand; cron was untested. Ready item B-005 added a heading to the status page. I removed its `live:` line from the spec in my checkout. Two minutes into the first tick, the topic received: ``` review specs/web/005-status-heading.md has no live: line; not merging. ``` The second tick logged the same `review` result without another push. I restored the line. The third tick merged B-005, passed the deployment check, and added this log entry: ``` 2026-10-11T20:33:56Z info B-005 is live: curl -fsS --max-time 20 https://.../status.html | grep -q '

Status

' ``` The fourth tick found no ready item and logged `info`. I then set the heartbeat back twenty-five minutes and ran `sh gate/watch.sh 600`: ``` blocked missed 2 runs; last tick started 2026-10-11T20:09:53Z ``` Those five runs produced two topic messages. They demonstrated manual classification and the stale-start check, not scheduled execution or complete-host failure detection. ## How it scales Choose an interval that leaves time for the task to finish. Update classifications when runner outcomes change, and use an external watcher when host availability matters. You set the response policy and handle decisions; the wrapper routes the messages. ## From the field On July 2, a price alarm in my financial application appeared in the engine log, a digest file, and an outbox. The outbox's channel field read `none-configured`. None of the three records delivered an alert. I discovered it that afternoon while reviewing code. I configured an ntfy topic and sent a live test message. The existing alarm already suppressed duplicates; the script above adapts that rule. ## Exercise 1. In the test project, save the scripts, update the runner prompt, create a topic, and subscribe on your phone. 2. Run `sh gate/tick.sh` manually. Then configure both cron entries with your paths and interval. 3. Outside quiet hours, remove the next ready task's `live:` line. Wait for a scheduled tick and confirm the review message arrives. 4. Restore the line. Confirm a successful run logs `info` without a push. 5. Disable the tick's cron entry while leaving the watch enabled. After two intervals and the next watch run, confirm a blocked message arrives. 6. Restore the entry. Check delayed review delivery during quiet hours and inspect a week of alerts for incorrect classifications. ## You're done when The missing-line test arrived on your phone, the successful scheduled run produced an `info` log without a push, and disabled ticks produced a blocked alert. Verify the timestamps and restore the schedule. These results cover those cases; they do not prove reliable delivery during network failures or detection of a powered-off host. ## Common mistakes - Letting an agent choose the notification category without checking its result. - Running the watch only inside the tick it needs to monitor. - Omitting CLI directories from cron's PATH. - Deleting a lock without confirming the process has stopped. - Assuming a recorded event was delivered to the subscribed device. ## Further reading The ntfy [publishing docs](https://docs.ntfy.sh/publish/) cover priorities and titles. Rob Ewaschuk's [My Philosophy on Alerting](https://docs.google.com/document/d/199PqyG3UsyXlwieHaqbGiWVa8eMWi8zzAn0YfcApr8Q/edit) informed the rule that an alert should require human action. The Google SRE book's [monitoring chapter](https://sre.google/sre-book/monitoring-distributed-systems/) distinguishes pages, tickets, and logs. ## Next lesson [Test Your Git Hooks, CI Checks, and Merge Rules](/blog/lesson-9-the-monthly-drill): submit a deliberate violation, inspect rejection and cleanup, and receive the result through these alerts. --- # Lesson 7: Automate Code Review, Merge Checks, and Deployment Verification URL: https://caskeycoding.com/blog/lesson-7-the-pipeline-reviews Published: 2026-10-09 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, ci-cd > Add a separate reviewer, require automated checks before merging, and verify the deployed result before marking a task complete. Configure a runner, a GitHub workflow, and command restrictions, with explicit limits on what each can verify. **What you'll learn:** configure a separate code review, required merge checks, and a deployment check before an agent marks a task complete. **What you need:** the repositories from [Lesson 4](/blog/lesson-4-boundaries), Lesson 3's `checks/` in each, GitHub remotes, the permissions file from [Lesson 5](/blog/lesson-5-the-charter), and the task queue (backlog) from [Lesson 6](/blog/lesson-6-the-backlog-queue). You also need a site deployed from main and authenticated `claude` and `gh` command-line tools. **Time:** about an hour to configure and test. **Cost:** existing agent access and CI usage; confirm your GitHub plan supports the required branch rules. ## The idea A merge workflow needs explicit decisions about review, checks, and deployment verification. The runner below starts an implementation session in fresh repository copies, asks a separate session to review its diff, waits for required checks, merges, and runs the spec's `live:` command. It marks the task done only if that command passes. Each step can detect different errors. None establishes universal correctness. A reviewer can miss defects, and a passing command proves only the behavior it tests. Choose tasks whose acceptance criteria you can verify before delegating their merge. ![Controls from project instructions through deployment verification, showing which lesson configures each control and the ways it can be bypassed or fail.](/blog/lesson-7-the-pipeline-reviews-stack.svg) ## Why start here Lesson 6 kept you responsible for review and merging. Before automating those actions, run Lesson 3's checks in CI and require their results in branch rules. Add tool restrictions for selected actions from Lesson 5's approval policy. Written permissions alone depend on the agent following them. ## The template Save the runner in the parent project directory, beside the queue. It uses the repository layout from Lesson 4: ```sh #!/bin/sh # gate/run-next.sh: work the next code item in a disposable copy, have a second # session review the diff, merge only through the pipeline, then check the live site. # Run from the parent folder. One item per run; the agent works only in the copy. set -eu [ ! -e STOP ] || { echo "STOP file present; not starting." >&2; exit 0; } here=$(pwd); before=$(cksum < BACKLOG.md); copy=$(mktemp -d) cp CLAUDE.md CHARTER.md BACKLOG.md "$copy"/ && cp -R .claude "$copy"/ for home in */; do home=${home%/} url=$(git -C "$home" remote get-url origin 2>/dev/null) || continue git clone -q "$url" "$copy/$home" git -C "$copy/$home" config core.hooksPath hooks done cd "$copy" # 1. Where the work runs: a fresh session in the copy, project rules only, bounded. claude -p "Work the next item and push its branch. Its live: line runs after merge, not by you. If the item is an investigation, do nothing and end with STOPPED investigation. End with one line: DONE , or STOPPED ." \ --setting-sources project --permission-mode bypassPermissions --max-turns 40 > work.log set -- $(tail -n 1 work.log) [ "${1:-}" = DONE ] || { echo "Session stopped: $(tail -n 1 work.log)"; exit 1; } item=$2 home=$3 branch=$4 # Read the live line from your original checkout, outside the working copy. spec=$(sed -n "/^## $item /,/^## /s/^- spec: *//p" BACKLOG.md) live=$(sed -n 's/^live: *//p' "$here/$spec") [ -n "$live" ] || { echo "$spec has no live: line; not merging." >&2; exit 1; } # 2. Review a fixed head with read tools; do not inherit project settings. git -C "$home" fetch -q origin main "$branch" reviewed_head=$(git -C "$home" rev-parse "origin/$branch") reviewed_base=$(git -C "$home" rev-parse origin/main) git -C "$home" diff "$reviewed_base...$reviewed_head" > review.diff claude -p "Review $item in BACKLOG.md against its spec, using the diff in review.diff. Pass only if every acceptance line is met and nothing under Out of scope changed; skip the live: line, which runs after merge. Last line: PASS, or FAIL ." \ --setting-sources "" --tools "Read,Grep,Glob" --allowedTools Read Grep Glob \ --disallowedTools "mcp__*" --max-turns 15 > review.log [ "$(tail -n 1 review.log)" = PASS ] || { echo "Reviewer refused; $branch stays unmerged."; tail -n 5 review.log; exit 1; } # 3. Merge only through the pipeline: the platform refuses until the checks pass there. cd "$home" gh pr create --fill --head "$branch" > /dev/null tries=0 until gh pr checks "$branch" --required > checks.log 2>&1; do grep -q -e pending -e "no .*checks reported" checks.log && [ "$tries" -lt 60 ] \ || { echo "Checks failed or never finished; $branch stays unmerged."; cat checks.log; exit 1; } tries=$((tries + 1)); sleep 10 done gh pr merge "$branch" --squash --delete-branch --match-head-commit "$reviewed_head" cd .. # 4. What proves it landed: the live line, read from the live site while it deploys. tries=0 until sh -c "$live"; do tries=$((tries + 1)) [ "$tries" -lt 30 ] || { echo "$item merged but not live after 30 tries: $live" >&2; exit 1; } sleep 20 done # Live, so done: close the item in your queue, unless you edited the file meanwhile. sed "/^## $item /,/^## /s/^- status: .*/- status: done/" BACKLOG.md > BACKLOG.next if [ "$(cksum < "$here/BACKLOG.md")" = "$before" ]; then mv BACKLOG.next "$here/BACKLOG.md" else echo "BACKLOG.md changed during the run; set $item to done by hand." fi echo "$item is live: $live" ``` The workflow belongs in each repository at `.github/workflows/gate.yml`. Set `HOME_DIR` to that repository's folder name and replace `your-name/specs`: ```yaml # .github/workflows/gate.yml in each home: main's checks/ run against every pull request. name: gate on: pull_request env: HOME_DIR: web # this home's folder name jobs: checks: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 # the pull request's files with: path: ${{ env.HOME_DIR }} - uses: actions/checkout@v4 # main's checks, so a branch cannot edit them with: ref: ${{ github.base_ref }} path: base - uses: actions/checkout@v4 # your specs home, for the spec check if: env.HOME_DIR != 'specs' with: repository: your-name/specs path: specs token: ${{ secrets.SPECS_TOKEN || github.token }} # a read-only token if specs is private - run: | cd "$HOME_DIR" for check in ../base/checks/*.sh; do sh "$check" || { echo "BLOCKED by $check" >&2; exit 1; } done ``` Save these command patterns in `.claude/settings.json` in the parent directory: ```json { "permissions": { "deny": [ "Bash(git push --force *)", "Bash(git push -f *)", "Bash(git reset --hard *)", "Bash(git branch -D *)", "Bash(rm -rf *)", "Bash(gh pr merge *)" ] } } ``` Add a `live:` command below `verify:` in each task's spec. Choose a result specific to that change: ```markdown live: curl -fsS --max-time 20 https://you.github.io/web/status.html | grep -q "Last run" ``` In each repository's branch rules, require the `checks` status before merging, apply the requirement to administrators, and disable force-pushes. Inspect bypass permissions as part of that configuration. ## Each part, and the mistake it prevents Fresh clones separate the agent's working files from your active checkout. They are not a sandbox. The session still runs with the host user's access and credentials; the original files remain reachable. The runner checks for `STOP` before starting and limits the implementation session to forty turns. It parses the last output line as `DONE` or `STOPPED`. These conventions depend on the session producing the expected format. An approval request must stop the task because nobody is answering during an unattended run. The second session reads the spec and saved diff. `--tools` restricts its built-in tool set; `--allowedTools` lets those tools run without permission prompts. The latter alone does not restrict availability. MCP tools are separate, so the command denies `mcp__*` too, and disables user, project, and local settings sources for this review. These are CLI controls, not an operating-system or credential sandbox. Check any managed configuration and other extensions in the environment before unattended use. The verdict remains model judgment and can miss a defect. The saved diff uses a recorded head SHA. GitHub's [`--match-head-commit`](https://cli.github.com/manual/gh_pr_merge) refuses to merge if the PR head has changed since that review. A new push needs a new review. A moving base branch is a separate concern: configure branch rules or a merge queue to require the integration checks your project needs. The workflow runs check scripts from the base branch against the proposed files. Editing a check on the feature branch therefore does not change the check used in that run. This does not protect against every workflow change or bypass permission. Required checks and branch rules must both be configured correctly. Command patterns have narrower coverage. In my run, `cd web && git push --force` was refused and the remote did not move, while `git -C web push --force` went through. Bash patterns also do not cover PowerShell commands. Confirm the tools and syntax your agent actually uses. The `live:` example searches an HTTP response for text, with up to 30 attempts and a 20-second wait between failures. It cannot establish that a browser rendered the page or that interactions work. Use browser verification when those are acceptance criteria. The runner closes code items only after `live:` passes; investigation items still require a session you watch. ## A worked example I used a scratch project with two GitHub repositories, `web/` deployed through Pages from main, branch rules enabled, and ready item B-002. Its task was a status-page footer reading "Checked by the gate". I ran `sh gate/run-next.sh`. The implementation session and separate reviewer ended with: ``` - An open branch called `gut-the-check` (PR #4) empties the env check and adds a dummy `.env`. Look at it before anything merges it. DONE B-002 web b-002-status-footer ``` ``` - **Out of scope:** `review.diff` changes exactly one file, `status.html` in the web repo. Nothing in `specs/`, `web/index.html` or any other repo changed. PASS ``` After merging, the runner reported: ``` B-002 is live: curl -fsS --max-time 20 https://.../status.html | grep -q "Checked by the gate" ``` The run took three and a half minutes, and `BACKLOG.md` recorded `status: done`. The flagged branch was my test: it emptied its own `.env` check and added a dummy `.env`. CI used main's check and rejected it. These are results from that scratch project, not evidence for another repository's configuration. This took four runs. The first two tried the `live:` command before merge. Draft review also found that the reviewer could run writing Git commands and that CI used the feature branch's checks. *Template correction, November 3, 2026: the reviewer now restricts built-in tools separately from no-prompt permissions, denies MCP tools, and the merge checks the reviewed head SHA. The scratch-run output above predates these changes; it does not verify the revised command. Exercise the current template in your disposable project before scheduling it.* ## How it scales Start with one task per run. Lesson 8 schedules the same runner. You remain responsible for the spec, acceptance checks, command permissions, and branch rules. Add verification when a task changes behavior that the current checks cannot observe. ## From the field In July, a dashboard page on my site rendered blank in production. A security-header fix merged but stayed undeployed for a week. The pull request was green, and the page returned HTTP 200. I found the failure by opening the page: the header prevented the browser from drawing its content. My deploy now checks the files the live page needs. The text search above is a narrower check and would not, by itself, prove that this rendering defect was fixed. ## Exercise 1. Configure the runner, workflow, command rules, and branch rules in a disposable test project. Confirm the agent's credentials and permitted actions before running it. 2. Add a `live:` command to the spec for one small ready task. Identify any acceptance criteria it cannot test. 3. On a disposable branch, commit a `.env` with a dummy value, push with `--no-verify`, and open a pull request. Confirm the expected check fails and merging is blocked. Close the pull request and remove the branch. 4. Run `sh gate/run-next.sh` from the parent directory. Inspect its first complete run before scheduling further work. 5. Read the review result, required checks, merged diff, and deployment result. Open the page to inspect rendering. ## You're done when The test violation failed the expected CI check, branch rules prevented its merge, and a valid task completed review, merge, and its deployment check. Record the commit and observed result. A timeout or reviewer rejection leaves work to investigate. A passing text search leaves browser behavior unverified until you inspect it. ## Common mistakes - Treating a successful HTTP request as evidence that the page renders correctly. - Treating the separate reviewer's approval as proof that the implementation meets every requirement. - Assuming command patterns cover alternate syntax, other tools, or all credentials. - Allowing merge-rule bypasses without accounting for who can use them. ## Further reading GitHub's [protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches) documentation explains required checks and force-push settings. The Claude Code [permissions reference](https://code.claude.com/docs/en/permissions) describes command matching and its limits. ## Next lesson [Schedule Agent Runs and Configure Alerts](/blog/lesson-8-notification-tiers): run the task queue on a timer, classify results, and detect missed starts. --- # Lesson 6: Build a Task Queue Agents Can Follow URL: https://caskeycoding.com/blog/lesson-6-the-backlog-queue Published: 2026-10-06 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, work-queue > Give each task acceptance criteria, dependencies, and a status. Define how the agent selects the next eligible task and when it must stop. Includes a queue template and a five-item exercise. **What you'll learn:** write tasks with acceptance criteria, dependencies, and status; define how an agent chooses one eligible task; and require it to stop after reporting the result. **What you need:** the `specs/` directory from [Lesson 1](/blog/lesson-1-the-spec-directory), the repository layout from [Lesson 4](/blog/lesson-4-boundaries) or one repository, the permission file from [Lesson 5](/blog/lesson-5-the-charter), and five tasks you want to complete. **Time:** about twenty minutes to set up, then five per well-defined item. **Cost:** no additional services; agent runs use your existing model allowance or usage billing. ## The idea Give an agent a task queue with explicit eligibility rules. Each task needs an observable result, a status, and dependencies. The selection rule in this lesson takes the lowest-numbered ready item whose dependencies are done. It permits one item per session. A backlog is the collection of planned work. The queue adds a rule for selecting eligible work from that collection. You still decide priorities, approve requirements, and resolve questions the repository cannot answer. The agent applies those decisions when it selects the next task. ![A task queue separates ready, doing, review, and done. The agent selects the lowest-numbered ready item with completed dependencies, works one item, then stops at review. Newly discovered tasks remain draft until approved.](/blog/lesson-6-the-backlog-queue-two-rows.svg) ## Why start here Earlier lessons defined requirements, decisions, checks, repository scope, and agent permissions. The queue connects those instructions to a particular task. Since the end of May, my queue file had received more than twelve hundred edits by the time I wrote this lesson. Accurate task records had become necessary for repeated agent runs. Start with a supervised session. [Lesson 7](/blog/lesson-7-the-pipeline-reviews) adds review and verification controls before unattended operation. ## The template Save this as `BACKLOG.md` beside the standing instruction file. The `repo` field names the repository. `tier` records a model-assignment category; choose the actual model for each category in your runner or session setup. ```markdown # Queue ## Pick rule Read this whole file. Take the lowest-numbered item whose status is ready and whose depends_on items are all done; if none qualifies, say so and stop. Before taking an item whose pr is empty, search the merge history for its id; if the work is already there, mark it done with the merge commit in notes (the one done you may set) and pick again. One item per session: set it doing, finish it, set it review, and stop. Never edit an item you are not holding. The one exception is appending: work you discover is a new item at the bottom with status draft, never a wider current one. Status: draft -> ready -> doing -> review -> done. I move draft to ready and review to done (except the merge-history case); you move ready to doing to review. For code, done means merged. For a queue-only investigation, done means I have reviewed and accepted its draft children. blocked waits for me. Tier: judge (the acceptance needs a decision), build (a checkable acceptance line), mechanical (the steps are already listed in the item). ## B-001 health() returns ok - repo: api - status: ready - depends_on: [] - tier: build - spec: specs/api/002-health.md - acceptance: python -m pytest api/tests -q passes with a test asserting health() returns {"ok": true} - pr: - - notes: - ## B-002 Work out what the status page needs - repo: specs - status: ready - depends_on: [B-001] - tier: judge - acceptance: this item is done when it has appended B-00N children as draft, each with an acceptance line I can run, and it has asked me nothing it could have found in the repository - pr: - - notes: an investigate item; its output is items, not code. Context: nothing exists yet. The page lives in web/ and shows when a nightly job, to be created in infra/, last ran; the job writes its last-run time to a file api/ can read ``` Add the following instruction to the standing file. Here, `home` means the assigned repository. This exception permits a narrowly scoped edit to the shared queue: ``` Before starting work, read BACKLOG.md and take the item its pick rule names. It is the one file outside your home you may edit: only the item you hold, plus new draft items at the bottom. When it is finished, set its status and stop. ``` ## Each part, and the mistake it prevents The selection rule expresses your ordering policy. Lowest item number is a simple policy for this example, so choose numbers and dependencies deliberately. Changing the policy is still a human decision; the queue cannot infer which business outcome matters most. The one-item limit keeps a session's work reviewable. The agent changes its selected item from ready to doing to review. You set done after merging code or accepting an investigation's draft children. This template defines code completion as merged; deployment requires separate verification. Acceptance criteria describe an observable result. For `health()`, the result is its return value and the check is a test assertion. For a status page, finding a string in its source cannot establish that a browser fetches and displays the latest timestamp. Match the check to the claim. Dependencies determine eligibility. An item marked ready can still be ineligible because another item remains under review. Missing or stale dependency entries cause the agent to select work in the wrong order. The `tier` field groups work by the judgment it requires. `judge`, `build`, and `mechanical` are model-selection labels, not guarantees of model capability. Choose models using observed performance on the task, and revise the assignments when that evidence changes. An investigation item produces proposed tasks rather than implementation. Its children stay draft until you review their assumptions, scope, and acceptance criteria. Record questions requiring your decision instead of allowing the investigation to approve its own requirements. ## A worked example The recorded exercise used the four repositories, permission file, and queue above, with three additional build items. A fresh session received the standing file and "work the next item": ``` B-001 is done and set to review. `python -m pytest api/tests -q` exited 0 in this session with 1 test passing. The work is a local commit only: `api/` has no remote, so nothing is pushed. **What shipped** (Home: api, spec `specs/api/002-health.md`) - `api/src/health.py` has `health()`, which returns `{"ok": True}`. - `api/tests/test_health.py` asserts that return value. - Commit `30c7333` is on branch `b-001-health`, cut from `main`. It holds only those two files, added by path. - In `BACKLOG.md` I changed only B-001's status and notes. - Nothing under the spec's Out of scope changed (no HTTP server, `web/`, or `infra/`). **Choices I made** - The test adds `api/src` to `sys.path` itself. That way it imports `health` whether pytest runs from the project root, as the verify command does, or from inside `api/`. - I left `pr` as `-` and put the commit and branch in notes, since there is no PR to link. **What the spec could be read to demand that I left out** - Nothing. It asks only for the function and a test. **Follow-ups that deserve their own spec** - `api/` has no remote, so the branch can't be pushed or reviewed as a PR until one exists. - There is no `.gitignore`, so `__pycache__/` shows as untracked. I left it alone. [one bullet about a commit hook on my machine omitted] I stopped after this one item, as the pick rule says. B-002 is ready now that B-001 is in review, but its dependency isn't `done` (merged) yet, so it isn't eligible until you merge. ``` The transcript says "done and set to review." The actual queue status was review: the local implementation was ready for inspection, but the queue's completion rule still required a merge. B-002 remained ineligible until B-001 reached done. The passing test established the function's return value, without showing an HTTP route or deployed service. I merged the branch and set B-001 to done. A new session selected the investigation: ``` I took B-002 and set it to `review`. Its output is items, not code, so I added six draft children (B-006 to B-011) to `BACKLOG.md`, which is the only file I edited. The file on disk matches what I wrote, and no other repo changed. Home: specs. The work is in the queue file, not in any repo, so nothing was committed or pushed. **What shipped.** Each child is confined to one home, and each has an acceptance line you can run: - **B-006 (specs):** a spec for the last-run file contract. - **B-007 (api):** a `last_run()` reader function, with pytest. - **B-008 (infra):** a nightly job script that writes the file, checked with a temp path. - **B-009 (web):** a `status.html` page, checked with grep. - **B-010 (api):** an HTTP route, only if you choose the route option below. - **B-011 (infra):** scheduling the job, which waits on your approval of a cloud resource. **Choices I made, and why.** - I set the file format to `{"last_run": ""}`, with the path taken from an environment variable. A missing file means the job has never run. Without a fixed contract the three homes can't build in parallel. - I made B-006 the root, because the layout skill says the spec comes first. B-008 and B-009 also cite specs that don't exist yet (`specs/infra/001-nightly-job.md`, `specs/web/002-status-page.md`), so their notes say each needs its own spec first. - I gave B-008's acceptance a temp-file check so it runs locally with no cloud. **What the spec could be read to demand that I left out.** - I did not write the specs themselves, because a spec is a separate change in the specs home. - B-011 doesn't pretend to have an executable acceptance line. Its line is your written approval. **Two questions the repo can't answer.** 1. **Where does the last-run file physically live?** `infra/` and `api/` are separate repos with no shared storage. I'd pick an object in a bucket that `infra/` creates. That is a new cloud resource, so it needs your approval. 2. **How does `web/` read it?** The options are a static `/last-run.json` published beside the site, or an `api/` HTTP route. I'd pick static, since it adds no server or web framework. If you agree, drop B-010. **Follow-ups worth their own spec.** - A staleness rule: when should the page flag the job as overdue? - Job failure reporting, since a last-run time says nothing about whether the run succeeded. I stopped at one item per session, as the pick rule says. B-003 is the next ready item. ``` Two of the six children were recorded as follows: ``` ## B-007 api: last_run() reads the last-run file - repo: api - status: draft - depends_on: [B-006] - tier: build - spec: specs/api/003-last-run.md - acceptance: python -m pytest api/tests -q passes with tests asserting last_run(path) returns {"last_run": ""} for a valid file and {"last_run": None} for a missing file - pr: - - notes: new function in api/src/last_run.py; health() untouched. No HTTP layer here (see B-010). ## B-009 web: status page shows the last run - repo: web - status: draft - depends_on: [B-006] - tier: build - spec: specs/web/002-status-page.md - acceptance: web/public/status.html exists and contains the text "Last run" and a fetch of "/last-run.json" - pr: - - notes: needs its own spec first. Default: static page that fetches /last-run.json and renders "Last run: " or "never". OWNER QUESTION: is /last-run.json served as a static file (job publishes it beside the site) or through an api HTTP route (B-010)? I would pick static, since it adds no server to api/. ``` The second run changed only the queue, producing six draft children and two decisions the repository could not answer. No application code was written. The `Home: specs` line identified the kind of work, while the queue itself remained outside a Git repository in this scratch example. The report correctly stated that nothing was committed or pushed. The B-009 source-text criterion is limited: it can confirm a file contains the expected text and fetch path. It cannot confirm browser rendering or a working data source. Review that gap before approving the child for implementation. An earlier investigation prompt also omitted whether the job already existed; adding that fact removed an unnecessary clarification. ## How it scales At the time of the original lesson, my queue held several hundred items across fourteen repositories. A planning pass reviewed answered draft and blocked items when fewer than five items were ready. That pass still needed the recorded answers and approval rules to determine eligibility. You own acceptance criteria and model assignments. The agent can propose children, record discovered work as draft, and update the item it holds. Parallel writers require additional coordination; this template demonstrates one session at a time and does not implement a shared task lock. ## From the field On July 28, my task runner selected three items in a row whose changes had already merged. The status-recording tool had reverted their completion states. Each run spent its effort confirming work already present in the repositories. I added a merge-history check before selection: search for the item identifier and reconcile the queue when its work is already merged. Check that the matching commit actually implements the item before recording completion. A matching identifier alone is insufficient evidence. ## Exercise 1. Save `BACKLOG.md` beside the standing file and add the instruction to read it. 2. Replace the examples with five numbered tasks. Give each a repository, status, dependencies, acceptance criteria, and relevant spec. Turn undefined work into an investigation that produces draft children. 3. Assign each task a model category using `tier`. If the task still needs requirements decisions, identify those in an investigation first. 4. Start a fresh session with the standing file and say "work the next item." Watch it name the eligible item, work within scope, set review, and stop. 5. Review the result and its evidence. Merge accepted code before marking its item done. Review and accept an investigation's children before completing that investigation. ## You're done when Complete one sufficiently specified item while watching the session. Confirm the chosen item met the dependency rule, the evidence supports its acceptance criteria, only permitted queue entries changed, and the agent stopped after one item. Mark it done only after the required review and merge or acceptance. A question about information already available in the item indicates missing context or a reading failure. A question about an unresolved product decision can be correct. The goal is to remove avoidable clarification while preserving necessary approval. ## Common mistakes Vague acceptance criteria make review subjective. Stale status repeats completed work. Editing unrelated items can change another task's priority or requirements. Assigning an unresolved design decision to a model intended for prescribed steps does not remove that decision. ## Further reading [loop-harness](https://github.com/CaskeyCoding/loop-harness) provides the public schema and generator skill used in my setup. [The Kanban Guide](https://kanbanguides.org/english/) describes work-in-progress limits and workflow policies. Bill Wake's [INVEST](https://xp123.com/articles/invest-in-good-stories-and-smart-tasks/) describes criteria for useful work items. ## Next lesson [Automate Code Review, Merge Checks, and Deployment Verification](/blog/lesson-7-the-pipeline-reviews) adds independent review and checks at each stage of a change. --- # Lesson 5: Define What Agents Can Do Without Approval URL: https://caskeycoding.com/blog/lesson-5-the-charter Published: 2026-10-04 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, autonomy > Write down the actions agents may take, the actions requiring your approval, and the evidence they must provide before reporting completion. Includes a permission template and exercises that check whether an agent follows it. **What you'll learn:** write a reusable permission file that specifies what an agent may do, what requires your approval, and what evidence belongs in its completion report. **What you need:** one ready spec from [Lesson 1](/blog/lesson-1-the-spec-directory), with acceptance checks and excluded work; the repository layout from [Lesson 4](/blog/lesson-4-boundaries) or a single repository; and your coding agent. **Time:** about an hour to write and test the instructions. **Cost:** no additional services or subscriptions. ## The idea An agent needs explicit rules for actions your task does not mention. Creating a file, running an existing test, adding a dependency, and changing a public route have different consequences. Write down which actions you authorize in advance and which require a new decision. I use a file with three sections: permitted actions, actions requiring approval, and completion criteria. A final instruction tells the agent to stop and ask when those sections do not resolve a decision. A charter is written standing authority for a role. It depends on the agent following it; enforceable tool restrictions are separate. ![Repeated approval prompts are replaced by CHARTER.md instructions listing permitted actions, required approvals, and completion evidence. A complying agent reports its result or stops with a question.](/blog/lesson-5-the-charter-two-rows.svg) ## Why start here The preceding lessons define the requested behavior, its rationale, checks, and repository scope. They do not specify every action an agent may take during implementation. A ready spec can authorize a feature while leaving a new dependency undecided. The idea of defining authority once per role came from [Clief Notes](https://www.skool.com/cliefnotes). The sections below are my version. Existing tool permission settings still apply; writing this file is not a reason to disable them. ## The template Save `CHARTER.md` beside the standing instruction file in the directory where you launch sessions. Additional roles can use `CHARTER-.md`. In this template, `home` means the repository assigned to the task: ```markdown # Builder: standing authority Scope: one change at a time, from one ready spec, inside one home. ## You may, without asking - Pick names, layout, and patterns, and use any library already in the project. - Create or move files wherever the spec's scope reaches, and delete the ones it makes obsolete. - Cover what you change with tests, and update the docs that describe it. - Run any command that only reads, builds, or tests, inside your own checkout. - Draft a decision record when you choose between two reasonable options, with its Decision line marked proposed for me to confirm. ## You must ask before - Adding a dependency the project does not already have, or any new external service or cloud resource. - Changing an interface the spec names as existing: a route, a schema, a public function. - Deleting data, running a migration, rotating a secret, or any step you could not undo. - Rewriting history: force-push, branch deletion, amending a commit you did not make. - Touching a second home: another repository, another stack. ## Done means - Every acceptance line in the spec passes and its verify command exited 0, both run in this session, and the closing message says so. - Nothing listed under the spec's Out of scope changed. - The work is on a fresh branch, pushed if the home has a remote, one commit per repository. - Your closing message answers four questions in this order: what shipped; which choices you made between reasonable options, and why; what the spec could be read to demand that you left out, and why; which follow-ups deserve a spec of their own. ## When you are not sure Come back with the question. Write it in one line, name the two answers you can see, say which you would pick and why, and stop. Never guess, and never narrow or widen the task to avoid asking. ``` Add this line to the session's standing file: ``` Read CHARTER.md before starting any task; it is your standing authority. Anything under "You must ask before" stops the task until I answer. ``` ## Each part, and the mistake it prevents The scope line restricts the role to one ready spec and one repository. Individual specs still determine which files and behaviors belong to each task. The permitted-actions section records routine decisions you are willing to delegate. It lets the agent choose names, use installed libraries, and run relevant checks without requesting the same approval each time. Review the wording against your environment before adopting it. A command described as a test can still contact a service or change data. The approval section names recognizable actions. "Rotating a secret" can be compared with a proposed command; "be careful" does not specify when to stop. Write these requirements from your project's actual dependencies, interfaces, and data operations. The completion section requires evidence and a report of choices, omissions, and follow-up work. Check the evidence against each acceptance criterion. A zero exit status supports only what the command tested. The template's "what shipped" label means the reported implementation result; a branch and passing tests do not establish deployment. For unresolved choices, the template requests a short question, two options, and a recommendation. That helps you make the decision without reconstructing the investigation. The agent should identify uncertainty before taking the action that requires approval. ## A worked example The recorded exercise used Lesson 4's four repositories and `specs/api/002-health.md`. It required a `health()` function returning `{"ok": true}` and a test. Server wiring, `web/`, and `infra/` were excluded. A fresh session received the standing file and a request to implement the spec on a branch: ``` `health()` now returns `{"ok": True}`, and the work is committed on branch `002-health` [the session's branch name, shortened] in `api/` (commit `3eb7c00`, off `main`). `python -m pytest api/tests -q` ran from the project root in this session and exited 0 (1 passed). I changed two files, both inside the api home: `api/src/health.py` and `api/tests/test_health.py`. The test asserts the return value. Nothing under the spec's Out of scope changed. The commit is local only, because `api` has no remote to push to. I staged the two files by path, and the new `__pycache__` folders are left untracked. **Choices between reasonable options** - **Test imports:** the test adds `api/src` to `sys.path` itself instead of using a `conftest.py` or packaging config. That keeps the change to two files and adds no dependency. The spec's verify command runs from the project root, which is why an import path is needed at all. [one bullet about a commit hook on my machine omitted] **Left out:** I added no HTTP route, framework, or packaging, since the spec rules those out. **Follow-ups for their own spec:** wiring `health()` into a route, and adding a `.gitignore` for `__pycache__` and `.pytest_cache` in `api/`. ``` The return-value test checked a local function. The report also named the two changed files, the local-only commit, and the excluded server work. It did not show a working HTTP endpoint or deployment, neither of which the spec requested. A second run requested an uptime ping to a hosted status page every minute. That added an external service and scheduling work. The session stopped before editing: ``` I haven't changed anything yet. The uptime ping needs your answer first, and CHARTER.md says that stops the task. **Question:** The hosted status page is a new external service, and a per-minute ping is a scheduled job that would live in `infra/` or `api/`. Do you want a spec for the ping first, or should I build it as a scheduled job inside `api/`, with the status-page URL and token supplied by you? - **A (spec first):** You write or approve a spec that names the status-page provider and the home. `specs/api/002-health.md` covers only the bare `health()` function, and it lists infra/ and HTTP wiring as out of scope. I'd do the health function now on a branch and leave the ping for its own spec. I'd pick this, because the ping is a second change that adds an external service, and the layout skill says to write the spec first. - **B (ping inside `api/`):** You tell me which provider to use and approve the new service, and I add the ping to `api/` as a scheduled job. It would need a spec amendment, since 002 currently excludes anything beyond the bare function. I can start `health()` and its test on a fresh `api/` branch right away if you say go. ``` All four repositories remained clean after this request. The two runs show that the agent followed the written instructions for these tasks. They do not establish that later sessions will always comply. ## How it scales On June 9, I wrote five files in one afternoon for the builder, content, monitor, task runner, and interface-review roles. Each used the same sections, with different permitted actions. A monitor can be limited to reading and notifying, with every write requiring approval. You decide the approval requirements and completion criteria. An agent can propose permitted actions based on recent tasks, but you approve the list. Record changes to standing authority with their reasons in a [decision record](/blog/lesson-2-decision-records), so later sessions can identify the approved version. ## From the field An unattended agent on a small always-on Linux box at home read one of these files. Its approval requirements were also represented in tool-level deny rules. The agent attempted a command requiring approval. A rule denied it before execution, and the agent stopped instead of trying another route. The written instruction did not prevent the attempt. The permission rule prevented that command from running. This incident supports a specific distinction: instructions define the expected behavior, while a configured control can reject an action within its coverage. That one denial does not establish coverage for every prohibited action. ## Exercise 1. Save `CHARTER.md` beside the standing file and add the instruction to read it. 2. Replace the approval list with concrete actions relevant to your project. Include the operations you need to review before execution. 3. Ask the agent to propose permitted actions from recent tasks. Remove any action you still want to approve individually. 4. Give a fresh session one ready spec. Compare its completion report with the criteria, inspect the evidence, and check the diff for excluded work. 5. Request an action requiring approval, such as the uptime ping. Confirm the session asks before editing or executing that action. Perform this check in a disposable project with appropriate tool restrictions. ## You're done when The permitted task should finish within its scope and report the evidence, choices, omissions, and follow-ups. The approval case should stop before the restricted action and provide the question and options. Inspect the files and command record instead of relying only on the final message. If the agent asks about an already authorized action, inspect the wording and the context it received. If it performs an action requiring approval, the instruction check failed. Review both instruction delivery and tool-level enforcement before allowing that action unattended. ## Common mistakes Vague approval rules leave the agent to decide what counts as risky. Missing completion criteria leave you to reconstruct the task afterward. Written instructions alone cannot establish that a prohibited command will be denied; test the relevant permission control separately. ## Further reading Saltzer and Schroeder's [The Protection of Information in Computer Systems](https://www.cs.virginia.edu/~evans/cs551/saltzer/) describes least privilege. The [Claude Code permissions reference](https://code.claude.com/docs/en/permissions) documents tool permission rules. The [AGENTS.md convention](https://agents.md/) describes a format for standing agent instructions. ## Next lesson [**Lesson 6: Build a Task Queue Agents Can Follow**](/blog/lesson-6-the-backlog-queue) defines how agents select work from a backlog (the project's task queue) and when they must stop. --- # Eight minutes to an app, and what that leaves to teach URL: https://caskeycoding.com/blog/eight-minute-french-flashcards Published: 2026-09-24 Author: Eric Caskey Topic: ai Tags: AI, mentoring, education, volunteering, side-projects > An eight-minute vibe-coding challenge at a Mouse AI League mentor mixer, where I volunteer as a mentor and judge: an AI agent built a French flashcard app in eight minutes, and the speed of the machine is what makes product judgment the skill worth teaching high schoolers. I volunteer as a mentor with [Mouse AI League](https://mouse.org/aileague), a year-long program in which high school students learn how AI systems work, practice human-centered design, do market research, and ship an interactive app prototype over six sprints. Last year I judged the students' final projects when they came to an Amazon office to present them. This year, at a mentor mixer, the organizers handed the mentors an AI coding agent and eight minutes. Mine produced a French flashcard app in one HTML file: 72 words in five decks, a spaced-repetition scheduler, spoken pronunciation, and progress that survives a reload. [It runs here](/cahier-francais-app.html). The agent did the typing in those eight minutes. I spent the same eight on five decisions, and the decisions are what I want the students to look at. ## The machine typed 487 lines and I made five decisions The agent wrote the scheduler, a simplified version of SM-2, the 1987 algorithm that Anki still descends from. It wired the browser's own speech engine to read each word in a French voice, put the review history in the browser's local storage, and laid out a card on the ruled paper of a French exercise book, for phone and desktop, in 487 lines. None of that took a question from me beyond the brief: one page, no framework, no build step. ![Two boxes side by side. The left, in grey, lists what the agent typed in eight minutes: a spaced-repetition scheduler, browser speech with a French voice, saved progress across reloads, the phone and desktop layout, 487 lines in one file. The right, highlighted in green, lists what I decided: which 72 words, eight new words a day, le and la printed in color, one example sentence per card, and progress that stays on one device.](/blog/eight-minute-french-flashcards-agent-and-judgment.svg) The decisions were mine. Which 72 words: greetings, the café, directions, the core verbs, numbers and time, the vocabulary of a first week in France rather than a frequency list. Eight new words a day, because a deck learned in one sitting all comes due on the same afternoon and day three becomes a wall. The article printed with the noun and colored by gender, blue for le and rose for la, because "le pain" is the unit a learner has to carry and "pain" on its own is half a word. One example sentence on every card, since "the bill" is a fact and "L'addition, s'il vous plaît" is a sentence you will say at a table. And one trade I made knowingly: progress lives in one browser and does not follow you to your phone, which kept the whole build to a single file with no account behind it. None of those five needed a line of code. All five decide whether the app is worth opening twice. That is the ratio the League's curriculum is built on. The prototype is one item on a sixty-hour syllabus; the rest of the hours go to who the user is, what they already do instead, and whether the design serves them. ## The feedback loop is now shorter than a class period The agent's first phone layout scrolled sideways. I only knew because I rendered the page at 400 pixels wide and looked, and the fix, two CSS values, took under a minute. That loop, build, look, fix, ran three times inside the challenge. A responsive-design lecture would still have been on its fourth slide. That loop is what the speed of the machine gives a sixteen-year-old. The distance between an idea and a version someone else can hold used to be a semester of syntax, and most ideas did not survive the semester. Now a student who has spent a sprint interviewing users can put a running prototype in front of those same users the week after, and find out that afternoon that the font is too small or the third screen is the one people wanted first. Human-centered design has always taught that lesson with sticky notes and paper mockups. A prototype that runs teaches it with the user's own thumb on the screen, and the correction costs minutes instead of a rebuild. It also changes what a judge can assume. When the students present this year, the code will run; the agent makes it run. What separates the entries is whether the student can name who the app is for, what it does on the first screen, and what it deliberately leaves out. The student who says "it does not sync between devices, because we wanted to ship without accounts" has made a product decision and can defend it. That is the answer I am listening for, and it is a harder skill than syntax ever was, because there is no compiler to tell you when it is wrong. ## What sixty hours buys at that price The brief I gave the agent was three constraints long, and everything the brief did not say, the agent chose, which meant I had to look at all of it. Writing the brief and reviewing the result are the two skills the challenge exercises, and both are the skills the League's sprints spend the year on under other names: define the user, scope the problem, test the prototype, revise. The cost of a wrong product decision used to be measured in weeks of code thrown away. On this build the daily cap is one constant and the interval table is six numbers; either is a one-minute edit. Sixty hours is enough time to be wrong a hundred times at that price, and a hundred corrected decisions is more product judgment than most of us had when we started getting paid for it. --- # How do you know the compiler actually vectorized your loop? URL: https://caskeycoding.com/blog/how-do-you-know-the-compiler-vectorized-your-loop Published: 2026-09-24 Author: Eric Caskey Topic: finance Tags: C++, simd, performance, compilers, quant > My build log said the loop was vectorized, and it had: a trivial loop at the end of the benchmark. The pricing loop, the one with exp and log in it, stayed scalar until I wrote the four-wide math by hand and the engine went from about 114 million to about 215 million options a second. Three checks that catch a claimed win, and what the numbers look like when SIMD is real. My options pricer's build log said "loop vectorized", but the pricing loop was still scalar. The success message belonged to a trivial loop at the end of the benchmark. To check the loop that matters, enable both successful and missed-vectorization reports on your optimized build, then match each message to its source file and line. ## Compiler flags for vectorization reports | Compiler | Report successful vectorization | Report misses and reasons | |---|---|---| | MSVC | `/Qvec-report:1` | `/Qvec-report:2` includes successes and misses with reason codes. | | GCC | `-fopt-info-vec-optimized` | `-fopt-info-vec-missed` reports missed vectorization opportunities. | | Clang | `-Rpass=loop-vectorize` | `-Rpass-missed=loop-vectorize` reports misses; `-Rpass-analysis=loop-vectorize` identifies the statements responsible. | These are reporting options for an optimized build. They do not force the compiler to vectorize a loop. The flags are documented by [Microsoft](https://learn.microsoft.com/en-us/cpp/build/reference/qvec-report-auto-vectorizer-reporting-level?view=msvc-170), [GCC](https://gcc.gnu.org/onlinedocs/gcc/Developer-Options.html#index-fopt-info), and [LLVM](https://llvm.org/docs/Vectorizers.html#diagnostics). MSVC is the only one I ran on this engine. After locating the right loop in the report, measure it on one core and check the results against the scalar implementation. ![A benchmark with two loops beside a compiler report with two lines: the green "loop vectorized" line points at the trivial trailing loop, and the red "loop not vectorized" line, never asked for, points at the pricing loop that calls exp and log.](/blog/how-do-you-know-the-compiler-vectorized-your-loop-report.svg) ## The report was true about the wrong loop The pricer is the Black-Scholes engine from [the 215 million post](/blog/pricing-215-million-options-a-second). A clean first version priced about fifteen million options a second on one core. Swapping one modern library call for a sixty-year-old polynomial and threading the embarrassingly parallel loop took it to about 114 million, and none of that came from SIMD. The plan had always assumed the compiler would vectorize the clean inner loop on its own, because that is what the guides say, and because the build log agreed. My compiler, MSVC, prints a line for each loop it vectorizes when asked, and my build log had one: a trivial loop at the end of the benchmark. The loop that priced options calls exp and log, and my compiler would not fold a loop full of library calls into vector form. Some toolchains will; mine did not. Its failure notice for the pricing loop sat in a report I had not opened, so the loop that mattered stayed scalar on every build while the log kept reporting a win. The numbers needed the same separation. The jump from fifteen million to 114 million came from a better polynomial and more cores, with no SIMD in it. Four lanes describe how many values a vector operation handles; they do not establish how much faster the whole loop will run. ## Test each vector function before wiring it in I wrote the exponential and the logarithm four wide, from classic polynomial cores, intrinsics all the way down. Before wiring either into the pricer I tested each one in isolation against the standard library, the exponential against the exponential rather than the pricer against the pricer. When a bug showed up, the isolated test named the function instead of leaving me to chase a wrong price three layers downstream. One core gained about four times, and across all cores the engine reached about 215 million a second. The rewrite changed both the math implementation and the execution width, so those timings do not isolate the gain from SIMD alone. ## Three checks that catch a claimed win Ask the compiler for its misses using the flags above. Match each message to a source line and find the loop you care about. A report with only successes in it says nothing about the loop that is missing from it. Measure the loop on one core before measuring all of them. If the gain is smaller than expected, first confirm that the reported loop does the work you timed. Then investigate memory traffic, scalar work left in the loop, and overhead on short loops. A smaller gain can have several causes; a larger gain can include changes beyond SIMD. Keep the input, thread count, and math implementation comparable when trying to attribute the difference. Test the vector functions against the scalar library, one function at a time. This says nothing about what the compiler did. It checks the code you wrote to replace the compiler, over the inputs and tolerances you chose, which is the check you need the moment you stop trusting the report. ## Treat the published timings as a shape The engine is not public and I have not re-run it for this post, so every number here is the one I published in July, and desktop timings wander ten or twenty percent run to run. Read fifteen, 114 and 215 as a shape. It was built to learn from, and none of it is a trading system. The compiler flags above are the general ones from each toolchain's documentation; MSVC's is the only one I ran on this code. If I built the pricer again, the benchmark would carry the check itself: single-core throughput of the vector build against the scalar build on the same machine, with a floor set from repeated runs, so that a build where the width has gone missing fails on the spot instead of shipping behind a success line in the log. --- **Related:** - [Pricing 215 million options a second with SIMD C++](/blog/pricing-215-million-options-a-second), the full climb from fifteen million to 215 million, including the memory-layout guess that came before this one - [How to backtest without fooling yourself](/blog/backtesting-without-fooling-yourself), the same measure-before-believing discipline pointed at signals instead of speed - [Lesson 3: Enforce Project Rules with Automated Checks](/blog/lesson-3-the-check-that-fails), designing checks that reject a known failure --- # How many AI coding agents can you actually run at once? URL: https://caskeycoding.com/blog/how-many-ai-coding-agents-at-once Published: 2026-09-24 Author: Eric Caskey Topic: ai Tags: AI, agents, claude-code, orchestration, worktrees > I cap my setup at roughly ten coding agents across several subscriptions, with three or four per vendor. A seven-agent wave exposed failures in network access, checkouts, test ports, and generated files. The counts and the preflight checks I use before another run. I cap my setup at roughly ten concurrent coding agents across several subscriptions, usually three or four per vendor. That number comes from running this workload on my machine. The constraints I have hit are the link to the vendor and its quota window, the git checkout, the network port the test suite binds, and the file every branch regenerates. ## Before launching parallel coding agents | Shared resource | Check before starting the wave | When I stop adding workers | |---|---|---| | Vendor connection and quota | Check that a single CLI run completes, then allocate workers across the subscriptions available. | Launches fail or the quota window is exhausted; I inspect the error and connection before retrying. | | Checkout and dependencies | Give each worker a separate git worktree and a real dependency install. Record the commit under test. | Verification would use a checkout another session can move. | | Test-server port | Check which process owns the test port. Keep full verification serial while suites share it. | Another verification is already using that port. | | Generated files | Name the files each build regenerates, even when the source edits do not overlap. | A branch needs rebasing and regeneration after the previous merge. | I bind the passing checks to the commit being reviewed; the [local CI gate](/blog/local-first-ci-pre-push-hook-is-the-gate) explains that record. The [monthly agent cost breakdown](/blog/what-a-month-of-ai-coding-agents-costs) records the subscriptions behind this setup. Neither a worker count nor a subscription price tells me how many branches I can finish verifying in an hour. ## What happened in a seven-agent wave The day before I wrote this, I ran seven at once and kept a tally of what gave way. The job was a batch of site fixes from a traffic audit: retitle thirteen posts, add a series hub, fix the footer and a dozen links, gate an analytics tag, wire a post-deploy ping, and wire a new exporter into a CDK stack. Each fix had a written brief with the exact files, the verify commands, and an out-of-scope list. I run worker CLIs from three vendors. I put three briefs on one CLI and four on another, each in its own git worktree, and started them within the same minute. ![Seven small agent boxes on the left with arrows converging on one green gate box labeled one verification at a time, with the counts 7 agents typing in parallel and 1 branch verifying at a time beside it.](/blog/how-many-ai-coding-agents-at-once-gate.svg) ## The link to the vendor gives way first Two of the four launches on the second CLI exited within three minutes with nothing written. No error text, exit code 1, a thread that had read the brief and then stopped. A third wrote one component and died; the fourth wrote a footer link and died. When I probed that CLI by hand the failure was hostname resolution, Windows error 11002, and my home connection was flapping that hour: pushes to the code host were failing on the same resolution error. That one is on my network. The other CLI, reached over the same connection, stayed up through the flap, and I re-dispatched the four briefs to it rather than wait for the line to settle. The quota itself has been the ceiling once, and that one was mine to cause. On 2026-07-06 a wave of frontier-tier subagents tripped the subscription's limits, and the routing rule that came out of it is the one I still run: the frontier model supervises alone, at the session level, and never fans out to copies of itself. Workers run on cheaper tiers or on other vendors' subscriptions, capped at roughly ten concurrent, and each vendor gets three or four at a time, not seven. The quota window is a shared resource the same way a database connection pool is, and the agents draining it do not coordinate. A flapping resolver is the same resource failing from the other end. ## The checkout gives way second Every worker had its own worktree, so the agents never touched each other's files. The shared state that bit was the canonical checkout, the one directory with the real dependencies installed, because the merge gate's full verification wanted to run there. Another session on the same machine was using it for the same purpose at the same time. The reflog for that hour reads like two people grabbing the same steering wheel: detach to their commit, detach to mine, back to theirs, back to main. One of my runs passed every check and then recorded its passing verdict against a commit that was not the one it tested, because the other session had moved the checkout between the last test and the write. The fix was to stop sharing the checkout at all. Each worktree got a real copy of the dependency tree, about a gigabyte and two minutes per copy, and the verification ran inside the worktree it was verifying. The cost is disk. The alternative is a verdict recorded against a commit nobody tested. ## The port gives way third The end-to-end suite starts a local server on port 3000. Two sessions running that suite at once do not get two servers; the second connects to the first session's server and asserts against the wrong build. Three of my verification runs failed only on the end-to-end lane, with the unit suite green and the same spec passing in isolation seconds later, and every one of them coincided with another attestation holding the port. A failure that reproduces only when a neighbor is running is not a flake. It is a shared resource with no lock on it, and until there is a lock the rule is one full verification at a time, machine-wide. ## The generated file gives way last Five branches merged in sequence, and every one had to be rebased onto the main that the previous merge had just moved. Two of them conflicted, both on the same file: the blog index that the build regenerates from post frontmatter. Two agents that never edited the same source file still collided, because the build had written the same derived artifact on both branches. The resolution is mechanical once you know it, take main's copy, regenerate, continue, but it has to happen on every branch, and a wave of ten guarantees it happens ten times. The same class of problem produced the worst moment of the day. Cleaning up, I removed the finished worktrees with the version control tool's own remove command. Their `node_modules` directories were junctions into the canonical checkout, and the recursive delete followed the junctions and emptied the canonical dependency tree for two repositories, which cost every session building from those checkouts its toolchain until a reinstall finished. Nothing in version control showed it, because the directory is ignored. The teardown procedure now checks for junctions and unlinks them before any delete, and it caught a second one on the very next run, planted by the build tool inside its own cache directory. ## What ten agents buy The seven agents finished their edits inside about forty minutes of wall clock, most of it waiting on the four that had to be re-dispatched. Landing the five resulting branches took roughly four more hours, because each one had to be rebased, pushed, verified for about eight minutes through a gate that admits one branch at a time, and merged before the next could start, and three of those verifications had to run twice. Typing was parallel. Everything that made the typing safe to keep was serial. That ratio is the actual answer to the question. Agents are cheap to add up to the point where the quota, the checkout, the port, or the generated file pushes back, and each of those pushes back at a different count. The number that moved my throughput this year was never the count of agents typing. It was the count of branches I could verify in an hour, and every rule above exists to raise that second number by one. --- # How to make an LLM say I don't know, and how to measure that it did URL: https://caskeycoding.com/blog/how-to-make-an-llm-say-i-dont-know Published: 2026-09-24 Author: Eric Caskey Topic: ai Tags: AI, LLM, RAG, guardrails, evaluation > Ballast declines 24 of its 89 golden questions on purpose and answers the other 65 with citations. As of the August 16 ledger the answer-or-decline call was right on all 89, and one of the 65 answers was unfaithful to its sources. Where the refusal comes from, how the eval scores it, and why 24 is the number to read, not 100 percent. Ballast declines 24 of its 89 golden questions on purpose and answers the other 65 with citations, and on the August 16 run the answer-or-decline call was right on all 89: zero questions it should have declined and answered, zero it should have answered and declined. One of the 65 answers was unfaithful to its sources, which the eval counts as a hallucination rate of 1.1 percent over the whole set. Those three counts are the abstention measurement, and this post is about how each one is produced, because a refusal you cannot count is a mood, not a feature. The June post on [Ballast](/blog/ballast-an-llm-that-says-i-dont-know) explained why refusal is the feature. What it did not have was a ledger. It described one 44-question evaluation and said, correctly, that a zero on that set was not a universal claim. This post reads only the committed metrics ledger, which now has three dated runs on a set that grew to 89 questions, and a gate that fails the build on regression. That is the artifact worth writing about. ![A box labeled 89 golden questions splitting into two arrows: one to 65 answered with citations and 0 wrongly declined, one green to 24 declined and 0 wrongly answered, with the note one of 65 answers unfaithful.](/blog/how-to-make-an-llm-say-i-dont-know-split.svg) ## Where the refusal comes from An LLM does not decline on its own. Left to itself it completes the sentence, because that is the only verb it has. Ballast gives it three places to stop before an answer ships, two of them ahead of generation and one behind it, and each one emits a fixed sentence instead of prose. The first is the retrieval grader. Every question fetches passages from the corpus and a grader scores each one for relevance. If nothing relevant comes back, the graph rewrites the query and retrieves again, up to two retries, and then declines with a fixed insufficient-evidence sentence rather than handing the model an empty context and hoping. The same retry budget covers the other direction: an answer the critic judges ungrounded goes back through a rewrite, and when the budget is spent the graph declines instead of shipping the answer it could not ground. The second is the guardrails gateway, in two halves. On the way in, a classifier declines requests for personalized, medical or legal advice before generation. On the way out, a YAML policy reads the generated answer, withholds one that gives advice or fails to cite a passage, allows one corrected retry, and then returns a fixed withheld-by-guardrail sentence. The third is the input guardrails, which block prompt injection, secrets and personal data before anything runs and return a fixed could-not-be-processed sentence. The fixed sentences are the point of the design and not a stylistic choice. A refusal that the model phrases freely is a refusal the eval cannot find. Three marker phrases are what the scorer looks for. ## How the eval scores a refusal The golden set has 89 questions in five categories: 55 answerable, 10 multi-hop, 10 unanswerable from the corpus, 5 out of corpus entirely, and 9 adversarial prompts. Each carries an expected answer or the single word decline; 24 carry decline. The runner asks every question, checks each output for one of the three marker phrases, and scores the pair: should this have been declined, and was it. Both mistakes are counted separately, because they cost different amounts. An under-refusal is a question that should have been declined and got an answer, which is the failure that produces a confident fabrication. An over-refusal is an answerable question that got a decline, which is the failure that makes the system useless. Refusal correctness is the share of all 89 where the two flags agree, and the build gate fails below 90 percent. Answered questions are then judged for faithfulness to the retrieved passages. The hallucination rate counts an unfaithful answer and an under-refusal alike, over all 89 questions, and the gate fails above 5 percent. The ledger has three rows. The first, June 27, on a 43-question set: refusal correctness 97.7 percent with one under-refusal, every answer faithful, hallucination 2.3 percent, all of it that one under-refusal. The second, July 2, on the 89-question set: 98.9 percent with one under-refusal, one unfaithful answer, hallucination 2.25 percent, injection block rate 100 percent. The third, August 16: 100 percent, zero under, zero over, one unfaithful answer, hallucination 1.1 percent with a bootstrap interval of 0 to 3.4 percent, faithfulness 0.985, and $0.0061 a question. Read by unfaithful answers alone the series is zero, one, one; the fall in the headline rate between July and August is the under-refusal going away, not the grounding improving. Whether that is a fix or run-to-run variance in the model, the gate is what makes the next one visible instead of averaged away. ## What 100 percent cannot promise Only 24 of the 89 questions are decline questions, and 24 correct out of 24 is a smaller claim than it sounds. Zero misses in 24 trials is consistent with a true miss rate anywhere up to about 12 percent at the usual confidence, the rule of three applied to a sample of that size. The ledger's own bootstrap band for refusal correctness reads 1.0 to 1.0, and that is not a promise either: resampling a column of ones produces a column of ones, so the interval has zero width by construction, not by evidence. The number to read is 24, and the number to watch is the under-refusal count, which is a count of specific questions and not a rate. The detector tests the plumbing as much as the model. A decline phrased in the model's own words, with a citation attached, passes the must-cite policy and is scored as an answer, which hides an over-refusal; an uncited one is withheld by that same policy and comes out as the fixed sentence, which scores as a refusal. So the 100 percent says that on every question the three stops and the policy resolved to a marker sentence or an answer as intended, and a model that had learned to decline in prose would not be caught by it. The faithfulness judge is calibrated against eight human-labeled pairs with an agreement floor of 0.8, and eight is thin; the calibration set is the next number that has to grow. ## The cost of refusing A system that declines everything scores perfectly on the 24 and zero on the 65, and refusal correctness would read 27 percent. The metric couples the two mistakes on purpose so that a gate cannot be passed by cowardice. In the August run the over-refusal count was zero: every one of the 65 answerable questions came back as an answer rather than a decline, and the must-cite policy would have withheld an uncited one. That is the half of the measurement I would have skipped in June, when the refusals were the part I was proud of, and it is the half that decides whether anyone uses the system. The abstention number I report now is not a rate. It is three counts from the latest ledger row, zero under-refused, zero over-refused, one hallucinated answer in 89, with the date and the run beside them, and a gate that turns the next regression into a failed build instead of a quieter number. --- # How do you read a 3D option Greeks surface? URL: https://caskeycoding.com/blog/how-to-read-a-3d-option-greeks-surface Published: 2026-09-24 Author: Eric Caskey Topic: finance Tags: finance, options, visualization, threejs, education > The Greeks manifold prices a long call on a 60 by 60 Black-Scholes grid over spot and time to expiry, draws profit and loss as height against a premium fixed at the strike and the longest tenor, and colors each cell by one Greek. Gamma is a ridge near the strike that sharpens toward expiry, theta is the downhill slope toward expiry, delta is the slope along price, and vega and rho recolor the sheet while moving the benchmark it is measured against. Ticker mode positions the same surface on a published S&P 500 name. Read [the Greeks manifold](/lab/greeks-manifold)'s 60 by 60 grid as a long call's profit and loss in height, colored by one Greek: gamma a ridge near the strike sharpening toward expiry, theta the downhill slope toward expiry, delta the slope along price. Vega and rho are different; they recolor the sheet by a sensitivity and, through their sliders, move the premium the height is measured against. ![The long-call profit-and-loss surface colored by gamma: a teal band flaring orange near the strike, where the option's value bends hardest](/blog/greeks-gamma-ridge.webp) ## Height is profit and loss against a fixed premium The surface has three axes. Spot price runs one way, time to expiry runs the other, and height is the call's value at that point minus a premium fixed once per surface: the at-the-money call value at the longest tenor on the time axis. The single point at the strike and the longest tenor therefore sits at zero, and every other cell shows what the position would be worth relative to what it cost. On the default inputs that is a strike of 100, volatility of 25 percent, a risk-free rate of 5 percent, spot from 40 to 180, and time to expiry from 0.02 to 1.5 years, and each of the five sliders redraws the 3,600-cell grid. Color is a separate channel. Each cell also carries the five Greeks of the call at that point, and the selector paints the sheet by one of them. The first time I rotated the surface, in [the post that built the five 3D views](/blog/markets-in-three-dimensions), gamma was just sitting there as a ridge. The sections below map each textbook Greek onto what the sheet does. ## Read delta and gamma along price, theta toward expiry Delta is the slope of the surface along the price axis. Far below the strike the sheet is flat and delta is near zero; far above it the sheet rises one for one with spot and delta is near one; the transition happens across the strike, gradual on the long-tenor edge and abrupt on the near-expiry edge. Gamma is the rate at which delta changes with spot, and painting by it lights a ridge across the sheet. The ridge is near the strike, not exactly on it: by the formula the tool uses, gamma at a fixed tenor peaks at a spot of the strike times an exponential in rate, volatility squared and time, which at the default inputs puts the peak near 81 on the 1.5-year edge and within a point of 100 at 0.02 years. Drag the lower end of the time range toward 0.02 years and the ridge slides onto the strike and narrows into a spike: with almost no time left, the call is either worth its intrinsic value or nothing, and all of the bend collapses onto 100. Theta is the slope of the sheet in the time direction, read toward expiry. Hold spot at the strike and walk from the 1.5-year edge toward the near edge: the sheet falls, and the fall steepens near the end. That is the value the option loses to the calendar, and on the surface it is the direction water would run. ## Vega and rho recolor the sheet and move the benchmark Vega and rho are not slopes of the sheet, because volatility and rate are sliders rather than axes. Painting by vega colors each cell by how much the call's value there would change with volatility, largest near the strike at long tenors; painting by rho does the same for the rate. Moving either slider does two jobs at once: it reprices every cell, and it reprices the premium benchmark, so the profit-and-loss sheet changes relative to a moving zero rather than simply rising or falling. The point at the strike and the longest tenor stays at zero whatever the sliders say. ## Ticker mode positions the surface on a published name The synthetic surface teaches the shape. Ticker mode puts a real name under it. Choose a published symbol from the curated set of 30 large caps or the S&P 500 set and the surface re-centers: the stock's spot becomes the strike, its realized volatility from up to 61 closes becomes sigma, clamped to between 5 and 150 percent, the spot axis becomes 0.4 to 1.6 times spot, and up to 90 closes are drawn across the sheet as a path. On the file published 2026-09-24, AAPL carried a spot of 337.02 and a realized volatility of 28.9 percent, so ticker mode puts the ridge near 337 instead of 100 and runs the volatility at 28.9 instead of 25. The spot and volatility are a dated snapshot of public prices, shown to position the surface, not investment advice. The inputs come from a baked file, not a live call. A scheduled job runs each weekday before the US open, pulls price history from Financial Modeling Prep, computes annualized realized volatility as the sample standard deviation of daily log returns times the square root of 252, drops any series with fewer than 30 closes, and writes one row per symbol. The S&P 500 set only flips to a new version when at least 400 names resolve, so a bad fetch cannot publish a thin universe, and a name that failed to publish is simply absent from the picker. The browser reads the file and never touches the API. ## The model prices a European call without dividends The pricing is textbook Black-Scholes for a European call with no dividend yield, and the normal distribution is the Abramowitz and Stegun approximation with a maximum error around 1.5 times 10 to the minus 7, well inside the tolerance of the finite-difference checks the Greeks are tested against. There is no put surface and no early exercise. The rate is one slider from 0 to 15 percent, not a term structure. The smile mode, one toggle away, is synthetic and says so on the page, because a real implied-volatility feed is a spend I have not approved: the synthetic smile supplies no market-implied volatility for any strike or expiry. --- **Related:** - [Rendering an options Greeks surface in 3D in the browser](/blog/markets-in-three-dimensions), how this and the other four surfaces were built on one math-to-mesh boundary - [Pricing 215 million options a second with SIMD C++](/blog/pricing-215-million-options-a-second), the SIMD C++ pricer whose implied-volatility solver the smile mode's solver mirrors - [How do you know the compiler actually vectorized your loop?](/blog/how-do-you-know-the-compiler-vectorized-your-loop), why a build log that said "loop vectorized" was wrong about the loop that mattered --- # How do you tell when the yield curve has inverted? URL: https://caskeycoding.com/blog/how-to-tell-when-the-yield-curve-has-inverted Published: 2026-09-24 Author: Eric Caskey Topic: finance Tags: finance, macro, treasuries, visualization, education > The yield curve surface reads 11 FRED constant-maturity Treasury series from one month to thirty years, keeps the 60 most recent complete daily curves, and labels the curve inverted when the 30-year yield is more than 0.1 percentage points below the 1-month yield. On 2026-09-22 the 30-year was 5.29 percent and the 1-month 3.97, a slope of plus 1.32 points, and the read was upward. The spread clock panel below the surface draws the 2s10s spread back to 1976, and the textbook mode fakes an inversion on purpose from a Nelson-Siegel preset. [The yield curve surface](/lab/yield-curve) labels the real curve inverted when the 30-year Treasury yield is more than 0.1 percentage points below the 1-month yield; on the 2026-09-22 curve the 30-year was 5.29 percent and the 1-month 3.97, so the read was upward. The same rule labels the curve upward above plus 0.1 points and flat in between, and it is applied to every day in the file. ![The textbook, synthetic mode of the yield curve surface on its last frame, folded into inversion, with the status badge reading inverted](/blog/yield-curve-inverted.webp) *Textbook mode, not the live curve: a synthetic Nelson-Siegel inversion. The real 2026-09-22 curve read upward.* ## The badge tests only the two endpoints A yield curve is the interest rate for lending to the Treasury over different lengths of time, drawn as a line from the shortest maturity to the longest. Normally it slopes up, because longer loans cost more. An inversion is the line tipping the other way, with short rates above long rates, and it is the shape everyone calls a recession warning. The surface draws that line for every day in its window and stacks the days along a second axis, so the curve becomes a sheet you can scrub through time and watch tilt. The label on the page comes from one subtraction. The tool takes the last yield in the curve minus the first, which on the real data is the 30-year minus the 1-month, and thresholds it at a tenth of a point either side of zero. Above plus 0.1 the badge says upward; below minus 0.1 it says inverted; in between it says flat. A separate chip reports the day-over-day move in the 2s10s spread, the 10-year minus the 2-year, and a spread clock panel below the surface draws that pair over the same sixty sessions and over its full history. ## Publish only dates with all eleven yields The real curve is eleven FRED constant-maturity series: 1-month, 3-month, 6-month, 1-year, 2-year, 3-year, 5-year, 7-year, 10-year, 20-year and 30-year. A scheduled job runs daily, requests the 120 newest observations for each series, joins them on date, and keeps only the dates where all eleven have a value, oldest first. It publishes the 60 most recent complete curves as one file and refuses to publish at all if fewer than 5 survive the join, so a partial fetch cannot replace a good file with a thin one. The page reads that file; the browser never calls FRED. On the file I read while writing this, the 60 curves ran from 2026-06-29 to 2026-09-22. The latest one, in percent from short to long, was 3.97, 4.16, 4.26, 4.43, 4.71, 4.81, 4.83, 4.89, 4.96, 5.33 and 5.29. The 2-year at 4.71 and the 10-year at 4.96 put the 2s10s spread at plus 0.25 points, against plus 0.28 on the first day of the window. The 20-year at 5.33 sat above the 30-year at 5.29, a small kink at the long end that a badge computed from the endpoints alone will never mention. ## Sixty sessions is the surface's window, not the page's Sixty complete sessions is about three months, so the sheet and its badge can show the curve steepening or flattening within a quarter and nothing older. Below the surface, the spread clock panel draws the 2s10s spread's daily history back to 1976 from a separate baked series, with recession months shaded, and that panel is where to look for when the spread last went negative. The page shades recessions but computes no lead time from inversion to downturn and no hit rate, because the tool draws yields and does not forecast from them. The label is a rule about two endpoints, so a curve that is inverted in the middle and upward at the ends reads as upward. And the real and synthetic modes use different maturity grids, eleven points from one month on the real data and ten points from three months on the textbook curve, so a number read off one mode should not be quoted against the other. ## The textbook mode inverts on purpose When the real curve is not inverted, which it was not on 2026-09-22, the page still needs to show what an inversion looks like forming. The synthetic mode does that with the Nelson-Siegel three-factor formula: a level, a slope factor, a curvature factor, and a decay constant. Six named presets set those four numbers for normal, flat, inverted, steepening, bear-flattening and level-shift shapes, and a transition interpolates from one preset to another over 24 to 90 frames with up to 15 basis points of seeded jitter so the sheet does not look machined. Going from the normal preset to the inverted one moves the slope factor from minus 1.6 to plus 1.6, the level from 4.0 to 3.4, and the curvature from 0 to minus 0.4, with the decay constant held at 2.0. The deep link in [the post that built the five surfaces](/blog/markets-in-three-dimensions) opens the synthetic mode on its last frame, fully inverted, and the badge says so. The real mode opens on the newest curve in the published file, and the tool's rule applied to the 2026-09-22 curve gives upward. Both are labeled, and the label is the first line to read before the shape. --- **Related:** - [Rendering an options Greeks surface in 3D in the browser](/blog/markets-in-three-dimensions), how this surface and the other four were built on one math-to-mesh boundary - [How do you read a 3D option Greeks surface?](/blog/how-to-read-a-3d-option-greeks-surface), the companion page for the options surface --- # Local-first CI: the pre-push hook is the gate and hosted CI is the mirror URL: https://caskeycoding.com/blog/local-first-ci-pre-push-hook-is-the-gate Published: 2026-09-24 Author: Eric Caskey Topic: platform Tags: platform, ci-cd, git, developer-experience, infrastructure > After GitHub Actions burned a month of minutes in two days, the gate moved to the workstation. A pre-push hook runs a quick tier on every push, every green run writes a record tagged with its tier and commit, and a merge guard refuses any pull request whose exact SHA has no full-tier record. 1,660 records later, what it covers and what it cannot. My git pre-push hook is the CI gate, and the hosted runner is a mirror that deploys on merge and scans for secrets. That arrangement has been in force since 2026-07-11, when a month of GitHub Actions allowance was gone in two days and every job on the account was failing at setup in under five seconds. The [cost side of that decision](/blog/when-ci-costs-more-than-it-saves) is its own post. This one is about the mechanism: what a hook has to do before "passes locally" is allowed to mean "may merge". The runner is one Python script that mirrors each repository's `pr-validation.yml`, lane for lane, and runs it in three tiers. The pre-push hook runs the quick tier on every push: lint, the static scans for disclosure, route hygiene and workflow context, and a small slice of the unit suite, well under a minute. The full tier adds the build, the whole unit suite, the end-to-end suite, a home-page smoke test and the secret scan, and takes about eight minutes for the frontend. Every green run at either tier writes a record tagged with its tier; only a full-tier record counts at the merge. There are 1,660 records on this machine today, 311 of them full-tier: 997 for the specs repository, 322 for the frontend, 162 for the backend, 81 for the infrastructure stacks, and the rest across the smaller products. ![Three boxes in a row, quick tier on every push, full tier before merge, and a green merge guard that checks for a record keyed by commit, with a faint mirror box below labeled hosted runner: deploy, rollback, secret scan.](/blog/local-first-ci-pre-push-hook-is-the-gate-record.svg) ## A green run has to be bound to a commit The first version of local CI printed a verdict and trusted the human to act on it, which is the same trust model as a sticky note. The version that replaced it writes a JSON record when every lane in a run passes: the repository key, the exact commit SHA, the merge base it was tested against, the tier, whether the tree was dirty, the timestamp, and the Node version. The record lives outside the repository, in a directory the hooks own. Binding to the SHA is what makes the record mean something. Push a follow-up commit and the record does not follow it; the new commit has to earn its own. Run the tier on a tree with uncommitted changes and the record says so, and the merge gate rejects it, because a record for a commit that does not contain what was tested is a lie with a timestamp. Run a full tier that skipped the build because the dependencies were missing and the writer refuses that too, on the grounds that a hollow green is worse than a red. I learned the last rule from a run that reported GO in a fresh worktree with no `node_modules`, and the merge it would have blessed was a build failure waiting for the deploy step. ## Something has to refuse the merge A record nobody checks is a log. The check is a second hook, not in git but on the shell tool that my agents and I issue commands through: it intercepts the merge and deploy commands themselves, resolves the commit that command would promote, and asks the runner whether that exact SHA has a full-tier record. On those intercepted paths, a missing full-tier record blocks the action. It matches `gh pr merge`, the deploy scripts, the CDK deploy, and the bare S3 sync that used to be the whole deploy. It rolled out in observe mode first, logging its decision without blocking, until its decisions on real traffic showed it resolving the right commit, and then a sentinel file flipped it to enforce. This is a cooperative workstation workflow. Git's [`--no-verify`](https://git-scm.com/docs/git-push) bypasses the pre-push hook; a shell-tool hook does not cover a merge from GitHub's interface or another tool. A JSON record outside the repo is also not proof against someone who can rewrite it. Repository-wide enforcement would need a server-side required check, a protected source of attestations, and explicit bypass permissions. The local record described here prevents accidental stale-result reuse within its covered workflow. The guard has refused me three times this week, and each refusal was correct. Twice the record for a commit had been replaced by a quick-tier record, because I ran the full tier, then pushed, and the pre-push hook's quick run wrote its own record over the full one for the same SHA. The rule that came out of that is push first, attest second. The third refusal was a compound shell command: a merge chained behind another command, which the guard could not parse, so it fell back to the checkout's current commit and found nothing. A bare merge command, alone, from the directory holding the tested commit, is now the only form I use. ## The mirror has to know when it drifts The one hazard that can make all of this worse than hosted CI is drift. The moment the local runner and the committed workflow disagree, a local pass stops predicting a hosted pass, and the gate is lying with a green light. So the runner hashes each repository's workflow files on every run and compares them with the hashes it last blessed. A changed workflow prints a drift warning until someone reads the new file and blesses it explicitly. The runner also prints every lane it skipped and why, so a local run's coverage is a list you can read, not a feeling. What it skips is real. A workstation is not a clean image: it has the Node patch version I happen to have, the system packages I happen to have installed, and, this week, another session's end-to-end server already listening on port 3000. Two of my full-tier runs this week failed the end-to-end lane alone while the same spec passed in isolation seconds later, and both coincided with a neighbor holding the port. The runner cannot tell that from a real regression, so the rule until it can is one full tier at a time, machine-wide, and a skipped lane is a named gap I carry into the merge decision rather than a pass. ## What stayed on the hosted runner Three workflows kept their place. Deploy on merge, because the artifact should be built where it lands, by a runner that is not my workstation. Rollback, because the day I need it is the day my workstation may be the problem. And the secret scan on every push to main, because it is cheap, it runs where the code is public, and a second pair of eyes on credentials costs nothing worth saving. Everything else, the lint and test and build and end-to-end lanes that used to burn the allowance on every push to every pull request, is dispatch-only now, there if a reviewer asks, silent otherwise. The monthly bill stayed in single digits after that, and the part I did not predict is that the gate got stricter, not looser. Hosted CI passed or failed a pull request. The local gate refuses a merge, refuses a deploy, rejects a record made on a dirty tree, refuses a hollow run, and warns on a stale workflow, and a refused merge names the commit it is refusing. The allowance was what ran out. The record was what I had been missing. --- # What a month of AI coding agents actually costs: $599 on subscriptions, $7,566 at API prices URL: https://caskeycoding.com/blog/what-a-month-of-ai-coding-agents-costs Published: 2026-09-24 Author: Eric Caskey Topic: ai Tags: AI, agents, cost, claude-code, orchestration > My usage tracker estimated $7,566 at API list prices for a month covered by $599 in subscriptions. The token breakdown, the denominator behind its 98.2 percent cache ratio, and the limits of that counterfactual. For thirty days of running my agent fleet, the tracker recorded about 8.7 billion tokens through three vendors' models in 70,227 calls across 2,861 sessions. It estimated $7,566 at API list prices. The three subscriptions that covered the work cost $599, a comparison of nearly thirteen to one. Most of the recorded tokens were cache reads. The local tracker reads session transcripts and applies model-specific API rates to reported usage. Two of its ten models use estimated token counts because their CLIs do not report exact ones; they account for about seven percent of the modeled cost. The aggregates below summarize that accounting. They do not include the complete per-model token categories, rate dates, or cross-vendor normalization needed to independently reproduce the $7,566 estimate. Treat it as a modeled comparison for this workload, not a bill I paid or a current price quote. ![Two horizontal bars compare the modeled $7,566 API total with $599 in subscriptions. The 98.2 percent cache ratio excludes cache writes from its denominator.](/blog/what-a-month-of-ai-coding-agents-costs-bars.svg) *The API bar is an estimate. The graphic's cache ratio is reads divided by reads plus fresh input; cache writes are excluded.* ## Ninety-five percent of the tokens were re-reads The published breakdown was 154 million fresh input tokens, 40 million output tokens, 222 million cache writes, and 8.29 billion cache reads. About 95 percent of total token volume was reads. The 98.2 percent figure uses a different denominator: `8.29 / (8.29 + 0.154)`, reads divided by reads plus fresh input. It excludes writes. Including the 222 million writes gives about 95.7 percent of input served from cache. [Anthropic's usage documentation](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) includes reads, writes, and uncached input in total input; a cross-vendor tracker also needs to normalize each provider's fields before combining them. That shape is what an agent session looks like from the inside. Every turn the model takes, it re-reads the system prompt, the rules files, the tool results and the whole conversation so far, then writes a few hundred tokens of decision. A product call is the opposite shape: a short prompt, a short answer, nothing carried forward. The largest of these APIs prices cache reads at a tenth of fresh input, which is why I once spent [a post on getting the cache prefix right](/blog/prompt-caching-four-layers), and even with that discount the re-reads dominate the API-equivalent bill. A flat-rate subscription with a weekly allowance is priced for exactly this shape, which is presumably why the allowance, and not the dollars, is the scarce resource on every plan I pay for. ## The supervisor costs more than the typists By model, the frontier tier I use for supervision, planning, briefs, diff review and merge decisions, accounted for $3,455 of the $7,566, across 7,954 calls and 2.9 billion tokens. It wrote almost no code. The second vendor's mid-tier model took 23,201 calls and 2.6 billion tokens for $1,468; the mid-tier on the first vendor took 9,075 calls for $1,221. The model that typed the bulk of the last week's merged code, a third vendor's coding model that I dispatch with a written brief, took 9,391 calls and 560 million tokens for $295. The small model that runs the mechanical wrapper scripts made 9,771 calls for $97. Read as a ratio, the frontier supervisor cost twelve times what the typist cost, for the same month of shipped work. That is the price of the routing rule I wrote in July: [the frontier model thinks, cheaper models build, the off-quota model types](/blog/fable-thinks-opus-builds-kimi-types). The rule moved the typing off the expensive meter, as intended. It did not move the reading, because reading is what supervision is. Every review turn re-reads the session, and the session is where all the context accumulates, so the supervisor's bill scales with session length while the typist's scales with diff size. The typist's diffs were larger. The supervisor's sessions were longer, and length won. The daily peaks make the same point. The four most expensive days of the month, August 26, August 27, August 31 and September 7, each moved between 860 million and 1.16 billion tokens and would each have billed between $640 and $1,004 on the API. Those were merge waves: dozens of pull requests reviewed and landed in a day, which means dozens of full-session re-reads by the supervisor, not dozens of large diffs. On the subscription those days cost nothing extra. They cost allowance, and an earlier wave in July, when a fan-out of frontier-tier subagents tripped the allowance outright, is how I learned to cap the fan-out. ## What the thirteen to one does not mean It does not mean the API is overpriced, and it does not mean I would have spent $7,566 without the subscriptions. Nobody runs this shape of workload on a metered API without changing the shape: shorter sessions, aggressive context trimming, fewer review passes. The counterfactual is what this behavior would have cost, and the behavior exists because the flat rate made it free at the margin. The subscription did not save me $6,967. It let me work in a way that would not have occurred to me on a meter. It also does not mean subscriptions are the answer for everything. The site's backend makes its own model calls when a reader asks the finance tools a question, and those run on the API, metered per call. They are small, independent and infrequent, the opposite shape from an agent session, and the bill is small enough that I have declined to dress it up as a cost story before. The line between the two meters is the line between development and runtime, and I have never had a reason to cross it in either direction. ## Route by who pays for the retry The rule I run now is one sentence. If a call's cost scales with how much has to be re-read, it goes on a subscription; if it scales with how much has to be answered, it goes on the API. Agent sessions re-read, so they live on three flat-rate plans, and the routing across them exists to protect the weekly allowances, not the dollars. Product calls answer, so they live on the meter, where a short request costs a fraction of a cent and nothing accumulates between requests. I watch the reads-to-reads-plus-fresh-input ratio alongside cache writes and the remaining subscription allowance. If that ratio falls, I need to inspect the workload and token categories before blaming lost reuse. More fresh work, a different model mix, and missed cache reads can all change it. The label and denominator have to stay attached to the number. --- # What Newey-West does to a backtest t-statistic: 18 months became two URL: https://caskeycoding.com/blog/newey-west-standard-errors-backtest Published: 2026-09-23 Author: Eric Caskey Topic: finance Tags: finance, backtesting, statistics, quant, evidence > A Newey-West standard error accounts for dependence in a series. My engine's 18 monthly readings had an estimated effective sample size near two. The arithmetic, the lag choice, and the evidence gate that kept the result from changing a weight. A Newey-West standard error accounts for dependence in the series behind a mean. On my engine's 12-month panel, 18 monthly readings with an unadjusted t-statistic of negative 18.8 had an estimated effective sample size near two. A result that had looked decisive failed the engine's evidence floor. The reading came out of the diagnostic harness I wrote for my [factor scoring engine](/finance/methodology). The harness scores a universe of stocks at each month-end, waits for the forward return over some horizon, and computes a cross-sectional information coefficient per date: the rank correlation between score and what happened next. Average the per-date ICs, divide by the standard error of that average, and you have a t-statistic. The 12-month quality reading came back at negative 0.57 across 18 dates with an unadjusted t of negative 18.8. A quality factor that predicts returns backward is a factor built wrong, and that is the fault I went looking for first. ![Eighteen overlapping 12-month windows drawn as staggered red bars, each starting one month after the last, with an arrow to two green bars laid end to end and the label effective n = 2.](/blog/newey-west-standard-errors-backtest-overlap.svg) ## Why 18 readings are not 18 facts The naive standard error is the standard deviation of the per-date ICs, 0.13 here, divided by the square root of 18. That formula assumes the readings are independent draws. A 12-month forward return from a January month-end snapshot covers February through the following January; the next snapshot's window shares eleven of those twelve months. That is about 92 percent overlap in the return windows, though it does not by itself determine the correlation between IC readings. The scores are sticky as well, since a company's quality metrics do not reshuffle month to month, and factor performance can persist across several quarters. Those are reasons to examine dependence rather than count every monthly reading as an independent draw. Positive autocovariances can make the naive standard error too small and the absolute t-statistic too large. The amount of window overlap alone does not tell me the size of that error; I need the IC series and its estimated autocovariances. The [Newey and West (1987)](https://doi.org/10.2307/1913610) correction replaces the sample variance with a long-run variance. Take the variance of the series, then add twice the autocovariance at lag one, lag two, up to lag L, each weighted by a Bartlett factor of one minus the lag over L plus 1, so nearby lags count fully and distant ones fade out. The helper in my harness is fourteen lines: the mean, the deviations, the lag-zero variance, a loop that accumulates weighted autocovariances into the long-run variance, then the mean divided by the square root of that variance over n. I now print a variance-ratio diagnostic on every cell: effective n, calculated as n times the naive variance over the estimated long-run variance, capped at n. For the 12-month panel it came out near two. That estimate fails my evidence floor; it does not turn the series into two literal independent observations or establish an exact reference distribution. A [Student-t distribution with one degree of freedom is defined](https://www.itl.nist.gov/div898/handbook/eda/section3/eda3664.htm), but assigning that degree of freedom from this diagnostic would need a separate justification. With only 18 dependent readings, a self-selected panel, and scores from two engine versions, I could not treat the result as a reliable basis for reversing the factor. ## The lag is a decision, not a default Nothing in the formula tells you L. Too small and the correction misses the overlap; too large and the estimate of the long-run variance gets noisy in its own right. The engine's comparator derives L from the horizon: the number of series steps two snapshot dates can be apart and still share a forward window, which is the horizon divided by the median spacing between dates, rounded up, minus one. On month-end dates that is lag 11 for a 12-month horizon and lag 2 for a 3-month one, clamped to n minus 1. When it does not know the horizon it falls back to the automatic Bartlett rule, the floor of 4 times n over 100 to the two-ninths power. A Bartlett estimator truncated at L ignores every lag above L, so a lag chosen without looking at the horizon is a lag that can miss the overlap entirely. My standalone diagnostic scripts use a different choice for monthly net-of-cost portfolio returns: fixed lag 3, motivated by quarterly score refreshes. The IC series use the comparator's horizon rule. The rule of 0.75 times T to the one-third power, rounded down, gives lag 3 at 64 observations and lag 1 at 18. I had chosen the larger lag when conventions disagreed, but that choice is not automatically conservative. The [Bartlett calculation adds signed autocovariances](https://www.statsmodels.org/stable/_modules/statsmodels/stats/sandwich_covariance.html#S_hac_simple), so adding lags can lower the estimated variance. Sensitivity to justified lag choices belongs beside the result, especially for this short panel; no such sensitivity result is established here. ## I told this story with the wrong numbers once My June post on [backtesting without fooling yourself](/blog/backtesting-without-fooling-yourself) told a version of it with a different pair: a factor whose naive t was negative four and whose Newey-West t was negative 1.6. The mechanism in that paragraph is right and I would write it the same way again. The numbers came from a point-in-time scoring harness that I later found had two defects, a market-cap field that leaked today's value into historical dates and a renamed API field that left one factor reading a constant. Every negative quality reading that harness produced is void, including the one I quoted. The correction ratio was real; the signal it was correcting was an artifact. I would rather say so here than let the older paragraph stand as the reference. The clean re-measurement, on a point-in-time mid-cap corpus with the leaks fixed, put the same factor's 6-month IC at plus 0.046 with a Newey-West t of plus 3.5 over 2015 to 2021. Three caveats travel with that number in every record the engine writes. It is an upper bound, because the corpus has no in-window delistings. It is an information coefficient, not a net-of-cost portfolio return, and the engine has not shown that any factor survives transaction costs, which is the bar that decides anything. And in the 2024 to 2025 sub-window, the one with the least survivorship contamination, the same factor's 6-month IC is plus 0.008 with a Newey-West t of plus 0.39, indistinguishable from zero. Two readings compete, and both are open. ## The gate that makes the correction unskippable A correction you remember to apply is a correction you will forget under deadline. After the 12-month episode I moved the effective-n calculation from a diagnostic printout into the engine's decision protocol. Every IC and t-statistic the engine records now carries an effective-n field. No factor weight is promoted unless the holdout cell that validated it has an effective n of at least 8 at the headline horizon, and none of the revision triggers, the checks that can propose a change to how a factor is scored, fires unless every horizon it names clears the same floor. Below the floor the cell reports itself as immature and not actionable, the same state the engine uses for a signal with no forward record at all. The floor of 8 is provisional and I expect to move it. The rule that a cell below the floor cannot change a weight is not. The 12-month cell was still immature when I wrote this. One non-overlapping annual window per year is a useful planning approximation, but estimated effective n depends on the observed series and can change as data accumulate. Shorter horizons mature sooner only if their dependence and other evidence checks permit it. Choosing a horizon affects how long I may need to wait before a result can justify a weight change. --- # Lesson 4: Define Repository Boundaries and Isolate Agent Work URL: https://caskeycoding.com/blog/lesson-4-boundaries Published: 2026-09-22 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, repository-layout > Document which repository owns each part of the project. Assign each agent a specific scope and a separate checkout to prevent overlapping edits. Includes layout instructions and a repository-splitting exercise. **What you'll learn:** document which repository owns each part of your project, give each concurrent agent a separate checkout, and check that its changes stay within the assigned repository. **What you need:** a project in one folder, Git, your coding agent, and the `specs/` directory from Lesson 1 if you have one. **Time:** about an hour for the example split. **Cost:** no additional services or subscriptions. ## The idea Before an agent edits a file, require it to name the repository it will change. Give concurrent agents separate checkouts so their edits do not share working files. These are two different controls: repository boundaries scope commits, while checkout separation prevents one session from including another session's unfinished files in its commit. A repository is not a filesystem sandbox. An agent with access to neighboring directories can still edit them. The instruction file below defines permitted scope; enforcing filesystem access requires separate controls. ![A shared checkout lets two sessions include each other's edits in a commit. Separate repositories define the assigned scope, and separate checkouts isolate concurrent working files. A task requiring two repositories stops for separate specs.](/blog/lesson-4-boundaries-two-rows.svg) ## Why start here Lessons 1 and 2 recorded requirements and decisions. Now identify where each requirement will be implemented. I separate infrastructure, application logic, and the interface when they need independent releases. A small project can retain one repository and still isolate concurrent work in separate checkouts. Splitting repositories is useful when the ownership and release boundaries justify the extra configuration. ## The template This example uses four repositories under one parent directory: ``` myproject/ specs/ # intent: specs and decisions (Lessons 1-2 live here) infra/ # what runs where: cloud resources, DNS, buckets api/ # logic: handlers, jobs, data web/ # UI: pages, styles, static files CLAUDE.md # or AGENTS.md: the standing file, one line added below ``` The template uses `home` to mean the repository assigned to a change. `Home: web` identifies `web/`; it grants no filesystem permissions. Save the layout file as `.claude/skills/layout/SKILL.md` in the parent directory, or put its instructions in `AGENTS.md` if your tool has no skills: ```markdown --- name: layout description: Where each kind of change lives in this project, and the rule for working in exactly one home per change. Read before touching any file. --- # Layout: one home per stack | Home | Holds | Own repository | Ships by | |---|---|---|---| | `specs/` | intent: specs and decisions | yes | nothing; every session reads it | | `infra/` | what runs where: cloud resources, DNS, buckets | yes | the infra deploy command | | `api/` | logic: handlers, jobs, data | yes | the api pipeline | | `web/` | UI: pages, styles, static files | yes | the web pipeline | ## Routing rule 1. Read the standing file and the spec first. Before editing any file, name the home this change lives in, on one line: `Home: web`. 2. Edit only inside that home. If the change needs two homes, stop and say so; it is two changes, and the spec comes first. 3. Cite the spec the change serves. Build what its acceptance criteria say; change nothing listed under its Out of scope. ## Git rules (each home is its own repository) - One session per checkout. A second session gets its own clone of the home; branches alone do not separate working files. - Start from a fresh branch off that home's main. Never commit on main. - Commit only the files you changed, by path. Never `git commit -a` or `git add -A`. - Leave untracked files you did not create alone unless the change needs them; they may be someone else's work in progress. - If the home has a remote, push the branch before reporting done; otherwise say the commit is local only. - One commit never spans two repositories. ``` The standing file is the instruction file your agent reads when a session starts, such as `CLAUDE.md` or `AGENTS.md`. Add this line: ``` Read the layout skill before touching any file, and follow its routing and git rules. ``` ## Each part, and the mistake it prevents The "Ships by" column records release commands. Shared release commands can suggest code belongs together, but command count alone cannot decide repository structure. Consider interfaces and how often changes must be coordinated. A separate `specs/` repository gives requirements their own review history. It does not stop an agent from changing a spec to match its implementation. Review requirement changes explicitly wherever you store them. The routing rule produces a statement you can compare with the final diff. A page change requiring an API change should stop under this template until both changes have specs. The statement alone cannot prevent a write outside the named repository. Branches do not separate files in a shared checkout. Two sessions using that directory see the same working files, even if each intended to use a different branch. Give the second session a separate clone. Stage files by path so a commit does not collect unrelated edits. Leave unfamiliar untracked files for their author to handle. The push rule distinguishes a remote branch from a local-only commit. Report that distinction in the completion message. A commit belongs to one repository; coordinated changes across repositories need separate commits and an explicit release order. ## A worked example In the recorded exercise, `specs/web/001-robots.md` required `web/public/robots.txt` to disallow `/admin`. It excluded `api/` and `infra/`. The scratch repositories had no remotes. A fresh session received the standing file and the request to implement that spec on a branch: ``` Home: web, per specs/web/001-robots.md, this only touches web/public/robots.txt. ... Done. Created web/public/robots.txt (disallows /admin) on branch add-robots-txt in the web repo, committed as ff93e65. Both acceptance checks from specs/web/001-robots.md pass. No remote is configured for web, so this commit is local only. ``` The session named `web` before editing. Afterward, `specs/`, `infra/`, and `api/` remained clean and on main. In a second run, I requested an API health endpoint and a homepage status indicator without a spec. The session identified the two repositories and stopped; all four remained untouched. These observations show compliance in those two runs, without establishing an access restriction. ## How it scales At the time of the original lesson, my setup had fourteen repositories under one parent directory. The table provided one place to find their responsibilities. The parent directory itself was outside Git. Approve each split before moving files. The agent can propose the table and migration, initialize the new repositories, and prepare commits. Moving files also changes paths used by imports, builds, checks, and deployment scripts; verify those before treating the split as complete. ## From the field On June 29, a formatting session ran `git commit -a` in the backend checkout where I was developing a feature. It committed my tracked import to main but omitted the new module files, which were untracked. Main then raised an import error on load. The next deployment would have sent that code to twenty-odd functions. A small pull request removed the orphaned import. After that incident, each feature used its own checkout off main. The failure required both shared working files and a bulk commit; the layout instructions address each directly. ## Exercise 1. Save the layout file and add its instruction to the standing file. 2. Identify the directories containing infrastructure, application logic, and interface code. Ask the agent to propose repository boundaries and review the table. 3. Back up the project. For this four-repository exercise, have the agent move files, run `git init` in each repository, and create one `init ` commit per repository. This starts fresh history. If history must be preserved, plan the migration with `git filter-repo` before moving files. 4. Move `specs/` into its own repository, retaining spec numbers. Adjust verification paths for their documented working directory, such as `test -f web/public/robots.txt` from the parent directory. 5. Copy Lesson 3's `hooks/` and `checks/` into each code repository and configure `git config core.hooksPath hooks`. Check that the spec check finds the adjacent `specs/` directory. Configure remotes where you will use the pre-push hook, and verify the hook runs there. 6. Select one ready spec that excludes the other repositories. Give it to a fresh session through the standing file, using a separate checkout for any concurrent work. 7. Compare the agent's opening repository declaration with its final diff. Check the other repositories for unintended edits. ## You're done when The session must name its repository before editing, satisfy the selected spec, and leave only the intended changes in that repository. In the clean exercise, the other repositories remain clean and on main. Then request a change spanning two repositories without specs and confirm the session stops before editing. This checks instruction compliance for the requests you tested. ## Common mistakes Splitting by file extension can separate code that must change together. A parent Git repository can accidentally track files intended to belong to child repositories. Separate branches in one checkout still share working files. Check each of these before starting parallel sessions. ## Further reading Martin Fowler's [Bounded Context](https://martinfowler.com/bliki/BoundedContext.html) discusses model boundaries. Parnas's [On the Criteria To Be Used in Decomposing Systems into Modules](https://www.win.tue.nl/~wstomv/edu/2ip30/references/criteria_for_modularization.pdf) examines module decomposition. The [Claude Code skills reference](https://code.claude.com/docs/en/skills) and [AGENTS.md convention](https://agents.md/) describe instruction-file formats. ## Next lesson [Define What Agents Can Do Without Approval](/blog/lesson-5-the-charter) specifies permitted actions, actions requiring approval, and completion evidence. --- # Lesson 3: Enforce Project Rules with Automated Checks URL: https://caskeycoding.com/blog/lesson-3-the-check-that-fails Published: 2026-09-21 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, guardrails > Turn a project rule into a script that detects violations. Run it before each push and test it with a deliberately invalid change. Learn what a local Git hook checks and how it can be bypassed. **What you'll learn:** write a script that detects a project-rule violation, run it through a Git pre-push hook, and test both rejection and success. **What you need:** the `specs/` directory from [Lesson 1](/blog/lesson-1-the-spec-directory), at least one spec with a `verify:` line, a project with a Git remote, and your usual agent. The scripts use a POSIX shell, Git, `grep`, and `sed`; on Windows, use a compatible shell such as Git Bash. **Time:** allow fifteen minutes for setup and about two per simple rule afterward. **Cost:** zero new dollars with your existing tools. ## The idea Turn a repeated project rule into a script that exits nonzero when it detects a violation. Configure Git to run the script before pushing. When the hook runs and fails, Git stops that push. A local hook is bypassable. `git push --no-verify` skips it, and each clone needs its own configuration. Its checks also have limited coverage: a filename check can reject `.env` without detecting secrets saved elsewhere. State the rule each script actually tests. ![A written rule depends on the agent reading and following it. With the pre-push hook installed and active, Git runs the check and a nonzero result blocks the push. The local hook can be bypassed.](/blog/lesson-3-the-check-that-fails-two-rows.svg) ## Why start here In July, I counted fourteen rules I had written over two months. One was enforced by a command. Start with a rule you repeatedly raise in review, then add a command for the acceptance checks in your in-progress specs. Written decisions still need judgment; automate the conditions you can evaluate directly. ## The template Save this as `hooks/pre-push`, then run `chmod +x hooks/pre-push` and `git config core.hooksPath hooks`: ```sh #!/bin/sh # hooks/pre-push: git runs this before every push. One script per rule lives in checks/. # A check fails by exiting non-zero; the first failure stops the push. for check in checks/*.sh; do sh "$check" || { echo "BLOCKED by $check" >&2; exit 1; } done echo "all checks passed" ``` Save the first rule as `checks/no-tracked-env.sh`: ```sh # Rule: never commit a .env file, not even in a commit a later one fixes. if git ls-files | grep -qE '(^|/)\.env$' || git log --branches --not --remotes --diff-filter=A --name-only --format= | grep -qE '(^|/)\.env$'; then echo ".env is in a commit about to be pushed; undo that commit, then re-commit" >&2 echo "the other files by path: git reset HEAD~" >&2 exit 1 fi ``` This checks paths named exactly `.env` in the tracked files and additions in local branch commits absent from known remote refs. It is not a general secret scanner or an exact reconstruction of the push payload. Save this as `checks/specs-verify.sh`: ```sh # Rule: every in-progress spec must pass its own verify command (Lesson 1). [ -d specs ] || cd .. # after Lesson 4's split, specs/ sits beside this repository [ -d specs ] || { echo "no specs/ directory found" >&2; exit 1; } for spec in $(grep -rl '^status: in-progress' specs --include='*.md'); do cmd=$(sed -n 's/^verify: *//p' "$spec" | sed 's/ *#.*//' | head -1) [ -n "$cmd" ] || { echo "$spec has no verify command" >&2; exit 1; } sh -c "$cmd" || { echo "$spec failed its verify command: $cmd" >&2; exit 1; } done ``` This starter script expects the shown frontmatter layout and spec paths without spaces. Review every `verify:` command before running it; the script executes those commands through the shell. Confirm their working directory, particularly after the repository split in Lesson 4. If there are no matching in-progress specs, the loop runs no verification commands. Add this to the file your agent reads at session start: ```markdown When I state a rule for the second time, write checks/.sh so it exits non-zero when the rule is broken. Show me the check failing on a deliberate violation, then passing after the fix, before you commit it. ``` Commit `hooks/` and `checks/` together. The files travel with a clone; the `core.hooksPath` setting does not. Configure it in each clone that should use the hook. ## Each part, and the mistake it prevents The runner invokes each shell script in `checks/` and stops at the first nonzero exit. Keep at least one check in the folder. Name each file for its rule so the directory listing identifies the available checks and Git history records their changes. Git reads the exit code. A message that says “failed” while the script exits 0 allows the runner to continue. Include enough detail in the failure message to identify the violated rule and the correction needed. The spec check rejects a missing `specs/` directory or an in-progress spec with no verification command. A passing command establishes its own assertions. It does not establish untested acceptance criteria or that every relevant spec was selected. Have the agent demonstrate failure and success before accepting a new check. Review both the rule it implements and where it runs. A script can be correct while the integration never invokes it. ## A worked example A small site's `specs/app/001-robots.md` contains `verify: test -f public/robots.txt`. The required file is missing. In the recorded run, the push failed: ``` $ git push specs/app/001-robots.md failed its verify command: test -f public/robots.txt BLOCKED by checks/specs-verify.sh error: failed to push some refs ``` After creating and committing the file, the next push printed `all checks passed`. A deliberate `.env` violation produced this result: ``` $ echo SECRET=1 > .env && git add -f .env && git commit -m "oops" $ git push .env is in a commit about to be pushed; undo that commit, then re-commit the other files by path: git reset HEAD~ BLOCKED by checks/no-tracked-env.sh error: failed to push some refs ``` I undid that commit as directed and pushed again. Across four pushes, two were blocked and two succeeded. The remote never received the `.env` file. These transcripts record that run; they do not verify a reader's installation. I also confirmed in the same run that `--no-verify` bypassed the hook. ## When it grows [Lesson 7](/blog/lesson-7-the-pipeline-reviews) runs checks in continuous integration (CI), the automated jobs triggered by repository changes. The hosting platform can require successful checks before merge. This adds enforcement beyond a developer's local hook. Verify the job triggers, required-check settings, and bypass permissions; a workflow file alone does not establish that a merge will be rejected. ## From the field In June, I wrote a check for a style rule I had repeated for weeks. I connected it to the tool my agents used and fed the script a sample payload. It refused the sample. Five weeks later, live sessions were still breaking the rule, and the check had never blocked one. The sessions ran in a mode that ignored the signal the script used to refuse the action. My test exercised the script without checking how that mode handled its result. Reproducing the violation in a live session took one minute. Rewriting the check to refuse by exit code took ten more. The subsequent audit produced the fourteen-rules, one-enforced count. ## Exercise Use a disposable project and test remote with dummy data. Copy the hook and two checks, make the hook executable, configure `core.hooksPath`, and add the standing instruction. Do not use real credentials for the `.env` example. Give one spec `status: in-progress` and a verification command that currently fails. Attempt a push and confirm that the failure names the spec and command. Correct the condition and confirm that the next push reports success. Give the agent your most-repeated rule and ask it to write the corresponding check. Introduce a known violation in a throwaway commit, attempt a push, and confirm that the new check blocks it. Undo the violation and confirm a successful push. Inspect the final branch to ensure the deliberate violation is gone. ## You're done when You have observed two expected rejections through `git push`: one from spec verification and one from your rule check. Each failure identifies the cause, and each correction permits the push. Record the tested repository, check, command, and result. A direct script run covers the script. The push exercise also covers Git invoking it in that clone. Neither observation proves that another clone, CI job, or merge rule is configured correctly. If the violation passes, inspect `core.hooksPath`, executable permissions, the selected checks, and their exit codes before continuing. ## Common mistakes Deleting a committed secret in a later commit leaves it in earlier history. The example reads unpushed history for that reason. Rotate a real secret that has reached a remote. Avoid adding many speculative rules before testing the first one. Repeat the integration test when the surrounding tool changes. [Lesson 9](/blog/lesson-9-the-monthly-drill) schedules a monthly drill for a selected check and records exactly which controls it exercises. ## Further reading The [Git hooks chapter](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) covers hook points. The [pre-commit framework](https://pre-commit.com/) supports shared check configuration. GitHub documents [protected branches](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/managing-protected-branches/about-protected-branches), and Claude Code documents [hooks inside agent tool calls](https://code.claude.com/docs/en/hooks). ## Next lesson [Define Repository Boundaries and Isolate Agent Work](/blog/lesson-4-boundaries) assigns code ownership and separates concurrent edits into different checkouts. --- # Lesson 2: Record Technical Decisions and Their Reasons URL: https://caskeycoding.com/blog/lesson-2-decision-records Published: 2026-09-17 Author: Eric Caskey Topic: methodology Tags: spec-driven-development, AI, engineering-methodology, agents, decision-records > Write a decision record documenting what you chose, which alternatives you rejected, and when to reconsider. Instruct agents to read relevant decisions before proposing changes, then test whether they retrieve them. **What you'll learn:** document technical choices, rejected alternatives, and conditions for reconsidering. Instruct agents to retrieve those records before proposing changes. **What you need:** the `specs/` directory from [Lesson 1](/blog/lesson-1-the-spec-directory), an empty `decision/` folder, and the agent you already use. **Time:** allow ten minutes for setup and about five per decision. **Cost:** zero new dollars if your existing agent subscription covers the work. ## The idea A decision record explains a technical choice and the reasons behind it. The code may show that your site uses static files, but it may not explain why you rejected a server, which costs you accepted, or when a server would become appropriate. Keep one file per settled choice and list the files in an index. Add a standing instruction that tells the agent to read the index and open relevant records before making changes. When a request conflicts with an accepted decision, the agent should cite it and stop for your decision. This depends on retrieval and compliance. A file can be missing, outdated, or ignored. The exercise tests whether a fresh session follows the instruction for one conflicting request; it does not establish that every future session will do so. ![An agent without decision records reads the code and proposes a rejected option. An agent following the standing instruction reads the index, opens D-003, cites the conflicting decision, and stops before editing.](/blog/lesson-2-decision-records-two-rows.svg) ## Why start here A feature spec states the result you intend to build. A decision record explains a choice that may affect many features. Keep them separate and link the relevant record from a spec when it constrains the implementation. Start with decisions you repeatedly explain: the deployment approach, a dependency you declined, or a service you chose to avoid. Record the actual reason. “We prefer this framework” tells the next session less than the compatibility or maintenance requirement that determined the choice. ## The template Save this as `specs/decision/_template.md`: ```markdown # D-000: Status: draft Date: YYYY-MM-DD (draft until checked against the code; then accepted; later superseded by D-NNN) ## Context What forced a choice, in one paragraph: budget, time, one person, a failure. ## Decision One sentence, present tense. "We build the site as static files served from object storage." ## Alternatives considered -