Lesson 9: Keep the system honest

What you'll learn: how to prove, once a month, that the barriers you built still refuse: one drill that plants a violation, watches the hook, the pipeline, and the merge refuse it, reports to Lesson 8's topic, and puts everything back. What you need: the gate from Lesson 7 and the scripts from Lesson 8, working; Lesson 3's checks/ in a home on GitHub. Time: twenty minutes once, then a minute a month. Cost: zero new dollars; each drill spends one short pipeline run.

The idea#

A check you never see fail is indistinguishable from a dead one. Lesson 3's check has refused nothing since the day you watched it refuse your planted .env, and every green since fits two states: a check that holds, and one that stopped running because the hook path moved, the workflow was renamed, or the branch rule was switched off in a settings page you have not opened since. Both are silent. The drill tells them apart: once a month, plant the violation the check exists to refuse, watch it refuse through the path the agent takes, and put everything back. The figure is Lesson 7's barriers with one column added, the signal that proves each is alive; a row with no date is one you have never seen refuse.

Why start here#

Lesson 3 owed you this schedule, Lesson 7 the column, and Lesson 8 somewhere to report. This is the last lesson because its loop keeps the other eight from decaying.

The template#

One script in gate/ beside the runner, two lines in the watch, one timer line. The drill:

#!/bin/sh
# gate/drill.sh <home>: break Lesson 3's check on purpose and watch each rung refuse.
# In a throwaway copy, plant a .env in <home> and push it: the hook must refuse. Push
# past the hook with --no-verify and open a pull request: the pipeline must fail it.
# Try to merge, then try again as an administrator: the platform must refuse both.
# Then remove what the drill made and confirm main did not move. One dated message
# through Lesson 8's notify either way.
set -u
cd "$(dirname "$0")/.."
home=${1:-web}; when=$(date -u +%Y%m%d-%H%M%S); branch=drill-$when
here=$(pwd); copy=$(mktemp -d); pr=; done=
log() { printf '%s %s\n' "$(date -u +%FT%TZ)" "drill $when: $*" >> "$here/gate/drills.log"; }
restore() {
  [ -z "$pr" ] || gh pr close "$pr" --delete-branch >/dev/null 2>&1
  git push -q --no-verify origin --delete "$branch" >/dev/null 2>&1
  cd "$here" && rm -rf "$copy"
}
fail() {   # a barrier that did not refuse, or a restore that did not hold
  log "FAILED: $1"; restore; done=1
  git -C "$home" ls-remote --exit-code origin "refs/heads/$branch" >/dev/null 2>&1 \
    && set -- "$1; $branch is still on origin"
  sh gate/notify.sh blocked "drill $when: $1"; exit 1
}
tries=0                                     # Lesson 8's lock, so a tick cannot merge mid-drill
until mkdir gate/running 2>/dev/null; do
  tries=$((tries + 1)); [ "$tries" -lt 20 ] || fail "a run held the lock for ten minutes"; sleep 30
done
trap '[ -n "$done" ] || restore; rmdir "$here/gate/running"' EXIT   # a killed drill still cleans up
for h in */; do                             # the runner's layout, so every check finds its neighbours
  h=${h%/}; url=$(git -C "$h" remote get-url origin 2>/dev/null) || continue
  git clone -q "$url" "$copy/$h" && git -C "$copy/$h" config core.hooksPath hooks || fail "could not clone $h"
done
cd "$copy/$home" || fail "no copy of $home"
main=$(git rev-parse origin/main)

# 1. The hook (skippable): the push must stop at checks/no-tracked-env.sh.
git checkout -q -b "$branch"
echo "DRILL=1" > .env && git add -f .env && git commit -q -m "drill $when: a .env the check must refuse"
out=$(git push -u origin "$branch" 2>&1) && fail "the hook let a .env through"
case $out in
  *"BLOCKED by checks/no-tracked-env.sh"*) log "the hook refused" ;;
  *) fail "the push stopped for another reason: $(printf '%s\n' "$out" | tail -n 1)" ;;
esac

# 2. The pipeline (enforced): past the hook with --no-verify, the checks status must go red.
git push -q --no-verify -u origin "$branch" || fail "could not push past the hook"
pr=$(gh pr create --fill --head "$branch" 2>/dev/null) || fail "could not open the pull request"
tries=0
while gh pr checks "$pr" --required > checks.out 2>&1 || grep -q -e pending -e "no .*checks reported" checks.out; do
  grep -q pending checks.out || grep -q "no .*checks reported" checks.out || break
  tries=$((tries + 1)); [ "$tries" -lt 60 ] || fail "the pipeline never finished"; sleep 10
done
grep -qE '^checks[[:space:]]+fail' checks.out || fail "the checks status did not fail: $(head -n 1 checks.out)"
log "the pipeline refused: $(grep -E '^checks[[:space:]]+fail' checks.out | head -n 1)"

# 3. The merge (enforced): the platform must refuse while the check is red, administrator or not.
for flag in "" --admin; do
  gh pr merge "$pr" --squash $flag > merge.out 2>&1 && fail "the merge went through${flag:+ with $flag}; revert it with a pull request"
  grep -qi -e "not mergeable" -e "status check" merge.out || fail "the merge failed for another reason: $(head -n 1 merge.out)"
  log "the merge was refused${flag:+ with $flag}: $(head -n 1 merge.out)"
done

# 4. Restore, and confirm it: no pull request, no branch, main where it was, no .env on it.
gh pr close "$pr" --delete-branch >/dev/null 2>&1 || fail "could not close the pull request"
[ "$(gh pr view "$pr" --json state --jq .state)" = CLOSED ] || fail "the pull request is still open"
git ls-remote --exit-code origin "refs/heads/$branch" >/dev/null 2>&1; code=$?
[ "$code" -eq 2 ] || { [ "$code" -eq 0 ] && fail "the drill branch is still on origin"; fail "could not read origin"; }
git fetch -q origin main || fail "could not fetch main"
[ "$(git rev-parse origin/main)" = "$main" ] || fail "main moved during the drill"
git ls-tree -r --name-only origin/main > tree.out || fail "could not read main's tree"
! grep -qE '(^|/)\.env$' tree.out || fail ".env is on main"
cd "$here" && rm -rf "$copy"; done=1
date +%s > gate/last-drill.new && mv gate/last-drill.new gate/last-drill
short=$(git -C "$home" rev-parse --short "$main")
log "restored; main at $short"
sh gate/notify.sh review "drill $when: hook, pipeline, and merge each refused a planted .env in $home; restored, main at $short"

Two lines at the end of gate/watch.sh, so a drill that stops running is noticed:

read -r drill < gate/last-drill 2>/dev/null || drill=0        # Lesson 9: a drill that stopped running
[ $(( $(date +%s) - drill )) -le $(( 35 * 86400 )) ] \
  || sh gate/notify.sh blocked "no drill in $(( ( $(date +%s) - drill ) / 86400 )) days"

The timer line, beside Lesson 8's two:

0 9 1 * *    cd /home/you/project && sh gate/drill.sh web  >> gate/timer.log 2>&1

Each part, and the mistake it prevents#

One violation, three rungs. The hook refuses the push, and the drill gets past it with --no-verify, the flag an agent would use: the skippable rung, skipped on purpose. The pipeline refuses the pull request. The platform refuses the merge, then refuses it again with the administrator flag the first refusal offers, because the branch rule includes administrators. Each refusal is matched by its own words, BLOCKED by checks/no-tracked-env.sh, a checks line reading fail, not mergeable, not by exit code alone.

The drill works in a throwaway copy of every home, as the runner does, so a check that looks for a neighbour finds it, and it takes the runner's lock first, so a tick cannot merge an item while the drill is comparing main. The restore is confirmed, not assumed: the pull request is closed, the branch is gone, main is at the commit the drill started from, and no .env is on it. A restore that fails any of these is blocked, the same tier as a barrier that let the violation through. A merge that went through is the one state the drill will not repair; a barrier that let a violation past is not one to automate around.

A pass is review, dated, the one exception to Lesson 8's rule that a pass is a log line, because a passed drill's silence and a dead drill's silence are the same silence.

When a drill fails, the fix goes in as an amendment, never a rewrite: a dated section at the bottom of the file that owns the barrier (the spec behind the check, the authority file, the decision record) holding what the drill expected, what it saw, and what you changed. The text above stays as written because, as with Lesson 2's records, a record you rewrite cannot show you were wrong. If the change can be a check, it becomes one, and the line in your standing instructions that described it is deleted.

A worked example#

The same scratch project as Lesson 7's, the drill saved and the two lines added to the watch. I ran sh gate/drill.sh web by hand, the timer line untested as in Lesson 8, and read gate/drills.log thirty-six seconds later:

2026-10-14T18:09:33Z drill 20261014-180930: the hook refused
2026-10-14T18:10:00Z drill 20261014-180930: the pipeline refused: checks  fail  8s  https://github.com/.../actions/runs/...  
2026-10-14T18:10:01Z drill 20261014-180930: the merge was refused: X Pull request ...#16 is not mergeable: the base branch policy prohibits the merge.
2026-10-14T18:10:03Z drill 20261014-180930: the merge was refused with --admin: GraphQL: Required status check "checks" is failing. (mergePullRequest)
2026-10-14T18:10:09Z drill 20261014-180930: restored; main at aae4bb8

The topic got one message, review tier, ending in restored, main at aae4bb8. Then I set gate/last-drill back forty days and ran the watch: blocked no drill in 40 days.

How it scales#

The next drill is a different row of the figure: a session told to force-push, which the deny rule should refuse, or a live: line pointed at text the page does not show. Each takes the same shape, a throwaway, the violation, the refusal in its own words, the restore confirmed. The column grows into a table with a row per capability and the signal that proves it is used, a harder question than alive: the first one I built had 114 rows, an empty proof column until the signals were wired, and then about two-thirds proved live. The dashed box holds what this series leaves out: a merge sorted by blast radius, a deploy gate after the merge, a watch on another machine, more than one agent at a time.

Who does the work: you fill in the dates and read one message a month; the drill breaks, watches, and restores, and the agent is not involved.

From the field#

After a machine migration in July, every guard in my pre-commit layer kept passing for sixteen days: deploy guards, secret scan, style scan, the rest of the layer. Each command pointed at the previous user profile's home folder, which no longer existed, and a hook that fails to start is non-blocking, so sixteen days of silence read as health. An audit found it, not an alarm. The fix that week rewrote the paths; the fix that lasted is a weekly job that shows every guard a planted violation and fails loudly when any stops rejecting it. The drill above is the same idea, cut down to one check.

Exercise#

1. Save the drill, add the two lines to the watch, and run sh gate/drill.sh web by hand. Read gate/drills.log, then the topic.
2. Open the pull request it closed and read the red check yourself, once.
3. Set gate/last-drill back forty days and run the watch. Blocked should arrive. Put it back.
4. Add the timer line.
5. Date every row of the figure you have seen refuse. The rest are the next drills, one a month.

You're done when#

The drill's message arrived with all three refusals and restored, its pull request is closed with the branch gone, and main's commit matches the message. Two results fall short: a red check in the log you cannot find on the pull request page means the drill is wrong, not the barrier; a pass in the log with no message means the fault is between the notify script and your phone, which the watch cannot see.

Common mistakes#

• Running the check script on a planted file. That proves the script; the agent pushes, opens a pull request, and merges, so the drill does too.
• Turning the barrier off so the drill passes. The amendment goes in the file; the barrier stays on.
• Running the drill in your checkout instead of a copy. A crash leaves a .env beside your work.

Twenty minutes once, a minute a month, and zero new dollars; from then on every green you see has a dated red behind it.

Further reading#

The Principles of Chaos Engineering is the long form of running the experiment where the real path is. MIT's 6.S191 Introduction to Deep Learning reinforcement learning lecture is where the agent-and-environment framing above comes from. GitHub's protected branches page covers the include-administrators setting the admin flag ran into.

After this lesson#

The series ends here, and the system still gets some of it wrong. Regressions while moving fast are the problem I have not solved: a change that passes every check and still breaks what an older lesson relied on. Unwired machinery is the standing temptation, and the empty proof column is where it shows.

One last word on what the nine lessons were about. Reinforcement learning is taught as two parts: an agent that picks actions, and an environment that holds the state and returns the reward. The attention goes to the agent. None of these lessons touched it. You did not choose the model, tune it, or write a cleverer prompt, and where a lesson gave the agent a line of instruction, the aim was to replace that line with a check. You changed the environment the model works in. The spec directory and the decision records are the state it can see. The check that fails and the gate that refuses are the reward, returned by the world instead of asserted in the prompt. The queue, the tiers, and the drill are how that environment keeps working while you are away. The model under your agent will be replaced, probably more than once a year, and none of what you built here leaves with it. The environment is the part you own, and it carries over to the next one. Asimov's laws lived inside the robot, as prose, and every story was about the day they did not hold. Yours live outside the agent, in a column you can read, and once a month you watch one of them refuse.