The Orange Pi That Maintains Itself

My Orange Pi 6 Plus runs local language models through Ollama, but CPU generation is slow enough that I use it mainly for retrieval and queued work. A 14B model produced roughly one or two tokens per second in my runs. Searching my own writing and generating a short answer was useful at that speed; waiting for a long chat response was not.

The hardware and workload in this setup#

Part	My setup	What it meant in use
Board	Orange Pi 6 Plus, twelve ARM cores	Ollama inference ran on the CPU.
Memory and storage	32GB RAM and an NVMe drive	Enough for the models and blog corpus I used here.
Models run	A 14B model through Ollama	I ran local inference on the CPU.
Generation	A few tokens per second; roughly one or two with a 14B model	Long interactive answers required patience. These are approximate observations from the original runs.
KV-cache quantization	Generation fell by more than half with it on	Turning it off more than doubled the speed in my runs.
Retrieval	Local embeddings over seventeen of my blog posts	Search found the relevant passages; the model wrote a short answer with source references.
Access and power	Headless on my LAN, no battery backup	I needed a way to recover remote access and handle interrupted writes.
Accelerators	GPU and NPU unused for inference in this setup	Their advertised capability did not describe the performance I was getting.

Before copying this setup, I would test one question against a small document set and check whether the cited passages support the answer. I would time retrieval separately from answer generation and record the model tag, quantization, context length, and runtime version with each run. The speeds above are enough to explain my workload choice; a reproducible comparison would need those details.

What it is bad at, and what it is good at#

Start with what it is bad at, because that part is quick. It is a poor interactive chatbot. Generation runs on the CPU at a few tokens a second. Watching a 14B model think at a token or two per second cures you of any idea that this replaces a hosted model for anything you are sitting there waiting on.

What it is good at is the work that does not depend on generation speed. The clearest win is retrieval. I pointed it at our own blog, seventeen posts, had it embed everything locally, and now I can ask questions and get answers grounded in our actual writing. "What is the rule about letting an LLM make decisions, and how is it enforced?" comes back citing the real posts that answer it. Embeddings are quick even on a CPU, and the slow step only has to write a short final answer, so the box's one weakness never lands on the workflow.

A question answered from my own writing, with the sources it pulled. The retrieval runs on the box; the slow model only writes the short final paragraph.

That reframed it for me. The local Ollama and retrieval path keeps the documents and inference on hardware we own. The resident Claude Code agent described below uses a hosted model and has a separate data boundary. The 32GB was enough for the models I tried, while generation speed determined which jobs were worth running. For the times a chat window is the right tool, there is an Open WebUI front door on the LAN, pointed at the local models.

I tried to make it faster and made it slower. I turned on KV-cache quantization, which stores the attention cache at lower precision to save memory, as described in Ollama's cache documentation. On this CPU-only box with memory to spare, it added overhead for a saving I did not need and cut generation by more than half. I only caught it because I measured before and after; turning it back off more than doubled the speed. I would measure that tradeoff again for each model and runtime version.

Keeping it alive#

A box that is always on has to survive being always on. The failure mode I actually hit was not a crash, it was a wedge: under full multi-core load, the part of SSH that negotiates a new connection gets starved, and the machine answers a ping but will not let you log in. No screen, no battery, no out-of-band console. The only fix was to walk over and pull the cord.

So a chunk of the work is unglamorous reliability plumbing. The login service gets priority to reduce the starvation risk I hit. A hardware watchdog is configured to reboot the box when it stops responding. A reboot is the one privileged action I have configured the box to take on its own. And because there is no UPS, the governing assumption for everything that writes to disk is that the power can disappear mid-write at any moment. State files are written to a temporary file and then renamed into place, with one previous good copy kept. This reduces partial-file risk; the write pattern alone does not establish recovery after every abrupt power failure. None of this is exciting. All of it helps make the box something I can leave running.

From a tool to a resident#

I gave the box a resident agent: a long-lived process whose job is to keep the node healthy and useful, re-invoked across reboots, with the filesystem as its only memory between runs. The brain is Claude Code running headless and calling a hosted model. Ollama serves the separate local workloads described above. The agent lives as its own unprivileged user, with its own login, walled off from my account and my credentials. What it can read is bounded by the actual file permissions on the box, and it is not in the sudoers file.

The agent is governed by a written constitution: a document it reads first on every run, before it touches anything. The constitution is all hard facts about this specific box and the rules that follow from them. Power loss is the normal shutdown, so writes must be crash-safe. The human is usually gone, so "ask" can never mean "block," and silence is never consent. Observe before you mutate: run a read-only census before you install or change so much as a single file. The first time the agent ran, it did exactly that, and it caught the constitution being wrong about its own hardware, an assumption left over from an earlier draft, and corrected the record from what it actually measured. That was the moment I started trusting it.

I use the same written-rule approach in my software projects. The spec-directory lesson shows how I keep intent, acceptance criteria, and approval in files that a fresh coding session can read.

A fence made of code, not prose#

A constitution is words, and a language model is very good at talking its way around words. So the rules that actually matter are enforced in code, underneath the model, on the action channel the fence intercepts, where prompting is not part of the decision.

The fence checks commands on the action channel it intercepts. It allows recognized read-only commands. Its deny and approval patterns cover bulk deletion, disk formatting, firewall and SSH configuration, remote pushes, cloud operations, and access to credentials that are not the agent's. When a command matches a denied pattern, the check blocks it regardless of the model's explanation. A harmful command in an unrecognized shape, or sent over another channel, is outside these checks. Running the agent as an unprivileged user restricts its permissions, but does not close those gaps. The approval queue governs requests actually routed through it.

When a request is routed for approval, the agent writes it to a queue, sends a push notification to my phone, and moves on to other work. There is usually no human there to wait for. A queued request stays unexecuted until I approve it. There is a small web console on the LAN, behind a password, where I can see the box's health, read what it has been doing, and approve or deny anything it has parked.

The control plane. Recognized privileged requests are routed here for approval, with the reason, for me to approve or deny.

It files its own backlog now#

The most recent piece is the one that made me write this. Every night, a scheduled job wakes up, takes a read-only census of the Pi, compares what it finds against the written record of what the box is supposed to be, and reconciles the two. If something has drifted, it notes it. If a task that was open turns out to be done, it marks it done. If it finds a genuine new gap, it writes a new item into the backlog, with its own acceptance criterion, in the same format I would have used. Then it attempts exactly one bounded local fix, through the same governed path as everything else, and commits the updated record.

The night I switched it on, it ran clean: no drift, one new fact recorded that I had not written down, two tasks correctly marked finished. The box now keeps its own to-do list. I review what it wrote in the morning the way I would review a careful junior's notes, which is to say: mostly nodding, occasionally correcting, never starting from scratch. The same fence checks apply to this job, and because the job commits its own changes, what lands still needs that review.

One night's run: it censuses the box, reconciles the record, and commits the change itself; recognized privileged requests are routed to the approval queue.

What it adds up to#

Step back and the through-line is simple. A small box that is a mediocre chatbot and an excellent private librarian, made useful for unattended work, then handed an agent with a restricted account and command checks. The fence is the part worth keeping. For commands it recognizes on the intercepted channel, code determines whether to allow, deny, or request approval.

What I have is a foundation I trust, not a finished worker. The box already does one real job end to end: it keeps its own house in order, catching its own drift and filing its own work. Expanding its authority takes more evidence than the record so far. It notices and files far better than it finishes. Closing that gap is what comes next.

What is next#

Three next steps, in order of how much I trust them today.

First, close the loop. Right now the nightly agent is much better at noticing and filing work than at finishing it. The set of fixes it will do unattended is deliberately narrow, and recognized privileged requests get routed to my approval queue. The next step is widening what it can finish on its own without widening what it can break, which is a question about the approval system, not about a smarter model.

Second, the accelerators. In the setup described here, the GPU and NPU are unused for inference; every token is computed on the ARM cores. The Vulkan driver worked, so getting a runtime to use the GPU was the next project I wanted to try. I had not integrated the vendor's NPU toolchain. Neither accelerator contributed to the generation speeds reported above.

Third, make the pattern portable. None of the safety machinery is specific to this board. The version I actually want runs anywhere: the same fence dropped onto any unattended machine. That is the aspiration; whether it would earn the same trust on another machine is its own question.

The useful version of this box is the one in the corner, on our own metal, doing real work and lately keeping itself in order while I am not looking. Running a model locally was never the surprising part. What still surprises me is that, fenced correctly, I would let it run itself.