When the Watcher Is the Problem

Earlier this month my whole setup told me its brain was dead.

I run a handful of AI assistants on one desktop computer at home, and a small check pings them every fifteen minutes to make sure they still answer. That evening the check started timing out. Seven seconds became a hundred and fifty. Then it paged me: the brain is down.

Except it was not. I could open a session and talk to it just fine. Everything worked when I touched it by hand. Only the automated check kept insisting the sky was falling.

The thing watching was the thing breaking

The cause turned out to be a tiny side project. I had a little game bot running for fun, and next to it a small script whose only job was to write down how the bot was doing. A status writer. A few lines of output every so often. About the most harmless thing you can build.

In three hours it grew to 26 gigabytes of memory.

The machine has 48. Once that one script was sitting on more than half of it, everything else slowed to a crawl, including the check that measures whether things are slow. So the alarm was real, in a sense. The computer really was struggling. It just blamed the wrong part.

I killed the script and memory dropped by more than half in a second. The brain had been fine the whole time.

Watchers are code too

Here is what stuck with me. I treat the stuff that does the work as serious. I test it, I fix it, I worry about it. The stuff that watches the work, the status pages and the little reporters and the health checks, I treat as furniture. I build it once and never look at it again.

But a watcher runs on the same machine, eats from the same memory, and fails in the same ways. And when it fails, it fails in the worst possible way, because it is the thing you trust to tell you the truth. A broken worker gets reported. A broken reporter reports something else.

I wrote recently about the assumption nobody checked. This was a cousin of that. My assumption was that anything small enough to be harmless stays small. It does not, unless something makes it.

What I am changing

The fix is not more monitoring. Adding a watcher to watch the watcher is how you end up with a tower of alarms nobody believes.

What I want instead is simpler.

Keep the watchers tiny and boxed in. The game bot is fun and I want it running all the time, but its reporter should live in its own little space with a hard ceiling, writing one small file and nothing else. If it ever misbehaves, it hits its own wall instead of taking the house down with it.

Look at the machine before believing the message. When an alarm says something is dead but it works when I poke it, the first question is now what else is eating the computer, not what is wrong with the thing named in the alarm.

Make the alert say what actually happened. “Brain down” sent me looking at the wrong thing. “The computer is out of memory and this is the process using it” would have pointed straight at the fix.

The quiet part of any setup

Every system I have built has a loud part and a quiet part. The loud part is the stuff I show people. The quiet part is the plumbing that keeps it honest. It is tempting to spend all my attention on the loud part, which is the same lesson I keep relearning in running a few bots and zero employees.

The quiet part deserves a look every so often too. Not a rebuild. Just a glance at how much room it is taking up, before it takes all of it.