One Agent, One Container: My Hermes Fleet Shrinks to Two

Overview

Diagram of three stages of the Hermes homelab fleet in November: seven agents each owning a corner, two merged agents sharing the hermes-lxc container on Grimlock, and two agents each in their own LXC container with their own gateway, user and tailnet node
Fewer agents, more walls: November in three steps.

A month ago I wrote about giving my Hermes agents a machine of their own: one unprivileged Incus LXC container on Grimlock, called hermes-lxc, with one gateway serving all my profiles. I finished that post with "Let's see how long it lasts!"

The answer is: about a month. hermes-lxc no longer exists. It was exported one last time and deleted yesterday. In its place there are now two containers, each one holding exactly one agent. And the number of agents I run for my homelab has dropped from a small army to just two.

This post is about both changes, because one led to the other.

Too many agents, too little visibility

By early November my homelab agents looked like this: one agent per machine (Cybertron, Pangolin, OptimusPrime, Grimlock), Megatron looking after the Unraid NAS, Ironhide doing the weekly UniFi network review, plus Bumblebee, my main assistant. It sounds very organised on paper. In practice it had two problems.

Nobody was watching the watchers. Each agent had its own cron jobs, its own reports and its own little failures. The worst example: Megatron's NAS survey was effectively blind from the 1st to the 16th of November. Nothing crashed, nothing alerted, it just quietly stopped seeing things. With seven agents each reporting on their own corner, a silent failure in one of them is very easy to miss.

Every host agent had the whole host as blast radius. The per-host agents ran as my own user on each machine, and my user is in the docker group, which is root in all but name. So a bad decision by the model, or an instruction injected through some web page or email it read, could in theory do anything to that machine. I had built isolation for the agents' homes in the last post, but not for what they could reach.

Two agents, bigger jobs

So I consolidated. Mid-November I merged all the admin agents into one, and kept my personal assistant separate. These are the two that remain:

AgentRoleWhat it absorbed
BumblebeeMy personal assistant: Java and Spring jobs, mail, the everyday stuffsome of Megatron's reporting
RatchetThe one and only homelab admin, for Cybertron, Pangolin, OptimusPrime and Grimlockthe four per-host agents, Ironhide (UniFi reviews), Megatron (the Unraid NAS, now via its GraphQL API) and a couple of OptimusPrime's digests

Ratchet is the interesting one, because I designed it to be careful rather than clever:

  • It is not me. It reaches every host over SSH as its own restricted ratchet account: no root, no docker group.
  • Its powers live in code, not in a prompt. The only things it may run with sudo are two small root-owned helper scripts: one that applies safe package upgrades and one that reports security status. The upgrade helper holds back the kernel, SSH, systemd, Docker and friends, and Ratchet never reboots a host. Before, "please don't upgrade the kernel" was a sentence in a prompt. Now it is a rule the agent cannot talk its way around.
  • Most of its jobs don't need an LLM at all. Host watch every 30 minutes, weekly maintenance, the weekly security report, the NAS watch: these are mostly plain scripts. The model steps in where reading and summarising actually helps. Cheaper, and much more predictable.

Fewer agents also means fewer bots to read on Telegram. One admin agent with one report is much easier to trust than seven small ones I only skim.

One container was still one blast radius

With two agents in place, I looked at hermes-lxc again and did not like what I saw. Both agents ran as the same ubuntu user, behind the same gateway, in the same container. Which means Bumblebee — the agent that reads my mail and browses the web, so the one most exposed to prompt injection — could read Ratchet's admin SSH key.

Or, as I wrote in my decision log at the time: a prompt injection or a mistake in Bumblebee had Ratchet's reach. Two containers, with two separate network identities, make that boundary real.

The family agents had already made this move a week earlier: on the 8th of November they each got their own container, capped at 2 CPUs and 6 GiB of RAM. So splitting my own two was mostly a matter of following the same recipe.

Splitting hermes-lxc

On the 17th of November I cloned hermes-lxc into hermes-bumblebee and hermes-ratchet, both still unprivileged Ubuntu 26.04 containers on Grimlock (Incus 7.0). Then I cleaned each clone up:

  1. Remove the other agent's profile.
  2. Reset the machine ID, so the two containers don't look like the same machine.
  3. Drop the copied Tailscale state and join each container to my tailnet as its own node, with its own tag and its own access rules.

That last point is a small U-turn. In the previous post I deliberately avoided giving the container its own identity on my private network, and used Incus proxy devices instead. This time a separate identity per agent is the whole point: the network itself now knows that Bumblebee and Ratchet are two different things, and lets them reach different places. Bumblebee, for example, does not get SSH to anything.

Two gotchas from the day:

  • incus copy froze the running container. Copying a running hermes-lxc froze it for about 17 minutes. Both agents were simply gone for that time. incus export, on the other hand, does not freeze a running container — I checked afterwards, because that is what my backups use.
  • One Telegram bot, one gateway. Two gateways polling the same bot token will fight over the incoming messages. So the old container had to be stopped before the clones started, which meant a few minutes of radio silence from both bots. Not a big deal for a homelab, but plan for it.

For the record: the final export of hermes-lxc was 10.1 GB. Bumblebee's profile alone is 3.7 GB. Ratchet's is 170 MB. Scripts are small, it turns out — a personal assistant's memories, sessions and tools are not.

Living with more containers

More containers could easily mean more chores, so I tried to keep everything boring and automatic:

  • Updates. A weekly timer updates Hermes in each container, one at a time, and stops at the first failure. The family containers update a day later, and only if Ratchet is already happily running the newest release. Ratchet is my canary.
  • Skills. Every container keeps a checkout of my shared skills repository and fast-forwards it every 15 minutes.
  • Logs. Each container ships its logs with Vector to VictoriaLogs, same as everything else.
  • Backups. The plan is the same incus export routine from the last post, just one file per agent now, twice a month, onto the NAS.

There is still some cleanup to do. Both clones started life as a full copy of the shared container, so each one still carries leftovers from the other agent that I am pruning bit by bit. The beauty of the new setup is that I can do that one agent at a time.

Lessons learned

  1. Fewer, bigger agents beat many small ones. Seven agents each owning a corner looked tidy, but silent failures hid in the gaps. One admin agent, with one report, is easier to trust.
  2. Put the rules in code, not in the prompt. If an agent must never upgrade the kernel or reboot a host, don't ask it nicely — give it a helper that cannot do it.
  3. Isolate by blast radius, not by convenience. The agent that reads email and the web should not share a user, a filesystem or a network identity with the agent that holds admin keys.
  4. Clone stopped, export running. incus copy of a running container can freeze it; incus export won't.

So: one agent, one container, two agents. I said "let's see how long it lasts" last time, and I learnt my lesson — so this time I will just say it feels right. For now.