Skip to content

Liber Quartus · Tome V

Reaper

An application cannot report that its own container keeps dying. Something outside it has to look.

≈6 min grim 1.0.101
Source

Every five minutes a deployed project tells the Monitoring HQ how it is doing: the queue flows, the scheduler ticked, the disk has room. That report comes from inside the app container, and it has a blind spot the size of the container. A worker that is killed for its memory every ninety seconds, a scheduler stuck in a restart loop, a service that exited last night — the app either does not know or is not there to say so.

The Reaper is the thing outside. It is a small service in the production stack that walks the stack's containers, asks the docker engine what it knows about each of them, and sends that to the HQ. It collects what died. It does nothing else.

What it is

A compose service named reaper, on the project's own image, started with php artisan monitoring:reap --loop. It shares the .env with app, so it knows the HQ and the token the project registered with, and it mounts the docker socket read-only. No other service in the stack sees that socket. The Reaper only ever asks; it starts, stops and execs nothing.

It runs as root inside its own container, because the socket belongs to root:docker on the host and the group's number differs from box to box; that changes nothing about what it can do — any user that can open the docker socket can do everything through it, which is why the socket is mounted here and nowhere else. It runs on 128 MB, waits for app to be healthy before it begins, and is restarted with the rest of the stack. grim deploy ships it like every other service. A stack from before it gets it with its next deploy.

What it sends

Once a minute, one look at the whole stack, to every HQ the project is registered with:

Field Means
services[].name The compose service: app, horizon, scheduler, reverb, nginx, redis, reaper itself. One-off compose run containers are skipped.
state, health What docker reports: running, restarting, exited; healthy, unhealthy, starting, or nothing for a service without a healthcheck.
restart_count How many times docker restarted the container since it was created. Growing between two looks is a restart loop.
oom_killed, exit_code Whether the kernel killed it for memory, and how it last exited.
started_at, image When it came up, and from which image.
host.disk, host.memory The box's docker partition and memory, in percent used.

The HQ derives one status from that: critical when a service is OOM-killed, not running, restarting between looks, or failing its healthcheck past its start period; ok otherwise. A critical look alerts the same way a critical report does, once, and a clean look after it sends the recovery.

When it is silent

The Reaper does not watch itself. The HQ does: a Reaper that reported once and then stops is noticed after fifteen minutes, like an application that stops reporting, and the alert says which of the two went quiet. A project whose Reaper never reported is not missed — an old stack, or one that turned it off, is not an outage.

Turning it off

"docker": { "services": { "reaper": false } }

in grim.json, then deploy. The service is left out of the compose and nothing mounts the socket. There is no reason to on a box you trust; the reason to is a box where the docker socket may not be handed to a container at all.

What it is not

It is not a remote control. It cannot restart a worker, clear a queue or pull an image, and it never will — the socket is read-only so that the answer to "what can a compromised Reaper do" stays "read". It is not a metrics agent either: no CPU graphs, no request rates. It reports state, and only the state that decides whether someone should be woken up.