Skip to content
GRIMOIRY

Check the CI runners

$ grim server:runner:status ci

Jobs sit in the queue with "Waiting for a runner to pick up this job", or a build died with no space left on the device. Both questions have the same first answer, and grim server:runner:status gives it: are the runner containers up, does GitHub see them, and how full is the disk.

It exits with a failure when fewer runners are up than were set up, so it also works as a check in cron or in monitoring.

Usage

$ grim server:runner:status        # every runner host this machine knows
$ grim server:runner:status ci     # one host in detail

Run it from anywhere. It knows the hosts that grim server:runner:setup registered in your ~/.grim/servers.json, and connects as the runner user with your SSH key.

Arguments and options

Show CI runner host status — containers, cache volume, disk, GitHub runner registration

Usage

grim server:runner:status [<host>]

Arguments

Argument Description
host Runner host or alias (omit to list all runner hosts) (optional)

What it actually does

Without an argument it connects nowhere. It prints a table of the registered runner hosts with address, alias, organization and replica count.

With a host it prints five sections.

  1. Runner containers. The output of docker compose ps for /opt/grim-runner/docker-compose.yml, then Running: 3/3. The second number is the replica count saved at setup. The line turns red when the first is lower.
  2. Cache volume. Whether grim-ci-cache exists.
  3. Disk usage. Used and total space on /.
  4. Docker disk. The docker system df table: images, containers, volumes and build cache, with what can be reclaimed.
  5. GitHub runners. One line per runner of the organization, asked through the gh CLI on your machine: name, online or offline, busy or idle, and labels.

The exit code follows the first section only. A full disk or a failed GitHub query does not change it.

Pitfalls

Green containers with no runner on GitHub

A replica keeps running while its registration fails, and retries every 30 seconds. 3/3 next to an empty GitHub section means the problem is between the box and GitHub, usually the App's installation or key. Read the replica's log on the box with docker compose -f /opt/grim-runner/docker-compose.yml logs.

The GitHub section needs admin:org

could not query GitHub is about the gh login on your machine, not about the runners. The command prints the fix:

$ gh auth refresh -h github.com -s admin:org

The list covers the whole organization

Runners from other hosts appear next to yours. The ones from this host are named grim-runner-1 and up.

A runner shown as busy is in the middle of a job, and an offline entry with a name of yours is what a replica leaves behind when it is killed mid-job. The replica removes that entry by itself on its next registration.

After it finishes

Nothing was changed. When the count is short, running grim server:runner:setup again with the same flags brings the missing replicas back. When the disk is the problem, docker system prune on the box frees what the fourth section listed as reclaimable.

Questions

Can I run it from cron?

Yes. The exit code is non-zero when fewer containers run than the saved replica count, and zero otherwise. The machine running it needs the servers.json entry and an SSH key the runner user accepts.

Why does it say the server is not a CI runner host?

ci is not a CI runner host (role=…) means the alias belongs to a server set up as something else, for example a production server. Aliases are yours to choose, so pick one per machine.

grim server:runner:setup

Raise a CI runner host

Turn a VPS into a host for ephemeral GitHub Actions runners that register to your organization through a GitHub App.

grim server:status

Look over a server

List the servers you have registered, or look at one of them: its containers, its disk, Traefik and its projects.

grim server:keys

Grant or revoke deploy access

List, add and remove the SSH keys that may log in as a server's deploy user, each under a label you choose.