Raise a CI runner host

$ grim server:runner:setup 203.0.113.10 --org=acme --app-id=123456 --app-key=~/keys/acme-ci.pem

Hosted CI minutes run out, or the builds need a Docker cache that survives from one job to the next. A VPS of your own solves both. grim server:runner:setup makes a bare machine into a host for GitHub Actions runners that belong to your organization.

The runners are ephemeral. Each one registers for a single job, runs it, and registers again as new, so no job inherits the leftovers of the one before. No personal token is stored on the machine. The runners identify themselves through a GitHub App, and its private key is the only secret on the box.

Usage

$ grim server:runner:setup 203.0.113.10 --org=acme --app-id=123456 --app-key=~/keys/acme-ci.pem
$ grim server:runner:setup 203.0.113.10 --org=acme --app-id=123456 --app-key=~/keys/acme-ci.pem --replicas=5
$ grim server:runner:setup 203.0.113.10 --user=ubuntu --alias=ci2 --org=acme --app-id=123456 --app-key=~/keys/acme-ci.pem

The first line gives you three runners under the alias ci. The second is the same command run again to scale to five. The third is a cloud image that does not let root in. Run it from anywhere; --org, --app-id and --app-key are required on every run, and the key file is checked on your machine before any connection is made.

Arguments and options

Provision a VPS as a self-hosted GitHub Actions runner host (ephemeral, GitHub App JIT)

Usage

grim server:runner:setup [--alias ALIAS] [--user USER] [--org ORG] [--app-id APP-ID] [--app-key APP-KEY] [--replicas REPLICAS] [--] <host>

Arguments

Options

Before you run it

The command does not create the GitHub App. You need one that is installed on the organization and allowed to manage its self-hosted runners, its numeric id, and a private key for it downloaded as a .pem file. You also need SSH access to the machine as root or as a user with sudo.

What it actually does

Each step is printed with its result, and the first failure stops the run.

1. Checks SSH as --user, then reads which OS the machine runs.
2. Installs Docker where it is missing.
3. Creates the runner user, in the docker group, with your own public key from ~/.ssh/id_ed25519.pub or ~/.ssh/id_rsa.pub. This account is for you and for grim server:runner:status, not for the jobs.
4. Fills /opt/grim-runner. The App key as grim-runner.pem, readable by root and the runner containers only, a register.sh that every replica starts with, and a docker-compose.yml with one service per replica, runner-1 to runner-N.
5. Creates the grim-ci-cache volume, pulls the official actions-runner image and starts the replicas. Removing replicas on a scale-down is part of the same step.
6. Installs a weekly refresh in /etc/cron.d/grim-runner-update. Every Monday at 04:00 it pulls the runner image and recreates the replicas, logging to /var/log/grim-runner-update.log.
7. Counts the running containers. Fewer than --replicas fails the run with Only 1 of 3 runner containers are running and the command to read their logs.
8. Asks GitHub how many runners are online, through the gh CLI on your machine. This step only warns when it cannot.
9. Remembers the server in your ~/.grim/servers.json with the role ci-runner, the organization, the App id and the replica count.

The summary ends with the labels the runners carry: self-hosted, linux and grim.

How a job reaches the box

A workflow asks for the runners by label.

jobs:
  test:
    runs-on: [self-hosted, linux, grim]

The replicas are named grim-runner-1 to grim-runner-N. They talk to the Docker daemon of the host, so a job's container: and services: run as containers on the VPS next to the runner, not inside it. The grim-ci-cache volume is shared by all replicas and survives restarts; a job that runs in a container can mount it by name.

Pitfalls

Running containers do not prove a registration

The containers start and stay up even when GitHub refuses them, because register.sh retries every 30 seconds for ever. Step 7 is green in that case. Look at the GitHub count in the summary, and when it is zero, read the logs on the box: GitHub App is not installed on org acme or a line starting generate-jitconfig HTTP points at the App, not at the server.

The GitHub check needs a scope gh does not have by default

gh api failed (needs gh auth refresh -h github.com -s admin:org) is a warning about your own machine. The runners are not affected. Run the command it names and the check works next time.

Scaling still needs all three App flags

--replicas=5 on its own stops with --org is required. Repeat the full command with the new number. Every step checks what is already there, so nothing is reinstalled.

Jobs can do anything Docker can

A job on these runners has the host's Docker socket, which is root on that machine in practice. Keep the box for CI only, never on a server that also runs an app, and do not let workflows from forks of public repositories onto it.

After it finishes

Check the host, then point a workflow at it.

$ grim server:runner:status ci

grim server:runner:status shows the containers, the disk and the runners as GitHub sees them.

Related spells