Skip to content
GRIMOIRY

Set a watch on a server

$ grim infra:aws-alarms --instance=vps1 --email=ops@example.com

A server made by grim infra:aws-ec2 runs with nobody watching it. If the instance hangs at night, the first to notice is a customer. grim infra:aws-alarms puts the two cheapest useful alarms on it and sends them to an email address, so the first to notice is you.

The alarms watch the machine from AWS's side. They know nothing about the app: a site that returns errors on a healthy instance stays quiet.

Usage

$ grim infra:aws-alarms                                                        # pick the instance from a list
$ grim infra:aws-alarms --instance=vps1 --email=ops@example.com                # by alias
$ grim infra:aws-alarms --instance=vps1 --email=ops@example.com --dry-run      # plan and estimate only
$ grim infra:aws-alarms --instance=i-0abc1234def567890 --region=eu-west-1 --email=ops@example.com

Run it from anywhere. It needs the aws CLI, terraform and working credentials for the account the instance lives in; grim infra:doctor checks them. The last line is an instance grim did not create: give the raw ID, and the region with it.

Arguments and options

Attach CPU + StatusCheckFailed CloudWatch alarms to an EC2 instance, with email notifications via SNS

Usage

grim infra:aws-alarms [--instance INSTANCE] [--email EMAIL] [--region REGION] [--cpu-threshold CPU-THRESHOLD] [--profile PROFILE] [--dry-run] [--force]

Options

Option Description
--instance=INSTANCE EC2 instance ID (e.g. i-0abc...) or alias from servers.json
--email=EMAIL Notification email — AWS will send a confirmation link
--region=REGION AWS region (defaults to instance region from servers.json or eu-central-1)
--cpu-threshold=CPU-THRESHOLD CPU% threshold for high-CPU alarm (default: 80)
--profile=PROFILE AWS CLI profile to use
--dry-run Show plan only, do not provision
--force Allow re-using an existing state directory

What it actually does

  1. Shows the AWS identity and asks whether to create resources in that account. The default answer is no.
  2. Finds the instance. The list holds the servers in ~/.grim/servers.json that were made by grim infra:aws-ec2 in the account you confirmed. An alias brings its instance ID and region along. A raw ID gets eu-central-1 unless you pass --region.
  3. Asks for the email and prints the plan with a monthly estimate from the price snapshot built into grim. With --dry-run it stops here.
  4. Runs Terraform in ~/.grim/infra/alarms-<instance-id>/ after a second confirmation. That creates the SNS topic grim-alarms, an email subscription to it, and the two alarms.
  5. Records the set under aws_resources in servers.json as alarms-<instance-id>, with the instance, topic, email, region, account and profile.

The two alarms

Alarm Fires when Missing data counts as
grim-<alias>-cpu-high Average CPU is above the threshold for three 5-minute periods in a row, so 15 minutes. The threshold is 80 percent, or --cpu-threshold. Fine
grim-<alias>-status-check-failed One 5-minute period in which an AWS status check failed: the instance is unreachable, hung, or its hardware is. A failure

Both mail you when they fire and again when they recover. There is no memory or disk alarm, because EC2 does not report those numbers without an agent on the machine.

AWS sends a confirmation link to the address first. Until someone clicks it, the alarms change state and no mail goes out. The command ends by reminding you, and grim cannot see whether the link was clicked.

Pitfalls

The alias is not offered

The list only shows instances whose recorded account matches the identity from step 1. --instance must be a grim-managed alias from servers.json or a raw EC2 instance ID (i-...). for an alias you know exists means you confirmed the wrong account. Run again with the --profile the server was created with.

The alarms already exist

Alarms already exist for instance i-0abc1234def567890. Pass --force or destroy first. means a set was made earlier. To change the threshold or the address, run grim infra:destroy alarms-i-0abc1234def567890 and create the set again.

A stopped instance raises the status alarm

Missing data counts as a failure for the status check, and a stopped instance reports nothing. If you stop a machine on purpose, expect the mail.

After it finishes

Click the link in the confirmation mail. The set now shows in grim infra:list with its estimate, and grim infra:destroy takes the same id to remove it.

$ grim infra:list
$ grim infra:destroy alarms-i-0abc1234def567890

grim infra:aws-ec2

Raise a server on AWS

Create one EC2 instance with a static address, a firewall and your SSH key in your AWS account, then hand it to grim server:setup.

grim infra:aws-budget

Set a spending alarm

Create a monthly AWS budget that mails you at 50, 80 and 100 percent of a limit, and when AWS forecasts the month will end above it.

grim infra:list

Survey what runs on AWS

List every server and AWS resource grim has created, with a monthly cost estimate for each and the ids that infra:destroy takes.

grim infra:destroy

Unmake what grim made

Remove a server or AWS resource that grim created, after checking the account and making you type its name, and stop it from being billed.

grim server:status

Look over a server

List the servers you have registered, or look at one of them: its containers, its disk, Traefik and its projects.