Debug production

Before you start

The project is deployed, as First deploy to a fresh VPS leaves it, and something is wrong with it: a 500, a job that never runs, a page that is slow since this morning.

Work from the project directory, on the machine that ran grim server:add-project. grim logs --remote and the three remote:* commands find the server only in ~/.grim/servers.json. A host in grim.json is not enough for them.

$ grim server:status

The project's server should be in that list. When the fault arrived with the last release and customers are waiting, take the release back first with Roll back a deploy and debug afterwards.

The ritual, in order

Six steps. The first four only read. The fifth restarts the app twice, and the sixth puts you inside production. Stop as soon as you know the cause.

1. I

   Look at the server

   A full disk or a container that keeps restarting explains more faults than any stack trace. grim server:status with the project's name connects to its server and shows both.

   $ grim server:status shop

   Read the status column of the container table. shop-app-1, shop-nginx-1 and shop-redis-1 should be Up, the app (healthy). Restarting or unhealthy is your answer for now. Then the disk line: above ninety percent MySQL and Docker both start to fail in odd ways. Then Traefik, which should not say not found.

   The last block, GRIM projects on this server:, may say (none registered) while the project runs. That list is read from your own machine and is not a statement about the server. The container table is.

2. II

   Read what the app logged

   On a server made by grim the app runs with LOG_CHANNEL=stderr. Laravel's exceptions go to the container's output, and grim logs reads that output. This is the production log.

   $ grim logs app --remote=production --tail=300
   $ grim logs -f app --remote=production

   Look for the exception class and the first frame of the trace that is in your own code. The second form stays open. Reload the failing page in a browser and watch the line arrive. For a job that never runs, ask for horizon or scheduler in place of app. The level is warning, so Log::info() calls are not in there. Step five changes that.

3. III

   Search the log on the server

   Three hundred lines are not always enough, and sending a day of log over SSH to grep it locally is slow. grim remote:exec runs a command on the host, in the project's directory. It quotes every word you type, so a pipe has to travel inside one quoted sh -c argument.

   $ grim remote:exec docker compose ps
   $ grim remote:exec sh -c 'docker compose logs --since=24h app | grep -c "SQLSTATE"'

   The first line prints where it is going: Running on shop (203.0.113.10):. Docker keeps at most three files of 50 MB per container, so a busy app's log reaches back days, not months.

4. IV

   Ask the app

   The log says what failed. The app can say why. grim remote:artisan is php artisan inside the deployed app container, one command at a time.

   $ grim remote:artisan migrate:status
   $ grim remote:artisan queue:failed
   $ grim remote:artisan tinker --execute="echo config('mail.mailers.smtp.host');"

   A migration still marked Pending means the deploy hook failed, which a deploy only warns about. A config value that differs from what you expected points at the server's .env, and Change config on a live app is the way to fix it. Leave config:clear and optimize:clear alone here. The container built its caches when it started, and an app without them is slower, not more correct.

5. V

   Turn the log up, then back down

   Optional. When the log is silent about the fault, let the app say more for a few minutes. grim env:set changes the level and recreates the PHP containers, which takes the site away for a few seconds each time.

   $ grim env:set LOG_LEVEL=debug production
   $ grim logs -f app --remote=production
   $ grim env:set LOG_LEVEL=warning production

   Reproduce the fault while the second command is open. Put the level back the same hour, because debug output fills the 150 MB of log quickly and pushes the useful lines out. Do not reach for APP_DEBUG=true. It shows the stack trace, and with it parts of the configuration, to every visitor who hits the error.

6. VI

   Go inside

   When single commands are not enough, grim remote:shell opens bash in the running app container, in /var/www/html, with the app's own credentials loaded.

   $ grim remote:shell
   Opening shell on shop (203.0.113.10) in container app...

   Look at what the image really contains, check permissions under storage/, run php artisan tinker and query the database through the models. There is no mysql service to enter. MySQL runs on the host. Whatever you edit in here is gone when the container is recreated, so use the shell to find the cause, and ship the fix as a release.

What you have now

A cause, or a much shorter list of suspects. Nothing on the server changed unless you ran step five, and that step ends with the level back at warning.

step 1      the server: containers, disk, Traefik
steps 2-3   the app's log, in Docker's output
steps 4-6   the app itself, through artisan and a shell

The fix goes out the normal way: a commit, then grim release. A wrong value in .env is fixed in place with grim env:set.

When it does not work

“Environment 'production' not configured for project 'shop'.”

grim logs --remote found no registration on this machine. It does not read the host from grim.json, unlike the env:* commands. Work from the machine that registered the project, or read the log yourself: ssh deploy@203.0.113.10, cd /opt/shop, docker compose logs --tail=300 app.

“No default environment configured for project "shop". Run grim server:add-project first.”

The same cause, reported by the remote:* commands.

The remote commands reach production and you wanted staging

grim remote:exec, grim remote:artisan and grim remote:shell take no environment. They reach the one that was registered first. grim logs --remote=staging and grim env:set … staging do take one. For the rest, connect yourself: ssh deploy@203.0.113.10, then cd /opt/shop-staging and use docker compose exec app ….

tail storage/logs/laravel.log finds nothing

No such file or directory is correct. The project's directory on the server holds docker-compose.yml and .env, and the code is inside the image. The log is the container's output. Use step two.

grim remote:artisan migrate --help prints grim's help

grim takes --help, -v and -q for itself wherever they stand. Put -- first and the rest goes to artisan untouched: grim remote:artisan -- migrate --help. The same holds for grim remote:exec.

The pipe ran on your machine

grim remote:exec docker compose logs app | grep ERROR sends the whole log over SSH and greps it locally. That works, slowly. With the pipe in quotes but without sh -c, the server is handed | as a plain word and Docker complains about an unknown service. Use the form from step three.

Questions

Where does grim error:test fit in?

It does not test your app. It sends one harmless report to the grim registry, to prove that crash reports from the grim command on this machine arrive. Run it when it was grim that crashed during a deploy and the maintainers say they never saw the error.

Is there a record of what I ran?

Not from grim. The commands run as the deploy user over SSH, and the server keeps no more than its own SSH log. Whoever holds a key for that user can do everything on this page.