upstream sent too big header while reading response header from upstream

Symptom

The browser shows 502 Bad Gateway while the app itself is healthy. The pages with the largest response headers fail first, typically those behind a login.

The message is in an nginx error log, and which one tells you where the fix goes.

$ grim logs nginx
$ grim logs nginx --remote=production
$ ssh root@203.0.113.10 tail /var/log/nginx/error.log

The first is the nginx inside your local stack, the second the one inside the stack on the server, the third the nginx on the server itself, in front of Traefik.

Cause

nginx reads the response headers of its upstream into one buffer, and by default that buffer is a single memory page of 4 KB. An app on grim sends more than that. The Content-Security-Policy header alone is close to 5 KB, and session cookies come on top, which is why signed-in pages fail first. A response whose headers do not fit is dropped and answered with a 502.

The configs grim writes today carry larger buffers at both places where nginx stands in the way: fastcgi_buffer_size 32k in the nginx of the stack, which talks to PHP, and proxy_buffer_size 32k in the vhost on the server, which talks to Traefik. The curse strikes where one of those files was written by a grim from before that change and never renewed.

Breaking the curse

Take the current templates first, then renew the file the log pointed at.

$ grim self-update

In the local stack, grim upgrade rewrites docker/nginx-dev.conf, and a restart makes nginx read it.

$ grim upgrade
$ grim down
$ grim up

In the stack on the server, grim server:upgrade uploads the current docker/nginx-production.conf and recreates the containers. The next deploy would upload the same file, so a release from the updated grim breaks the curse as well.

$ grim server:upgrade shop --restart

In the vhost on the server, run grim server:add-project again with the same arguments as the first time. It keeps the .env, the database and the certificate, and writes the vhost anew.

$ grim server:add-project shop --host=vps1 --domain=shop.example.com

Verify

Ask for a page that failed, signed in, and watch the log that carried the message while you do. The 502 is gone and no new line appears.

$ curl -s -o /dev/null -w '%{http_code}\n' https://shop.example.com/admin
$ grim logs nginx --remote=production --tail=20

On the server you can also look for the directive itself.

$ ssh deploy@203.0.113.10 grep fastcgi_buffer_size /opt/shop/docker/nginx-production.conf
    fastcgi_buffer_size 32k;