Using nghttp2 to work around nginx bugs.
For the most part nginx has been my default go-to choice. While it has its quirks and strange design choices usually doesn't get in the way. Lately their refusal to backport bug fixes from their development version to the stable version has been causing some problems.
Specifically, the "stable" version of nginx can't do HTTP/2 POST on any current browser without nginx dropping and resetting the connection.
This affected sites like https://bugs.freebsd.org/ - if you attempted to process a bug, your good deed was punished by a connection reset. This was directly caused by nginx.
They fixed this in their development tree and you can back-port the fix yourself. The issue and patch in question is referenced here: https://trac.nginx.org/nginx/ticket/959#comment:18 It boggles my mind that they don't seem to think that HTTP/2 POST not working in their stable branch isn't worth fixing. Turning off HTTP/2 and falling back to SPDY is no longer an option because they removed SPDY support.
Luckily there is a new kid on the block and now there is another option to carrying an nginx patch set yourself. The front-end TLS/HTTP/2 proxy server that comes with the nghttp2 library package Just Works(TM). Unlike nginx, it does both SPDY and HTTP/2 at the same time and a few other nice things like dealing with old pre-ALPN clients in a sensible way.
It is simple to use and as a bonus, it plays nicely with Varnish. Here is how I've been using it:
- Remove TLS/SSL/HTTP/2 from nginx, reconfigure the main nginx servers (or Varnish) to listen on localhost:83.
- Fire up nghttpx (the front-end proxy) to relay from the wild internet on 443 to localhost:83
Sample nghttpx.conf - it is quite simple:
# The basics
backend=127.0.0.1,83
frontend=*,443
# TLS settings
ciphers=ECDHE-RSA-.... (I use mozilla's recommended cipher settings)
private-key-file=/path/to/privkey.pem
certificate-file=/path/to/fullchain.pem
# weaken the defaults - re-enable tls-1.0
tls-proto-list=TLSv1.2,TLSv1.1,TLSv1.0
# Logs
accesslog-file=/var/log/nghttpx-access.log
errorlog-file=/var/log/nghttpx-error.log
# and a few extra settings
strip-incoming-x-forwarded-for=yes
add-x-forwarded-for=yes
strip-incoming-forwarded=yes
add-forwarded=by,for,host,proto
# Don't hash forwarded-for stuff, use the actual information
forwarded-by=ip
forwarded-for=ip
# HSTS settings etc
add-response-header=Strict-Transport-Security:max-age=31536000; includeSubdomains
# other security headers here
For nginx.conf, all I needed for using correct IP addresses in logs is something like:
set_real_ip_from 127.0.0.1;
real_ip_header X-Forwarded-For;
real_ip_recursive on;
I have been able to use nghttp2's nghttpx with varnish to completely eliminate nginx from the pipeline. Well, almost - nginx has been relegated to serving letsencrypt cookies but I have other options for that if I cared.
It does OCSP stapling by default and generally has a decent set of default TLS settings - if anything, it is a bit stricter out of the box than I would like. This is a welcome development and choosing whether to relax TLS settings a nice problem to have.
It is important to note that nghttpx does not claim to be focused on peak performance. However, it is no slouch either. It has been more than enough for anything I would expect it to handle.
It is very nice to have another option for a solid, modern TLS/SPDY/HTTP2 terminator. At the very least it might give the nginx folks a wake-up call so they think a little more carefully about holding back on fixes for user-affecting bugs.
The nghttp2 project is https://nghttp2.org/ You will discover that it is very well documented and has a solid HTTP/2 client and server library implementation in addition to the proxy server I described above. It is well worth a look.
Update: It was pointed out to me that how nginx defines their branches was lost in the noise. Here is their description. This is not in sync with what some packagers do, notably FreeBSD, who mis-package the "mainline" version as "nginx-devel". On the other hand, Nginx say: We recommend that in general you deploy the NGINX mainline branch at all times. Unfortunately this is not obvious unless you go looking for it. As for myself, I'm going to stick with nghttp2 as a small, futile protest against monoculture.