This needs a patch in caddy to work properly, but we want something like
it anyway.

See https://github.com/mholt/caddy/pull/1058