Empty reply from server with Varnish & hitch on separate cache server

Question

After installation of varnish & hitch on ubuntu 20.04 server, getting following error:

curl: (52) Empty reply from server

Tutorial I am following:

[https://docs.varnish-software.com/tutorials/hitch-letsencrypt/][1]

In my case, I have two servers s1 & s2 both have ubuntu 20.04 server installed. local static ip of s1:

192.168.1.105

local static ip of s2:

192.168.1.106

On s1 virtualmin pannel is installed & every thing working fine. So i decided to make a saparate cache server & installed varnish & hitch as:

sudo apt install varnish -y 

sudo apt install hitch -y

changes in default.vcl file:

    backend default {
    .host = "192.168.1.105";
    .port = "80";

   }

hitch.conf:

# Run 'man hitch.conf' for a description of all options.
    frontend = {
        host = "*"
        port = "443"
    }
    backend = "[127.0.0.1]:8443"    # 6086 is the default Varnish PROXY port.
#workers = 4                     # number of CPU cores

#daemon = on

# We strongly recommend you create a separate non-privileged hitch
# user and group
#user = "hitch"
#group = "hitch"

# Enable to let clients negotiate HTTP/2 with ALPN. (default off)
# alpn-protos = "h2, http/1.1"

# run Varnish as backend over PROXY; varnishd -a :80 -a localhost:6086,PROXY ..
#write-proxy-v2 = on             # Write PROXY header

#pem files

    pem-file = "/etc/letsencrypt/live/example.com/hitch-bundle.pem"

unit file:

    [Unit]
Description=Varnish HTTP accelerator
Documentation=https://www.varnish-cache.org/docs/6.1/ man:varnishd

    [Service]
    Type=simple
    LimitNOFILE=131072
    LimitMEMLOCK=infinity
    ExecStart=/usr/sbin/varnishd -j unix,user=vcache -F -a :80 -a localhost:8443,proxy -T localhost:6082 -f /etc/varnish/default.vcl -S /etc/varnish/secret -s malloc,3G
    ExecReload=/usr/share/varnish/varnishreload
    ProtectSystem=full
    ProtectHome=true
    PrivateTmp=true
    PrivateDevices=true

    [Install]
    WantedBy=multi-user.target

Note: I have no idea about networking, so main router connected with second router and both s1 & s2 directly connected with second router. Also dmz host ip changed from s1 ip to s2 ip.

Result:

curl -i https://example.com
curl: (52) Empty reply from server.

upon This command varnishlog -g request -q "ReqUrl eq ’/’" VSM: Could not get hold of varnishd, is it running?

Please guide thanks.

Answer 1

Fix empty reply by enabling PROXY protocol

Uncomment the following line in your hitch.conf:

#write-proxy-v2 = on

This will enable the use of the PROXY protocol to communicate with Varnish. This protocol passes more information about the original client (the one that connected with Hitch) to Varnish via TLV attributes.

You are connecting to port 8443 from Hitch, which is Varnish's PROXY protocol port as described by the following varnishd runtime parameters:

-a localhost:8443,proxy

Right now Varnish is expecting PROXY information to be present in the TCP packets, but couldn't find that information. Hence the empty reply.

Make Varnish protocol aware

If your application forces HTTPS redirection, it is important to make Varnish aware of the protocol that is used. It'll either be HTTP or HTTPS.

If you use Hitch over the PROXY protocol, we can use vmod_proxy to identify these connections and create proper cache variations.

Here's the VCL code for that:

vcl 4.1;

import proxy;

sub vcl_recv {
    if (proxy.is_ssl()) {
        set req.http.X-Forwarded-Proto = "https";
    } else {
        set req.http.X-Forwarded-Proto = "http";
    }
}

sub vcl_backend_response {
    if(beresp.http.Vary) {
        if(beresp.http.Vary !~ "X-Forwarded-Proto") {
            set beresp.http.Vary = set beresp.http.Vary + ", X-Forwarded-Proto";
        }
    } else {
        set beresp.http.Vary = "X-Forwarded-Proto";
    }
}

This code will store an HTTP version and an HTTPS version of each response.

Bonus: enable HTTP/2

While you're at it, you might as well enable HTTP/2 by uncommenting the following line in your hitch.conf file:

# alpn-protos = "h2, http/1.1"

To ensure HTTP/2 is enabled in Varnish, please add the following runtime parameter to varnishd in your systemd configuration:

-p feature=+http2

Restart both Hitch & Varnish and Hitch will be able to terminate TLS for HTTP/2 requests.