Nginx ssl termination and ssl_preread

Question

I'm trying to use nginx to reverse proxy both an xmpp and an http connection, having the same port, same ip, same server name in such a way that:

• http connections to https://example.com:443/ are proxied to localhost:8080
• XMPP connections to example.com:443 are proxied to localhost:5222

This is working perfectly if I don't enable SSL termination.

The logic works by leveraging ssl_preread, and ssl_preread_alpn_protocols variables but when I enable ssl termination, the ssl_preread_* variables are always empty.

In other words:

• listen 443 ssl; works, because ssl_preread_alpn_protocols is different between xmpp calls and http calls
• listen 443 ssl; does not work, because ssl_preread_alpn_protocols is always empty

Following, an abstract of the configuration:

stream  {
    map $ssl_preread_alpn_protocols $upstream_by_alpn {
        "" xmpp;
        ~xmpp xmpp;
        default http;
    }
    upstream xmpp   {
        server 127.0.0.1:5222;
    }
    upstream http   {
        server 127.0.0.1:8443;
    }
    server  {
        listen 443 ssl; #listen 443; works / listen 443 ssl; does not work
        ssl_certificate /path/to/fullchain.pem;
        ssl_certificate_key /path/to/privkey.pem;
        proxy_pass $upstream_by_alpn;
        ssl_preread on;
    }
}
http {
    server {
        server_name example.com;
        listen 8443;
        location /  {
            proxy_pass http://127.0.0.1:8080/;
        }
    }
}

Questions:

• Is there a way to make ssl_preread_* variables work when the ssl directive is applied?
• Is there a better way to identify http and non-http requests?
• Is there a better approach to proxy pass http and xmpp and enable ssl offload and keeping the same server, ip and port?

PS: I know that by using a different port or domain i wouldn't have such problems, but this is not an option for now.

Tried to use the ssl_preread_protocol, ssl_preread_alpn_protocols, ssl_preread_server_name but it's always empty, when the ssl termination is enabled.

Tried to use the $protocol, but it's always TCP

Answer 1

I never set up xmpp, but I think you need to enable SSL on the destination server, not during ClientHello. Here's what the official Nginx documentation says about it:

The ngx_stream_ssl_preread_module module (1.11.5) allows extracting information from the ClientHello message without terminating SSL/TLS, for example, the server name requested through SNI or protocols advertised in ALPN.

It turns out that the ssl directive and certificates must be removed from the stream and used in the http block.

stream  {
    map $ssl_preread_alpn_protocols $upstream_by_alpn {
        "" xmpp;
        ~xmpp xmpp;
        default http;
    }
    upstream xmpp   {
        server 127.0.0.1:5222;
    }
    upstream http   {
        server 127.0.0.1:8443;
    }
    server  {
        listen 443; #listen 443;
        proxy_pass $upstream_by_alpn;
        ssl_preread on;
    }
}
http {
    server {
        server_name example.com;
        listen 8443 ssl;
        ssl_certificate /path/to/fullchain.pem;
        ssl_certificate_key /path/to/privkey.pem;
        location /  {
            proxy_pass http://127.0.0.1:8080/;
        }
    }
}

You also need to enable ssl on the xmpp server.