Config reverse proxy for multiple nginx services in docker swarm mode

Question

I am working on config a docker swarm cluster， and my original plan is to set 3 replicas of nginxs work as my reverse proxies and load balancers. I think I locate the problme, which the 3 replicas of nginx will make request to my container, and the container sees as invalid request.

The root error(I think)

Invalid HTTP_HOST header: 'archive.tomhanfriends.com,archive.tomhanfriends.com,archive.tomhanfriends.com'. The domain name provided is not valid according to RFC 1034/1035.

Am I missing some configurations or settings in order to make the nginx replicas work inside of a docker swarm cluster? The following are my setups.

---

The docker swarm cluster

The Nginxs Docker

version: '3.8'
services:
  nginx:
    image: docker.io/nginx:latest
    restart: unless-stopped
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - ./nginx/data/conf/conf.d:/etc/nginx/conf.d
      - ./nginx/data/conf/ssl:/etc/nginx/ssl
      - ./nginx/data/logs:/var/log/nginx
      - ./nginx/data/www:/usr/share/nginx/html
    deploy:
      replicas: 3
      placement:
        constraints:
          - node.role == manager
      restart_policy:
        condition: on-failure
        delay: 5s
        max_attempts: 3
        window: 120s
    networks:
      - infrastructure
      - ops
networks:
  infrastructure:
    external: true
  ops:
    external: true

The Nginx Conf

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    tcp_nopush on;
    tcp_nodelay on;
    types_hash_max_size 2048;
    server_tokens off;

    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;
    gzip on;

    gzip_vary on;
    gzip_proxied any;
    gzip_comp_level 6;
    gzip_buffers 16 8k;
    gzip_http_version 1.1;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

    proxy_connect_timeout 300s;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    proxy_buffer_size 128k;
    proxy_buffers 4 128k;
    proxy_busy_buffers_size 256k;
    proxy_temp_file_write_size 256k;
    proxy_headers_hash_max_size 1024;
    proxy_headers_hash_bucket_size 128;
    proxy_redirect off;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    include /etc/nginx/conf.d/*.conf;
}

The archivebox nginx conf

server {
    listen 80;
    listen 443 ssl;
    charset utf-8;
    server_name archive.tomhanfriends.com;
    .... omit for ssl
    location / {
        **proxy_pass http://ops_archivebox:8000/;**
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;proxy_set_header Host $host;
        proxy_set_header Host $http_host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Scheme $scheme;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_redirect off;
    }

    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   html;
    }

}

I have checked the following

• [x] the dns config works, ping works
• [x] the nginx config file check works, nginx -t
• [x] the docker swarm service works, in the same network, able to talk
• [x] the docker compose for nginx works
• [x] the docker compose for archivebox works`

Answer 1

Figured out, answer some questions if someone find it here.

The main error:

Invalid HTTP_HOST header: 'archive.tomhanfriends.com,archive.tomhanfriends.com,archive.tomhanfriends.com'. The domain name provided is not valid according to RFC 1034/1035.

which figured out according to the second commend, need to double check the nginx cfg file.

About multiple nginx container in docker swarm:

Works like a charm, which I think the ingress of docker swarm will handel the traffic

About the reverse proxy to container and port:

In simple docker mode, insite a same network, use the containder_name:port_expose, the swarm mode, use the service_name:port_expose to pass the proxy.