Cloudflare, One Tunnel for Multiple Servers with Different Services

Question

I've registered and configured my domain's name servers to use cloudflare.

I've configured a bunch of web apps on "Server A" to run through a cloudflare tunnel. Everything works beautifully, and I can access my apps on subdomain.domain.whatever without problems.

I'm using docker compose to run the tunnel service:

  tunnel:
    image: cloudflare/cloudflared
    restart: always
    command: tunnel run
    environment:
      - TUNNEL_TOKEN=my_token_here

Then exposing my apps to the public hostname:

Now, my challenge is when trying to run a copy of "my-service", or any service (not necessary replicas) on "Server B", this server can be local or remote.

I'm using the exact same tunnel configuration locally.

The tunnel service is running on the same docker network as the services I'm trying to expose.

The services themselves are healthy and accessible on localhost or through port forwarding.

On cloudflare, both servers "Connectors" appear under the same tunnel:

Whenever I try to access anything from "Server B", I run into the infamous error message:

Edit 1: Interesting findings today, either everything on "Server A" works, or everything on "Server B", but never both. Like a failover for the whole server, but not for services within the server.

In the example below I can access either:

Services A, B, and C, or

Services C, D and E.

but never A, B, C, D, and E.