Traefik cannot Issue Lets Encrypt Cert for gitlab container on different port

Question

im running a gitlab-ee docker container behind a traefik v1 docker container. My gitlab is supposed to run on the domain gitlab.dev.example.com:65443 the port 65443 is being forwarded to 443 on my server within my router (i have other stuff running on my 443 and 80 port). my traefik dashboard is running on traefik.dev.example.com:65443/dashboard/

Now when i want to get a lets encrypt certificate with traefik it tries to get it for the domain gitlab.dev.example.com and fails with "Unable to obtain ACME certificate for domains "gitlab.dev.example.com" [...]"

if i visit https://gitlab.dev.example.com:65443 it opens my gitlab container but with the "Traefik Default Cert". I cannot push or pull from these gitlab repositories because the SSL Certificates are self signed hence why i want to get a lets encrypt one.

i swapped out the actual domain with "example.com" obviously

my traefik.toml:

debug = false

logLevel = "ERROR"
  defaultEntryPoints = ["https", "http"]

[web]
  address = ":80"

[docker]
  endpoint = "unix:///var/run/docker.sock"
  domain = "traefik.dev.example.com" //swapped the url out
  watch = true
  exposedByDefault = false

# Force HTTPS
[entryPoints]
  [entryPoints.http]
  address = ":80"
    [entryPoints.http.redirect]
    entryPoint = "https"
  [entryPoints.https]
  address = ":443"
  [entryPoints.https.tls]

# Enable retry sending a request if the network error
[retry]

# Let's encrypt configuration
[acme]
   email="[email protected]" //swapped the email out
   storage="acme.json"
   entryPoint="https"
   acmeLogging=true
   OnHostRule=true
[acme.httpChallenge]
   entryPoint = "http"

now to my docker-compose files. i have a seperate docker-compose.yml for each container my docker-compose.yml for the traefik container:

version: "3.2"

services:
  reverse-proxy:
    image: traefik:alpine
    command: --api --docker --logLevel=error
    restart: unless-stopped
    container_name: docker-traefik
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./traefik.toml:/etc/traefik/traefik.toml
      - ./acme.json:/acme.json
    environment:
      - "TZ=Europe/Berlin"
    networks:
      - traefik_proxy
      - default
    ports:
      - "443:443"
      - "80:80"
    logging:
      driver: "json-file"
      options:
        max-file: "3"
        max-size: "5m"
    labels:
      - traefik.backend=traefik-proxy
      - traefik.frontend.rule=Host:traefik.dev.loropserver.de
      - traefik.docker.network=traefik_proxy
      - traefik.port=8080
      - traefik.enable=true
      - traefik.frontend.auth.basic=lorop:$$apr1$$dHnqprRX$$DjIWIaE97EnMoxwu6o/14.
networks:
  traefik_proxy:
    external:
      name: traefik_proxy
#  default:
#    driver: bridge

my docker-compose.yml for the gitlab container:

version: '3.5'

services:
  gitlab:
    image: 'gitlab/gitlab-ee:latest'
    container_name: gitlab
    restart: unless-stopped
    hostname: 'gitlab.dev.example.com'
    environment:
      GITLAB_OMNIBUS_CONFIG: |
        external_url 'https://gitlab.dev.example.com';
        //some more configs
    ports:
      - '22:22'
      - '5005:5005'
    volumes:
      - './volumes/gitlab/config:/etc/gitlab'
      - './volumes/gitlab/logs:/var/log/gitlab'
      - './volumes/gitlab/data:/var/opt/gitlab'
      - /etc/localtime:/etc/localtime:ro
      - './certs:/etc/gitlab/trusted-certs'
    networks:
      - traefik_proxy
    labels:
      - 'traefik.enable=true'
      - 'traefik.port=65443'
      - 'traefik.docker.network=traefik_proxy'
      - 'traefik.backend=gitlab'
      - 'traefik.frontend.rule=Host:gitlab.dev.example.com'
      - 'traefik.http.routers.entrypoints=websecure'

networks:
  traefik_proxy:
    external: true