Another: istio/envoy 503 "cluster_not_found" issue

Question

I have a service which I can reach from any pod in my cluster using:

curl my-service.my-namespace.svc.cluster.local:8080

but when I try to reach it via gateway/virtualservice, I am getting a "503 cluster_not_found" (response_flags: NC).

Also when querying clusters from the admin endpoint (localhost:15000/clusters) does not show any of "my-service", "my-namespace" or "my-app" (but it does show others).

I just can't see what I am missing to make istio/envoy "see" my service.

I went through my configuration many times and everything looks correct to me. I also ready through every single SO post about envoy and 503s + exhausted all meaningful search results. If anyone has an idea, that would be great!

istio: 1.18.2 envoy: 1.26.4

Gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
  creationTimestamp: "2023-10-27T01:26:47Z"
  generation: 52
  labels:
    app.kubernetes.io/managed-by: Helm
    release: my-release
  name: my-gateway
  namespace: my-namespace
  resourceVersion: "147090413"
  uid: 3d589050-3438-4bc3-42fd-d2ca7caa82c3
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - my-service.my-domain.com
    port:
      name: http8080
      number: 8080
      protocol: HTTP

Service

apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2024-03-11T07:31:10Z"
  labels:
    app: my-app
    app.kubernetes.io/managed-by: Helm
  name: my-service
  namespace: my-namespace
  resourceVersion: "147107337"
  uid: 8a0e4e17-1dc4-48c1-9918-c689b22a3233
spec:
  clusterIP: 172.20.XXX.XXX
  clusterIPs:
  - 172.20.XXX.XXX
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http8080
    port: 8080
    protocol: TCP
    targetPort: 8080
  selector:
    app: my-app
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Virtualservice

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  creationTimestamp: "2024-03-11T12:17:39Z"
  generation: 2
  name: my-virtualservice
  namespace: my-namespace
  resourceVersion: "147102662"
  uid: 4766efa0-a5a5-4354-b80c-ba6ed5fcc65c
spec:
  exportTo:
  - '*'
  gateways:
  - my-gateway
  - mesh
  hosts:
  - my-service.my-domain.com
  http:
  - match:
    - port: 8080
    route:
    - destination:
        host: my-service.my-namespace.svc.cluster.local
        port:
          number: 8080

Answer 1

It turned out we had an environment variable set which prevented publishing to our gateway - which I didn't know about .

The variable was controlling whether we were setting discoverySelectors [1] or not.

Not deleting my question to leave the boilerplate examples here.

[1] https://istio.io/latest/docs/reference/config/istio.mesh.v1alpha1/