Kafka Bootstrap broker <lb>:9092 (id: -1 rack: null) disconnected

Question

I have kafka set up on EKS cluster using the strimzi set up. https://strimzi.io/install/latest?namespace=kafka https://strimzi.io/examples/latest/kafka/kafka-persistent-single.yaml When I try to access the kafka using the servicename from inside the cluster I am able to connect. But when I try from outside the cluster or even from inside the cluster using the LB address (attached to istio ingressgateway ) and the port, I get error.

bin/kafka-console-producer.sh --bootstrap-server :9092 --topic test

[2024-01-10 13:51:11,754] WARN [Producer clientId=console-producer] Bootstrap broker :9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

Kafka version : 3.6.1 Istio : 1.15.6

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"kakfa-gateway","namespace":"kafka"},"spec":{"selector":{"istio":"ingressgateway"},"servers":[{"hosts":["internal-ac6cbcd571e9143c78a2890d16feedfd-1934725189.ap-south-1.elb.amazonaws.com"],"port":{"name":"kafka-tcp","number":9092,"protocol":"TCP"}}]}}
  creationTimestamp: "2024-01-10T05:59:41Z"
  generation: 3
  name: kakfa-gateway
  namespace: kafka
  resourceVersion: "253310486"
  uid: 6757f20f-2ec9-4599-9fca-ff9fa047cd95
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - LB
    port:
      name: kafka-tcp
      number: 9092
      protocol: TCP

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"VirtualService","metadata":{"annotations":{},"name":"kafka-vs","namespace":"kafka"},"spec":{"gateways":["kafka-gateway"],"hosts":["*"],"tcp":[{"match":[{"port":9092}],"route":[{"destination":{"host":"my-cluster-kafka-external-bootstrap","port":{"number":9094}}}]}]}}
  creationTimestamp: "2024-01-10T05:59:41Z"
  generation: 12
  name: kafka-vs
  namespace: kafka
  resourceVersion: "253338379"
  uid: ed7f54bb-72a9-40ff-87d2-5faa5b99c83f
spec:
  gateways:
  - kafka-gateway
  hosts:
  - '*'
  tcp:
  - match:
    - port: 9092
    route:
    - destination:
        host: my-cluster-kafka-bootstrap
        port:
          number: 9094

added kafka listeners

    listeners:
    - name: plain
      port: 9092
      tls: false
      type: internal
    - name: tls
      port: 9093
      tls: true
      type: internal
    - configuration:
        brokers:
        - advertisedHost: LB
          advertisedPort: 9092
          broker: 0
      name: external
      port: 9094
      tls: false
      type: internal

I have tried using the listener type as nodeport and even with that internally using the node IP and nodePort from inside the cluster its working but when I use the LB address and the port in the --bootstrap-server I get connection error as mentioned above.

I have also added the port in istio-ingressgateway svc and the port is opened on the LB and I am able to telnet from outside.

Please let me know what else needs to be done to make the outside connectivity work using the istio ingressgateway with the LB.

I cannot use the kafka listener type as LB as it will create a new LB which I do not want unless I can use the same LB.

Outside connection to kafka not working and all details mentioned in description.