Kubernetes: Haproxy ingress isn't taking on any addresses

Question

I'm running Kubernetes v1.25.9+rke2r1. I have a Metallb correctly setup with an external ip address and haproxy ingress controller. I need to expose a cockroachdb instance via ingress and I set up that config in the crdb instance correctly and the ingress spins up correctly. but without any ip address. And therefore when I click the page goes into timeout, if I telnet the port it goes into timeout aswell.

k get ingress -A
NAMESPACE              NAME              CLASS     HOSTS                                                     ADDRESS   PORTS   AGE
cockroachdb-instance   sql-cockroachdb   haproxy   cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it             80      15m

From the haproxy pod logs I don't see any errors, to my knowledge the ingress and the corresponding service are setup correctly. What can I do in order to further debug this or fix this?

k describe ingress sql-cockroachdb -n cockroachdb-instance
Name:             sql-cockroachdb
Labels:           app.kubernetes.io/component=database
                  app.kubernetes.io/instance=cockroachdb
                  app.kubernetes.io/managed-by=cockroach-operator
                  app.kubernetes.io/name=cockroachdb
                  app.kubernetes.io/part-of=cockroachdb
                  app.kubernetes.io/version=v23.1.4
                  crdb=test
Namespace:        cockroachdb-instance
Address:          
Ingress Class:    haproxy
Default backend:  <default>
Rules:
  Host                                                     Path  Backends
  ----                                                     ----  --------
  cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it  
                                                              cockroachdb-public:sql (10.222.19.88:26257,10.222.196.89:26257,10.222.30.93:26257)
Annotations:                                               crdb.io/last-applied:
                                                             UEsDBBQACAAIAAAAAAAAAAAAAAAAAAAAAAAIAAAAb3JpZ2luYWyMk0GT0zAMhf+LzrG3pbvQ5ki5cIEZYLgwHGRbaTxxbK+sFDqd/HfGbelsoYe9aZSn7+kpyRFGEnQoCO0RAhoKpV...
                                                           field.cattle.io/publicEndpoints:
                                                             [{"addresses":[""],"port":80,"protocol":"HTTP","serviceName":"cockroachdb-instance:cockroachdb-public","ingressName":"cockroachdb-instance...
Events:
  Type    Reason  Age                From                Message
  ----    ------  ----               ----                -------
  Normal  CREATE  28m                ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  CREATE  28m                ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  UPDATE  27m (x2 over 27m)  ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb
  Normal  UPDATE  27m (x2 over 27m)  ingress-controller  Ingress cockroachdb-instance/sql-cockroachdb

If I try to port-forward the service that handles cockroachdb I get this error:

E0711 09:59:38.779413   43555 portforward.go:409] an error occurred forwarding 8081 -> 26258: error forwarding port 26258 to pod 8a774d302b846fdbdd7bbc6b3f35144d9712f15de9ab72d4ef9e0c8cdfa8ee85, uid : failed to execute portforward in network namespace "/var/run/netns/cni-f1c36dc8-3647-e00e-b67e-7abb019b236b": read tcp4 127.0.0.1:33424->127.0.0.1:26258: read: connection reset by peer

This is the Kind: CrdbCluster - The installation of cockroachdb was done via kubectl apply with all default values.

apiVersion: crdb.cockroachlabs.com/v1alpha1
kind: CrdbCluster
metadata:
  # this translates to the name of the statefulset that is created
  name: cockroachdb
spec:
  dataStore:
    pvc:
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: "20Gi"
        volumeMode: Filesystem
  resources:
    requests:
      # This is intentionally low to make it work on local k3d clusters.
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 2
      memory: 4Gi
  tlsEnabled: true
   ingress:
    #ui:
    #  ingressClassName: haproxy
    #  annotations:
    #    key: value
    #  host: cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it
    sql:
      ingressClassName: haproxy
      annotations:
        #key: value
      host: cockroachdb.dev-service-stage.k8s.mi1.prod.cloudfire.it
# You can set either a version of the db or a specific image name
# cockroachDBVersion: v23.1.4
  image:
    name: cockroachdb/cockroach:v23.1.4
  # nodes refers to the number of crdb pods that are created
  # via the statefulset
  nodes: 3
  additionalLabels:
    crdb: test

Answer 1

The missing IP in the ingress list is because you have the error about the port-forward the service.

The reason you have that error, and the connection is refused, is there is no process listening on the port you're forwarding (26258) and this may be related either to:

• you didn't change the default DB port to the port you want to forward (26258)
• from the Dockerfile you're exposing the port 26258 even if your DB is still on the default port 26257

Answer 2

The issue was with metallb and a custom resource

while adding this to the manifests, everything was provisioned successfully:

apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: advertisement
  namespace: metallb
spec:
  ipaddresspools:
  - first-pool