Kubernetes Nginx Controller - "Service rancher does not have any active Endpoint"

Question

The RKE2 install goes great, but once I install Rancher via Helm and try to reach the management UI, I'm unable to reach and receive a "404 Not Found" error.

root@rancher-master-01:~# kubectl -n kube-system logs -f rke2-ingress-nginx-controller-46nh6
W0228 16:56:38.654557       8 controller.go:1112] Service "cattle-system/rancher" does not have any active Endpoint.
W0228 16:56:38.654630       8 controller.go:1333] Error getting SSL certificate "cattle-system/tls-rancher-ingress": local SSL certificate cattle-system/tls-rancher-ingress was not found. Using default certificate


root@rancher-master-01:~# kubectl get endpoints,svc,ingress -n cattle-system
NAME                        ENDPOINTS                                            AGE
endpoints/rancher           10.42.2.3:80,10.42.3.7:80,10.42.4.7:80 + 3 more...   63m
endpoints/rancher-webhook   10.42.4.10:9443                                      61m
endpoints/webhook-service   10.42.4.10:8777                                      61m

NAME                      TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)          AGE
service/rancher           ClusterIP   10.43.185.148   <none>        80/TCP,443/TCP   63m
service/rancher-webhook   ClusterIP   10.43.102.206   <none>        443/TCP          61m
service/webhook-service   ClusterIP   10.43.74.170    <none>        443/TCP          61m

NAME                                CLASS    HOSTS                          ADDRESS                                                                      PORTS     AGE
ingress.networking.k8s.io/rancher   <none>   odsrancher.x.x.com   172.22.137.148,172.22.137.149,172.22.137.150,172.22.137.151,172.22.137.186   80, 443   63m

Answer 1

There is a detailed troubleshooting procedure described for this issue in the official rancher documentation.

According to the documentation there are many reasons for NGINX or ingress controller not to forward traffic to your rancher instance, while the most commonly this issue is caused due to ssl misconfiguration.

The basic things to check here are:

1. Is rancher running
2. Cert CN is “Kubernetes Ingress controller Fake Certificate”

Are the rancher pods running?

When doing the rancher installation you might have created a namespace called cattle-system, using the kubectl command, check whether the pods in this namespace are in running state. If you find any failed or crashed pods check the logs of those particular pods and also you can use the kubectl describe command.

Cert CN is “Kubernetes Ingress controller Fake Certificate”

In your browser verify the certificate details if you find anything like Common Name is Kube Ingress Controller fake certificate something might have gone wrong while reading or issuing your ssl certificate. As per the official documentation if you are using LetsEncrypt certificates it will take some time for issuing the certificate. Apparently you are using custom generated or rancher generated certificates, follow the steps mentioned in the troubleshooting doc for more details.

Note: I’m not including every detail here because this is provided using official document and it's already detailed there. Hope this helps you in resolving the issue.