Rancher 2 RKE cluster component kube-api-auth is throwing an error with v1Authenticate request

Question

I’m facing a strange error in my RKE1 k8s cluster. Running 1.24.13 on vsphere. kube-api-auth is constantly throwing these errors in the logs:

time="2023-06-08T09:21:09Z" level=info msg="Starting Rancher Kube-API-Auth service on 127.0.0.1:6440"
time="2023-06-08T09:21:09Z" level=info msg="Running in single server mode, will not peer connections"
time="2023-06-08T09:21:09Z" level=info msg="Starting workload controllers"
time="2023-06-08T09:21:09Z" level=info msg="Starting cluster.cattle.io/v3, Kind=ClusterUserAttribute controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting catalog.cattle.io/v1, Kind=ClusterRepo controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=Role controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting /v1, Kind=Secret controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRoleBinding controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=RoleBinding controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting rbac.authorization.k8s.io/v1, Kind=ClusterRole controller"
time="2023-06-08T09:21:09Z" level=info msg="Starting cluster.cattle.io/v3, Kind=ClusterAuthToken controller"
time="2023-06-08T09:21:24Z" level=info msg="Processing v1Authenticate request..."
time="2023-06-08T09:21:24Z" level=error msg="found 1 parts of token"
time="2023-06-08T09:22:24Z" level=info msg="Processing v1Authenticate request..."
time="2023-06-08T09:22:24Z" level=error msg="found 1 parts of token"
time="2023-06-08T09:24:00Z" level=info msg="Processing v1Authenticate request..."
time="2023-06-08T09:24:00Z" level=error msg="found 1 parts of token"

I couldn't find anything in the Rancher support pages related to this or here on stackoverflow. Any help would be appreciated.

Answer 1

You had a similar error message in vmware-tanzu/kubeapps issue 1874, pointing to issue 2111.

It suggests checking the token format. The thread points to jwt.io to decrypt a JWT token and its three parts: header, payload, and signature.
Make sure that token is included in the Authorization header in the format Bearer <token>.

Looking at RKE1 default Kubernetes services, I would also check the logs of:

• the kube-apiserver: ssh <user>@<master-node-ip> + docker ps + docker logs <container-id-or-name>
• the Authentication provider.
  It depends on which authentication service you have configured.
  For instance: kubectl get pods -n <namespace-where-keycloak-is-running> + kubectl logs <pod-name> -n <namespace-where-keycloak-is-running>

---

I have Rancher UI login configured through Microsoft Azure AD and also Argo UI is configured to use Azure AD.

Is there anything specific there that could be causing this error to appear?

The fact that the Rancher UI login and Argo UI are both configured to use Microsoft Azure AD as the authentication provider could be relevant to the issue.

Azure AD issues tokens for authentication, and it is essential that these tokens are correctly configured. Verify that the token configuration in Azure AD matches what the Rancher and Argo UIs expect. For instance, check if the tokens have the correct claims, audience, and issuer.
If you have Conditional Access policies set up in Azure AD, ensure that they are not inadvertently blocking access for Rancher or Argo.

Furthermore, for integration with external applications like Rancher and Argo, you might have Azure AD Application Registrations. Ensure that these are configured correctly, including permissions and API permissions.

As usual, check the logs of Rancher and Argo for any authentication-related errors.
Additionally, review the logs or audit trails in Azure AD to see if there are any failed authentication attempts or issues with token issuance.