Kubelet API access from k8s operator

Question

I playing with operator-sdk and trying to issue an API call to Kubelet on a specific node.

My API calls result in result code 403 - Forbidden.

Below is the relevant code piece:

clientset, err := kubernetes.NewForConfig(r.Config)
    if err != nil {
        logger.Info("Failed to get clientset")
        return ctrl.Result{}, err
    }
    logger.Info("Successfully got clientset")

    restClient := clientset.CoreV1().RESTClient()

    for _, container := range pod.Spec.Containers {
        kubeletAPI := fmt.Sprintf("http://localhost:8080/api/v1/nodes/%s/proxy/checkpoint/%s/%s/%s", pms.Spec.SourceNode, pms.Spec.Namespace, pms.Spec.PodName, container.Name)
        result := restClient.Post().AbsPath(kubeletAPI).Do(context.TODO())
        rawBody, _ := result.Raw()
        if result.Error() != nil {
            logger.Info("Failed to checkpoint container", "namespace", pms.Spec.Namespace, "podName", pms.Spec.PodName, "containerName", container.Name)
        } else {
            logger.Info("Succeded to checkpoint container")
        }

        logger.Info("Checkpoint result", "result", result, "response", string(rawBody))

    }

Below is the role.yaml:

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  creationTimestamp: null
  name: manager-role
rules:
- apiGroups:
  - ""
  resources:
  - endpoints
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/proxy
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - pods
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - services
  verbs:
  - create
  - delete
  - get
  - list
  - watch
- apiGroups:
  - migration.pantarhei.ai
  resources:
  - podmigrations
  verbs:
  - create
  - delete
  - get
  - list
  - patch
  - update
  - watch
- apiGroups:
  - migration.pantarhei.ai
  resources:
  - podmigrations/finalizers
  verbs:
  - update
- apiGroups:
  - migration.pantarhei.ai
  resources:
  - podmigrations/status
  verbs:
  - get
  - patch
  - update

Below is the response:

2023-09-25T17:34:07Z    INFO    Checkpoint result       {"controller": "podmigration", "controllerGroup": "migration.pantarhei.ai", "controllerKind": "PodMigration", "PodMigration": {"name":"podmigration-sample","namespace":"pantarhei-operator-system"}, "namespace": "pantarhei-operator-system", "name": "podmigration-sample", "reconcileID": "237abbaf-36ac-4562-9182-b183ba8e6e70", "result": {}, "response": "{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"message\":\"forbidden: User \\\"system:serviceaccount:pantarhei-operator-system:pantarhei-operator-controller-manager\\\" cannot post path \\\"/http:/localhost:8080/api/v1/nodes/pantarhei-dp1/proxy/checkpoint/pantarhei-operator-system/ubuntu/ubuntu-container\\\"\",\"reason\":\"Forbidden\",\"details\":{},\"code\":403}\n"}

I cannot understand what permission I am missing.

Answer 1

kubelet agents only communicate with the k8s API server. Whatever the thing you are trying to get done with kubelet, should be possible with k8s API server access. In the operator-pattern, the operator pods don't do any direct communication with kubelet.

See CNCF Operator White Paper 's security section,