k8s docker hub login credentials is not working

Question

The problem is this: I have a cluster on Kubernetes with containerd as container runtime and Docker Hub Registry, where I store my private images, also master node on linux centos 7.

I logged in to Docker using the docker login and my PAT (personal access token), after which I checked the work of docker and pulled my several private images without any problems. Next, I created a secret "dockerhub-credentials" for my k8s in the same namespace where I plan to pull the images.

# kubectl create secret docker-registry dockerhub-credentials \
    --docker-server=https://index.docker.io/v1/ \
    --docker-username=<username> \
    --docker-password=<dockerhubPAT> \
    --docker-email=<[email protected]> -n <mynamespace>

after that, I was able to pull public images via k8s deployment without any problems, but I can't pull private ones from my own account in docker hub. my sample deployment in k8s:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-world-deployment
  namespace: <mynamespace>
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-world
  template:
    metadata:
      labels:
        app: hello-world
    spec:
      containers:
      - name: hello-world-container
        image: <dockerhub-username>/<dockerhub-repository-name>:<image-tag>
      imagePullSecrets:
      - name: dockerhub-credentials

The errors is:

# kubectl get pods -n <mynamespace>
NAME                                     READY   STATUS             RESTARTS   AGE
hello-world-deployment-79dbffcd7-xq5br   0/1     ImagePullBackOff   0          146m

# kubectl logs hello-world-deployment-79dbffcd7-xq5br -n <mynamespace>
Error from server (BadRequest): container "hello-world-container" in pod "hello-world-deployment-79dbffcd7-xq5br" is waiting to start: trying and failing to pull image

and describe:

# kubectl describe pod hello-world-deployment-79dbffcd7-xq5br -n <mynamespace>
Name:             hello-world-deployment-79dbffcd7-xq5br
Namespace:        <mynamespace>
Priority:         0
Service Account:  default
Node:             worker-1/192.168.10.8
Start Time:       Tue, 19 Sep 2023 22:35:44 +0600
Labels:           app=hello-world
                  pod-template-hash=79dbffcd7
Annotations:      <none>
Status:           Pending
IP:               10.44.0.2
IPs:
  IP:           10.44.0.2
Controlled By:  ReplicaSet/hello-world-deployment-79dbffcd7
Containers:
  hello-world-container:
    Container ID:   
    Image:          <dockerhub-username>/<dockerhub-repository-name>:<image-tag>
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-ntlw7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  kube-api-access-ntlw7:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason   Age                    From     Message
  ----     ------   ----                   ----     -------
  Warning  Failed   29m (x13 over 135m)    kubelet  Error: ErrImagePull
  Warning  Failed   20m (x2 over 37m)      kubelet  Failed to pull image "<dockerhub-username>/<dockerhub-repository-name>:<image-tag>": failed to pull and unpack image "docker.io/<dockerhub-username>/<dockerhub-repository-name>:<image-tag>": failed to read expected number of bytes: unexpected EOF
  Warning  Failed   10m (x13 over 135m)    kubelet  Failed to pull image "<dockerhub-username>/<dockerhub-repository-name>:<image-tag>": failed to pull and unpack image "docker.io/<dockerhub-username>/<dockerhub-repository-name>:<image-tag>": failed to copy: httpReadSeeker: failed open: server message: invalid_token: authorization failed
  Normal   BackOff  9m49s (x55 over 135m)  kubelet  Back-off pulling image "<dockerhub-username>/<dockerhub-repository-name>:<image-tag>"

and here's my secret:

# kubectl describe secrets dockerhub-credentials -n <mynamespace>
Name:         dockerhub-credentials
Namespace:    <mynamespace>
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/dockerconfigjson

Data
====
.dockerconfigjson:  223 bytes

JSON output:

# kubectl get secret dockerhub-credentials -n <mynamespace> -o json
{
    "apiVersion": "v1",
    "data": {
        ".dockerconfigjson": "<HERE'S-BASE64-ENCODED-DATA-FROM-DOCKERCONFIG.JSON>"
    },
    "kind": "Secret",
    "metadata": {
        "creationTimestamp": "2023-09-19T16:35:32Z",
        "name": "dockerhub-credentials",
        "namespace": "<mynamespace>",
        "resourceVersion": "3557701",
        "uid": "e453ef1d-f4df-4c2a-9c08-f4e1271f8b03"
    },
    "type": "kubernetes.io/dockerconfigjson"
}

Btw, I replaced all real data with syntax to make it more abstract

I've tried different authorization methods and read a lot of discussions on stackoverflow and other resources. I've been stuck at this basic stage for several days now. Sorry, I'm just learning (also, this is my first stackoverflow problem). I am ready to provide almost any information if someone can solve this problem! I would be grateful for any help!

Answer 1

It might be due to docker login.

Add a command in your deployment to docker login with your credentials.