`ErrImageNeverPull` error when trying to start a pod

Question

I have created a local kubernetes cluster using K3S. Following are the nodes in my cluster.

hpc-14@cluster-pc2:~/software/jasminegraph$ kubectl get nodes -o wide
NAME           STATUS   ROLES                  AGE   VERSION        INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
cluster-pc04   Ready    <none>                 34m   v1.28.5+k3s1   10.8.100.10   <none>        Ubuntu 22.04.2 LTS   6.2.0-33-generic   containerd://1.7.11-k3s2
cluster-pc2    Ready    control-plane,master   49m   v1.28.5+k3s1   10.8.100.2    <none>        Ubuntu 22.04.3 LTS   6.5.0-14-generic   docker://25.0.1
cluster-pc1    Ready    <none>                 33m   v1.28.5+k3s1   10.8.100.11   <none>        Ubuntu 22.04.3 LTS   6.2.0-34-generic   containerd://1.7.11-k3s2

Here, the contol-plane,master node is started with --docker flag. So the CONTAINER-RUNTIME is docker there as expected.

I'm in master node and I've built jasminegraph image there. I can see the docker image is listed when I run docker images.

hpc-14@cluster-pc2:~/software/jasminegraph$ docker images
REPOSITORY                         TAG                    IMAGE ID       CREATED          SIZE
jasminegraph                       latest                 8364e6010bb3   36 minutes ago   3.15GB
rancher/klipper-helm               v0.8.2-build20230815   5f89cb8137cc   5 months ago     256MB
rancher/local-path-provisioner     v0.0.24                b29384aeb4b1   10 months ago    40.1MB
rancher/mirrored-metrics-server    v0.6.3                 817bbe3f2e51   10 months ago    68.9MB
rancher/mirrored-coredns-coredns   1.10.1                 ead0a4a53df8   11 months ago    53.6MB
rancher/mirrored-pause             3.6                    6270bb605e12   2 years ago      683kB

When I tried to start a deployment being inside the master node pods are not started and I can see following error.

hpc-14@cluster-pc2:~/software/jasminegraph$ kubectl get pods
NAME                                              READY   STATUS              RESTARTS   AGE
jasminegraph-master-deployment-57574bd64d-tclrv   0/1     ErrImageNeverPull   0          11s

I have specified the imagePullPolicy: Never as well.

How can I get this working?. Should I need to have CONTAINER-RUNTIME as docker for each node?. Should I need to transfer the built docker image for other nodes?.

Answer 1

After spending hours on debugging being able to fix the problems.

• ImageInspectError in master node - Got this fixed by downgrading the docker version to 24.0.7 from 25.0.1 previously used.
• Still the images are being inaccessible by other nodes and caused ErrImageNeverPull when pods got scheduled to the other nodes. Since, I was using a local image I had to make other nodes' CONTAINER-RUNTIME to docker and transfer the build docker image to other nodes.