containerd can't pull image from nexus

Question

We have upgraded kops kubernetes cluster from v1.19.2 (docker://19.3.15 ) to v1.24.11 (containerd://1.6.18) but after the upgrade we are not able to pull docker images from our nexus repository.

It’s giving error:

Failed to pull image "myregistry.com/infra-tools/busybox:latest": rpc error: code = NotFound desc = failed to pull and unpack image "myregistry.com/infra-tools/busybox:latest": failed to resolve reference "myregistry.com/infra-tools/busybox:latest": myregistry.com/infra-tools/busybox:latest: not found

Image is present and I can pull/push using docker.

From worker node:

ctr --debug images pull myregistry.com/infra-tools/busybox:latest
DEBU[0000] fetching                                      image="myregistry.com/infra-tools/busybox:latest"
DEBU[0000] resolving                                     host=myregistry.com
DEBU[0000] do request                                    host=myregistry.com request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/1.6.6 request.method=HEAD url="https://myregistry.com/v2/infra-tools/busybox/manifests/latest"
DEBU[0000] fetch response received                       host=myregistry.com response.header.cache-control="post-check=0, pre-check=0" response.header.connection=keep-alive response.header.content-length=2031 response.header.content-security-policy="sandbox allow-forms allow-modals allow-popups allow-presentation allow-scripts allow-top-navigation" response.header.content-type=text/html response.header.date="Fri, 10 Mar 2023 10:54:29 GMT" response.header.expires=0 response.header.pragma=no-cache response.header.server=nginx response.header.x-content-type-options=nosniff response.status="404 Not Found" url="https://myregistry.com/v2/infra-tools/busybox/manifests/latest"
INFO[0000] trying next host - response was http.StatusNotFound  host=myregistry.com
ctr: failed to resolve reference "myregistry.com/infra-tools/busybox:latest": myregistry.com/infra-tools/busybox:latest: not found

Is anyone facing the same issue?

Answer 1

This problem appeared in versions before nexus 3.2.0, and the problem disappeared after upgrading to a newer version.

Reason:
containerd submits HTTP HEAD request while pulling image, but old nexus version returns 404 response to HEAD request.

Reproduce problem mock request:

curl -v -X HEAD -H "Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" "https://<docker-registry>/v2/<image-name>/manifests/<image-tag>"