unexpected EOF when pulling from private docker registry with two repositories

Question

Setup:

• I am hosting a private docker registry by using the official registry (as a docker container) from docker (https://hub.docker.com/%5C_/registry).

• On that registry, I have two repos repo1 and repo2

• They can be seen as different flavours of one application and therefore share some layers

• I am pushing images from a separated build server to that registry

• I know related issues (Docker pull fails with unexpected EOF) but neither do they present a solution to the root cause nor does the work-around reliable fix my problem

Problem:

• When pulling from that registry, I (sometimes) get:

docker pull registry.foobar.com/repo1/repo1:latest 
latest: Pulling from repo1/repo1
125a6e411906: Already exists 
ab029ed1e0ab: Already exists 
bd96b0d396b3: Pull complete 
e0135e3ddcf1: Pull complete 
6a8cac0382d1: Pull complete 
756f32f3ded1: Downloading [==================================================>]  2.291kB/2.291kB
unexpected EOF 

• Same issue exists then on the other repository:

docker pull registry.foobar.com/repo2/repo2:latest 
latest: Pulling from repo2/repo2
<different SHA>: Already exists 
<different SHA>: Already exists 
<different SHA>: Pull complete 
<different SHA>: Pull complete 
<different SHA>: Pull complete 
756f32f3ded1: Downloading # <<< same sha as the other repo [==================================================>]  2.291kB/2.291kB
unexpected EOF 

Investigation (so far):

• When logging into the registry and checking on a filesystem level, I saw that the failing layer does not even exists on the registry (therefore EOF makes 100% sense...)

• I deleted the corresponding layer by first retrieving the full sha256 from the manifest and then executing:

curl --user "$USER:$PASS" -H 'Accept: application/vnd.docker.distribution.manifest.v2+json' -k -X DELETE "https://registry.com/v2/repo1/repo1/blobs/sha256:THE_SHA"

• after deleting the layer and re-pushing afterwards I can see:

  docker push registry.foobar.com/repo1/repo1:latest
  The push refers to repository [registry.foobar.com/repo1/repo1]
  c9a6d53cb9c5: Mounted from repo2/repo2 # <<<<< this caught my attention!
  231dc2de5585: Layer already exists 
  4c868cb538dc: Layer already exists 
  7a3259a58e8b: Layer already exists 
  a4ed6cf37dad: Layer already exists 
  256d88da4185: Layer already exists 
  latest: digest: sha256:904655eda0e83255442c57011f1b0cb37d045612925d1893fa530136f234104c size: 1583
  

• However the re-push does not solve the problem during pulling

I am somehow expecting the shared layers between both repositories and that the affected layer is not even transferred to the registry. But so far I have no real proof nor a solution for the issue.

I can docker pull without issues.

Answer 1

Your registry is corrupt, and you need to correct that corruption by deleting blobs from the registry state that do not exist in the blob store. That's done by inspecting the image to get the full blob digest of the failing blobs:

docker buildx imagetools inspect $image

And then deleting those blobs, from every repository where they may be located, using the registry API:

curl --user "$USER:$PASS" -k -X DELETE "https://${registry}/v2/${repo}/blobs/sha256:${digest}"