filesystem layer verification failed for digest for native docker

Question

I'm running docker on Ubuntu 22.04 (CLI, not Docker-Desktop) and I keep getting layer verification error. See the example below:

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Downloading [==========================>                        ]  15.44MB/29.12MB
f715c8c55756: Download complete 
b11a1dc32c8c: Downloading  2.693MB
f29e8ba9d17c: Verifying Checksum 
78af88a8afb0: Waiting 
b74279c188d9: Waiting 
6e3e5bf64fd2: Waiting 
b62a2c2d2ce5: Waiting 
eba91ca3c7a3: Waiting 
d4a24cdf2433: Waiting 
b20f8a8dfd5c: Waiting 
e0731dd084c3: Waiting 
0361da6a228e: Waiting 
filesystem layer verification failed for digest sha256:f29e8ba9d17cfa147141648b72ff8ab49a86234dfe1194f6220690939f1daa3c

mjozwikowski:~$ docker -D pull postgres:latest
latest: Pulling from library/postgres
648e0aadf75a: Pull complete 
f715c8c55756: Pull complete 
b11a1dc32c8c: Pull complete 
f29e8ba9d17c: Pull complete 
78af88a8afb0: Pull complete 
b74279c188d9: Pull complete 
6e3e5bf64fd2: Pull complete 
b62a2c2d2ce5: Pull complete 
eba91ca3c7a3: Verifying Checksum 
d4a24cdf2433: Download complete 
b20f8a8dfd5c: Download complete 
e0731dd084c3: Download complete 
0361da6a228e: Download complete 
filesystem layer verification failed for digest sha256:eba91ca3c7a37844775569d1771c8acfab80b32d9c24f4a0b5b998d91911d747

I've tested this command multiple times. This issue occurs on different layers (although some tend to have more occurrences) and once in a while (like one in twenty) everything works. But on other images (i.e. mcr.microsoft.com/mssql/server:2019-CU18-ubuntu-20.04 I wasn't able to successfully finish the pull.

It's now a fresh installation. I've completely removed docker along with its configs and reinstalled it all. /var/lib/docker has been removed, so were /etc/docker/daemon.json and ~/.docker/config.json. I don't use proxy, the internet connection is stable, and this is not a VM - it's a native Ubuntu installation. I've tried docker-desktop and there everything is fine, so it's not a repository issue nor a connection one.

Starting the service doesn't report errors:

sie 05 14:02:03 systemd[1]: Starting Docker Application Container Engine...
░░ Subject: A start job for unit docker.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has begun execution.
░░ 
░░ The job identifier is 3852.
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.041443800+02:00" level=info msg="Starting up"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.042248549+02:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.055564872+02:00" level=info msg="Loading containers: start."
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.383176470+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 192.168.128.0/24. Daemon option --bip can be used to set a preferred IP address"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.461435658+02:00" level=info msg="Loading containers: done."
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.474691599+02:00" level=info msg="Docker daemon" commit=a61e2b4 graphdriver=overlay2 version=24.0.5
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.474754307+02:00" level=info msg="Daemon has completed initialization"
sie 05 14:02:03 dockerd[8422]: time="2023-08-05T14:02:03.491109520+02:00" level=info msg="API listen on /run/docker.sock"
sie 05 14:02:03 systemd[1]: Started Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished successfully.
░░ 
░░ The job identifier is 3852.

I'd appreciate all and any help. I'm running out of ideas.

Answer 1

It seems to be a known and random error. There are some issue open on GitHub about that, such as https://github.com/GoogleContainerTools/kaniko/issues/1962.

The 2 possibile workaround, that not solve the error but only mitigate it, seems to be:

• removing all the images, for example removing the /var/lib/docker folder, and retry (but you said you already did it)
• disabling cache, by setting the --cache=true flag