K3s docker registry service: push fail with error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found"

Question

I deployed a docker registry inside K3s, but i'm not able to upload images on it.

I have a K3s Cluster running with K3d. On this cluster i want to deploy a docker registry. The Cluster is reachable on localhost via port 8090. To deploy the registry, i created a folder with the name registry with following manifests:

1. deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: registry
  namespace: registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: registry
  template:
    metadata:
      labels:
        app: registry
    spec:
      containers:
      - name: registry
        image: registry:2
        ports:
        - containerPort: 5000
          name: web
          protocol: TCP
        readinessProbe:
          httpGet:
            port: web
            path: /

2. ingressroute.yaml

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry
  namespace: registry
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`registry.test.local`)
    kind: Rule
    services:
    - name: registry
      port: 5000

3. namespace.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: registry

4. service.yaml

apiVersion: v1
kind: Service
metadata:
  name: registry
  namespace: registry
spec:
  type: ClusterIP
  selector:
    app: registry
  ports:
    - protocol: TCP
      port: 5000

5. kustomization.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - namespace.yaml
  - deployment.yaml
  - service.yaml
  - ingressroute.yaml

Then i applied this configuration to my cluster via kubectl apply -k registry

When i use curl to receive the content of the registry i get a valid response:

$ curl http://registry.test.local:8090/v2/_catalog
{"repositories":[]}

When i try to push a image on the registry i get following error message:

$ docker push registry.test.local:8090/hello-world
Using default tag: latest
The push refers to repository [registry.test.local:8090/hello-world]
ac28800ec8bb: Preparing 
error parsing HTTP 404 response body: invalid character 'p' after top-level value: "404 page not found\n"

When i shutdown the K3S Test Cluster and start the docker registry via docker run it is also working:

# Start registry with:
docker run -p 8090:5000 registry:2
 
# Push Image with
docker push registry.test.local:8090/hello-world
Using default tag: latest
The push refers to repository [registry.test.local:8090/hello-world]
ac28800ec8bb: Layer already exists 
latest: digest: sha256:d37ada95d47ad12224c205a938129df7a3e52345828b4fa27b03a98825d1e2e7 size: 524

What is wrong in the configuration above and needs to be modified to behave as expected?

Thanks a lot, Rob

Answer 1

Quick fix

You need TLS, to your ingress add new TLS route

apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry-https
  namespace: registry
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`registry.test.local`)
    kind: Rule
    services:
    - name: registry
      port: 5000
  tls:
    domains:
    - main: registry.test.local

Then if you want test it with curl use https and -k to ignore Certificate verification

curl -k https://registry.test.local:8090/v2/_catalog

---

More details

Docker is not allowing pushing images without any encryption, this is why when you try to push it's failing as your ingress can handle only HTTP. To make it work you need to make your port 8090 listen for HTTPS connection by adding tls section, but it makes you unable to use HTTP at all.

When there is not tls and you call

curl http://registry.test.local:8090/v2/_catalog

You will get {"repositories":[]} (as you wrote), but when you do

curl -k https://registry.test.local:8090/v2/_catalog

You will get 404 page not found exactly what docker push command gave you in console

You may ask Why registry work for both cases?, registry just kind of in both modes a the same time, because when it's directly exposed to the internet it will get HTTPS connection from Docker Client, but when it's behind Reverse Proxy, it's common to forward requests as plain HTTP

---

To make it works with HTTP and HTTPS at the same time you need two routes

---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry-http
  namespace: registry
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`registry.test.local`)
    kind: Rule
    services:
    - name: registry
      port: 5000
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: registry-https
  namespace: registry
spec:
  entryPoints:
    - web
  routes:
  - match: Host(`registry.test.local`)
    kind: Rule
    services:
    - name: registry
      port: 5000
  tls:
    domains:
    - main: registry.test.local

And that's it!