Basic Auth doesn't work in kubernetes ingress

Question

I have created pypiserver in kubernetes cluster, I have used https://hub.docker.com/r/pypiserver/pypiserver docker image. I need to create basic auth for the server which I created. I used this method https://kubernetes.github.io/ingress-nginx/examples/auth/basic/

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: pypiserver
  labels:
    app: pypiserver
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: 'true'
    ingress.kubernetes.io/auth-type: basic
    ingress.kubernetes.io/auth-secret: secret
    ingress.kubernetes.io/auth-realm: "Authentication Required - ok"
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /
        backend:
          serviceName: pypiservice
          servicePort: 8080
  tls:
  - hosts:
    - example.com
    secretName: secret-tls

But my host name would be "www.example.com/8080" and I don't see ingress has any pod in kubernetes cluster. Ingress is running fine but I don't get auth for this host. (And also I have http://IP adress:8080 which I converted to domain through cloudflare)

Please let me know what am I doing wrong?

Answer 1

I don't know exactly what is your nginx ingress controller version, but I can share what worked for me. I've reproduced it on my GKE cluster.

I installed my nginx ingress controller following this guide. Basically it came down to running the following commands:

If you're using GKE you need to initialize your user as a cluster-admin with the following command:

kubectl create clusterrolebinding cluster-admin-binding \
  --clusterrole cluster-admin \
  --user $(gcloud config get-value account)

The following Mandatory Command is required for all deployments.

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.1/deploy/static/mandatory.yaml

I'm using 1.13 version on my GKE so this tip is also applied in my case:

Tip

If you are using a Kubernetes version previous to 1.14, you need to change kubernetes.io/os to beta.kubernetes.io/os at line 217 of mandatory.yaml, see Labels details.

But I dealt with it quite differently. Basically you need your Nodes to have kubernetes.io/os=linux label so you can simply label them. Following command will do the job:

kubectl label node --all kubernetes.io/os=linux

Then we're heading to Provider Specific Steps which in case of GKE came down to applying the following yaml:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.27.1/deploy/static/provider/cloud-generic.yaml

Then you may want to verify your installation:

To check if the ingress controller pods have started, run the following command:

kubectl get pods --all-namespaces -l app.kubernetes.io/name=ingress-nginx --watch

or simply run:

kubectl get all -n ingress-nginx

It will also tell you if all the required resorces are properly deployed.

Next we need to write our ingress (ingress object/resource) containing basic-auth related annotations. I was following same tutorial as mentioned in your question.

First we need to create our auth file containing username and hashed password:

$ htpasswd -c auth foo
New password: <bar>
New password:
Re-type new password:
Adding password for user foo

Once we have it, we need to create a Secret object which then we'll use in our ingress:

$ kubectl create secret generic basic-auth --from-file=auth
secret "basic-auth" created

Once it is created we can check if everything went well:

$ kubectl get secret basic-auth -o yaml
apiVersion: v1
data:
  auth: Zm9vOiRhcHIxJE9GRzNYeWJwJGNrTDBGSERBa29YWUlsSDkuY3lzVDAK
kind: Secret
metadata:
  name: basic-auth
  namespace: default
type: Opaque

Alright, so far so good...

Then we need to create our ingress resource/object.

My ingress-with-auth.yaml file looks slightly different than the one in the instruction, namely I just added kubernetes.io/ingress.class: nginx to make sure my nginx ingress controller is used rather than built-in GKE solution:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-with-auth
  annotations:
    kubernetes.io/ingress.class: nginx
    # type of authentication
    nginx.ingress.kubernetes.io/auth-type: basic
    # name of the secret that contains the user/password definitions
    nginx.ingress.kubernetes.io/auth-secret: basic-auth
    # message to display with an appropriate context why the authentication is required
    nginx.ingress.kubernetes.io/auth-realm: 'Authentication Required - foo'
spec:
  rules:
  - host: foo.bar.com
    http:
      paths:
      - path: /
        backend:
          serviceName: pypiserver
          servicePort: 80

In your example you may need to add nginx prefix in your basic-auth related annotations:

ingress.kubernetes.io/auth-type: basic
ingress.kubernetes.io/auth-secret: secret
ingress.kubernetes.io/auth-realm: "Authentication Required - ok"

so it looks like this:

nginx.ingress.kubernetes.io/auth-type: basic
nginx.ingress.kubernetes.io/auth-secret: secret
nginx.ingress.kubernetes.io/auth-realm: "Authentication Required - ok"

First I used the address listed in my ingress resource (it doesn't appear there any more once I added kubernetes.io/ingress.class: nginx annotation in my ingress definition:

$ kubectl get ingress 
NAME                HOSTS         ADDRESS   PORTS   AGE
ingress-with-auth   foo.bar.com             80      117m

When I tried to access pypi-server using this IP it brought me directly to the page without a need of any authentication. But it looks like if you didn't define proper ingress class, the default is used instead so in practice your ingress definition with auth-basic details isn't taken into consideration and isn't passed to the nginx ingress controller we installed in one of the previous steps.

So what IP address should be used to access your app ? Run the following command which will show you both CLUSTER-IP (can be accessed within your cluster from any Pod or Node) and EXTERNAL-IP of your nginx ingress controller:

$ kubectl get service --namespace ingress-nginx 
NAME            TYPE           CLUSTER-IP   EXTERNAL-IP     PORT(S)                      AGE
ingress-nginx   LoadBalancer   10.0.3.220   35.111.112.113   80:30452/TCP,443:30006/TCP   18h

You can basically host many different websites in your cluster and all of them will be available through this IP. All of them can be available on default http 80 port (or https 443 in your case). The only difference between them will be the hostname that you pass in http header of your http request.

Since I don't have a domain pointing to this external IP address and can't simply access my website by going to http://foo.bar.com I need to pass somehow the hostname I'm requesting from 35.111.112.113 address. It can be done in a few ways:

I installed in my Google Chrome browser ModHeader extension which allows me to modify my http request headers and set the hostname I'm requestig to any value I want.

You can do it also using curl as follows:

curl -v http://35.111.112.113 -H 'Host: foo.bar.com' -u 'foo:bar'

You should be prompted for authentication.

If you don't provide -u username:password flag you should get 401 Authorization Required.

Basically hat's all.

Let me know if it helped you. Don't hesitate to ask additional questions if something isn't completely clear.

One more thing. If something still doesn't work you may start from attaching to your nginx ingress controller Pod (check your Pod name first by running kubectl get pods -n ingress-nginx):

kubectl exec -ti -n ingress-nginx nginx-ingress-controller-pod /bin/bash

and checking the content of your /etc/nginx/nginx.conf file. Look for foo.bar.com (or in your case example.com). It should contain similar lines:

auth_basic "Authentication Required - foo";
            auth_basic_user_file /etc/ingress-controller/auth/default-ingress-with-auth.passwd;

Then check if the file is present in the indicated location /etc/ingress-controller/auth/default-ingress-with-auth.passwd.

One note to your Service definition. The fact that pypiserver container exposes specifically port 8080 doesn't mean that you need to use this port when accessing it via ingress. In Service definition the port exposed by the Container is called targetPort. You need to specify it when defining your Service but Service itself can expose completely different port. I defined my Service using following command:

kubectl expose deployment pypiserver --type=LoadBalancer --port=80 --target-port=8080

Note that the type should be set to NodePort or LoadBalancer. Then in your ingress definition you don't have to use 8080 but 80 which is the port exposed by your pypiserver Service. Note that there is servicePort: 80 in my ingress object/resource definition. Your example.com domain in cloudflare should point with it's A record to your nginx ingress controller LoadBalancer Service IP (kubectl get svc -n ingress-nginx) without specifying any ports.