Can't resolve dns in kubernetes

Question

I use next command to check dns issue in my k8s:

kubectl apply -f https://k8s.io/examples/admin/dns/dnsutils.yaml
kubectl exec -i -t dnsutils -- nslookup kubernetes.default

The nslookup result is:

;; connection timed out; no servers could be reached

command terminated with exit code 1

dnsutils.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: dnsutils
  namespace: default
spec:
  containers:
  - name: dnsutils
    image: gcr.io/kubernetes-e2e-test-images/dnsutils:1.3
    command:
      - sleep
      - "3600"
    imagePullPolicy: IfNotPresent
  restartPolicy: Always

NOTE: it's a machine which default disable all ports, so I ask our IT admin already open the port based on next doc check-required-ports, I'm not sure if this matters.

And use next I could get the pod ip of coredns.

kubectl get pods -n kube-system -o wide | grep core
coredns-7877db9d45-swb6c                                 1/1     Running   0          2m58s   10.244.1.8       node2   <none>           <none>
coredns-7877db9d45-zwc8v                                 1/1     Running   0          2m57s   10.244.0.6       node1   <none>           <none>

Here, 10.244.0.6 is my master while 10.244.1.8 is my working node.

Then if I directly specify coredns pod ip:

master node ok:

kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.244.0.6
Server:         10.244.0.6
Address:        10.244.0.6#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

work node not ok:

# kubectl exec -i -t dnsutils -- nslookup kubernetes.default 10.244.1.8
;; connection timed out; no servers could be reached

command terminated with exit code 1

So, the question narrow down to why COREDNS on work node not works? Anything I need to pay attention?

Environment:

• OS: ubuntu18.04
• K8S: v1.21.0
• Cluster boot command:

kubeadm init --pod-network-cidr=10.244.0.0/16
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

Answer 1

Finally, I find the root cause, this is hardware firewall issue, see this:

Firewalls
When using udp backend, flannel uses UDP port 8285 for sending encapsulated packets.
When using vxlan backend, kernel uses UDP port 8472 for sending encapsulated packets.
Make sure that your firewall rules allow this traffic for all hosts participating in the overlay network.
Make sure that your firewall rules allow traffic from pod network cidr visit your kubernetes master node.

• When nslookup client on the same node of dns server, it won't trigger firewall block, so everything is ok.
• When nslookup client not on the same node of dns server, it will trigger firewall block, so we can't access dns server.

So, after open the ports, everything ok now.

Answer 2

In my case I was using Amazon's Kubernetes (EKS) and this error was due to a bad security group. Using the default launch template for the AWS managed nodegroup fixed the issue.