coredns giving "network is unreachable" errors

Question

I've got a cluster with Flannel as the CNI. As a check to make sure everything was working, I deployed some dnsutils pods, but they can't seem to do any dns resolution:

[root@master-node ~]# kubectl exec -ti dnsutils-deployment-7d49b8f56-4n6tk -- nslookup kubernetes.default
;; connection timed out; no servers could be reached

command terminated with exit code 1

They also can't ping anything

[root@master-node ~]# kubectl exec -ti dnsutils-deployment-7d49b8f56-4n6tk -- ping localhost
ping: icmp open socket: Operation not permitted
command terminated with exit code 2

Checking the logs for coredns, it seems like it's having trouble too:

[root@master-node ~]# kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
[ERROR] plugin/errors: 2 5778069689412134934.833992230232747685. HINFO: dial udp [2602:807:900e:95::2]:53: connect: network is unreachable
[ERROR] plugin/errors: 2 5778069689412134934.833992230232747685. HINFO: dial udp [2602:807:900e:95::2]:53: connect: network is unreachable
[ERROR] plugin/errors: 2 5778069689412134934.833992230232747685. HINFO: read udp 10.244.0.3:40699->192.168.40.60:53: read: no route to host
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.6
linux/amd64, go1.17.1, 13a9191
[ERROR] plugin/errors: 2 2300225452912156849.7915040186128921085. HINFO: dial udp [2602:807:900e:95::2]:53: connect: network is unreachable
[ERROR] plugin/errors: 2 2300225452912156849.7915040186128921085. HINFO: dial udp [2602:807:900e:95::2]:53: connect: network is unreachable

2602:807:900e:95::2 is my master node's dns server (in resolv.conf), but that shouldn't really be a problem for the nslookup because it's doing a local lookup, I think. It seems like maybe none of the pods are being allowed to communicate at all. I can't figure out how to go about fixing this. I tried deleting and redeploying flannel and dnsutils but that didn't help.

Answer 1

Install the Calico to improve the coredns...

kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml 
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml

this will get some features like environment vars that are used by coredns.