Kubernetes does not recognize modification of service

Question

How can I force Kubernetes to "recognize" the modification of a service?

I eventually change the type/spec of an service from

kind: Service
apiVersion: v1
metadata:
  name: service-a
  namespace: namespace-a
spec:
  type: ClusterIP
  selector:
    app: service-a-pods
  ports:
    - name: web
      port: 80

to

kind: Service
apiVersion: v1
metadata:
  name: service-a
  namespace: namespace-a
spec:
  type: ExternalName
  externalName: service-b.namespace-b.svc.cluster.local
  ports:
    - name: web
      port: 80

This works fine, but when I revert the change in Kubernetes, the service-a still keeps forwarding to the service-b in namespace-b. Even hard deleting and recreating service-a does not help.

It seems like the CoreDNS is not updated or something like this. The only work-around I found, is to recreate service-b in namespace-b. Is this the expected behavior?

Answer 1

It is due to the cache of DNS records maintained by coredns in your k8s cluster. You can reduce the following ttl value to zero if you need the immediate effect of your service resource changes:

Run following to edit the Corefile (coredns configuration file):

kubectl edit cm coredns -n kube-system

then update the TTL under kubernetes plugin:

apiVersion: v1
data:
  Corefile: |
    .:53 {
        errors
        health {
           lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
           pods insecure
           fallthrough in-addr.arpa ip6.arpa
           ttl 0 # <-- set this ttl value to zero
        }
        prometheus :9153
        forward . /etc/resolv.conf {
           max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
  ...

Note: This might have a performance impact on DNS resolutions in your cluster. You can reduce it to an appropriate value as well.