Does kubernetes use conntrack twice when a pod talks to a pod through a service where the destination pod is on the same host?

Question

From what I understand, when a POD talks to a Service the IP tables have been updated by a CNI provider (this may be specific to some but not all CNI providers). The iptables have basically provided a virtual IP that then round robins or distributes (somehow) to backend ephemeral pods. Those pods may live on the same host or another host in the cluster. At this point (again based on the CNI) conntrack is used to keep src and dst straight as it remaps the svc-ip to the dest-ip of the POD. What I'm wondering though, is if the dest pod is on the same host, I'm not certain how it is routed on the return path. I would suspect via the service still and then possibly using conntrack for the return path.

Does kubernetes use conntrack twice when a pod talks to a pod through a service where the destination pod is on the same host?

Answer 1

I will use Calico as an example while trying to explain this topic.

Conntrack is only required from the source pod to the service endpoints. You have to track it to send rest of flow packets to the same destination endpoint. The return path always has only one option: from the destination pod to the source pod which means that even if conntrack is used here it doesn't change anything as the return path is managed by the NAT table.

Also worth mentioning:

For both iptables and IPVS mode, the response time overhead for kube-proxy is associated with establishing connections, not the number of packets or requests you send on those connections. This is because Linux uses connection tracking (conntrack) that is able to match packets against existing connections very efficiently. If a packet is matched in conntrack then it doesn’t need to go through kube-proxy’s iptables or IPVS rules to work out what to do with it.

You can use conntrack command line interface in order search, list, inspect and maintain the connection tracking subsystem of the Linux kernel.

For example:

conntrack -L will show the connection tracking table in /proc/net/ip_conntrack format:

 # conntrack -L
 tcp      6 431982 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34846 dport=993 packets=169 bytes=14322 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34846 packets=113 bytes=34787 [ASSURED] mark=0 secmark=0 use=1
 tcp      6 431698 ESTABLISHED src=192.168.2.100 dst=123.59.27.117 sport=34849 dport=993 packets=244 bytes=18723 src=123.59.27.117 dst=192.168.2.100 sport=993 dport=34849 packets=203 bytes=144731 [ASSURED] mark=0 secmark=0 use=1
conntrack v0.9.7 (conntrack-tools): 2 flow entries have been shown.

A practical example would be when you change Calico's policy to disallow a flow that was previously allowed. Calico only needs to check its policies for the first packet in an allowed flow—between a pair of IP addresses and ports—and then conntrack automatically allows further packets in the same flow, without Calico rechecking every packet. If packets were recently exchanged on the previously allowed flow, and so there is conntrack state for that flow that has not yet expired, that conntrack state will allow further packets between the same IP addresses and ports, even after the Calico policy has been changed. To avoid that you could delete the relevant conntrack states manually with conntrack -D and than use conntrack -E in order to observe the connection events with a new Calico policy.

Sources:

• Linux Conntrack

• Connection tracking

• Kube-proxy modes

• Packet flow

I hope this helps.