I have a Ubuntu (22.04.3) VM I'll refer to as machine1 that is running Docker (community version 25.0.0). On machine1 I have a container that contains a Golang application. That application needs to be able to reach to another VM (machine2) on the same subnet (over HTTP port 8080). I have been trying to enable nftables on machine1 but have run into a specific problem that I have not been able to fix yet.
Once I enable nftables, my Golang application in my docker container on machine1 can no longer reach to "http://machine2:8080/app". But, it does work if I use FQDN in URL like: "http://machine2.full.domain:8080/app". The same thing also works if I use machine2's IP in URL like: "http://10.1.XX.XXX:8080/app".
The Golang app that is running inside a docker container on machine1 is testing connectivity to machine2 using http package like:
http.NewRequest("http.MethodGet", "http://machine2:8080/app", nil)
The container on machine1 is connected to a docker network.
Here are my nftables rules for machine1 (from /etc/nftables.conf):
table inet filter {
chain input {
#block all inbound traffic
type filter hook input priority filter; policy accept;
ip saddr 127.0.0.0/8 counter packets 53160 bytes 3216598 drop
ip6 saddr ::1 counter packets 729 bytes 58320 drop
#allow subnet inbound 80, 443, 8080
ip saddr SUBNET_CONTAINING_MACHINE1_AND_MACHINE2 tcp sport { 80, 443, 8080 } accept
#allow my workstation inbound SSH
ip saddr WORKSTATION_IP tcp dport { 22 } accept
ip daddr MACHINE1_IP tcp sport { 53, 88, 389, 445, 636, 3268, 3269 } tcp dport 1-65535 accept
ip daddr MACHINE1_IP udp sport { 53, 88, 389, 445, 636, 3268, 3269 } udp dport 1-65535 accept
ip daddr MACHINE1_IP tcp sport 443 tcp dport 1-65535 accept
log prefix "Dropped: " level debug
}
chain forward {
type filter hook forward priority filter; policy accept;
}
chain output {
type filter hook output priority filter; policy accept;
}
}
With nftables running on machine1 with the above rules, I see the following error in /var/log/syslog when my app attempts to connect to machine2 (using "http://machine2:8080/app"):
Jan 21 21:46:23 machine1 dockerd[2513158]: time="2024-01-21T21:46:23.264171605-05:00" level=error msg="[resolver] failed to query DNS server: 127.0.0.53:53, query: ;machine2.\tIN\t A" error="read udp 127.0.0.1:59459->127.0.0.53:53: i/o timeout"
When I stop nftables on machine1 the same test works fine. Additionally, with nftables running on machine1 it works as long as I use "http://machine2.full.domain:8080/app" or "http://10.1.XX.XXX:8080/app".
In my nftables rules on machine1 I have this line that should log all blocked traffic:
log prefix "Dropped: " level debug
I don't see any dropped traffic log entries when I attempt the connectivity test from my app to machine2.
Using GPT's help, I've tried adding various nftables rules on machine1 related to allowing DNS traffic. None of those have helped so I've backed them all out.