DEVHIDE
Login Or Sign up

VPC Peering Connection stopped working after installing Docker

61 Views Asked by At

I have two VPCs connected through Peering Connection. Each VPC has one EC2 instance. The Peering is working as I can ping from one EC2 instance to another using their private IP.

Now I need to have Docker installed in one of the EC2. After installing Docker, I cannot ping anymore to the other EC2 instance. The VPC Reachability Analyzer shows the connection is reachable. After analyzing I have found that Docker has added following rules in the iptables:

$ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (4 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             <internal-ip>.<region>.compute.internal  tcp dpt:commplex-main
ACCEPT     tcp  --  anywhere             <internal-ip>.<region>.compute.internal  tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (4 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

From the Packet filtering and firewalls documentation which is normal.

So I tried to add the following rules to enable access from anywhere:

$ sudo iptables -I INPUT -j ACCEPT
$ sudo iptables -I OUTPUT -j ACCEPT

Followed by:

$ sudo iptables-save

Despite these rules being added the ping didn't work. Then I removed the Docker and iptables and rebooted the instance. I got the ping working again.

Then reinstall the Docker again. But before enabling the docker.service I added:

{
  "iptables": "false"
}

in the /etc/docker/daemon.json by following the instruction given here to prevent Docker from manipulating iptables.

But still no luck. Any advice or pointers would be much appreciated.

Thanks.

PS: I am using Amazon Linux 2023

$ cat /etc/os-release
NAME="Amazon Linux"
VERSION="2023"
ID="amzn"
ID_LIKE="fedora"
VERSION_ID="2023"
PLATFORM_ID="platform:al2023"
PRETTY_NAME="Amazon Linux 2023"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2023"
HOME_URL="https://aws.amazon.com/linux/"
BUG_REPORT_URL="https://github.com/amazonlinux/amazon-linux-2023"
SUPPORT_END="2028-03-01"

Update

The output of docker network ls:

ETWORK ID     NAME                                             DRIVER    SCOPE
5823629b2038   bridge                                           bridge    local
83fd27326747   docker-registry-server_docker-registry-network   bridge    local
efa259e6d263   docker-registry_docker-registry-network          bridge    local
8582e4f3a279   host                                             host      local
2924559cf5fd   ldap_default                                     bridge    local
52352b7d663b   none                                             null      local

And the output of ifconfig:

br-2924559cf5fd: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.24.0.1  netmask 255.255.0.0  broadcast 172.24.255.255
        ether 02:42:db:c0:08:51  txqueuelen 0  (Ethernet)
        RX packets 1537  bytes 352781 (344.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1450  bytes 249327 (243.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-83fd27326747: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.20.0.1  netmask 255.255.0.0  broadcast 172.20.255.255
        ether 02:42:81:61:bc:25  txqueuelen 0  (Ethernet)
        RX packets 1537  bytes 352781 (344.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1450  bytes 249327 (243.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br-efa259e6d263: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.23.0.1  netmask 255.255.0.0  broadcast 172.23.255.255
        inet6 fe80::42:58ff:fec1:7e37  prefixlen 64  scopeid 0x20<link>
        ether 02:42:58:c1:7e:37  txqueuelen 0  (Ethernet)
        RX packets 1537  bytes 352781 (344.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1450  bytes 249327 (243.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        ether 02:42:ee:ef:e3:6d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens5: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9001
        inet 192.168.36.110  netmask 255.255.128.0  broadcast 192.168.127.255
        inet6 fe80::2c:d5ff:fea2:64b8  prefixlen 64  scopeid 0x20<link>
        ether 02:2c:d5:a2:64:b8  txqueuelen 1000  (Ethernet)
        RX packets 1537  bytes 352781 (344.5 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1450  bytes 249327 (243.4 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 8893  bytes 4936039 (4.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 8893  bytes 4936039 (4.7 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth572770b: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::649c:6cff:fe7a:e2d0  prefixlen 64  scopeid 0x20<link>
        ether 66:9c:6c:7a:e2:d0  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 13  bytes 1046 (1.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth85112bc: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::d05a:4eff:fe46:2da0  prefixlen 64  scopeid 0x20<link>
        ether d2:5a:4e:46:2d:a0  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 14  bytes 1156 (1.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
docker amazon-vpc iptables aws-vpc-peering
Original Q&A
1

There are 1 best solutions below

3
Tapas Bose On

After having a conversation with RichardRublev and erik258, I realized that the IP address of the EC2 I am trying to connect has some kind of conflict with the networking configuration of the EC2 where the Docker is installed.

As I am using kOps to create a cluster, I have the option to use another CIDR block for the second VPC (which has EC2 without Docker). So using a different CIDR block such as 182.20.0.0/16 has resolved the problem.

I thank both of you RichardRublev and erik258 for helping me out.

Related Questions in DOCKER

Related Questions in AMAZON-VPC

Related Questions in IPTABLES

Related Questions in AWS-VPC-PEERING

Trending Questions

Popular # Hahtags

javascript python java c# php android html jquery c++ css ios sql mysql r reactjs node.js arrays c asp.net json

Popular Questions

.

Copyright © 2021 Jogjafile Inc.