Lambda in one security group cannot talk to RDS in another security group in a peered VPC

Question

Here's my architecture:

1. My Lambda in Lambda_VPC with SecurityGroup (Lambda_SG);
2. My RDS in RDS_VPC with SecurityGroup (RDS_SG);
3. I've created a peering connection with Lambda_VPC and RDS_VPC called peering_cxn;
4. I've modified the route tables for both Lambda_VPC and RDS_VPC to include this peering_cxn following this link;
5. I've added Lambda_SG as one of the inbound rules in RDS_SG on port 8192 with custom TCP as traffic type
6. DNS resolution enabled on both VPCs;

But somehow my lambda still cannot call RDS, it keeps timing out.

A couple things to note:

1. The endpoint and credentials that my Lambda uses to connect with RDS is correct as I was able to confirm the connection after briefly adding 0.0.0.0/0 as one of the inbound rules for RDS_SG. Of course I don't want to resort to 0.0.0.0/0 as a resolution as it opens our system to the whole internet and protected only by the DB credentials;
2. Also, I'm able to connect to this RDS instance via exactly the same credentials through a SQLClient directly from my Macbook.

Based on my understanding, with this peering VPC in the same account in the same region, Lambda_SG added as an inbound rule for RDS_SG, it should be working, but it's not yet.

Any suggestions would be greatly appreciated!

UPDATE:

1. I launched an EC2 instance in Lambda_VPC with Lambda_SG, connected to this EC2 instance and tried to connect to RDS:

sh-5.2$ telnet cluster-abc.cluster-abc987-dbd.us-west-2.rds.amazonaws.com 8192
Trying 12.345.398.11...
telnet: connect to address 12.345.398.11: Connection timed out

2. I've gone into DNS settings of the peering connection and checked both boxes to:

1. Allow accepter VPC (Lambda_VPC) to resolve DNS of requester VPC (RDS_VPC) hosts to private IP.
2. Allow requester VPC (RDS_VPC) to resolve DNS of accepter VPC (Lambda_VPC) hosts to private IP.

UPDATE 2: 12.345.398.11 is a public IP address, indicating it didn't try to resolve to a private address as DNS propogation takes time. So after one night, I connected with the ec2 instance and tried to connect RDS again:

sh-5.2$ telnet cluster-abc.cluster-abc987-dbd.us-west-2.rds.amazonaws.com 8192
Trying 10.0.41.180...
telnet: connect to address 10.0.41.180: Connection timed out

This time, this IP: 10.0.41.180 is within the CIDR range of the RDS_VPC of: 10.0.0.0/16. But still timed out.

Answer 1

The fact that the connection works when RDS_SG allows incoming traffic from 0.0.0.0/0 but not from Lambda_SG suggests that the connection is being made to the Amazon RDS instance's private IP address. If so, then the Security Group will not recognise the incoming connection as coming from Lambda_SG because the traffic has gone out of Lambda_VPC via the Internet rather than the VPC Peering Connection.

To confirm this, you can attempt to resolve the RDS instance's DNS name within Lambda_VPC.

To resolve it, you will need to enable private DNS resolution.

From Enable DNS resolution for a VPC peering connection - Amazon Virtual Private Cloud:

To enable a VPC to resolve public IPv4 DNS hostnames to private IPv4 addresses when queried from instances in the peer VPC, you must modify your existing peering connection.

Both VPCs must be enabled for DNS hostnames and DNS resolution.

...

To ensure that queries from the peer VPC resolve to private IP addresses in your local VPC, choose the option to enable DNS resolution for queries from the peer VPC. This option is Requester DNS resolution or Accepter DNS resolution, depending on whether the VPC is the requester or accepter VPC.

Once this is activated, your existing configuration should work correctly.