I have cross site replication enabled using percona mongodb operator, mongo version 6.0.4-3 we are using oracle load balancer to expose the pods and we noticed a drop happens suddenly in the connection and when we check the logs we found the below:
{"t":{"$date":"2023-09-12T19:15:01.043+00:00"},"s":"I", "c":"CONNPOOL", "id":22572, "ctx":"ShardRegistry","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.11.10.18:27017","error":"ConnectionPoolExpired: Pool for 10.11.10.18:27017 has expired."}}
{"t":{"$date":"2023-09-12T19:19:31.387+00:00"},"s":"I", "c":"CONNPOOL", "id":22572, "ctx":"ShardRegistry","msg":"Dropping all pooled connections","attr":{"hostAndPort":"10.11.10.1:27017","error":"ConnectionPoolExpired: Pool for 10.11.10.1:27017 has expired."}}
please note the IPs mentioned in the logs is for the DR site.
here the graph from the LB: LB graph
I noticed the drop happened every 2 hours also number of connections varies from 3K to 5K
any ideas?
I tried to find and understand why mongo drops all the connections but still no luck