JGRP000006: failed accepting connection from peer Socket

Question

I am running ActiveMQ Artemis 2.29.0 on AKS. There are 2 pods running on cluster mode. I have configured cluster using JGroup KUBE_PING. The clusters are formed, and working as expected. But I am getting some error at random time of day.

java.lang.IllegalStateException: BaseServer.NioConnection.readPeerAddress(): cookie read by 10.1.3.51:7900 does not match own cookie; terminating connection
    at org.jgroups.blocks.cs.NioConnection.readPeerAddress(NioConnection.java:387) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection._receive(NioConnection.java:242) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection$Reader._run(NioConnection.java:495) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection$Reader.run(NioConnection.java:483) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
2023-07-19 14:52:55,852 WARN  [org.jgroups.blocks.cs.NioServer] failed handling message
java.lang.IllegalStateException: BaseServer.NioConnection.readPeerAddress(): cookie read by 10.1.3.51:7900 does not match own cookie; terminating connection
    at org.jgroups.blocks.cs.NioConnection.readPeerAddress(NioConnection.java:387) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection._receive(NioConnection.java:242) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection$Reader._run(NioConnection.java:495) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.NioConnection$Reader.run(NioConnection.java:483) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
2023-07-19 14:52:55,853 WARN  [org.jgroups.protocols.TCP] JGRP000006: failed accepting connection from peer Socket[addr=/10.1.2.237,port=57314,localport=7800]: java.net.SocketException: 10.1.3.51:7800: BaseServer.TcpConnection.readPeerAddress(): cookie sent by /10.1.2.237:57314 does not match own cookie; terminating connection
java.net.SocketException: 10.1.3.51:7800: BaseServer.TcpConnection.readPeerAddress(): cookie sent by /10.1.2.237:57314 does not match own cookie; terminating connection
    at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:254) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:55) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:115) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:99) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]
2023-07-19 14:52:55,867 WARN  [org.jgroups.protocols.TCP] JGRP000006: failed accepting connection from peer Socket[addr=/10.1.2.237,port=57328,localport=7800]: java.net.SocketException: 10.1.3.51:7800: BaseServer.TcpConnection.readPeerAddress(): cookie sent by /10.1.2.237:57328 does not match own cookie; terminating connection
java.net.SocketException: 10.1.3.51:7800: BaseServer.TcpConnection.readPeerAddress(): cookie sent by /10.1.2.237:57328 does not match own cookie; terminating connection
    at org.jgroups.blocks.cs.TcpConnection.readPeerAddress(TcpConnection.java:254) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpConnection.<init>(TcpConnection.java:55) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpServer$Acceptor.handleAccept(TcpServer.java:115) ~[jgroups-5.2.0.Final.jar:5.2.0.Final]
    at org.jgroups.blocks.cs.TcpServer$Acceptor.run(TcpServer.java:99) [jgroups-5.2.0.Final.jar:5.2.0.Final]
    at java.lang.Thread.run(Thread.java:833) [?:?]

Jgroup.xml

<config xmlns="urn:org:jgroups"
 xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
 <TCP
 external_addr="${JGROUPS_EXTERNAL_ADDR:match-interface:eth0}"
 bind_addr="site_local,match-interface:eth0"
 bind_port="${TCP_PORT:7800}"
 recv_buf_size="5M"
 send_buf_size="1M"
 thread_naming_pattern="cl"
 thread_pool.min_threads="2"
 thread_pool.max_threads="500"
 thread_pool.keep_alive_time="30000" />
 <RED/>

 <org.jgroups.protocols.kubernetes.KUBE_PING namespace="${KUBERNETES_NAMESPACE:activemq}"
 labels="app=activemq-active-active" masterHost="kubernetes.default.svc" masterPort="443" masterProtocol="https"
 caCertFile="/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"
 saTokenFile="/var/run/secrets/kubernetes.io/serviceaccount/token"
 />

 <MERGE3 max_interval="30000"

 min_interval="10000"/>
 <FD_SOCK external_addr="${JGROUPS_EXTERNAL_ADDR}"
 start_port="${FD_SOCK_PORT:9000}"/>
 <FD_ALL timeout="30000" interval="5000"/>
 <VERIFY_SUSPECT timeout="1500" />
 <BARRIER />
 <pbcast.NAKACK2 xmit_interval="500"
 xmit_table_num_rows="100"
 xmit_table_msgs_per_row="2000"
 xmit_table_max_compaction_time="30000"
 use_mcast_xmit="false"
 discard_delivered_msgs="true" />
 <UNICAST3
 xmit_table_num_rows="100"
 xmit_table_msgs_per_row="1000"
 xmit_table_max_compaction_time="30000"/>
 <pbcast.STABLE desired_avg_gossip="50000"
 max_bytes="8m"/>
 <pbcast.GMS print_local_addr="true" join_timeout="3000"/>
 <MFC max_credits="2M"
 min_threshold="0.4"/>
 <FRAG2 frag_size="60K" />
 <pbcast.STATE_TRANSFER />
 <CENTRAL_LOCK />
 <COUNTER/>
</config>

I tried searching for it, but could not understand. This same error is also coming when using FILE_PING with shared storage.

Can someone help me with this? Any suggestions would be appreciated.