wso2mi 4.2.0 : Continous Socket connection errors in logs when pod count >1

Question

wso2mi 4.2.0 : Continous Socket connection errors in logs when pod count >1

54 Views Asked by SriniK At 29 February 2024 at 05:15

Environment: wso2mi 4.2.0, JDK 17, Rocky 8.9(RHEL)

We have deployed the wso2mi with a wrapper in k8s. Single pod instance we do not see any errors. But when the replica/pod count is increased to 2 we see below errors in logs. Below errors are printing twice for every 5 seconds.

INFO Logs:

[2024-02-07 10:23:27,878] ERROR {SourceHandler} I/O error: Connection reset java.net.SocketException: Connection reset
at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
at java.base/java.lang.Thread.run(Thread.java:840)

DEBUG LOGS:

[2024-02-21 11:33:41,826] DEBUG {JsonReadOnlyStream} - <init>
[2024-02-21 11:33:41,832] DEBUG {JsonReaderDelegate} - #JsonReaderDelegate. Setting XMLStreamReader: org.apache.synapse.commons.staxon.core.json.JsonXMLStreamReader
[2024-02-21 11:33:41,832] DEBUG {JsonReaderDelegate} - #getLocalName. old=jsonObject, new=jsonObject
[2024-02-21 11:33:41,834] DEBUG {OMSourcedElementImpl} - forceExpand: changing prefix from  to 
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getLocalName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggerName, new=loggerName
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getLocalName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=loggingLevel, new=loggingLevel
[2024-02-21 11:33:41,834] DEBUG {JsonReaderDelegate} - #getName. old=jsonObject, new=jsonObject
[2024-02-21 11:33:41,834] DEBUG {AddressingHelper} - [MessageContext: logID=634fdab486ade390d77c61f5a397f27a11a51b39fbe16bfe] isReplyRedirected: ReplyTo is null. Returning false
[2024-02-21 11:33:41,834] DEBUG {MessageProcessorSelector} - Message format is: application/json; message formatter returned by AxisConfiguration: org.wso2.micro.integrator.core.json.JsonStreamFormatter@4c7b692a
[2024-02-21 11:33:41,835] DEBUG {BuilderUtil} - Input contentType (application/json)
[2024-02-21 11:33:41,835] DEBUG {BuilderUtil} - CharSetEncoding defaulted (UTF-8)
[2024-02-21 11:33:41,835] DEBUG {JsonStreamFormatter} - #writeTo. Wrote JSON payload to output stream. MessageID: urn:uuid:6f4c8123-2615-40bf-b526-29431a3987e5
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Produce output
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: HTTP/1.1 200 OK
[2024-02-21 11:33:41,835] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Consume input
[2024-02-21 11:33:41,836] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Close connection
[2024-02-21 11:33:41,836] DEBUG {SourceHandler} - http-incoming-33080: Keep-Alive connection was closed:  Remote Address : /127.0.0.1:48336
[2024-02-21 11:33:41,836] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-33080
[2024-02-21 11:33:41,836] DEBUG {LoggingNHttpServerConnection} - http-incoming-33080: Shutdown connection
[2024-02-21 11:33:42,942] DEBUG {SSLIOSession} - I/O session http-incoming-33081-1 10.255.18.41:8253<->10.255.12.1:49604[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set timeout 180000
[2024-02-21 11:33:42,942] DEBUG {SSLIOSession} - I/O session http-incoming-33081-1 10.255.18.41:8253<->10.255.12.1:49604[ACTIVE][r:][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Set attribute CONNECTION_INFORMATION
[2024-02-21 11:33:42,943] ERROR {SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:840)

what could be the problem?

Tried adding role binding at cluster to service. But still this problem exists. Not seen this problem with wso2mi 4.1.0, RockyLinu 8.8(RHEL 8.8). Tried wso2mi 4.20 with RockyLinux 8.8(RHEL 8.8), no luck.

Original Q&A

There are 1 best solutions below

**Kolakaluri Sukeerth** · Answer 1 · 2024-03-14T07:49:50.910000

@arunan

In deployment.toml of wso2mi, I have added these configurations to disable the keepalive. However, I am still facing the same issue.

[transport.http]

socket_timeout = 180000 # timeout in milliseconds

disable_connection_keepalive = true

connection_timeout = 90000 # in milliseconds

Logs:

[2024-03-14 07:44:11,406] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-133

[2024-03-14 07:44:11,406] DEBUG {LoggingNHttpServerConnection} - http-incoming-133: Shutdown connection

[2024-03-14 07:44:11,406] DEBUG {SSLIOSession} - I/O session http-incoming-133-85 10.255.8.115:8253<->10.255.18.1:54714[ACTIVE][r:r][ACTIVE][r][NOT_HANDSHAKING][0][0][0]: Shutdown

[2024-03-14 07:44:11,406] DEBUG {SourceHandler} - http-incoming-133: Keep-Alive connection was closed: Remote Address : /10.255.18.1:54714

[2024-03-14 07:44:11,406] DEBUG {SourceConnections} - Shutting down connection forcefully http-incoming-133

[2024-03-14 07:44:11,406] DEBUG {LoggingNHttpServerConnection} - http-incoming-133: Shutdown connection

[2024-03-14 07:44:11,406] DEBUG {SSLIOSession} - I/O session http-incoming-133-85 10.255.8.115:8253<->10.255.18.1:54714[CLOSED][][CLOSED][r][NOT_HANDSHAKING][0][0][0]: Shutdown

wso2mi 4.2.0 : Continous Socket connection errors in logs when pod count >1

There are 1 best solutions below

Related Questions in WSO2-MICRO-INTEGRATOR

Trending Questions

Popular # Hahtags

Popular Questions