WSO2 API-M: What is Synapse and why it spits following error: I/O error: Connection reset java.net.SocketException: Connection reset?

Question

We have standard deployment of WSO2 API-Manager with LoadBalancer (HAProxy) with minimal configurations.
Configured/Published only one endpoint with one resource and configured OAuth2 for it.
Logs got flooded with following errors:

ERROR {org.apache.synapse.transport.passthru.SourceHandler} - I/O error: Connection reset java.net.SocketException: Connection reset
    at java.base/sun.nio.ch.SocketChannelImpl.throwConnectionReset(SocketChannelImpl.java:394)
    at java.base/sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:426)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.receiveEncryptedData(SSLIOSession.java:482)
    at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:544)
    at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:120)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
    at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
    at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
    at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
    at java.base/java.lang.Thread.run(Thread.java:833)

API-M is in idle state (no requests passing to/through it (for now (for testing purposes) API-M deployed at separate server with no clients hitting it)).
Why we are getting such errors (though we do not configured anything related to Synapse) ? And what is Synapse anyway?

EDIT:
It turns out that if LoadBalancer (HAProxy) turned off then such errors no longer appears...
But why? And how to mitigate/fix this?

Answer 1

Synapse is the underlying mediation engine WSO2 APIM uses. In your case, it's accepting incoming requests. Since you have a LB fronting APIM the above error could occur due to the periodical health checks the LB does. You can confirm this by checking the time difference between the error logs. If it has the same pattern it means something is periodically calling APIM.

To get rid of this error you can properly set the health checks to do proper HTTP request to check the health of the server. Health check details are available here