We are seeing below errors intermittently while calling APIs from our service. The issue is happening only during load testing - Error was received while reading the incoming data. The connection will be closed.
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] io.netty.util.IllegalReferenceCountException: refCnt: 0, decrement: 1
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.util.internal.ReferenceCountUpdater.toLiveRealRefCnt(ReferenceCountUpdater.java:83)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.util.internal.ReferenceCountUpdater.release(ReferenceCountUpdater.java:147)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.buffer.AbstractReferenceCountedByteBuf.release(AbstractReferenceCountedByteBuf.java:101)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.http.DefaultHttpContent.release(DefaultHttpContent.java:92)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.util.ReferenceCountUtil.release(ReferenceCountUtil.java:90)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at reactor.netty.channel.FluxReceive.onInboundNext(FluxReceive.java:380)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at reactor.netty.channel.ChannelOperations.onInboundNext(ChannelOperations.java:411)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at reactor.netty.http.client.HttpClientOperations.onInboundNext(HttpClientOperations.java:734)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1382)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1256)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1296)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
2023-11-29T12:07:17.968+05:30 [APP/PROC/WEB/25] [OUT] at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.core.publisher.Operators$BaseFluxToMonoOperator.onSubscribe(Operators.java:1988)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.core.publisher.FluxPeek$PeekSubscriber.onSubscribe(FluxPeek.java:171)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.core.publisher.FluxMap$MapSubscriber.onSubscribe(FluxMap.java:92)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.netty.channel.FluxReceive.startReceiver(FluxReceive.java:170)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at reactor.netty.channel.FluxReceive.lambda$subscribe$2(FluxReceive.java:148)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:403)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
2023-11-29T12:07:45.403+05:30 [APP/PROC/WEB/18] [OUT] at java.base/java.lang.Thread.run(Unknown Source)
The issue started appearing after we have upgraded spring boot version to 3.0.3 and java version to 17 from java 8.
We have tried to increase the buffer size using application.properties file as well as by using ExchangeStrategies. After increasing the buffer size we see a reduction in frequency of error, but error is still persisting intermittently-
spring.codec.max-in-memory-size: 20MB
WebClient.builder()
.exchangeStrategies(ExchangeStrategies.builder()
.codecs(configurer -> configurer.defaultCodecs().maxInMemorySize(MAX_BUFFER_SIZE))
.build())