Something occasionally happened for one random instance of 5 available instances few times in a day. Worker threads get stucked, after exhausting all the worker threads service fail to respond a health check and get restarted. We observe that problem occurs with POST/PUT/PATCH requests. Those stucked requests runs for tens of minutes and then are reported as failed with status code 400 || 500. Blocked threads grows quickly and in few minutes could exhauste all pool, and they grow only on a single instance other 4 works fine.
Spring boot version 3.1.4 / 3.0.11 Spring Cloud Gateway.(Netty) <->(http2 SSL) Spring based service with REST endpoint(Undertow).
We add an Undertow StuckThreadDetector and it reports a stacktrace like:
java.lang.Throwable: null
at [email protected]/java.lang.Object.wait(Native Method)
at [email protected]/java.lang.Object.wait(Object.java:338)
at io.undertow.server.protocol.framed.AbstractFramedStreamSourceChannel.awaitReadable(AbstractFramedStreamSourceChannel.java:354)
at org.xnio.conduits.StreamSourceChannelWrappingConduit.awaitReadable(StreamSourceChannelWrappingConduit.java:71)
at org.xnio.conduits.ConduitStreamSourceChannel.awaitReadable(ConduitStreamSourceChannel.java:151)
at io.undertow.channels.DetachableStreamSourceChannel.awaitReadable(DetachableStreamSourceChannel.java:77)
at io.undertow.server.HttpServerExchange$ReadDispatchChannel.awaitReadable(HttpServerExchange.java:2305)
at org.xnio.channels.Channels.readBlocking(Channels.java:345)
at io.undertow.servlet.spec.ServletInputStreamImpl.readIntoBuffer(ServletInputStreamImpl.java:201)
at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:176)
at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:162)
at io.undertow.servlet.spec.ServletInputStreamImpl.read(ServletInputStreamImpl.java:153)
at [email protected]/java.io.FilterInputStream.read(FilterInputStream.java:82)
at [email protected]/java.io.PushbackInputStream.read(PushbackInputStream.java:135)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver$EmptyBodyCheckingHttpInputMessage.<init>(AbstractMessageConverterMethodArgumentResolver.java:331)
at org.springframework.web.servlet.mvc.method.annotation.AbstractMessageConverterMethodArgumentResolver.readWithMessageConverters(AbstractMessageConverterMethodArgumentResolver.java:172)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.readWithMessageConverters(RequestResponseBodyMethodProcessor.java:163)
at org.springframework.web.servlet.mvc.method.annotation.RequestResponseBodyMethodProcessor.resolveArgument(RequestResponseBodyMethodProcessor.java:136)
at org.springframework.web.method.support.HandlerMethodArgumentResolverComposite.resolveArgument(HandlerMethodArgumentResolverComposite.java:122)
at org.springframework.web.method.support.InvocableHandlerMethod.getMethodArgumentValues(InvocableHandlerMethod.java:179)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:146)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:884)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1081)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:974)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1011)
java.lang.Throwable: null
at java.lang.Object.wait(Object.java)
at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:329)
at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:313)
at org.xnio.conduits.StreamSinkChannelWrappingConduit.awaitWritable(StreamSinkChannelWrappingConduit.java:99)
at org.xnio.conduits.ConduitStreamSinkChannel.awaitWritable(ConduitStreamSinkChannel.java:134)
at io.undertow.channels.DetachableStreamSinkChannel.awaitWritable(DetachableStreamSinkChannel.java:87)
at io.undertow.server.HttpServerExchange$WriteDispatchChannel.awaitWritable(HttpServerExchange.java:2141)
// ... 183 frames truncated
This guys mention similar issue https://access.redhat.com/solutions/6988036. - but there are no info how they fix it.
We add an Undertow StuckThreadDetector to catch the thread's stacktrace. We are going to try to BlockingReadTimeoutHandler BlockingWriteTimeoutHandler to fail fast when there are a problems in undertow awaitReadable/awaitWritable, but we are still looking for a root cause of such a random issue.
Ohhh... this tricky issue with Underthow... When I used Tomcat as an application server - there were no problems, however there was a necessity to change it to Underthow...
Hope, you will find a solution!