Jetty 12 idle timeout limited to 30 seconds when upgrading to Java 21 in Google App Engine Standard Environment

Question

I used to work with Google App Engine Standard Environment with Java 8 runtime for several years, after the end of support, early this year (2024), i've successfully migrated to Java 21 runtime. Since Java 21, i've experienced an error message when a process lasts longer than 30 seconds:

HTTP ERROR 500 java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms URI: http://XXXXX.appspot.com/servlet STATUS: 500 MESSAGE: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms CAUSED BY: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms Caused by: java.util.concurrent.TimeoutException: Idle timeout expired: 30001/30000 ms at org.eclipse.jetty.io.IdleTimeout.checkIdleTimeout(IdleTimeout.java:167) at org.eclipse.jetty.io.IdleTimeout.idleCheck(IdleTimeout.java:113) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583)

One of the great advantages of having migrated from Java 8, is that previously HTTP requests lasted only 1 minute, now it has a timeout of 10 minutes. But this timeout of 30 seconds seems to occur in the Jetty 12 "layer". Having measure how long my process lasts, it does everything it has to do, lasting the request more than a minute, perfectly, this error message appear not 30 seconds after, but after the process finished no matter how long it takes, i.e., it will show up even after 5 minutes saying "30000 ms timeout". What i mean with that is that the process loads the info perfectly, so i just have to go back and reload the page and my info is there:

enter image description here

My theory is that this error occurs at the Jetty "layer", i.e., for me right now, there is no way i can access or increase this timeout parameter. First off, this error does not show anywhere in my Google Cloud's Log Explorer. I've tried to configure some tags in my appengine-web.xml file, but nothing worked:

<system-properties>
  <property name="appengine.api.urlfetch.defaultDeadline" value="10"/>
  <property name="org.eclipse.jetty.io.IdleTimeout._idleTimeout" value="50000"/>
  <property name="JAVA_USER_OPTS" value="-Dorg.eclipse.jetty.io.IdleTimeout._idleTimeout=50000"/>
</system-properties>
  
<env-variables>
  <env-var name="JETTY_PROPERTIES" value="-Dorg.eclipse.jetty.io.IdleTimeout._idleTimeout=50000"/>
  <env-var name="JETTY_PROPERTIES" value="-Djetty.http.idleTimeout=60000" />
  <env-var name="JETTY_ARGS" value="idleTimeout=36000" />
  <env-var name="JETTY_PROPERTIES" value="-Dserver.jetty.connection-idle-timout=36000" />
  <env-var name="JETTY_ARGS" value="server.jetty.connection-idle-timout=36000" />
</env-variables>

Also in my web.xml file:

<context-param>
  <param-name>org.eclipse.jetty.server.IdleTimeout</param-name>
  <param-value>35000</param-value>
</context-param>

My pom.xml file "trying" to override the Jetty classes:

    <!-- https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server -->
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-server</artifactId>
        <version>12.0.6</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-io -->
    <dependency>
        <groupId>org.eclipse.jetty</groupId>
        <artifactId>jetty-io</artifactId>
        <version>12.0.7</version>
    </dependency>
    
    <!-- https://mvnrepository.com/artifact/org.eclipse.jetty.ee10/jetty-ee10-servlet -->
    <dependency>
        <groupId>org.eclipse.jetty.ee10</groupId>
        <artifactId>jetty-ee10-servlet</artifactId>
        <version>12.0.6</version>
    </dependency>

I've found this post describing a similar issue, suggesting to access to the instances of the Jetty server or something: https://github.com/spring-projects/spring-boot/issues/38960

In my Log Explorer, each time i call my servlet, i got some logs saying that Jetty server has initialized:

Started JettyServerConnectorWithReusePort@XXXXXXXXXX{HTTP/1.1, (http/1.1)}{0.0.0.0:8081} jetty-12.0.6; built: 2024-01-30T02:43:24.387Z; git: XXXXXXXXXX; jvm 21.0.2+13-Ubuntu-122.04.1 Started cgarj.JettyServletEngineAdapter$@XXXXXXXXXX{STARTING}[12.0.6,sto=0] @4698ms

Then i've tried several ways of getting access to that server instance from my servlet, my last try is this:

public void doPost(HttpServletRequest req, HttpServletResponse resp) throws IOException 
{
    java.util.Enumeration<String> attributeNames = req.getServletContext().getAttributeNames();
    while (attributeNames.hasMoreElements()) {
        String attributeName = attributeNames.nextElement();
        Object attributeValue = getServletContext().getAttribute(attributeName);
        log.warning(attributeName + ": " + attributeValue);
            
        if (attributeName.equals("org.eclipse.jetty.server.Executor"))
        {
            // Get all fields of the object
            Field[] fields = attributeValue.getClass().getDeclaredFields();

            // Print the properties
            for (Field field : fields) {
                field.setAccessible(true); // Make the field accessible if it's private
                try {
                    Object value = field.get(attributeValue);
                    log.warning(field.getName() + ": " + value);
                } catch (IllegalAccessException e) {
                    e.printStackTrace();
                }
            }
        }
    }
}

The output, from the log explorer:

com.google.apphosting.runtime.jetty.APP_VERSION_CONTEXT_ATTR: com.google.apphosting.runtime.AppVersion@6f2b44ba
org.eclipse.jetty.tlds: [jar:file:///workspace/WEB-INF/lib/org.glassfish.web.jakarta.servlet.jsp.jstl-3.0.1.jar!/META-INF/fmt.tld, 
org.eclipse.jetty.util.DecoratedObjectFactory: org.eclipse.jetty.util.DecoratedObjectFactory[decorators=2]
org.eclipse.jetty.originAttribute: origin
org.apache.tomcat.InstanceManager: org.apache.tomcat.SimpleInstanceManager@XXXXXX
org.eclipse.jetty.quickstart.xml: /workspace/WEB-INF/quickstart-web.xml
org.eclipse.jetty.server.Executor: QueuedThreadPool[qtp1105322512]@XXXXX{STARTED,0<=0<=100,i=0,r=-1,t=-46773ms,q=0}[ReservedThreadExecutor@XXXXXXX{reserved=0/1,pending=0}]
jakarta.servlet.context.tempdir: /tmp/jetty-0_0_0_0-0-workspace-_-any-
org.eclipse.jetty.ee10.quickstart.QuickStartDescriptorProcessor: org.eclipse.jetty.ee10.quickstart.QuickStartDescriptorProcessor@XXXXX
org.apache.jasper.compiler.TldCache: org.apache.jasper.compiler.TldCache@XXXXX
org.apache.jasper.runtime.JspApplicationContextImpl: org.apache.jasper.runtime.JspApplicationContextImpl@XXXXX

The attributes of the instance of org.eclipse.jetty.server.Executor:

LOG: org.slf4j.jul.JDK14LoggerAdapter@693f642d
NOOP: org.eclipse.jetty.util.thread.QueuedThreadPool$$Lambda/0x00003e2948164ff8@5c84e324
_counts: 0|0
_evictThreshold: 4624122529
_threads: []
_joinLock: org.eclipse.jetty.util.thread.AutoLock$WithCondition@436a9f55
_jobs: []
_threadGroup: null
_threadFactory: QueuedThreadPool[qtp1105322512]@41e1e210{STARTED,0<=0<=100,i=0,r=-1,t=-46777ms,q=0}[ReservedThreadExecutor@17bd3b7c{reserved=0/1,pending=0}]
_name: qtp1105322512
**_idleTimeout: 60000**
_maxThreads: 100
_minThreads: 0
_reservedThreads: -1
_tryExecutor: ReservedThreadExecutor@17bd3b7c{reserved=0/1,pending=0}
_priority: 5
_daemon: false
_detailedDump: false
_lowThreadsThreshold: 1
_budget: org.eclipse.jetty.util.thread.ThreadPoolBudget@31ae5249
_stopTimeout: 5000
_virtualThreadsExecutor: null
_maxEvictCount: 1
_runnable: org.eclipse.jetty.util.thread.QueuedThreadPool$Runner@772be7f0

As you can see above, the property _idleTimeout has a value of 60,000 ms, my guess is that even if i got access to increase it, it would change a thing.