High Latency and Thread Starvation Issues in Spring Boot App

Question

I have a Spring Boot application with an API endpoint that pushes data to a Kafka topic. However, when the application experiences high load, it leads to significant latency issues, and many requests become thread-starved, eventually resulting in disconnections due to response timeouts.

The application utilizes CompletableFuture to push data asynchronously to Kafka. While this approach works well under normal circumstances, it struggles to handle the load efficiently, leading to degraded performance and thread starvation.

I have observed that during high load, only 70-80 threads are created, even though the maximum limit of threads is set to 300. This thread limitation further contributes to the latency and thread starvation problems.

Here is my controller

  @RequestMapping(value = "/pushtokafka", method = RequestMethod.POST, consumes = MimeTypeUtils.APPLICATION_JSON_VALUE)
    public EventResponse publishEventToKafka(@RequestParam Map<String, Object> map, @RequestBody(required = false) Payload payload) {
        CompletableFuture.runAsync(() -> {
            if (map.containsKey("abc")) {
                eventProducerService.pushtoKafka1(payload);
            } else {
                eventProducerService.pushtoKafka2(payload);
            }
        }, taskExecutor);

        return new EventResponse("success", "Successfully pushed to kafka");
    }

I have implemented custom thread pool
 @Bean
    public Executor taskExecutor() {
        ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
        executor.setCorePoolSize(8); // set the core pool size
        executor.setMaxPoolSize(20); // max pool size
        executor.setQueueCapacity(Integer.MAX_VALUE);
        executor. Initialize();
        return executor;
    }

It's a blocker for us.

My spring boot versions is 2.2.6.RELEASE my tomcat version is 9.0.65

I have changed these properties

server.tomcat.max-threads=300
server.tomcat.connection-timeout=120000
server.tomcat.accept-count=1000
server.tomcat.max-connections=-1
server.tomcat.min-spare-threads=200 

after applying these properties , i am still getting the same issue.

I wanted to know why new threads is not getting created , if there is load . [enter image description here][1]

kibana Transactions image [1]: https://i.stack.imgur.com/vaCsC.png