Bad performance with hibernate with second level cache when query cache enabled

Question

In the application I am working on has multiple cachings used along with second level caching using ehcache.

<prop key="hibernate.cache.region.factory_class">ehcache</prop>
                <prop key="hibernate.cache.use_second_level_cache">true</prop>
                <prop key="hibernate.cache.use_query_cache">true</prop>
                <prop key="hibernate.session.events.log.LOG_QUERIES_SLOWER_THAN_MS">100</prop>
                <prop key="net.sf.ehcache.configurationResourceName">/ehcache.xml</prop>

While performing stress test we identified that hibernate performing worst. As you can observe in the below hibernate session takes lot of time, while queries are quick by themselves.

By digging further into the problem, we found disabling query caching, problem got resolved. To understand why query caching causing the problem, we enabled hibernate debug level logs and found that TimestampsCacheEnabledImpl class which is updating time stamps of query spaces is the culprit. Strangely, we are not using query caching on most of the table queries and yet TimestampsCacheEnabledImpl doing timestamp invalidation and other stuff for those tables, which is taking a lot of time.

How can I solve this problem? Should I just abandon the query caching and depend on redis for caching?