Why is EhCache slowing down Gets when clustering?

Question

I'm using EhCache in my Java web-application successfully. I'm now experimenting with a Terracotta server, because I want to distribute the application on more than one server instance but I'm puzzled.

I would expect the PUT times be slower, when I add a cluster as a resource (keeping the heap resource as it is). But shouldn't the GET times stay more or less the same? The GET times (as measured by JMX MBeans) are increasing by 10-100x when I add the cluster as a resource (as I said, I keep the heap resource as-is).

My understanding from the documentation is that upon PUT, the authorative tier (cluster in this case) needs to be informed, so it can evict all upper tiers. But upon repeating GETs, only the heap tier should be considered (when big enough - which is the case here, it's just an experiment yet).

Did I get something wrong here or are my assumptions correct and I have an error somewhere?

<cache alias="publisherCache" uses-template="eternal-10000">
    <resources>
        <heap>200</heap>
        <terracotta:clustered-dedicated unit="MB">15</terracotta:clustered-dedicated>
    </resources>
</cache>

vs.

<cache alias="publisherCache" uses-template="eternal-10000">
    <resources>
        <heap>200</heap>
    </resources>
</cache>

produces 1600us vs. 16us on average over 2000 GETs.

Answer 1

Your configuration seems fine. The first get on an entry will be slower because you have a cache miss on the heap tier.

However, all the following ones should be as fast as usual.

Can you please tell me if that's the case?