We have 20 cache in Apache Ignite cache and each having backup count 2.
Each cache represent to one table.
Affinity key for every cache is customer id.
Architecture is designed in such way that one customer all cache belong to one node in ignite cluster, to avoid the multiple node querying.
Where as first time cache loading take too much time for 4 caches as these 4 table having around 30+ millions of record and each table having max 7 column, most of the columns type are numbers and and few having date.
Cache loading took 2 hr. Is there any efficient way to load cache?
Also data is not distributed evenly, one node having large data where as other having low record count which sometime leads to heap issue.
This is IMO an example of a poor choice of an affinity key because of the exact problems that you site. In order to get the benefits of parallelism you want / need your data for various customers to be spread out as evenly as possible among your cluster nodes. A good affinity key has many distinct values not 5, 10, or 20, but thousands or millions. But given two tables that are to be joined the joined records of the two tables will be on the same host. Note that this is very different than having 1 or 2 customer's data on a single host. When data is spread out evenly but can be joined efficiently then all hosts in the cluster can efficiently work together to answer questions about any customer and in this scenario you get the benefits of each host working on a smaller set of data. Net / net is that your issues are due to a poor choice of affinity key. Here's an article that might help: https://www.gridgain.com/resources/blog/micro-learning-unit-colocation-and-data-affinity-apache-ignite