I'm attempting to implement Infinispan distributed synchronous caching in my Spring Boot applications. I anticipate that data should be accessible and modifiable across all nodes within a cluster. However, I'm encountering a problem where cache entries added in one node are not accessible for reading or writing from other nodes. Below is my cache bean configuration.
I'm trying to access and evice pair-executions-cache in my uibs but when I'm autowring org.springframework.cache.CacheManager bean. I get following exception on startup sometimes.
confBuilder.expiration().lifespan(8, TimeUnit.HOURS);
confBuilder.clustering().cacheMode(CacheMode.DIST_SYNC);
confBuilder.build();```
2024-02-23 11:17:55,625 recon-ui-service WARN [main] CONTAINER: ISPN000189: While stopping a cache or cache manager, one of its components failed to stop
java.util.concurrent.CompletionException: org.infinispan.commons.CacheException: Initial state transfer timed out for cache pair-executions-cache on uibs-app-6495d4dc6d-kchkc-30778
at java.base/java.util.concurrent.CompletableFuture.reportJoin(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.join(Unknown Source)
at org.infinispan.manager.DefaultCacheManager.terminate(DefaultCacheManager.java:781)
at org.infinispan.manager.DefaultCacheManager.stopCaches(DefaultCacheManager.java:842)
at org.infinispan.manager.DefaultCacheManager.internalStop(DefaultCacheManager.java:818)
at org.infinispan.manager.DefaultCacheManager.stop(DefaultCacheManager.java:794)
at org.infinispan.manager.DefaultCacheManager.close(DefaultCacheManager.java:1164)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.DisposableBeanAdapter.invokeCustomDestroyMethod(DisposableBeanAdapter.java:281)
at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:215)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:587)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1152)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:580)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:559)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:1152)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:520)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:1145)
at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1106)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:596)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
at com.timestreamgroup.reconstream.ui.service.ReconUiServiceApplication.main(ReconUiServiceApplication.java:37)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache pair-executions-cache on uibs-app-6495d4dc6d-kchkc-30778
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:249)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1018)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:698)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:644)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:533)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:511)
at com.timestreamgroup.reconstream.common.config.CacheConfig.buildCache(CacheConfig.java:118)
at com.timestreamgroup.reconstream.common.config.CacheConfig.pairExecutionsCache(CacheConfig.java:193)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e.CGLIB$pairExecutionsCache$8(<generated>)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e$$FastClassBySpringCGLIB$$6bd07379.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e.pairExecutionsCache(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
... 15 common frames omitted
2024-02-23 11:17:55,642 recon-ui-service INFO [main] CLUSTER: ISPN000080: Disconnecting JGroups channel recon-infinispan
2024-02-23 11:17:55,720 recon-ui-service INFO [main] LocalContainerEntityManagerFactoryBean: Closing JPA EntityManagerFactory for persistence unit 'default'
2024-02-23 11:17:55,724 recon-ui-service INFO [main] HikariDataSource: HikariPool-1 - Shutdown initiated...
2024-02-23 11:17:55,731 recon-ui-service INFO [main] HikariDataSource: HikariPool-1 - Shutdown completed.
2024-02-23 11:17:55,734 recon-ui-service INFO [main] StandardService: Stopping service [Tomcat]
2024-02-23 11:17:55,817 recon-ui-service INFO [main] ConditionEvaluationReportLoggingListener:
Error starting ApplicationContext. To display the conditions report re-run your application with 'debug' enabled.
2024-02-23 11:17:55,839 recon-ui-service ERROR [main] SpringApplication: Application run failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'pairExecutionsCache' defined in class path resource [com/timestreamgroup/reconstream/common/config/CacheConfig.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.infinispan.Cache]: Factory method 'pairExecutionsCache' threw exception; nested exception is org.infinispan.commons.CacheException: Initial state transfer timed out for cache pair-executions-cache on uibs-app-6495d4dc6d-kchkc-30778
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:658)
at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:638)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1334)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1177)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:564)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:524)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:335)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:234)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:333)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:208)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:944)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:918)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:145)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:754)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:434)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:338)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1343)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:1332)
at com.timestreamgroup.reconstream.ui.service.ReconUiServiceApplication.main(ReconUiServiceApplication.java:37)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:88)
Caused by: org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.infinispan.Cache]: Factory method 'pairExecutionsCache' threw exception; nested exception is org.infinispan.commons.CacheException: Initial state transfer timed out for cache pair-executions-cache on uibs-app-6495d4dc6d-kchkc-30778
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185)
at org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:653)
... 27 common frames omitted
Caused by: org.infinispan.commons.CacheException: Initial state transfer timed out for cache pair-executions-cache on uibs-app-6495d4dc6d-kchkc-30778
at org.infinispan.statetransfer.StateTransferManagerImpl.waitForInitialStateTransferToComplete(StateTransferManagerImpl.java:249)
at org.infinispan.cache.impl.CacheImpl.start(CacheImpl.java:1018)
at org.infinispan.cache.impl.AbstractDelegatingCache.start(AbstractDelegatingCache.java:512)
at org.infinispan.manager.DefaultCacheManager.wireAndStartCache(DefaultCacheManager.java:698)
at org.infinispan.manager.DefaultCacheManager.createCache(DefaultCacheManager.java:644)
at org.infinispan.manager.DefaultCacheManager.internalGetCache(DefaultCacheManager.java:533)
at org.infinispan.manager.DefaultCacheManager.getCache(DefaultCacheManager.java:511)
at com.timestreamgroup.reconstream.common.config.CacheConfig.buildCache(CacheConfig.java:118)
at com.timestreamgroup.reconstream.common.config.CacheConfig.pairExecutionsCache(CacheConfig.java:193)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e.CGLIB$pairExecutionsCache$8(<generated>)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e$$FastClassBySpringCGLIB$$6bd07379.invoke(<generated>)
at org.springframework.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:244)
at org.springframework.context.annotation.ConfigurationClassEnhancer$BeanMethodInterceptor.intercept(ConfigurationClassEnhancer.java:331)
at com.timestreamgroup.reconstream.common.config.CacheConfig$$EnhancerBySpringCGLIB$$2b08c18e.pairExecutionsCache(<generated>)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:154)
... 28 common frames omitted```