Solr 9.2.1, it appears at any time only one Thread is executing and all others are blocked, though started 20 threads to index. We have 50 million objects to index and it will take 1 year.
Any suggestions to improve?
******All threads except one are blocked as below
"qtp1155757579-16" #16 prio=5 os_prio=0 cpu=1070.17ms elapsed=337.00s tid=0x00007f915acc4000 nid=0xc716e waiting on condition [0x00007f840985a000]
java.lang.Thread.State: WAITING (parking)
at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
- parking to wait for <0x00007f8e1249aa60> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:194)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt([email protected]/AbstractQueuedSynchronizer.java:885)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared([email protected]/AbstractQueuedSynchronizer.java:1009)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared([email protected]/AbstractQueuedSynchronizer.java:1324)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock([email protected]/ReentrantReadWriteLock.java:738)
at org.apache.solr.update.VersionInfo.lockForUpdate(VersionInfo.java:127)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionAdd(DistributedUpdateProcessor.java:351)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:236)
at org.apache.solr.update.processor.UpdateRequestProcessor.processAdd(UpdateRequestProcessor.java:55)
at com.ptc.solr.update.processor.PTCUpdateRequestProcessor.processAdd(PTCUpdateRequestProcessor.java:194)
at org.apache.solr.handler.loader.JavabinLoader$1.update(JavabinLoader.java:123)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readOuterMostDocIterator(JavaBinUpdateRequestCodec.java:342)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readIterator(JavaBinUpdateRequestCodec.java:286)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:338)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec$StreamingCodec.readNamedList(JavaBinUpdateRequestCodec.java:236)
at org.apache.solr.common.util.JavaBinCodec.readObject(JavaBinCodec.java:303)
at org.apache.solr.common.util.JavaBinCodec.readVal(JavaBinCodec.java:283)
at org.apache.solr.common.util.JavaBinCodec.unmarshal(JavaBinCodec.java:193)
at org.apache.solr.client.solrj.request.JavaBinUpdateRequestCodec.unmarshal(JavaBinUpdateRequestCodec.java:126)
******The only executing thread
"qtp1155757579-25" #25 prio=5 os_prio=0 cpu=3635.83ms elapsed=336.95s tid=0x00007f915acd3800 nid=0xc7178 runnable [0x00007f8409611000]
java.lang.Thread.State: RUNNABLE
at sun.nio.fs.UnixNativeDispatcher.unlink0([email protected]/Native Method)
at sun.nio.fs.UnixNativeDispatcher.unlink([email protected]/UnixNativeDispatcher.java:156)
at sun.nio.fs.UnixFileSystemProvider.implDelete([email protected]/UnixFileSystemProvider.java:236)
at sun.nio.fs.AbstractFileSystemProvider.delete([email protected]/AbstractFileSystemProvider.java:105)
at java.nio.file.Files.delete([email protected]/Files.java:1142)
at org.apache.lucene.store.FSDirectory.privateDeleteFile(FSDirectory.java:344)
at org.apache.lucene.store.FSDirectory.deleteFile(FSDirectory.java:309)
at org.apache.lucene.store.NRTCachingDirectory.deleteFile(NRTCachingDirectory.java:128)
- locked <0x00007f8e11678d10> (a org.apache.lucene.store.NRTCachingDirectory)
at org.apache.lucene.store.LockValidatingDirectoryWrapper.deleteFile(LockValidatingDirectoryWrapper.java:37)
at org.apache.lucene.index.IndexFileDeleter.deleteFile(IndexFileDeleter.java:763)
at org.apache.lucene.index.IndexFileDeleter.deleteFiles(IndexFileDeleter.java:757)
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:655)
at org.apache.lucene.index.IndexFileDeleter.decRef(IndexFileDeleter.java:686)
at org.apache.lucene.index.IndexWriter.decRefDeleter(IndexWriter.java:5851)
- locked <0x00007f8e1180b968> (a org.apache.solr.update.SolrIndexWriter)
at org.apache.lucene.index.StandardDirectoryReader.lambda$doClose$1(StandardDirectoryReader.java:466)
at org.apache.lucene.index.StandardDirectoryReader$$Lambda$910/0x00007f83f55bdc58.close(Unknown Source)
at org.apache.lucene.index.StandardDirectoryReader.doClose(StandardDirectoryReader.java:482)
at org.apache.lucene.index.IndexReader.decRef(IndexReader.java:265)
at org.apache.lucene.index.IndexReader.close(IndexReader.java:411)
- locked <0x00007f90153053a0> (a org.apache.lucene.index.StandardDirectoryReader)
at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:687)
at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:381)
at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:366)
at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:246)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:2329)
at org.apache.solr.update.UpdateLog.openRealtimeSearcher(UpdateLog.java:745)
- locked <0x00007f8e11df7a20> (a org.apache.solr.update.UpdateLog)
at org.apache.solr.update.DirectUpdateHandler2.deleteByQuery(DirectUpdateHandler2.java:590)
- locked <0x00007f8e0d55a1a8> (a java.lang.Object)
at org.apache.solr.update.processor.RunUpdateProcessorFactory$RunUpdateProcessor.processDelete(RunUpdateProcessorFactory.java:86)
at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDelete(DistributedUpdateProcessor.java:276)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doLocalDeleteByQuery(DistributedUpdateProcessor.java:1011)
at org.apache.solr.update.processor.DistributedUpdateProcessor.versionDeleteByQuery(DistributedUpdateProcessor.java:980)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:938)
at org.apache.solr.update.processor.DistributedUpdateProcessor.doDeleteByQuery(DistributedUpdateProcessor.java:913)
at org.apache.solr.update.processor.DistributedUpdateProcessor.processDelete(DistributedUpdateProcessor.java:830)
at org.apache.solr.update.processor.UpdateRequestProcessor.processDelete(UpdateRequestProcessor.java:59)
at org.apache.solr.handler.loader.JavabinLoader.delete(JavabinLoader.java:230)
at org.apache.solr.handler.loader.JavabinLoader.parseAndLoadDocs(JavabinLoader.java:140)
at org.apache.solr.handler.loader.JavabinLoader.load(JavabinLoader.java:74)
Expect the 20 threads working in parallel