the logs for the pods contains the following error, not able to use Pinot now. Resource usage looks good to me, and deleting/restarting pod did not help. How to solve this and let Pinot back to work? What other issues can it have?
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f4aa82f8bb2, pid=1, tid=97
#
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.16+8) (build 11.0.16+8)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.16+8, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x162bb2]
#
# Core dump will be written. Default location: /opt/pinot/core.1
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
[thread 107 also had an error][thread 104 also had an error]
[thread 101 also had an error]
[thread 98 also had an error]
[thread 103 also had an error]
[thread 99 also had an error]
[thread 91 also had an error]
[thread 94 also had an error]
[thread 96 also had an error]
[thread 100 also had an error]
[thread 93 also had an error]
[thread 106 also had an error][thread 90 also had an error]
[thread 95 also had an error]
[thread 102 also had an error]
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f4aa81bb602]
Got a similar error after deleting the pod. It seems like a segment issue. .
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGBUS (0x7) at pc=0x00007f29d9471ddd, pid=1, tid=106
#
# JRE version: OpenJDK Runtime Environment 18.9 (11.0.16+8) (build 11.0.16+8)
# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.16+8, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x162ddd]
#
# Core dump will be written. Default location: /opt/pinot/core.1
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
Exception in thread "bridge_c4c20cc81ed34dadaef62b43e29fe40c__0__86__20240224T0904Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code
at org.apache.pinot.segment.local.realtime.impl.dictionary.BaseOffHeapMutableDictionary.expand(BaseOffHeapMutableDictionary.java:263)
at org.apache.pinot.segment.local.realtime.impl.dictionary.BaseOffHeapMutableDictionary.expand(BaseOffHeapMutableDictionary.java:297)
at org.apache.pinot.segment.local.realtime.impl.dictionary.BaseOffHeapMutableDictionary.indexValue(BaseOffHeapMutableDictionary.java:402)
at org.apache.pinot.segment.local.realtime.impl.dictionary.StringOffHeapMutableDictionary.index(StringOffHeapMutableDictionary.java:52)
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.updateDictionary(MutableSegmentImpl.java:582)
at org.apache.pinot.segment.local.indexsegment.mutable.MutableSegmentImpl.index(MutableSegmentImpl.java:520)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.processStreamEvents(LLRealtimeSegmentDataManager.java:581)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager.consumeLoop(LLRealtimeSegmentDataManager.java:434)
at org.apache.pinot.core.data.manager.realtime.LLRealtimeSegmentDataManager$PartitionConsumer.run(LLRealtimeSegmentDataManager.java:629)
at java.base/java.lang.Thread.run(Thread.java:829)
[thread 107 also had an error]
[thread 96 also had an error]
[thread 100 also had an error]
[thread 108 also had an error]
[thread 91 also had an error]
[thread 92 also had an error]
[thread 95 also had an error]
[thread 103 also had an error]
[thread 94 also had an error]
[thread 104 also had an error]
[thread 97 also had an error]
[thread 102 also had an error]
[thread 99 also had an error]
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007f29d9334602].