java -XX:OnError for post mortem?

Question

I have a java tool running on a raspi. I ssh into the raspi and start that tool using

java -jar name.jar &

After a while (hours or days), the process doesn't run anymore. I have pretty extensive logging in my code, but my log doesn't show any error. So the question would be how I can analyze the situation? I thought using -XX:OnError method, but what would best to specify? Any other ideas what I can do?

Update: an hs_err_pid file I am not able to find. What is the working directoy when I start the program like that? I have scanned the directory from where I started the java app, /var and /tmp and /home/pi.

Update 2: Working directory is shown as /home/pi there was no err pid file. I am running it now as

java -XX:OnError="/home/pi/Server/deah.sh" -XX:ErrorFile=/home/pi/Server/hs_err_pid%p.log -jar /home/pi/Server/myjavatool-0.1.2.jar &

Can I "simulate" a crash that I see if the err file is being created? a kill -9 doesn't do the trick.

Answer 1

So I can think of a number of possibilities.

1. The JVM is panic / crashing. If this happens, then a crash file should be created somewhere. So ... wait until it happens again. (I am not sure if the OnError handler gets called in that case.)

2. The application itself is causing the JVM to exit; e.g. by calling System.exit(...) somewhere. If that is the case, then you shouldn't get a crash file and (I think) that the OnError handler won't be run.

3. Something external to the application is killing it. If it is a SIG_KILL (-9) signal, then the JVM won't get a chance to do anything. There won't be a crash dump, the OnError won't be run, there will be nothing in the application logs, and JVM shutdown hooks won't be run.

   What could cause a SIG_KILL? One possibility is some other application. A second possibility is the OOM killer. That is a builtin function of (many) Linux based systems that reacts to excessive virtual memory paging activity by sending a SIG_KILL to the process that appears to be the cause. If the OOM killer is doing this, then you should see a log entry in the system log files.

At this point, the OOM killer seems the most likely culprit.

More information:

• Linux's OOM Process Killer

---

Can I "simulate" a crash that I see if the err file is being created? a kill -9 doesn't do the trick.

I don't think so. A kill -9 sends a SIG_KILL which can't be caught by the target process. The JVM would not have a chance to generate a crash file.

If you want to be sure that no err file is being created, use the find command to search the entire file system; e.g.

 $ sudo find / -name hs_err_\*