high number of lwps in solaris 11 associated to weblogic instances

Question

I have a weblogic 11g cluster, with 2 physical servers. The instances are balanced with F5, but on server 2, the six weblogic instances have a high number of lwps, more than 20000. A thread dump of a weblogic instance was exported, and many occurrences of a log4j class can be seen. I need to correct this problem, because server 2 reaches 100,000 and the machine crashes. Thanks, I look forward to your suggestions.

1. Server 2 (number lwps)

[user@serverwls2 ~]$ prstat -a 1 1 | egrep "java|Total"
 25685 soracle   6040936K 5590440K sleep    59    0  10:43:47 1.279% java/6404
 29776 soracle   6033888K 5656640K sleep    59    0  10:43:17 1.019% java/6603
  8190 soracle   5501688K 3571712K sleep    59    0   0:41:39 0.058% java/4029
  8633 soracle   5650160K 4005776K sleep    59    0   1:18:00 0.046% java/5186
  8635 soracle   5630848K 4220944K sleep    59    0   1:17:02 0.043% java/4963
  8634 soracle   5643592K 4078840K sleep    59    0   1:14:07 0.043% java/5160
Total: 96 processes, 33568 lwps, load averages: 1.64, 1.61, 1.63

2. Extract thread dump (pid 29776) export with jstack

    2022-05-17 16:53:00
    Full thread dump Java HotSpot(TM) 64-Bit Server VM (24.331-b06 mixed mode):
    "Attach Listener" daemon prio=3 tid=0x0000000115d3b800 nid=0x3443 waiting on condition [0x0000000000000000]
       java.lang.Thread.State: RUNNABLE
       Locked ownable synchronizers:
            - None
    "Thread-13074" daemon prio=3 tid=0x0000000115d39800 nid=0x3442 waiting on condition [0xfffffffc09cff000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
       Locked ownable synchronizers:
            - None
    "Thread-13072" daemon prio=3 tid=0x0000000116e36000 nid=0x3440 waiting on condition [0xfffffffc09eff000]
       java.lang.Thread.State: TIMED_WAITING (sleeping)
            at java.lang.Thread.sleep(Native Method)
            at org.apache.log4j.helpers.FileWatchdog.run(FileWatchdog.java:104)
       Locked ownable synchronizers:
            - None

3. Occurrence number in thread dump

    [user@serverwls2 ~]$ grep "org.apache.log4j.helpers.FileWatchdog.run" pid29776_dump.txt | wc -l
        6450

Regards.

Answer 1

You may disable configureAndWatch as a workaround to your FileWatchdog issue.

Also, review the log4j.xml for permissions, it looks like cannot be read.