Observed an issue in which system is throwing - "too many open files" after working fine for few hours.
Observed that there are many tcp connection stuck in "CLOSE_WAIT" state.
sudo lsof | grep ":http (CLOSE_WAIT)" | wc -l -> 16215.
Number is increasing with time and in few hours it would cross max limit allowed.
Also ran netstat command -
"netstat -ant | awk '{print $6}' | sort | uniq -c | sort -n" and output is -> 122 CLOSE_WAIT.
Why output from netstat command is way lower than lsof command. Both are returning close wait connections and should have given approx same value.
Once i got to know that connection to specific service is causing this issue, then what should i do identify the exact code where this is happening ? I went through the client code for connecting to service and i don't see any connection leakage.