Cause of 502 Bad Gateway where PHP Reloaded and many child processes spawned and exited quickly

Question

We just had an issue on 2 servers (both unrelated, beefy servers, and one not public-facing) at the same time we had 502 Gateway where I had to manually reload PHP to fix it.

Looking back in the PHP logs, I noticed PHP was reloaded (trying to work out why - maybe automated PHP version update by Runcloud?) and then many PHP Child processes were spawned and exited, each exiting in under 1 second each - very strange as all historical processes exited after 1500+ seconds as normal.

Server 1 had lots that said: "child XXXX exited on signal 6 (SIGABRT - core dumped) after 0.848228 seconds from start"

Server 2 (not open to public) had lots that said: "child XXXX exited with code 1 after 0.036807 seconds from start"

I can't seem to figure out what has caused it. I've looked through traffic logs and nothing looks too crazy, and I can't find anything that would've caused the PHP restart, nor has it been regular in the past that we haven't triggered ourselves via deployment.

I can only put it down to maybe RunCloud running an update on our servers and it breaking - still can't explain why after a PHP reload, the child processes went crazy and broke both servers resulting in 502, and then my manual reload of PHP which fixed it and everything is back to normal.

Super super strange, any ideas?

Answer 1

1. SIGABRT is PHP or one of its PECL extensions crashing with "undefined behavior".
2. Since you get these errors in FPM, that is definitely not a cronjob to blame.
3. You can disabling some extensions such as APCu and/or opcache/preloading. But that's just a blind trial which may or may not work. What you need to find the cause for certain is to collect core dump and analyze it.