Node.js server stops responding to requests

56 Views Asked by At

I have written a Node.js (14.17.1) web application using Sails.js (0.12.14) which internally uses Express.js (3.21.3). This application has been in production since more than 5 years (whith only minor version bumps to Node.js and Sails.js).

Once in a while, this server stops responding to requests. I have used logging in my controller to identify that it is not stopping anywhere in the code that I have written. I have also confirmed it by attaching a debugger.

I will give an example using a simple get API request which returns static JSON.

contactDetails: function (req, res) {
    return res.json({
        support: {
            email: '[email protected]',
            phone: '0-1111-222-3333',
        },
        pricing: {
            email: '[email protected]',
            phone: '0-1111-222-3334',
        },
    });
}

where contactDetails is bound to get /api/v2/contact.

In my middleware (http.js in Sails.js framework), I have modified the res.json() function to log a line after it is done, in this way:

res.originalJson = res.json
res.json = function() {
    var ret = res.originalJson(...arguments)
    console.log("res.json() done")
}

When I encounter this issue, I can see in the logs res.json() done. So I know that my function has run completely. Once I restart this Node.js instance, everything works fine, until the time this issue recurs.

I have also taken a wireguard capture of two ports, one running an instance where this issue is occurring (8091) and another where this is not happening (8092). Here is the screenshot (I have removed source and destination columns as both are 127.0.0.1): wireguard capture comparison

Here, I can see that the whole data has already been pushed from the server to client (in 689 bytes sent in [PSH, ACK]) which the client has acknowledged. But in the instance where this issue is occurring, a final [PSH, ACK] is missing, which looks like this: wireguard capture of final PSH,ACK This leads me to believe that somehow the Node.js server is not closing the connection properly.

I also checked the active connections to port 8091 using sudo netstat -anp | grep :8091.

  1. When the client (curl) is hung on the request but after res.json() has been called:
tcp        0      0 127.0.0.1:45558         127.0.0.1:8091          ESTABLISHED 974453/curl         
tcp6       0      0 :::8091                 :::*                    LISTEN      874776/node         
tcp6       0      0 127.0.0.1:8091          127.0.0.1:45558         ESTABLISHED 874776/node
  1. After I kill the client:
tcp        0      0 127.0.0.1:45558         127.0.0.1:8091          TIME_WAIT   -                   
tcp6       0      0 :::8091                 :::*                    LISTEN      874776/node
  1. After about a couple of minutes:
tcp6       0      0 :::8091                 :::*                    LISTEN      874776/node

Can anybody tell me why this issue might be happening? Is there any limits that I need to increase? Let me know if you need more details.

0

There are 0 best solutions below