Nodejs log4js on AWS ec2 stopped writing to log, but app is still running

27 Views Asked by At

We have a nodejs app with log4js and pm2 on an EC2 instance. the logs are compressed each hour if they reach X size by the logrotate service.

Last week we had a problem, the app seems to be running - but there is no logs (one minute before it worked well). the logrotate service compiles an empty file each hour instead of compressing a big file. We think it happens after users upload files, but we couldn't reproduce it (we have some logic for resizing files in our app).

When we make requests to the app, it returns the correct response but writes nothing to the log file!

The problem was solved only after terminating the instance and uploading another one by AutoScaling.

PM2:

PORT=8081 pm2 start server.js --name api -o /dev/null -e /dev/null

LOG4JS:

const log4js = require('log4js');
const logDirectory = require('config').get('log.directoryPath');

log4js.configure({
    appenders: {
        outLog: {
            type: 'file',
            filename: `${logDirectory}/api-out.log`,
            layout: {
                type: 'pattern',
                pattern: '%[[%d{dd/MM/yy hh:mm:ss.SSS}]%] %m'
            }
        },
        errorLog: {
            type: 'file',
            filename: `${logDirectory}/api-error.log`,
            layout: {
                type: 'pattern',
                pattern: '%[[%d{dd/MM/yy hh:mm:ss.SSS}]%] %m'
            }
        },
        errors: {
            type: 'logLevelFilter',
            level: 'error',
            appender: 'errorLog'
        }
    },
    categories: {
        default: {
            appenders: ['outLog', 'errors'],
            level: 'debug'
        }
    }
});

LOGROTATE:

               missingok
               notifempty
               rotate 5
               sharedscripts
               compress
               copytruncate
               dateext
               dateformat -%Y-%m-%d-%s+
               olddir /var/log/nodejs/rotated
               lastaction
                 BUCKET="prod-logs"
                 ......
                 read DAY MONTH YEAR <<< `date "+%d %m %Y"`
                 FORMAT=`date "+%Y-%m-%d"`
                 aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/...../" --region $REGION --exclude "*" --include "*.log-$FORMAT*"
               endscript
                     
               /var/log/nodejs/*out.log {
                 size 10M
               }

               /var/log/nodejs/*error.log {
                 size 2M
               }

Can someone help here? any idea?

0

There are 0 best solutions below