We have a nodejs app with log4js and pm2 on an EC2 instance. the logs are compressed each hour if they reach X size by the logrotate service.
Last week we had a problem, the app seems to be running - but there is no logs (one minute before it worked well). the logrotate service compiles an empty file each hour instead of compressing a big file. We think it happens after users upload files, but we couldn't reproduce it (we have some logic for resizing files in our app).
When we make requests to the app, it returns the correct response but writes nothing to the log file!
The problem was solved only after terminating the instance and uploading another one by AutoScaling.
PM2:
PORT=8081 pm2 start server.js --name api -o /dev/null -e /dev/null
LOG4JS:
const log4js = require('log4js');
const logDirectory = require('config').get('log.directoryPath');
log4js.configure({
appenders: {
outLog: {
type: 'file',
filename: `${logDirectory}/api-out.log`,
layout: {
type: 'pattern',
pattern: '%[[%d{dd/MM/yy hh:mm:ss.SSS}]%] %m'
}
},
errorLog: {
type: 'file',
filename: `${logDirectory}/api-error.log`,
layout: {
type: 'pattern',
pattern: '%[[%d{dd/MM/yy hh:mm:ss.SSS}]%] %m'
}
},
errors: {
type: 'logLevelFilter',
level: 'error',
appender: 'errorLog'
}
},
categories: {
default: {
appenders: ['outLog', 'errors'],
level: 'debug'
}
}
});
LOGROTATE:
missingok
notifempty
rotate 5
sharedscripts
compress
copytruncate
dateext
dateformat -%Y-%m-%d-%s+
olddir /var/log/nodejs/rotated
lastaction
BUCKET="prod-logs"
......
read DAY MONTH YEAR <<< `date "+%d %m %Y"`
FORMAT=`date "+%Y-%m-%d"`
aws s3 sync /var/log/nodejs/rotated/ "s3://$BUCKET/...../" --region $REGION --exclude "*" --include "*.log-$FORMAT*"
endscript
/var/log/nodejs/*out.log {
size 10M
}
/var/log/nodejs/*error.log {
size 2M
}
Can someone help here? any idea?