I can't collect logs with Fluent Bit

140 Views Asked by At

I want to use the "td-agent-bit" service to collect the logs of a node application (app1) into a log file (namely "/etc/td-agent-bit/file.log"). In the "td-agent-bit.conf" file I left the [SERVICE] section unchanged, but I changed the [INPUT] and [OUTPUT] sections:

...
[INPUT]
    Name forward
    Listen localhost
    Port 24224

[OUTPUT]
    Name        file
    Match       *
    Path        /etc/td-agent-bit/file.log

The "file.log" exists and has all the permissions because I launched the command

sudo chmod 777 /etc/td-agent-bit/file.log

The "td-agent-bit" service runs properly. I cheked its status through the following command:

sudo service td-agent-bit status

The problem is that after running app1 the log file remains empty. This is the code of app1:

const FluentLogger = require('fluent-logger');

// Initialize Fluent Logger
FluentLogger.configure('app1', {
    host: 'localhost', // Fluent Bit host
    port: 24224, // Fluent Bit port
    timeout: 3.0, // Timeout in seconds
});

// Send log messages
FluentLogger.emit('mytag', { message: 'First log' }, () => {
    console.log('First log sent successfully');
});

FluentLogger.emit('mytag', { message: 'Second log' }, () => {
    console.log('Second log sent successfully');
});

// Close the connection after 5 seconds
setTimeout(() => {
    FluentLogger.end(); // Close the connection
    console.log('Connection closed');
}, 5000);
1

There are 1 best solutions below

0
nix86 On

My approach was wrong. The application must write its logs to a file, not directly to Fluent Bit. This is a program that writes its logs into a file. I produces 5 logs, one per second.

const fs = require(`fs`);

var i = 0;
var log;
const interval = setInterval(() => {
  if (i < 5) {
    log = `log number ${i}`;
    console.log(log);
    fs.appendFileSync(`input.log`, `${log}\n`);
    i++;
  } else {
    clearInterval(interval); //Stop the interval when it reaches 5 logs
  }
}, 1000);

Then the input and output sections of the "td-agent-bit.conf" file must be set like this:

[INPUT]
    Name              tail
    Tag               mylogs
    Path              /home/fluent/app1/input.log
    Mem_Buf_Limit     5MB
    Skip_Long_Lines   Off
    Refresh_Interval  10

[OUTPUT]
    Name        file
    Match       *
    Path        /etc/td-agent-bit/logs #This is the path of a directory, not a file

The Path property of the input must be set to the file into which the application writes it logs. The path property of the output must be set to the directory, not the file, into which Fluent Bit will rewrite the logs.