I was wondering if anyone out there who has had some extensive experience with Calyptia Fluentd might be able to help me with an issue I'm having. I've kind of reached a dead end.
I am running Calyptia Fluentd version 1.15.3 on four Rocky Linux 8.8 servers. The servers themselves are virtual servers not physical servers. Each server has two CPUs.
I have a simple configuration that only uses record_modifier filters. These filters simply split incoming log events into fields using a delimiter.
I'm seeing an issue where the CPU utilization(according to the "top" command) slowly but steadily increases over time. When Calyptia Fluentd is first started up on the servers, the CPU utilization is around 1% on each server. By the end of the day it will be at around 10%. By the end of the next day it will be at around 20%. After a number of days it will eventually reach 100% CPU utilization.
Does anyone have any thoughts/ideas as to why this could be happening? To deal with this issue for the time being we have setup cron jobs to periodically restart Calyptia Fluentd on the servers but obviously that is more of a temporary "hack" and not a permanent solution.