I have installed td-agent and I am trying to upload data to Elasticsearch. Below is the td-agent.conf file:
<source>
@type tail
path /home/rocket/PycharmProjects/EFK/log.json
pos_file /home/rocket/PycharmProjects/EFK/log.json.pos
format json
time_format %Y-%m-%d %H:%M:%S
tag log
</source>
<match *log*>
@type elasticsearch
host 35.171.30.19
port 9200
user elastic
password XXXXXX
index_name test
</match>
Below is the error I am getting:
2023-01-30 14:13:47 +0000 [info]: starting fluentd-1.15.3 pid=5105 ruby="2.7.6"
2023-01-30 14:13:47 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2023-01-30 14:13:47 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:48 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:48 +0000 [info]: adding match pattern="*log*" type="elasticsearch"
2023-01-30 14:13:48 +0000 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-01-30 14:13:48 +0000 [error]: Worker 0 finished unexpectedly with status 2
2023-01-30 14:13:48 +0000 [info]: Received graceful stop
2023-01-30 14:13:49 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: parsing config file is succeeded path="/etc/td-agent/td-agent.conf"
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-calyptia-monitoring' version '0.1.3'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '5.2.4'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-flowcounter-simple' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-kafka' version '0.18.1'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-metrics-cmetrics' version '0.1.2'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-opensearch' version '1.0.8'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-prometheus' version '2.0.3'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-prometheus_pushgateway' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-record-modifier' version '2.1.1'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '2.4.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-s3' version '1.7.2'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-sd-dns' version '0.1.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-systemd' version '1.0.5'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-td' version '1.2.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-utmpx' version '0.5.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluent-plugin-webhdfs' version '1.5.0'
2023-01-30 14:13:49 +0000 [info]: gem 'fluentd' version '1.15.3'
2023-01-30 14:13:49 +0000 [info]: using configuration file: <ROOT>
<source>
@type tail
path "/home/rocket/PycharmProjects/EFK/log.json"
pos_file "/home/rocket/PycharmProjects/EFK/log.json.pos"
format json
time_format %Y-%m-%d %H:%M:%S
tag "log"
<parse>
time_format %Y-%m-%d %H:%M:%S
@type json
unmatched_lines
time_type string
</parse>
</source>
<match *log*>
@type elasticsearch
host "35.179.40.29"
port 9200
user "elastic"
password xxxxxx
index_name "test"
</match>
</ROOT>
2023-01-30 14:13:49 +0000 [info]: starting fluentd-1.15.3 pid=5116 ruby="2.7.6"
2023-01-30 14:13:49 +0000 [info]: spawn command to main: cmdline=["/opt/td-agent/bin/ruby", "-Eascii-8bit:ascii-8bit", "/opt/td-agent/bin/fluentd", "--log", "/var/log/td-agent/td-agent.log", "--daemon", "/var/run/td-agent/td-agent.pid", "--under-supervisor"]
2023-01-30 14:13:49 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: #0 init worker0 logger path=nil rotate_age=nil rotate_size=nil
2023-01-30 14:13:49 +0000 [info]: adding match pattern="*log*" type="elasticsearch"
2023-01-30 14:13:50 +0000 [error]: #0 config error file="/etc/td-agent/td-agent.conf" error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
2023-01-30 14:13:50 +0000 [error]: Worker 0 finished unexpectedly with status 2
2023-01-30 14:13:50 +0000 [info]: Received graceful stop
So the error says error_class=Fluent::ConfigError error="Using Elasticsearch client 8.4.0 is not compatible for your Elasticsearch server. Please check your using elasticsearch gem version and Elasticsearch server."
So it's an issue between the Elastic plugin version and Elasticsearch server version. But I am unable to find anywhere which version is supported and how to install it.
Below is how I have installed td-agent in Ubuntu 18.04.
curl -fsSL https://toolbelt.treasuredata.com/sh/install-ubuntu-bionic-td-agent4.sh | sh