I've been facing a probem since a while now. My filebeat plugin does not harvest the fields that i ask him to harvest in my conf file. I'm using filebeat 7.6.0
My conf:
filebeat.inputs:
- type: log
paths:
- /var/log/user.log
- /var/log/slapd.log
output.kafka:
hosts: ["kafka1:9092"]
topic: 'log'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 10000000
The log output:
|2020-02-14T07:55:58.664Z|INFO|crawler/crawler.go:72|Loading Inputs: 1,
|2020-02-14T07:55:58.665Z|INFO|log/input.go:152|Configured paths: [/var/log/user.log /var/log/slapd.log],|
|2020-02-14T07:55:58.665Z|INFO|input/input.go:114|Starting input of type: log; ID: 6297130742941599674 ,|
|2020-02-14T07:55:58.665Z|INFO|crawler/crawler.go:106|Loading and starting Inputs completed. Enabled inputs: 1,
|2020-02-14T07:56:00.664Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":250,"time":{"ms":8}},"total":{"ticks":390,"time":{"ms":16},"value":390},"user":{"ticks":140,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":6},"info":{"ephemeral_id":"a601336e-8252-460f-9a25-f05dad5851b2","uptime":{"ms":480275}},"memstats":{"gc_next":8594432,"memory_alloc":5169696,"memory_total":17158072},"runtime":{"goroutines":20}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":0.21,"15":1.06,"5":0.88,"norm":{"1":0.105,"15":0.53,"5":0.44}}}}}}
I've been trying to solve it by making it harvest various files. But without success so far. I always get this "filebeat":{"harvester":{"open_files":0,"running":0}}
Thanks!
Probably rule out that the issue is not with Kafka output. For testing, set up the FIlebeat file output and check whether you are getting any data or not.