I am trying to use filebeat, logstash, kibana and elasticsearch to read a log file on my windows laptop. I have docker for desktop installed. Please find below my project directory:
The docker compose file is shown below:
version: '3'
services:
elasticsearch:
image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
container_name: elasticsearch
environment:
- discovery.type=single-node
- bootstrap.memory_lock=true
- "ES_JAVA_OPTS=-Xms512m -Xmx512m"
- xpack.security.enabled=false
- network.host=0.0.0.0
ulimits:
memlock:
soft: -1
hard: -1
stdin_open: true
ports:
- 9200:9200
- 9300:9300
networks:
- elk
kibana:
image: docker.elastic.co/kibana/kibana:8.12.2
container_name: kibana
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
ports:
- "5601:5601"
networks:
- elk
depends_on:
- elasticsearch
logstash:
image: docker.elastic.co/logstash/logstash:8.12.2
container_name: logstash
environment:
- ELASTICSEARCH_HOSTS=http://elasticsearch:9200
volumes:
- ./logstash/config/logstash.conf:/usr/share/logstash/pipeline/logstash.conf
command: logstash -f /usr/share/logstash/config/logstash.conf
networks:
- elk
depends_on:
- elasticsearch
filebeat:
image: docker.elastic.co/beats/filebeat:8.12.2
command: filebeat -e -strict.perms=false
container_name: filebeat
user: root
volumes:
- ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml
- /var/run/docker.sock:/var/run/docker.sock
- /var/lib/docker/containers:/var/lib/docker/containers
- /var/log:/var/log
networks:
- elk
depends_on:
- elasticsearch
networks:
elk:
driver: bridge
The filebeat.yml is shown below:
filebeat.inputs:
- type: log
enabled: true
paths:
- D:/Project/elk/elk-metric-demo/logs/metric-app.log
output.logstash:
hosts: ["logstash:5000"]
The logstash.conf is shown below:
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "%{[@metadata][beat]}-%{[@metadata][version]}"
}
}
When I perform a docker compose up command all service start except logstash with the following error message:
2024-03-23 11:53:47 [2024-03-23T07:53:47,472][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused>}
2024-03-23 11:53:47 [2024-03-23T07:53:47,475][WARN ][logstash.licensechecker.licensereader] Attempted to resurrect connection to dead ES instance, but got an error {:url=>"http://elasticsearch:9200/", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError, :message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused"}
2024-03-23 11:53:47 [2024-03-23T07:53:47,483][INFO ][logstash.licensechecker.licensereader] Failed to perform request {:message=>"Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused", :exception=>Manticore::SocketException, :cause=>#<Java::OrgApacheHttpConn::HttpHostConnectException: Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused>}
2024-03-23 11:53:47 [2024-03-23T07:53:47,484][WARN ][logstash.licensechecker.licensereader] Marking url as dead. Last error: [LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError] Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused {:url=>http://elasticsearch:9200/, :error_message=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused", :error_class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::HostUnreachableError"}
2024-03-23 11:53:47 [2024-03-23T07:53:47,486][WARN ][logstash.licensechecker.licensereader] Attempt to fetch Elasticsearch cluster info failed. Sleeping for 0.02 {:fail_count=>1, :exception=>"Elasticsearch Unreachable: [http://elasticsearch:9200/][Manticore::SocketException] Connect to elasticsearch:9200 [elasticsearch/172.25.0.2] failed: Connection refused"}
2024-03-23 11:53:47 [2024-03-23T07:53:47,507][ERROR][logstash.licensechecker.licensereader] Unable to retrieve Elasticsearch cluster info. {:message=>"No Available connections", :exception=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::NoConnectionAvailableError}
2024-03-23 11:53:47 [2024-03-23T07:53:47,508][ERROR][logstash.licensechecker.licensereader] Unable to retrieve license information from license server {:message=>"No Available connections"}
2024-03-23 11:53:47 [2024-03-23T07:53:47,524][ERROR][logstash.monitoring.internalpipelinesource] Failed to fetch X-Pack information from Elasticsearch. This is likely due to failure to reach a live Elasticsearch cluster.
2024-03-23 11:53:47 [2024-03-23T07:53:47,533][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/usr/share/logstash/config/logstash.conf"}
2024-03-23 11:53:47 [2024-03-23T07:53:47,534][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-03-23 11:53:47 [2024-03-23T07:53:47,594][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2024-03-23 11:53:47 [2024-03-23T07:53:47,603][INFO ][logstash.runner ] Logstash shut down.
2024-03-23 11:53:47 [2024-03-23T07:53:47,610][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
2024-03-23 11:53:47 org.jruby.exceptions.SystemExit: (SystemExit) exit
2024-03-23 11:53:47 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:808) ~[jruby.jar:?]
2024-03-23 11:53:47 at org.jruby.RubyKernel.exit(org/jruby/RubyKernel.java:767) ~[jruby.jar:?]
2024-03-23 11:53:47 at usr.share.logstash.lib.bootstrap.environment.<main>(/usr/share/logstash/lib/bootstrap/environment.rb:90) ~[?:?]
2024-03-24 05:23:07 Using bundled JDK: /usr/share/logstash/jdk
...
2024-03-24 05:23:38 [2024-03-24T01:23:38,849][INFO ][logstash.config.source.local.configpathloader] No config files found in path {:path=>"/usr/share/logstash/config/logstash.conf"}
2024-03-24 05:23:38 [2024-03-24T01:23:38,851][ERROR][logstash.config.sourceloader] No configuration found in the configured sources.
2024-03-24 05:23:38 [2024-03-24T01:23:38,936][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600, :ssl_enabled=>false}
2024-03-24 05:23:38 [2024-03-24T01:23:38,955][INFO ][logstash.runner ] Logstash shut down.
2024-03-24 05:23:38 [2024-03-24T01:23:38,966][FATAL][org.logstash.Logstash ] Logstash stopped processing because of an error: (SystemExit) exit
2024-03-24 05:23:38 org.jruby.exceptions.SystemExit: (SystemExit) exit
While starting logstash in the terminal I can get access to elastic search as shown below:
However the logstash container cannot start successfully because it is unable to connect. Any advice why i am doing wrong please?