Is there a way to prioritize and send logs to elasticsearch inorder to optimize the log load on the server

Question

Need to read critical, error and warning in all application logs in my environment. If a critical happens after 1000 warnings, I want the Logstash to send it right away, giving priority order highest for critical, then for error and least for the warnings generated.

What are the options available here and How can it be done?

• Is there an inbuilt feature in Logstash for it?
• Is there another integration that can help achieve it? What about RabbitMQ? Can RabbitMQ directly send it to elastic search without Logstash?
• Is there another module of elastic or any other software that can be integrated to elastic search to achieve this end goal?

Preference for lightweight and fast solution.

I tried optimizing Logstash. Heard About RabbitMQ (Like to try it)

Answer 1

Logstash itself doesn't have built-in prioritization to send the logs. Basically, it works like first in first out.

RabbitMQ can be used as a message broker to decouple log producers from log consumers. You can send logs to RabbitMQ and then consume and process them based on their severity.

Kafka acts as a distributed message broker. It can handle large volumes of logs and provides fault tolerance. You can create Kafka topics for each severity level to maintain separation.

Kafka vs. RabbitMQ: Both Kafka and RabbitMQ are message brokers, but Kafka is often chosen for its durability, fault tolerance, and support for high-throughput scenarios. RabbitMQ, on the other hand, might be simpler to set up initially and may be preferred in scenarios where guaranteed message delivery is critical.

Sample Logstash conf for rabbitmq input and elasticsearch output:

input {
  rabbitmq {
    id => "input_rabbitmq"
    exchange => "logstash-exchange"
    threads => 2
  }
}
filter {}
output {
    elasticsearch {
        hosts => "localhost:9200"
        data_stream => "true"
    }
}

Question: filebeat => Logstash => Elasticsearch data pipeline sends the document near-real time. Are you experiencing any delay between the log generation and indexing?