Can OpenTelemetry Collector replace Filebeat and Metricbeat

1.3k Views Asked by At

I am in the process of implemented traces. We already have Filebeat for logs and Metricbeat for metrics. Filebeat and Metricbeat writes directly to Elasticsearch and are running as daemon sets.

I know the OPTL Collector can do Traces and Metric, but is it mature enough to handle logs? Also file and metric beat are very mature products, So is it save to replace them? It would be nice to only have one product to work with.

1

There are 1 best solutions below

0
Павел Лозовиков On

Opentelemetry collector can deal with logs. Here is link from NewRelic which shades some light on it: Enrich your logs with the OpenTelemetry Collector

Have it replace Filebeat and Metricbeat? Yes and no. Remember that all Beats are elk-targeted solutions which has features like adding tags or some ECS fileds (or your own one) to enrich your log documents. Using OTEL collector, you will send pure logs, but you probably using Logstash, which for sure expects documents which adheres to the specification in which they will be stored in ELK. Documents - the only way to store data in index. Read about it here: The Complete Guide to the ELK Stack ("Documents" section) So here you need to translate them which will take computational resources and unneeded effort.

Also, OTEL collector is a good way in case of separation of concerns - keeping there your telemetry data and backend receivers servers configs, to decouple it from your microservices. ELK-Beats solutions can be also treated such way, but again, they are more targeted to ELK ecosystem. p.s. Elasticsearch + Kibana having APM integration with OTEL colelctor, but I can not tell much here because I failed to connect it together :(

My final thought on opentelemetry to have 1 fits-all-solution: No, it is currently not possible and you probably do not need it. OTEL sdks is still partially in not very mature state, your microservices are also different and requiring different integrations and having different deployment procedures. What I can propose here is to try this and that, fail, write down your observations, fail again and again to have more mature understanding of what you more or less need. So, be able to instrument your systems with different tools and keep your hand on pulse of the whole Observability news - it's road only started ;)