How do I ship the metrics of Elasticsearch (CPU utilization, cluster health, and others) on ElasticCloud to Google Cloud Monitoring?

Question

I'm using ElasticCloud (also known as Elasticsearch Service) on Google Cloud [1] as a search engine.

In my architecture, an application server is deployed on Google Cloud Run, and it communicates to Elasticsearch deployed by ElasticCloud.

Currently, I'm utilizing Google Cloud Monitoring [2] for the observability of the application server, and Stack Monitoring [3] for Elasticsearch.

I wish to integrate the metrics of Elasticsearch to Google Cloud Monitoring to look into the metrics of the application server and Elasticsearch together on the same dashboard.

However, as far as I investigated, Google Cloud Monitoring offers only two ways to fetch the metrics from Elasticsearch.

1. Installing "Ops agent" to the VM where Elasticsearch is running [4]
2. Construct a Kubernetes cluster with Elasticsearch Exporter and PodMonitoring custom resource [5]

The first option seems impossible because I'm not running Elasticsearch on my VM. The second option seems reasonable but too complicated because my system does not have any Kubernetes cluster.

Is there any other good solution to ship Elasticsearch metrics to Google Cloud Monitoring?

Any knowledge or ideas would help me a lot, thanks!

Answer 1

You might want to check Fluentd with Google Cloud Monitoring output plugin. This tool can be used to collect metrics from Elasticsearch and forward them to GCM.

You might also want to check an open-source tool like Telegraf or managed services like Datadog and Dynatrace