Is it possible to take action on an application using datadog or Prometheus?

Question

I want to take some action on the app server based on the server utilisation. The monitoring on the server is done by datadog. So is it possible to take an action on the server using datadog ?

Answer 1

The datadog agent you deployed has no power to run scripts or take action. It is purely a monitoring/data collection tool.

---

However one of the things your monitors in the Datadog application can do is trigger events when they go into an alert state. There are lots of integrations: creating a ticket in Jira, posting a message to Slack, triggering an SNS topic.

What I recommend you try to do, is to create some kind of job or script that can be triggered externally, like a lambda function, or a jenkins job, or anything really. When the monitor goes off you can use a webhook to trigger that script to do whatever you define. Here is a blog post showing how twilio sent out a text message by connecting their api to a webhook.

Answer 2

You can configure Prometheus alertmanager. This can give a callback to webhook configured and this webhook can then execute action upon invocation.

https://prometheus.io/docs/alerting/latest/configuration/