dont alert using prometheus/alertmanager at particular time (maintance window) if machine goes down

Question

I have ec2 machine on which i have setup alerting (if it goes down) using tags promethues/alertmanager. The maintance is scheduled fot that machine at 6 am in the morning twice a week

I dont want to get alerts for those machines in that time. How can this be achieved ?

PD is used for alerting (i dont want to be done at service level).

Answer 1

As far as I know, there are three ways to avoid alerting at maintenance time:

1. Using service discovery: depending on the service discovery you are using, you can remove the services from Prometheus during the maintenance time. Then, they won't trigger any alert at Prometheus level.
2. Using silence in AlertManager: alert manager let you schedule silences that will automatically snooze alerts. Depending on your version or alert manager you may use the v1 API not very well documented or the v2 API that uses open API. This requires an external scheduler and a bit of tinkering (curl is usually enough).
3. Using inhibition: the principle is to define a rule that triggers a silent alert during maintenance time and use it for snoozing other alerts. See my answer in this question. This solution doesn't require external components but complexifies your configuration.

Which solution is the best depends on your use case and environment. Giving a few cases:

• If you use a dynamic service discovery (say consul), it should be part of the normal workflow and comes at no cost.

• If you are on a traditional setup with known servers and operations done by hand, then scripts to snooze alerts can be integrated in your playbook. It becomes part of the normal workflow of update.

• If you are in an environment where there is a strong separation between the team handling alerts and the ones doing upgrades, then using a Prometheus only solution, with well scheduled maintenance time, would probably better. The monitoring team being supposed to be expert on the tool and can handle the added complexity.