Trying to suppress Sensu alarms being resolve 30 secs after triggered

Question

I have a Sensu deployed connected to pager duty. I can see that there are some Sensu alarms that are triggered but getting resolved after 30 seconds. Is there any way to "really" trigger an alarm only if it not resolved after [60 seconds]?

Answer 1

Sounds like a prime case for occurrence filtering in Sensu. Since those filters are provided natively (see the docs here), you could just set your config to something like:

{
  "handlers": {
    "email": {
      "type": "pipe",
      "command": "pagerdurty.rb",
      "filters": ["occurrences"]
    }
  }
}

And your check config like:

{
  "checks": {
    "check-http": {
      "command": "check-http.rb -u https://localhost:8080/api/v1/health",
      "subscribers": ["web_application"],
      "interval": 30,
      "handlers": ["pagerduty"],
      "occurrences": 2,
      "refresh": 500
    }
  }
}

And that would do what you'd need.