Threshold value for Google monitoring alert policy

149 Views Asked by At

I have created Google Monitoring alert policy for Google monitoring SLO( 95% to total https requests for 7 days should have less than 5s latency and should receive alert when burn_rate consumes 2% of total 5% failure allowed in 24 hours.)

question- Not sure What should be exact threshold_value for alert policy. I have set it to 2.5 now.

source code-

resource "google_monitoring_custom_service" "lb-backend-latency" {
  service_id = "lb-backend-latency"
  project = var.project_id
  display_name = "Load Balancer Backend Latency service for SLO"
}
resource "google_monitoring_slo" "lb_backend_latency_slo" {
  project = var.project_id
  service = google_monitoring_custom_service.lb-backend-latency.service_id
  slo_id = "slo-number-of-requests-faster-than-5s"
  display_name = "95 % of valid HTTPS requests served faster than 5s"

  goal = 0.95
  rolling_period_days = 7

  request_based_sli {
    distribution_cut {
          distribution_filter = "metric.type=\"loadbalancing.googleapis.com/https/backend_latencies\" resource.type=\"https_lb_rule\" metric.label.\"response_code_class\"=\"200\" resource.label.\"backend_target_name\"=\"${var.backend_name}\""
          range {
            max = 5000
            min = 0
          }
       }
  }
user_labels = {
    slo_label       = "slo-1"
  }
}

#monitoring alert policy

resource "google_monitoring_alert_policy" "lb_backend_latency_slo_alert" {
  project               = var.project_id
  combiner              = "OR"
  display_name          = "lb_backend_latency_slo_alert_policy"
  notification_channels = [ google_monitoring_notification_channel.sl3_notification_channel.id]
  conditions {
    display_name = "Burn Rate condition for response total latency SLO"
    condition_threshold {
      filter          = "select_slo_burn_rate(\"projects/${var.project_id}/services/${google_monitoring_custom_service.lb-backend-latency.service_id}/serviceLevelObjectives/${google_monitoring_slo.lb_backend_latency_slo.slo_id}\",\"3600s\")"
      duration        = "300s"
      comparison      = "COMPARISON_GT"
      threshold_value = 2.5
    }
  }
}

tried to put threshold value = 2.5 value. but not sure how to calculate exact threshold value in case of this alert

0

There are 0 best solutions below