How to add http headers to blackbox exporter probes using relabeling

1k Views Asked by At

I use Prometheus with blackbox exporter to monitor web health and SSL certificate health of multiple targets. Since I have a lot of targets, I put them to a blackbox.json file so I can keep config files clean. I add appropriate labels to the targets so prometheus knows which job to use. Now I need to add a few more targets (IP addresses) but some probes are failing because the target server refuses connections if no hostname specified. I can't specify the domain name instead of the IP because some targets are behind load balancers.

The job from prometheus.yml

  - job_name: blackbox_http
    file_sd_configs:
      - files:
          - /etc/prometheus/blackbox.json
    metrics_path: /probe
    relabel_configs:
      - source_labels:
          - __address__
        target_label: __param_target
      - source_labels:
          - module
        target_label: instance
      - source_labels:
          - module
        target_label: __param_module
      - source_labels:
          - __param_target
        target_label: instance
      - source_labels: [__name]
        target_label: __param_name
      - replacement: localhost:9115
        target_label: __address__

Example of targets from blackbox.json

[
  {
    "labels": {
      "module": "https_2xx",
      "__shard": "sd",
      "job": "https_2xx",
      "project": "project_name",
      "service": "service_name",
      "VirtualServer": "some_identifier"
    },
    "targets": [ "https://10.10.0.1" ]
  },
  {
    "labels": {
      "module": "http_2xx",
      "__shard": "nc",
      "job": "http_2xx",
      "project": "project_name",
      "service": "service_name",
      "VirtualServer": "another_identifier"
    },
    "targets": [ "http://10.10.0.2" ]
  }
]

The module in blackbox.yml

modules:
   https_2xx:
     prober: http
     timeout: 5s
     http:
       valid_status_codes: [200, 403, 401, 404]
       method: GET
       no_follow_redirects: false
       fail_if_ssl: false
       fail_if_not_ssl: false
       tls_config:
         insecure_skip_verify: true
       preferred_ip_protocol: "ip4"
   http_2xx:
     prober: http
     timeout: 5s
     http:
       valid_status_codes: [200, 403, 401, 404]
       method: GET
       no_follow_redirects: false
       fail_if_ssl: false
       fail_if_not_ssl: false
       preferred_ip_protocol: "ip4"

I tried to add another label to targets and change the job like this

  {
    "labels": {
      "module": "http_2xx",
      "__shard": "sd",
      "job": "http_2xx",
      "project": "project_name",
      "service": "service_name",
      "VirtualServer": "some_identifier",
      "dns_name": "example.com"
    },
    "targets": [ "http://10.10.0.1" ]
  } 

The corresponding job

  - job_name: blackbox_http
    file_sd_configs:
      - files:
          - /etc/prometheus/blackbox.json
    metrics_path: /probe
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - source_labels: [module]
        target_label: __param_module
      - source_labels: [dns_name]
        target_label: __param_hostname
      - target_label: __address__
        replacement: localhost:9115

Basically, I need probes to take headers from labels and add them to http requests, so I can easily add other targets without changing configs all the time. How can I do that? Is that even possible?

0

There are 0 best solutions below