I use Prometheus with blackbox exporter to monitor web health and SSL certificate health of multiple targets. Since I have a lot of targets, I put them to a blackbox.json file so I can keep config files clean. I add appropriate labels to the targets so prometheus knows which job to use. Now I need to add a few more targets (IP addresses) but some probes are failing because the target server refuses connections if no hostname specified. I can't specify the domain name instead of the IP because some targets are behind load balancers.
The job from prometheus.yml
- job_name: blackbox_http
file_sd_configs:
- files:
- /etc/prometheus/blackbox.json
metrics_path: /probe
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- module
target_label: instance
- source_labels:
- module
target_label: __param_module
- source_labels:
- __param_target
target_label: instance
- source_labels: [__name]
target_label: __param_name
- replacement: localhost:9115
target_label: __address__
Example of targets from blackbox.json
[
{
"labels": {
"module": "https_2xx",
"__shard": "sd",
"job": "https_2xx",
"project": "project_name",
"service": "service_name",
"VirtualServer": "some_identifier"
},
"targets": [ "https://10.10.0.1" ]
},
{
"labels": {
"module": "http_2xx",
"__shard": "nc",
"job": "http_2xx",
"project": "project_name",
"service": "service_name",
"VirtualServer": "another_identifier"
},
"targets": [ "http://10.10.0.2" ]
}
]
The module in blackbox.yml
modules:
https_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [200, 403, 401, 404]
method: GET
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
tls_config:
insecure_skip_verify: true
preferred_ip_protocol: "ip4"
http_2xx:
prober: http
timeout: 5s
http:
valid_status_codes: [200, 403, 401, 404]
method: GET
no_follow_redirects: false
fail_if_ssl: false
fail_if_not_ssl: false
preferred_ip_protocol: "ip4"
I tried to add another label to targets and change the job like this
{
"labels": {
"module": "http_2xx",
"__shard": "sd",
"job": "http_2xx",
"project": "project_name",
"service": "service_name",
"VirtualServer": "some_identifier",
"dns_name": "example.com"
},
"targets": [ "http://10.10.0.1" ]
}
The corresponding job
- job_name: blackbox_http
file_sd_configs:
- files:
- /etc/prometheus/blackbox.json
metrics_path: /probe
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- source_labels: [module]
target_label: __param_module
- source_labels: [dns_name]
target_label: __param_hostname
- target_label: __address__
replacement: localhost:9115
Basically, I need probes to take headers from labels and add them to http requests, so I can easily add other targets without changing configs all the time. How can I do that? Is that even possible?