Prometheus not reading blackbox targets from file_sd_configs

Question

I'm configuring blackbox for monitoring my websites. Prometheus static config targets are working but are also little bit messy. I would to put all my targets into file but its not working at all.

docker-compose:

version: '2.1'

volumes:
  prometheus_data: {}

services:
  prometheus:
    mem_limit: 1000m
    image: prom/prometheus
    container_name: prometheus
    volumes:
      - ./prometheus/:/etc/prometheus/
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=200h'
    links:
      - 'blackbox:blackbox'
    expose:
      - 9090
    labels:
      container_group: monitoring

  blackbox:
    image: prom/blackbox-exporter
    container_name: blackbox
    expose:
      - 9115
    volumes:
      - ./blackbox/:/etc/blackbox/
    command: --config.file=/etc/blackbox/blackbox.yml
    labels:
      container_group: monitoring

prometheus.yml

scrape_configs:

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
       file_sd_configs:
      - files: ['/blackbox/blackbox_targets.yml']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox:9115

blackbox_targets.yml

- targets: ['http://google.com']
  labels:
    group: 'localhost'
    instance: 'localhost'

Without blackbox_targets.yml can see targets in prometheus bat leter on are missing.

Answer 1

In your prometheus.yml config file, the file location /blackbox/blackbox_targets.yml is invalid. Depending on where the file actually lives on the host comnputer, you need to make sure that a) it's mapped into the container and b) that you use the right path in the config file.

For instance, if the file lives in ./prometheus/ folder on your host computer then the path should be /etc/prometheus/blackbox_targets.yml instead.

Answer 2

I resolved it using the below configuration in values.yaml file of kube-prometheus-stack.

prometheusSpec:
     configMaps: ['node-cm']
     additionalScrapeConfigs:
      -   job_name: 'node-metrics'
          file_sd_configs:
          - files:
             - '/etc/prometheus/configmaps/node-cm/targets.json'

Initially, i was using file_sd_config inside scrape_configs instead of additionalScrapeConfigs and it was not working.