I have a type of remote app (.NET 8 console apps) that I want a Prometheus instance to discover/scrape.
- Apps are short- and long-lived (seconds to days)
- Many instances can run simultaneously
- Are started manually
- May run on different networks
- Can crash, be stopped or complete
- Can expose a Prometheus metrics endpoint: prometheus-net
What is the appropriate architecture to discover/scrape these app instances?
I tried using PushGateway, however, it is unsuitable due to it never forgetting series pushed to it and exposing them indefinitely to Prometheus. When an app instance is closed (either crashed, stopped or completed), I want some of its metrics to be forgotten.
Potential solution: Use PushGateway and utilize push_time_seconds to disregard stale data (as suggested by @markalex).