I have been working with the Kubernetes service discovery configuration (kubernetes_sd_configs)
in Promtail very successfully to pull the stderr and stdout streams into Loki for display on Grafana dashboards.
However it seems that if I want to watch specific log files within the containers (/var/log/apps/error.log) I have to use a static_configs and specify the targets explicitly.
I have been able to get around this by running Promtail as a sidecar in each pod, pushing the specified files back to the loki instance.
I also saw a workaround that involved using a pv to write the logs to the node however this is not possible as I am limited to one namespace within a multi tenant cluster.
So my requirement is to combine the service discovery configuration of promtail with the static configs ability to read specific log files within the pod. Has anyone had any success in this area?
You can use
kubernetes_sd_config, it can discover the log files on k8 nodes, but first you need to mount the collecting logs volumes onto the host.