loki promtail configuration changes to exclude all level=info lines from all pods except few

Question

I need help with promtail configuration where I want to drop all "level=info" lines from all pods, except 2 pods where all lines are needed including level=info.

Can someone please help me with it? I am very new to loki-stack and I am not sure how the configuration works and what is the precedence if I add multiple rules in scrape configs.

Thanks in advance.

Answer 1

Finally I was able to achieve this :

snippets:
  pipelineStages:
    - match:
        selector: '{app!="<podname>"} |~ "^.*level=info .*$"'
        #except for one pod, drop all lines with string level=info.
        action: drop
    - match:
        selector: '{app="<podname>"} |~ "^.*\\ \"[A-Za-z]+\\ .*HTTP.*\"\\ [2,3]\\d\\d .*$"'
        #In one specific pod, drop all log lines with with successful 2xx/3xx response codes for calls.
        action: drop

Referred : https://grafana.com/docs/loki/latest/send-data/promtail/stages/ But the documentation should have stated that we can use logql with selector to get the desired filtering. Since I was new to grafana/loki/promtail stack, I didn't recognise it at first from the documentation. It took me sometime to figure it out.

However, the error thrown by promtail is very very bad and does not reflect the exact problem at all. I forgot to add escape char for " in regex expression, and the error I got was :

caller=main.go:170 msg="error creating promtail" error="failed to make file target manager: invalid match stage config: invalid selector syntax for match stage: parse error at line 1, col 31: syntax error: unexpected IDENTIFIER, expecting != or !~ or |~ or |="

After wasting couple of hours I realised this mistake and fixed it.

PS: Not sure why the question was marked for close?