Filter logs fluent-bit on regex parsing

18 Views Asked by At

I have a fluentbit running that scrapes json logs from a dir:

[PARSER]
    Name        json
    Format      json
    Time_Key    time
    # Time_Format %llu
    Time_Keep   On
[PARSER]
    Name    extract
    Format  regex
    Skip_Empty_Values On
    Regex   ^(?<event>[^,]*),(?<app>[^,]*),(?<user>[^,]*)$
{"level":33,"time":1710879528,"msg":"app_opened,"test_dashboard",user1"}

Which works fine. The logs are now send upstream to get stored. However, it can happen that a faulty configured log message gets saved in the same dir, too. Fluent-bit, unfortunately, does not discard un-parsable logs but tries to dump those as well.

Coming to my question: How can I select only those logs msg that got correctly parsed?

Config.yaml:

service:
    flush: ${flush_interval}
    grace: ${flush_interval}
    log_level: info
    http_server: On
    http_listen: 0.0.0.0
    http_port: 2020

pipeline:
  inputs:
    - name: tail
      Read_from_Head: True
      Mem_Buf_Limit: 2mb
      parser: json
      tag: userlogs
      path: ${input_path}
  filters:
    - name: parser
      match: '*'
      parser: extract
      Key_Name: msg
      Reserve_Data: True
      Preserve_Key: False
  outputs:
    # - name: file
    #   match: login
    #   path: /var/log/userlogs_output
    - name: http
      match: '*'
      host: ${rec_host} 
      port: ${rec_port}
      uri: ${rec_endpoint}
      Retry_Limit: False
      format: json
      header_tag:  FLUENT-TAG
      log_response_payload: False
0

There are 0 best solutions below