How to create a new field with a substring of another field with logstash

49 Views Asked by At

I use filebeat to upload logs to elasticsearch and I wish to use logstash to transform the documents before sending them to ES. I'd like to add a field called my.service that would parse the value of the field log.file.path. Basically, what I want is the following :

"log.file.path": "/var/log/myservice.log" 
"my.service": "myservice"

I've tried the following filters on logstash but none seems to work :

filter {
    if ("" in  [log.file.path]){
        grok {
            match => { "log.file.path" => "%{GREEDYDATA}/(?<my.service>).log"}
        }
    }
}


filter {
    if ("" in  [log.file.path]){
        grok {
            match => { "log.file.path" => "%{GREEDYDATA}/%{GREEDYDATA:my.service}.log"}
        }
    }
}
1

There are 1 best solutions below

0
Val On

The problem are the dotted field name, in Logstash nested fields need to be specified by square brackets, just do it this way:

filter {
    if [log][file][path] {
        grok {
            match => { "[log][file][path]" => "%{GREEDYDATA}/%{GREEDYDATA:myservice}.log"}
        }
        mutate {
            rename => {
                "myservice" => "[my][service]"
            }
        }
    }
}

When the following JSON document comes in (as a one-liner)

{
  "log": {
    "file": {
      "path": "/var/log/myservice.log"
    }
  }
}

The output will be

{
  "log": {
    "file": {
      "path": "/var/log/myservice.log"
    }
  },
  "my": {
    "service": "myservice"
  }
}