Importing Well-Structured JSON Data into ElasticSearch via Cloud Watch

Question

Is is there known science for getting JSON data logged via Cloud Watch imported into an Elasticsearch instance as well structured JSON?

That is -- I'm logging JSON data during the execution of an Amazon Lambda function.

This data is available via Amazon's Cloud Watch service.

I've been able to import this data into an elastic search instance using functionbeat, but the data comes in as an unstructured message.

        "_source" : {
          "@timestamp" : "xxx",
          "owner" : "xxx",
          "message_type" : "DATA_MESSAGE",
          "cloud" : {
            "provider" : "aws"
          },
          "message" : ""xxx xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx    INFO    {
  foo: true,
  duration_us: 19418,
  bar: 'BAZ',
  duration_ms: 19
}
""",

What I'm trying to do is get a document indexed into elastic that has a foo field, duration_us field, bar field, etc. Instead of one that has a plain text message field.

It seems like there are a few different ways to do this, but I'm wondering if there's a well trod path for this sort of thing using elastic's default tooling, or if I'm doomed to one more one-off hack.

Answer 1

Functionbeat is a good starting point and will allow you to keep it as "serverless" as possible.

To process the JSON, you can use the decode_json_fields processor.

The problem is that your message isn't really JSON though. Possible solutions I could think of:

1. A dissect processor that extracts the JSON message to pass it on to the decode_json_fields — both in the Functionbeat. I'm wondering if trim_chars couldn't be abused for that — trim any possible characters except for curly braces.
2. If that is not enough, you could do all the processing in Elasticsearch's Ingest pipeline where you probably stitch this together with a Grok processor and then the JSON processor.
3. Only log a JSON message if you can to make your life simpler; potentially move the log level into the JSON structure.