Removing empty buckets from aggregation

272 Views Asked by At

I am trying to create a number of aggregations, where each contains only results with the field I filter on. I am able to do this, but the problem is that it also returns empty buckets and I am wanting to foreach over them - I do not want an action to occur on empty results.

I currently have something like this:

"aggs" : {
              "colour_agg" : {
                "filters": {
                  "filters": [
                    { "term" : { "COLOUR" : "Red" }},
                    { "term" : { "COLOUR" : "Blue" }},
                    { "term" : { "COLOUR" : "Yellow" }}
                  ]
                },
                "aggs" : {
                  "colour_agg" : {
                    "terms" : {
                      "field" : "COLOUR",
                      "min_doc_count" : 150
                    },
                    "aggs" : {
                      "hex_agg" : {
                        "terms" : {
                          "field" : "HEX"
                        }
                      }
                    }

which then returns this:

"colour_agg" : {
              "buckets" : [
                {
                  "doc_count" : 116,
                  "colour_agg" : {
                    "doc_count_error_upper_bound" : 0,
                    "sum_other_doc_count" : 0,
                    "buckets" : [ ]
                  }
                },
                {
                  "doc_count" : 256,
                  "colour_agg" : {
                    "doc_count_error_upper_bound" : 0,
                    "sum_other_doc_count" : 0,
                    "buckets" : [
                      {
                        "doc_count" : 256,
                        "key" : "Blue",
                        "hex_agg" : {
                          "doc_count_error_upper_bound" : 0,
                          "sum_other_doc_count" : 0,
                          "buckets" : [
                            {
                              "doc_count" : 256,
                              "key" : "0000ff"
                            }
                          ]
                        }
                      }
                    ]
                  }
                },
                {
                  "doc_count" : 0,
                  "event_agg" : {
                    "doc_count_error_upper_bound" : 0,
                    "sum_other_doc_count" : 0,
                    "buckets" : [ ]
                  }
                }
              ]
            }
          }

This means that when I loop over the colour_agg buckets, my action fires 3 times. In this scenario, I would want it to fire only once. I only want there to be a bucket created when there are 150 results found.

How can I remove the empty buckets from the colour_agg result? I have tried using min_doc_count at the filter level but this doesn't work, and by the time it is applied lower down the tip-level filter has already created empty buckets. I do not seem to be able to reverse this order, and have seen that post_filter (if a workable solution here) brings a performance hit with it.

Thanks.

Full watcher:

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": "ABC",
          "body": {
            "size": 0,
            "query": {
              "bool": {
                "must": { 
                  "match": { 
                    "COMPONENT": "69"
                  }
                },
                "filter": { 
                  "range": {
                    "@timestamp": {
                      "gte": "now-24h"
                    }
                  }
                }
              }
            },
            "aggs" : {
              "colours" : {
                "filters": {
                  "filters": [
                    { "term" : { "COLOUR" : "Red" }},
                    { "term" : { "COLOUR" : "Blue" }},
                    { "term" : { "COLOUR" : "Yellow" }}
                  ]
                },
                "aggs" : {
                  "colour_agg" : {
                    "terms" : {
                      "field" : "COLOUR",
                      "min_doc_count" : 150
                    },
                    "aggs" : {
                      "hex_agg" : {
                        "terms" : {
                          "field" : "HEX"
                        }
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "compare": {
        "ctx.payload.hits.total": {
          "gt": 0
        }
      }
    },
    "actions": {
    "teams_message" : {
    "foreach": "ctx.payload.aggregations.colours.buckets",
      ...
    }
  }
}
0

There are 0 best solutions below