Using a search template in an ingest pipeline

Question

Can an ElasticSearch ingest pipeline use a search template as its script?

Specifically, I'd like to configure an ingest pipeline so that whenever data of a particular type come in, we query ElasticSearch for some related data, and populate a field (or fields) on the original data based on the results, before loading it in.

I see that ingest pipelines can use scripts (https://www.elastic.co/guide/en/elasticsearch/reference/master/script-processor.html), and that scripts can include search templates (https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html), but I haven't been able to find any information on potentially combining the two.

Answer 1

Actually, the feature you're asking for is coming in 7.5 via the new enrich processor, which kind of provides index-time JOIN capability.

The main idea is to set up an enrich policy that will source data from your related indexes into a new "enrich index" and then you can leverage that "enrich index" in your ingest pipeline using an enrich processor in order to enrich your documents with related fields.

So, without going in too many details, here is how it works in practice:

1. You have an index A with fields (a, b, c, d) that you'd like to use for enriching your incoming documents
2. You define an enrich policy based on that index A and the "join" field a
3. You define an ingest pipeline with an enrich processor that will try to match field z of the incoming document against field A.a of the enrich index
4. If a match is found, your incoming document will get fields b, c and d from the index A. Note that it will also get the match field a that you can remove using a remove processor if needed.

That's pretty much what you expect. You can find a complete example here. At the beginning, it will work for exact matches (i.e. term query) and geo matches (i.e. geo_shape query), but they will probably add new kind of matches (like range matches) in the near future.

Answer 2

So the ingest script pipeline is already the combination your looking for, you should use the if option, read about it here.

Your pipeline should look something like this:

{
    "script": {
        "if": "ctx.type == 'thisType'",
        "source": """
                  //calculation here
                  ctx.newField = value;
                  """
    }
}

I recommend after you create your pipeline you test it using simulate as it will make your life much easier.

EDIT:

Thanks to @val that cleared up some confusion on my part. So you can't really do exactly what you want however i recommend you read about enrich pipeline. with some setup you might be able to make it happen.