Logstash bulk update documens in ElasticSearch with WHERE conditon

79 Views Asked by At

I have 2 tables in MySQL like this

Table DEPARTMENT

Id Name
1 Department 1
2 Department 2

Table STAFF

Id Department_Id Name
1 1 Staff 1
2 1 Staff 2
3 2 Staff 3
4 1 Staff 4
  • STAFF table has about 10 million records.

All STAFF's informations has been pushed by Logstash to ElasticSearch. Each document in ElasticSearch now only have 3 fields are Staff_Id, Staff_Name and Department_Name. Something like this:

{
    "Staff_Id": 1,
    "Staff_Name": "Staff 1",
    "Department_Name": "Department 1"
}

Because of practical needs, I need to add one more field called Department_Id to each document. Note that this field (Department_Id) does not exist on existing documents.

I am a newbie to both Logstash and ElasticSearch. How can I do this with Logstash? Interpreted in the SQL way would be:

  • SELECT * FROM DEPARTMENT;
  • UPDATE STAFF SET Department_Id = XXX WHERE Department_Name = YYY

Note that DEPARTMENT table has about 100.000 records and ElasticSearch has about 10 million documents. Can you take a look?

0

There are 0 best solutions below