I have 2 tables in MySQL like this
Table DEPARTMENT
| Id | Name |
|---|---|
| 1 | Department 1 |
| 2 | Department 2 |
Table STAFF
| Id | Department_Id | Name |
|---|---|---|
| 1 | 1 | Staff 1 |
| 2 | 1 | Staff 2 |
| 3 | 2 | Staff 3 |
| 4 | 1 | Staff 4 |
STAFFtable has about 10 million records.
All STAFF's informations has been pushed by Logstash to ElasticSearch. Each document in ElasticSearch now only have 3 fields are Staff_Id, Staff_Name and Department_Name. Something like this:
{
"Staff_Id": 1,
"Staff_Name": "Staff 1",
"Department_Name": "Department 1"
}
Because of practical needs, I need to add one more field called Department_Id to each document. Note that this field (Department_Id) does not exist on existing documents.
I am a newbie to both Logstash and ElasticSearch. How can I do this with Logstash? Interpreted in the SQL way would be:
- SELECT * FROM DEPARTMENT;
- UPDATE STAFF SET Department_Id = XXX WHERE Department_Name = YYY
Note that DEPARTMENT table has about 100.000 records and ElasticSearch has about 10 million documents.
Can you take a look?