I have this csv structure dataset:
| S1 | S2 | ... | Sm | |
|---|---|---|---|---|
| A1 | 1.6 | 2.4 | ... | 0.5 |
| A2 | 1.8 | 0.1 | ... | 3.2 |
| .. | ... | ... | ... | ... |
| An | 0.3 | 4.1 | ... | 2.6 |
I want ingest a document for each cell in the csv like this to be ingested in kafka:
{ "sample_id": "Sm" "attribute:_id": "An" "expression": 2.6 }
This files have a size from 1Mb to 700Mb aprox.
What will be the best option to ingest this csv files in kafka to be indexed later in elasticsearch?