Ingest Kafka csv files cell by cell

18 Views Asked by At

I have this csv structure dataset:

S1 S2 ... Sm
A1 1.6 2.4 ... 0.5
A2 1.8 0.1 ... 3.2
.. ... ... ... ...
An 0.3 4.1 ... 2.6

I want ingest a document for each cell in the csv like this to be ingested in kafka:

{ "sample_id": "Sm" "attribute:_id": "An" "expression": 2.6 }

This files have a size from 1Mb to 700Mb aprox.

What will be the best option to ingest this csv files in kafka to be indexed later in elasticsearch?

0

There are 0 best solutions below