pandas to convert sqs message to csv

511 Views Asked by At

I'm new to Dremio, and I was following this SQS + S3 + Dremiotutorial to learn more about Dremio. In one of the code snippets, it is mentioned that get_messages_from_queue will create a CSV file and which is later used in the upload_file method to upload into S3. However I'm missing that portion of the command which converts into CSV, can anyone help me how to create CSV using pandas? I'm new to Pandas, still learning.

SQS message body looks like this

"Body": "{\"holiday\":\"None\",\"temp\":288.28,\"rain_1h\":0.0,\"snow_1h\":0.0,\"clouds_all\":40,\"weather_main\":\"Clouds\",\"weather_description\":\"scattered clouds\",\"date_time\":\"2012-10-02 09:00:00\",\"traffic_volume\":5545}"
1

There are 1 best solutions below

2
Marios Nikolaou On

Add sqs message to dictionary and load it to panda Dataframe. Finally use to_csv to export csv file.

import pandas as pd

df = pd.DataFrame(sqs_message)
df.to_csv('sqs_messages.csv', index = False)  # pass path and file name of csv.