csv file convert to io.BytesIO object, then stream to blob storage ,meets value type error:a bytes-like object is required, not '_io.TextIOWrapper'

Question

I am trying to stream a csv to azure blob storage, the csv is generated directly from python scripts without local copy, i have the following code, df is the csv file:

    with open(df,'w') as f:
       stream = io.BytesIO(f)
       stream.seek(0)
       block_blob_service.create_blob_from_stream('flowshop', 'testdata123', stream)

then i got the error massage:

  stream = io.BytesIO(f)  TypeError: a bytes-like object is required, not '_io.TextIOWrapper'

i think the problem has been the format incorrect, can you please identify the problem. thanks.

Answer 1

You opened df for write, then tried to pass the resulting file object as the initializer of io.BytesIO (which is supposed to to take actual binary data, e.g. b'1234'). That's the cause of the TypeError; open files (read or write, text or binary) are not bytes or anything similar (bytearray, array.array('B'), mmap.mmap, etc.), so passing them to io.BytesIO makes no sense.

It looks like your goal is to read from df, and you shouldn't need io.BytesIO at all for that. Just change the mode from (text) write, 'w', to binary read, 'rb'. Then pass the resulting file object to your API directly:

with open(df, 'rb') as f:
   block_blob_service.create_blob_from_stream('flowshop', 'testdata123', f)

Update: Apparently df was your actual data, not a file name to open at all. Given that, you should really skip the stream API (which is pointless if the data is already in memory) and just use the bytes based API directly:

block_blob_service.create_blob_from_bytes('flowshop', 'testdata123', df)

or if df is str, not bytes:

block_blob_service.create_blob_from_bytes('flowshop', 'testdata123', df.encode('utf-8'))