Python Azure Blob Direct Upload

Question

I am trying to read a csv from Azure blob into Python as a stream and write it back to Azure blob directly. Read operation works perfectly fine butwriting output stream just writes an empty file into the blob. The following code works until print(df) but not after that.

Below is the code:

Code:

from io import BytesIO, StringIO

with BytesIO() as input_blob:   

  with BytesIO() as output_blob:

    block_blob_service = BlockBlobService(account_name='aaaccc', account_key='*/*/*--')

    block_blob_service.get_blob_to_stream('test', 'Source.csv', input_blob)

    input_blob.seek(0)

    df=pd.read_csv(input_blob)

    print(df)

    copyfileobj(input_blob, output_blob)

    block_blob_service.create_blob_from_stream('test', 'OutFilePy.csv', output_blob)

what does copyfileobj function do? can you edit your question and paste the definition of that? — Saher Ahwal
– Saher Ahwal, Commented Apr 24, 2018 at 18:26
@Saher, copyfileobj copies the input stream to output stream.. I am trying to write the copied output_blob into blob storage but it just writes an empty file — AngiSen
– AngiSen, Commented Apr 25, 2018 at 1:26
I think the problem may be input_blob's cursor is at EOF after pd.read_csv. A input_blob.seek(0) after read_csv maybe helpful. — Sraw
– Sraw, Commented Apr 25, 2018 at 1:32
Fantastic Sraw, it works!! one last question... what if i have to make some operations on dataframe and write the modified dataframe as stream into blob? — AngiSen
– AngiSen, Commented Apr 25, 2018 at 1:49
@Sraw You could move your comment to the answer to help more community members to find, thanks. — Joy Wang
– Joy Wang, Commented Apr 25, 2018 at 2:11

Sraw · Accepted Answer · 2018-05-02 06:00:26Z

5

The problem is that after pd.read_csv, the cursor of input_blob is at EOF. So copyfileobj just copy nothing to output_blob.

You can just add an input_blob.seek(0) after read_csv to fix this problem.

answered May 2, 2018 at 6:00

Sraw

20.6k11 gold badges60 silver badges93 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Azure Blob Direct Upload

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related