How can I upload a dataframe as csv to azure by Python?

Question

I am working with Python and Pyspark, and I want to upload a CSV file to an azure blob storage. I have already a dataframe generated by code: df. What I want to do is the next:

# Dataframe generated by code
df

# Create the BlockBlockService that is used to call the Blob service for the storage account
block_blob_service = BlockBlobService(account_name='name', account_key='key') 

container_name ='results-csv'

d = {'one' : pandas.Series([1., 2., 3.], index=['a', 'b', 'c']), 'two' : pandas.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
df = pandas.DataFrame(d)


writer = pandas.ExcelWriter(df, engine='xlsxwriter')

a = df.to_excel(writer, sheet_name='Sheet1', index=False, engine='xlsxwriter')


block_blob_service.create_blob_from_stream(container_name, 'test', a)

I get the error:

ValueError: stream should not be None.

So I want to upload the content of the dataframe as a blob to the storage location provided above. Is there any way to do that without first generating a CSV file in my local computer?

Whatever how you create that CSV file, you can just save it into a BytesIO, it is almost the same as save to a file. And then you can upload it as stream or bytes. — Sraw
– Sraw, Commented Jun 19, 2018 at 7:46
You can use a = df.to_csv() and block_blob_service.create_blob_from_text(container_name, "test.csv", a) — Sraw
– Sraw, Commented Jun 19, 2018 at 8:16

Deepak Yadav · Accepted Answer · 2021-04-23 06:55:54Z

3

What we intent to do is using dataset.to_csv function create a file stream and then send that stream to azure blob. The alternative to this is we directly store the the string dataset to azure. Code :

    blob_client = service.get_blob_client(container=container_name, blob=local_file_name)
    print(str(dataset.to_csv()))
    blob_client.upload_blob(str(dataset.to_csv()))

This will store the file into blob. Any other solution is not working as of now. Still the issue being now the data is blob is not in csv format that part we still need to figure out.

Edit : Added the code to send it in csv format

edited Apr 23, 2021 at 6:55

answered Apr 23, 2021 at 6:50

Deepak Yadav

6407 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I upload a dataframe as csv to azure by Python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related