Write Python DataFrame as CSV into Azure Blob

Question

I have got two questions on reading and writing Python objects from/to Azure blob storage.

Can someone tell me how to write Python dataframe as csv file directly into Azure Blob without storing it locally?

I tried using the functions create_blob_from_text and create_blob_from_stream but none of them works.

Converting dataframe to string and using create_blob_from_text function writes the file into the blob but as a plain string but not as csv.
```
df_b = df.to_string()
block_blob_service.create_blob_from_text('test', 'OutFilePy.csv', df_b)  
```
How to directly read a json file in Azure blob storage directly into Python?

See this to send the data as csv to blob : stackoverflow.com/questions/50923055/… — Deepak Yadav
– Deepak Yadav, Commented Apr 23, 2021 at 7:00

Akavall · Accepted Answer · 2021-06-07 15:58:28Z

22

Can someone tell me how to write Python dataframe as csv file directly into Azure Blob without storing it locally?

You could use pandas.DataFrame.to_csv method.

Sample code:

from azure.storage.blob import (
    BlockBlobService
)
import pandas as pd
import io

output = io.StringIO()
head = ["col1" , "col2" , "col3"]
l = [[1 , 2 , 3],[4,5,6] , [8 , 7 , 9]]
df = pd.DataFrame (l , columns = head)
print(df)
output = df.to_csv (index_label="idx", encoding = "utf-8")
print(output)

accountName = "***"
accountKey = "***"
containerName = "test1"
blobName = "test3.json"

blobService = BlockBlobService(account_name=accountName, account_key=accountKey)

blobService.create_blob_from_text('test1', 'OutFilePy.csv', output)

Output result:

2.How to directly read a json file in Azure blob storage directly into Python?

Sample code:

from azure.storage.blob import (
    BlockBlobService
)

accountName = "***"
accountKey = "***"
containerName = "test1"
blobName = "test3.json"

blobService = BlockBlobService(account_name=accountName, account_key=accountKey)

result = blobService.get_blob_to_text(containerName,blobName)

print(result.content)

Output result:

Hope it helps you.

edited Jun 7, 2021 at 15:58

Akavall

86.8k58 gold badges214 silver badges260 bronze badges

answered Apr 25, 2018 at 8:47

Jay Gong

23.8k2 gold badges30 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

N.Fisher Over a year ago

When I store the df.to_csv in a variable, it stores it in a local directory and the variable is of None type. Am I missing something?

Palash Mondal Over a year ago

If you would like to save the output to a subfolder then make this change: blobService.create_blob_from_text('test1', 'folder1/folder2/OutFilePy.csv', output)

Dimiter Shalvardjiev · Accepted Answer · 2021-08-09 19:23:23Z

10

The approved answer did not work for me, as it depends on the azure-storage (deprecated/legacy as of 2021) package. I changed it as follows:

from azure.storage.blob import *
import dotenv
import io
import pandas as pd

dotenv.load_dotenv()
blob_block = ContainerClient.from_connection_string(
    conn_str=os.environ["CONNECTION_STRING"],
    container_name=os.environ["CONTAINER_NAME"]
    )
output = io.StringIO()
partial = df.DataFrame()
output = partial.to_csv(encoding='utf-8')
blob_block.upload_blob(name, output, overwrite=True, encoding='utf-8')

answered Aug 9, 2021 at 19:23

Dimiter Shalvardjiev

1091 silver badge3 bronze badges

1 Comment

genegc Over a year ago

More info here: github.com/Azure/azure-sdk-for-python/tree/main/sdk/storage/…

Khudoyberdi Rahimov · Accepted Answer · 2022-07-25 20:18:43Z

There was update in BlobServiceClient. create_blob_from_text method is no longer supported. Now you can use get_blob_client to get or create the blob file. Blob need not exist:

output = dataframe.to_csv(index_label="idx", encoding="utf-8")

blob_service = BlobServiceClient.from_connection_string(
   f"DefaultEndpointsProtocol=https;AccountName={ACCOUNT_NAME};AccountKey= 
{ACCOUNT_KEY};EndpointSuffix=core.windows.net"
)

container_client = blob_service.get_container_client(DEST_CONTAINER)
blob_client = blob_service.get_blob_client(container=DEST_CONTAINER, 
blob="kcScenarioTest/"+str(current_time.microsecond)+".csv") 

blob_client.upload_blob(output,overwrite=True,content_settings=ContentSettings(content_type="text/csv"))

KavlinClein · Accepted Answer · 2023-01-13 18:38:13Z

2

Here's an example of writing a Python DataFrame into Azure Blob Storage without storing it locally. It doesn't require String.IO and uses the ContainerClient instead of BlockBlobService.


import pandas as pd

def write_csv(env, df_path, df):
    container_client = ContainerClient(
        env['container_url'],
        container_name=env['container_name'],
        credential=env['container_cred']
    )

    output = df.to_csv (index_label="idx", encoding = "utf-8")
    print(output)
    blob_client = container_client.get_blob_client(df_path)
    blob_client.upload_blob(output, overwrite=True)

    return 'success'

answered Jan 13, 2023 at 18:38

KavlinClein

211 bronze badge

Comments

Man_Like_Dave · Accepted Answer · 2023-02-20 14:18:33Z

0

So you need a BytesIO file to upload to the blob, using the upload_blob method from azure.storage.blob module. You will also need to create a cotainer_client from the same module

blob_report_name = 'OutFilePy.csv'
stream_file = BytesIO()
df_b.to_csv(stream_file)  
file_to_blob = stream_file.getvalue()
blob_client = container_client.get_blob_client(blob_report_name)
blob_client.upload_blob(data=file_to_blob, overwrite=True)

answered Feb 20, 2023 at 14:18

Man_Like_Dave

802 silver badges11 bronze badges

Collectives™ on Stack Overflow

Write Python DataFrame as CSV into Azure Blob

5 Answers 5

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related