2

I am using the latest Azure Storage SDK (azure-storage-blob-12.7.1). It works fine for smaller files but throwing exceptions for larger files > 30MB.

azure.core.exceptions.ServiceResponseError: ('Connection aborted.', timeout('The write operation timed out'))

from azure.storage.blob import BlobServiceClient, PublicAccess, BlobProperties,ContainerClient

    def upload(file):
        settings = read_settings()
        connection_string = settings['connection_string']
        container_client = ContainerClient.from_connection_string(connection_string,'backup')
        blob_client = container_client.get_blob_client(file)
        with open(file,"rb") as data:
            blob_client.upload_blob(data)
            print(f'{file} uploaded to blob storage')
    
    upload('crashes.csv')
Improve this question
4

1 Answer 1

Reset to default
3

Seems everything works for me by your code when I tried to upload a ~180MB .txt file. But if uploading small files work for you, I think uploading your big file in small parts could be a workaround. Try the code below:

from azure.storage.blob import BlobClient

storage_connection_string=''
container_name = ''
dest_file_name = ''

local_file_path = ''

blob_client = BlobClient.from_connection_string(storage_connection_string,container_name,dest_file_name)

#upload 4 MB for each request
chunk_size=4*1024*1024  

if(blob_client.exists):
    blob_client.delete_blob()
    blob_client.create_append_blob()

with open(local_file_path, "rb") as stream:
    
    while True:
            read_data = stream.read(chunk_size)
            
            if not read_data:
                print('uploaded')
                break 
            blob_client.append_block(read_data)

Result:

enter image description here

Improve this answer
Sign up to request clarification or add additional context in comments.

10 Comments

Thanks for your inputs ! But as per the SDK spec the API should handle the chunk internally. So trying to figure out why the API not working as expected! I think as mentioned above SDK team already has got a similar issue active.
@DevMonk, I see, let's see what is the root reason.
@DevMonk,bu the way, does the solution: set max_single_put_size to a smaller value works for you?
those params didn't resolve the issue. I am going with your approach of chunking. (But the underlying issue is still there, which Azure SDK team needs to resolve)
I decided to use max_block_size and max_single_put_size, and this seems to lessen the timeouts when LTE network is slow.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.