Python requests.get() for Azure blob

Question

I am trying to download blobs from Azure storage, and it would be preferred to me to use Python's requests.get(). If I try it with get_blob_to_path() from azure, it works, but not with requests.get().

I use it as follows:

requests.get('https://<account_name>.blob.core.windows.net/<container_name>/<blob_name>')

I have also tried to just copy the whole URL from Azure Storage Explorer.

I get the following error from one account:

gaierror: [Errno -2] Name or service not known

and the following result (not displayed as an error) from another account:

<Response [404]>

What can cause this error / <Response [404]> and how to fix it? Is it some issues with permissions for connections through requests.get()?

Ivan Glasenberg · Accepted Answer · 2019-07-15 05:30:46Z

3

The error is due to you did not specify the permission to access blob storage.

Besides change the public access level to container or blob(screenshot as below) as @Martin mentioned in his post, you have the other 2 ways for the permission issue.

Method 1:You can generate a SAS URL for the blob. Nav to azure portal -> click the "..." symbol of blob you want to download -> select Generate SAS. After the SAS URL generated, you can use the SAS URL for blob downloading. The screenshot below shows how to generate SAS URL:

Then you can write code like below:

#use the SAS URL
r = requests.get('https://yy3.blob.core.windows.net/aa1/w2.JPG?xxxx')
open("d:\\temp\\mytest222.jpg","wb").write(r.content)

Method 2:Please take use of Get Blob rest api, and the sample code below is working for me.

import requests
import datetime
import hmac
import hashlib
import base64

storage_account_name = 'xxxx'
storage_account_key = 'xxxxx'
blob_name = 'your_blob_name,like w2.jpg, note it is case sensitive'
container_name='the container name'
api_version = '2018-03-28'
request_time = datetime.datetime.utcnow().strftime('%a, %d %b %Y %H:%M:%S GMT')

string_params = {
    'verb': 'GET',
    'Content-Encoding': '',
    'Content-Language': '',
    'Content-Length': '',
    'Content-MD5': '',
    'Content-Type': '',
    'Date': '',
    'If-Modified-Since': '',
    'If-Match': '',
    'If-None-Match': '',
    'If-Unmodified-Since': '',
    'Range': '',
    'CanonicalizedHeaders': 'x-ms-date:' + request_time + '\nx-ms-version:' + api_version + '\n',
    'CanonicalizedResource': '/' + storage_account_name + '/'+container_name + '/' + blob_name
}

string_to_sign = (string_params['verb'] + '\n' 
                  + string_params['Content-Encoding'] + '\n'
                  + string_params['Content-Language'] + '\n'
                  + string_params['Content-Length'] + '\n'
                  + string_params['Content-MD5'] + '\n' 
                  + string_params['Content-Type'] + '\n' 
                  + string_params['Date'] + '\n' 
                  + string_params['If-Modified-Since'] + '\n'
                  + string_params['If-Match'] + '\n'
                  + string_params['If-None-Match'] + '\n'
                  + string_params['If-Unmodified-Since'] + '\n'
                  + string_params['Range'] + '\n'
                  + string_params['CanonicalizedHeaders']
                  + string_params['CanonicalizedResource'])

signed_string = base64.b64encode(hmac.new(base64.b64decode(storage_account_key), msg=string_to_sign.encode('utf-8'), digestmod=hashlib.sha256).digest()).decode()

headers = {
    'x-ms-date' : request_time,
    'x-ms-version' : api_version,
    'Authorization' : ('SharedKey ' + storage_account_name + ':' + signed_string)
}

url = ('https://' + storage_account_name + '.blob.core.windows.net/'+container_name+'/'+blob_name)

r = requests.get(url, headers = headers)

#specify where to download and the new file name
open("d:\\temp\\mytest111.jpg","wb").write(r.content)

print("ok")

edited Jul 15, 2019 at 5:30

answered Jul 15, 2019 at 3:28

Ivan Glasenberg

30.3k2 gold badges56 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Valeria Over a year ago

For the first method, is there an automatic generation of SAS URLs? I will have to download hundreds of blobs, it is definitely not a good idea for me to press buttons manually. Or for the second method, is it possible to construct Authorization header without account key (with a connection string instead)? I do not have required permissions for making the storage public / see the primary/secondary keys.

Ivan Glasenberg Over a year ago

@Valeria, for method 1, do you have permission to create an account-level SAS via ui(by nav to azure portal -> your storage account->left pane -> Shared access signature)? if yes, then you can use this account level SAS for all the blobs download. For method 2, if you have a connection string(if I understand correctly), the account key of course includes in the connection string(the connection string should be DefaultEndpointsProtocol=https;AccountName=xx;AccountKey=xxx;EndpointSuffix=core.windows.net).

Valeria Over a year ago

For method 2: I am not sure about details since I am new to Azure, but I was told it is a "connection string with SAS". Basically, it is of this format: learn.microsoft.com/en-us/azure/storage/common/…. It has SharedAccessSignature (sv, ss, sig etc.) and BlobEndpoint. I tried to use sig as signed_string from the code, but I get the same error as in the beginning ([Errno -2] Name or service not known).

Ivan Glasenberg Over a year ago

can you try put this url("your_storage_account.blob.core.windows.net/your_container/…? + your_connection_string") in requests.get(url)? to see if any error occurs? or if some error occurs, do you mind email this connection string to me?

Ivan Glasenberg Over a year ago

@Valeria, do you have any more issues?

|

Martin Esteban Zurita · Accepted Answer · 2019-07-15 00:34:56Z

1

You are not authenticating this request, so I have made a public blob storage container to test this, and it works so I think its a permission's issue. Is the blob storage public as shown here? https://learn.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources

Here is my code, test and see if it works:

import requests
req = requests.get('https://publicstoragefallout.blob.core.windows.net/publiccont/icon.png')
open('icon.png', 'wb').write(req.content)

Note that it also works if you use http instead of https.

Hope this helped!

answered Jul 15, 2019 at 0:34

Martin Esteban Zurita

3,18416 silver badges24 bronze badges

Collectives™ on Stack Overflow

Python requests.get() for Azure blob

2 Answers 2

7 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related