How to append data in blob storage in Azure using Python

In Azure, you can store your data in various storage options provided by Azure like Blob, Table, CosmosDB, SQL-DB, etc. In this shot, we will learn how to append data to a text file stored in Azure Blob Storage.

“Azure Blob storage is Microsoft’s object storage solution for the cloud. Blob storage is optimized for storing massive amounts of unstructured data. Unstructured data is data that doesn’t adhere to a particular data model or definition, such as text or binary data.” - Azure Blob Storage documentationhttps://learn.microsoft.com/en-us/azure/storage/blobs/

Create an Azure Storage account

Let us first create an Azure Storage account. Follow the steps mentioned below:

  1. Search Storage Accounts in the Azure Portal.
  2. Click New to create a new storage account.
  3. Fill in all the details like the resource group, the subscription, the name of your storage account, region, etc.
  4. Now, after the successful creation of your storage account, open your storage account and click on Access Keys from the left navigation pane to get your storage account credentials. Copy them and save them for later use.

The blob that is appended with data is called an append blob. An append blob is made up of blocks and is optimized for append operations. Each block or data that you want to append in the blob can be of a different size, up to a maximum of 4 MB, and an append blob can include up to 50,000 blocks. Therefore, the maximum size of an append blob is slightly more than 195 GB (4 MB X 50,000 blocks).

Install the required package

Now, we are ready to connect this storage account using Python. Let us first install the required packages. Run the following command:

pip install azure

Code

Now, see the code to connect the Storage account with our Python script.

from azure.storage.blob import AppendBlobService
def append_data_to_blob(data):
  service = AppendBlobService(account_name="<STORAGE_ACCOUNT_NAME>", 
            account_key="<STORAGE_ACCOUNT_KEY>")
  try:
    service.append_blob_from_text(container_name="<CONTAINER_NAME>", blob_name="<FILE_NAME>", text = data)
  except:
    service.create_blob(container_name="<CONTAINER_NAME>", blob_name="<FILE_NAME>")
    service.append_blob_from_text(container_name="<CONTAINER_NAME>", blob_name="<FILE_NAME>", text = data)
  print('Data Appended to Blob Successfully.')
append_data_to_blob('Hello Azure Blob Storage!')

Explanation

  • In line 1, we import the required package.
  • In line 3, we create a function append_data_to_blob() that accepts the data we need to append to our blob.
  • In line 4, we create a connection to our storage account in Azure. Note that we pass the <STORAGE_ACCOUNT_NAME> and <STORAGE_ACCOUNT_KEY>, which we have already saved after the creation of our storage account.
  • In line 7, we try to append the data to a file <FILE_NAME> that is present inside our blob container <CONTAINER_NAME>. However, we haven’t created any containers yet; so, we need to keep this statement in a try-except block.
  • In line 9, we create a container by using the create_blob() function and passing the <CONTAINER_NAME> and <FILE_NAME>. You can specify any file name. For example, if you want to append your data to the log.txt file; then, you need to replace <FILE_NAME> with log.txt. The <CONTAINER_NAME> could be any name unique to your storage account.
  • Then, in line 10, we append the data to the file <FILE_NAME>.
  • In line 11, we print the statement.
  • Finally, in line 13, we call our function and pass the data.

So, in this way, we can append data generated at regular intervals and store them in the blob storage.

Free Resources

License: Creative Commons-Attribution-ShareAlike 4.0 (CC-BY-SA 4.0)