Copy blobs from source container and upload to target container async

Question

I am trying using azure.storage.blob.aio to upload blobs in one container to another container but event if the function is work as expected, I keep got the warning message:

Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f6a2ebf6e50>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x7f6a2e3553a0>, 32966.677798886)]']
connector: <aiohttp.connector.TCPConnector object at 0x7f6a2ebf6eb0>
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f6a2ec03400>
Unclosed connector
...

What wrong with my code, and how I can fix above message?

async def upload_blob(blob_args):
    # srource blob
    src_container_client = ContainerClient.from_connection_string(blob_args["src_connect_str"], blob_args["src_container"])

    # destination blob
    target_container_client = ContainerClient.from_connection_string(blob_args["target_connect_str"], blob_args["target_container"])
  
    tasks = []
    print(f"\nReading blobs from the container:")
    async for source_blob in src_container_client.list_blobs():
        # push data into specified stream
        source_blob_client = src_container_client.get_blob_client(source_blob.name)
        stream_downloader = await source_blob_client.download_blob()
        stream = await stream_downloader.readall()
            
        # create the BlobClient from the ContainerClient to interact with a specific blob
        target_blob_client = target_container_client.get_blob_client(source_blob.name)

        print(f"\t- Transfering {source_blob.name}.")
        tasks.append(asyncio.create_task(target_blob_client.upload_blob(stream)))

    await asyncio.gather(*tasks)
    print("Finished")

if __name__ == "__main__":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(upload_blob(blob_args))

Jim Xu · Accepted Answer · 2020-10-15 06:21:31Z

1

Regarding the waring, you do not run async context manager with your ContainerClient.

For example

import io
from azure.storage.blob.aio import ContainerClient
import asyncio


async def copy_blobs():
    # srource blob
    source_container_client = ContainerClient.from_connection_string(
        conn_str, 'upload')

    # destination blob
    target_container_client = ContainerClient.from_connection_string(
        conn_str, 'copy')
    async with source_container_client, target_container_client:
        tasks = []
        print(f"\nReading blobs from the container:")
        async for source_blob in source_container_client.list_blobs():
            source_blob_client = src_container_client.get_blob_client(source_blob.name)
            stream_downloader = await source_blob_client.download_blob()
            stream = await stream_downloader.readall()
            
            tasks.append(asyncio.create_task(target_blob_client.upload_blob(stream)))
            
            print(f"\t- Transfering {source_blob.name}.")

        await asyncio.gather(*tasks)
        print("Finished")

if __name__ == "__main__":
    asyncio.set_event_loop(asyncio.new_event_loop())
    loop = asyncio.get_event_loop()
    loop.run_until_complete(copy_blobs())

edited Oct 15, 2020 at 6:21

answered Oct 15, 2020 at 2:36

Jim Xu

23.2k2 gold badges24 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Tan Sang Over a year ago

by the way, I found out a bug in my code, it will create a file in target container without data, please check out code section in my question again, and update your code (for s.o who want to re-use it). Thank you in advance

Collectives™ on Stack Overflow

Copy blobs from source container and upload to target container async

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related