Asynchronous running of Python functions

Question

I have a function that will (based on parameters) move/rename image files from a mapped card reader drive in Windows to a server drive after the file is renamed using a template and appending the incrementing number from the for loop.

There are usually three cards of photos that will be sent to the destination folder. Currently each card is being processed one after the other which because of the file sizes and traveling over the network can take quite some time.

is there a way to have a function that would receive a list of the mapped card drives (not more than 3) and then run the rename function simultaneously for each card.

My poor attempt to illustrate what I am trying to do follows:

def collectCards(cards):
    for card in cards:
         #my goal would be to run each instance of the following function asynchronously
         drv =renameImages(card)

def renameImages(cardDrive):
    #perform renaming functions
    return count_of_renamed_images

Maciej Gol · Accepted Answer · 2013-08-10 19:33:12Z

2

You could try using multiprocessing with processes (Pool) or threads (pool.ThreadPool). In both cases, the only difference is in the import - the API stays the same:

from multiprocessing import Pool

my_pool = Pool(3)
# `cards` is a list of cards. Send rename requests to workers.
my_pool.map(renameImages, cards)

# Close the pool and wait for processes to close.
my_pool.close()
my_pool.join()

The number in Pool(3) indicates number of worker processes - more means greater number of concurrent renameImages functions running. Bear in mind that multiprocessing requires card objects to be able to be pickled. If renameImages is not heavy on memory, you could try using ThreadPool - card objects are then shared between threads.

answered Aug 10, 2013 at 19:33

Maciej Gol

15.9k4 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Asynchronous running of Python functions

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related