How to perform file copying in multiple threads or processor in python?

Question

I"m trying to copy files in python , it'll take long time in sequential approach so I wanted to do it in multiple threads. Below is my code to copy files in sequential

for file in files:
    shutil.copy(file,destination_path)

this took around 1.2 sec to complete I tried to implement multithreading as below

import multiprocessing
copy_instance(src,dest):
    shutil.copy(src,dest)
for file in files:
    p1 = multiprocessing.process(target=copy_instance,args=(file,destination_path)
    p1.start()

this took 168.8 sec to complete

Doing it in multiprocessing taking way more time then sequential approach , what am I doing wrong? how can I correctly implement multithreading to speed up my copying process? any help or suggestion on this will be very helpful, thanks

Copying is almost certainly bound by storage access time, unless you are copying to and from a ramdisk: parallelisation is unlikely to help. — Amadan
– Amadan, Commented Feb 3, 2022 at 4:42
You are not multithreading but multiprocessing. It can be much slower, more if you are running windows — azelcer
– azelcer, Commented Feb 3, 2022 at 4:43

azelcer · Accepted Answer · 2022-02-03 13:13:13Z

1

Spawning new processes is slow, particularly in windows. Instead of multiprocessing you should use multithreading, which in CPython can be good for I/O (and is bad for CPU bound processes):

import shutil
import itertools
from concurrent.futures import ThreadPoolExecutor

def copy_instance(args):
    shutil.copy(*args)
    
#   ... snipped definition of files and destination

with ThreadPoolExecutor() as pool:
    pool.map(copy_instance, zip(files, itertools.repeat(destination)))

Nevertheless, as noted in the comments, depending on you storage you probably won't see any difference.

answered Feb 3, 2022 at 13:13

azelcer

1,5331 gold badge6 silver badges7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to perform file copying in multiple threads or processor in python?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related