Copy files to another folder using python

Question

Lets say, i have a source folder with 10k file, i want to copy 1k files to another folder. Tried the below methods, it worked but, is there any way to do it more efficiently ?

sourceFiles = os.listdir("/source/")
destination = "/destination/"

for file in sourceFiles[0 : 1000]:
    shutil.copy(file, destination)

Here what i feel is, i am loading 10k files into a list variable and iteration through each element in the list for 1k times, loading unwanted data into the RAM, which doesn't look good for me. Is there any better way to do the same ?

You just load 10k strings, which is absolutly OK, you just keep 9k string unused but you don't have to care about — azro
– azro, Commented Dec 24, 2019 at 11:46
sourceFiles just holds paths to your source files, not the files themselves. 10k strings in a variable won't bother your RAM, as azro commented. The actual 'work' your computer has to do is the file IO from shutil.copy but I guess there's no way around this. — FObersteiner
– FObersteiner, Commented Dec 24, 2019 at 11:56
ok, got that. Do i need to iteration through all the elements in the list and move files one by one? is there any way to move them in bulk(in a single go), in my case 1k? — Surya
– Surya, Commented Dec 24, 2019 at 11:58

accdias · Accepted Answer · 2019-12-24 12:24:20Z

1

If you are using Python 3, pathlib.Path.iterdir is a better option:

from pathlib import Path

source = Path('/source')
target = Path('/destination')

counter = 0

for obj in source.iterdir():
    if obj.is_file():
        obj.rename(target / obj.name)
        counter += 1
    if counter > 1000:
        break

It uses a generator and the syntax is cleaner IMHO.

It is also better on memory efficiency. Look:

Python 3.7.5 (default, Dec 15 2019, 17:54:26) 
[GCC 9.2.1 20190827 (Red Hat 9.2.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sys import getsizeof
>>> from os import listdir
>>> from pathlib import Path
>>> files = listdir('/usr/bin')
>>> usrbin = Path('/usr/bin')
>>> getsizeof(files)
26744
>>> getsizeof(usrbin.iterdir())
128
>>>

edited Dec 24, 2019 at 12:24

answered Dec 24, 2019 at 12:01

accdias

5,3523 gold badges24 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Surya Over a year ago

it looks good, i can see lot of difference in the memory. Thank You very much.

axolotl Over a year ago

OP asked about copying. Your answer does the equivalent of mv.

accdias Over a year ago

@axolotl, Take a look again. What OP really wanted was a more efficient memory-wise way to iterate over files in a directory, hence his acceptance of the answer. Take a look also at OP's latest comment, where he states "any way to move them in bulk".

axolotl Over a year ago

Ah. I was going off of the question title and the MWE code, since that's what brought me here (and so I expected to find an ans with copying :) ). Whatever floats OP's boat.

Collectives™ on Stack Overflow

Copy files to another folder using python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related