Ways to filter files in directory and join directory path - Python

Question

Given a suffix and a directory path, I need to extract the full path of the files in the directory that ends with a given suffix.

Currently, I'm doing it as such:

import os
dir_path = '/path/to/dir'
suffix = '.xyz'
filenames = filter(lambda x: x.endswith(suffix), os.listdir(dir_path))
filenames = map(lambda x: os.path.join(dir_path, x), filenames)

I could also do it with glob:

import glob
dir_path = '/path/to/dir'
suffix = '.xyz'
glob.glob(dir_path+'*.'+suffix)

I understand that there's also pathlib that can check for suffixes using PurePath but I'm not sure what is the syntax for that.

Are there other ways of achieving the same filtered list of full paths to the files?

msvalkon · Accepted Answer · 2016-10-06 08:08:18Z

4

You can use a list comprehension to build the result in one go:

>>> [os.path.join(os.sep, x, dir_path) for x in os.listdir(dir_path) 
if x.endswith(suffix)]
['/home/msvalkon/foo.txt', 
 '/home/msvalkon/output.txt',
 '/home/msvalkon/remaining_warnings.txt',
 '/home/msvalkon/test.txt',
 '/home/msvalkon/hdr_chksum_failure.txt']

If dir_path is always an absolute path, you can use os.path.abspath(x) in place of the os.path.join().

For a large directory, it may be wise to use os.scandir which returns an iterator. This will be way faster.

>>> [entry.path for entry in os.scandir(dir_path) if entry.name.endswith(suffix)]
 ['/home/msvalkon/foo.txt', 
  '/home/msvalkon/output.txt',
  '/home/msvalkon/remaining_warnings.txt',
  '/home/msvalkon/test.txt',
  '/home/msvalkon/hdr_chksum_failure.txt']

edited Oct 6, 2016 at 8:08

answered Oct 6, 2016 at 7:12

msvalkon

12.1k2 gold badges46 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

alvas Over a year ago

Short question, given 1,000,000 files in the directory, would filter + map be faster than list comprehension, or the reverse is true? Or would they be the same?

msvalkon Over a year ago

Given that filter() will create one list, and map() another, I would wager that the list comprehension is faster. Though, with such numbers, it may be wiser to find out a way to use a generator.

alvas Over a year ago

Actually, they won't be creating lists, both filter and map returns generators.

msvalkon Over a year ago

@alvas that's true for python 3, but not for python 2. In python 3 both return an iterator, in python2 both construct the resulting list. See my update with os.scandir().

Collectives™ on Stack Overflow

Ways to filter files in directory and join directory path - Python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related