embarrassingly parallel problem in python

Question

I have 634 *.npy files, each contain a 2D numpy array of shape (8194, 76). I want to use STL decomposition on each column five time with different frequencies. so what I'm trying to do is:

for file in files:
    for column in columns:
        for freq in frequencies:
            res = STL(file[:,column], period = freq)
            decomposed = np.vstack((res.trend, res.seasonal, res.resid)).T
    np.save(decompoesd)

finally the shape of decomposed should be (8194,1140). How can I parallize this? as it would take over 2 months to run in serial implementation.

Sorry for the late feedback @ArtiomKozyrev, It worked perfectly by parallelizing file level, thank you so much for your help. I made simulate_cpu_bound(file) takes a file process it and write it to a disk. I will try next parallelize the CPU heavy task itself, any hint on how to group the res output from each processor and write it to a disk using single process then only after that, all other processors take the next file and do the same — Tamer Adel
– Tamer Adel, Commented Dec 11, 2021 at 16:28

Artiom Kozyrev · Accepted Answer · 2021-12-08 15:30:51Z

1

You can do something like that:

from concurrent.futures import ProcessPoolExecutor


FILES = ["a", "b", "c", "d", "e", "f", "g", "h"]


def simulate_cpu_bound(file):
    2 ** 100000000  # cpu heavy task
    # or just use time.sleep(n), where n - number of seconds
    return file


if __name__ == '__main__':
    with ProcessPoolExecutor(8) as f:
        res = f.map(simulate_cpu_bound, FILES)

    res = list(res)

    print(res)

edited Dec 8, 2021 at 15:30

answered Dec 8, 2021 at 12:38

Artiom Kozyrev

3,9062 gold badges18 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Booboo Over a year ago

multithreading is definitely not what you want to use for a CPU heavy task unless that task is implemented, for example, as a CPU-language -implemented function that releases the Global Interpreter Lock (GIL). Did you try timing calling simulate_cpu_bound 8 times in a loop and then timing your multithreading version? You will not see much of a difference. Of course, if you change the function to just the sleep version you will because sleep is not CPU-intensive and does release the GIL.. There I would expect a time reduction of approximately 8-fold.

Artiom Kozyrev Over a year ago

@Booboo My mistake, I even did not pay attention that imported ThreadPoolExecutor instead of ProcessPoolExecutor, thanks. I made the required changes.

Collectives™ on Stack Overflow

embarrassingly parallel problem in python

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related