Parallel processing for loop in dataframe

Question

Want to parallel process the below code. Due to some reason I've to subset and then apply function. Note that the subset size will not be consistent.

for i in range(0, df['col1'].max()+1):
    subset = df[ df['col1'] == i ]
    subset_result = func(subset)
    result = result.append(subset_result)

The below can be used for equal split of dataframe. How can I use it for unequal subsets.cores=mp.cpu_count() df_split = np.array_split(df, cores, axis=0) # create the multiprocessing pool pool = Pool(cores) # process the DataFrame by mapping function to each df across the pool df_out = np.vstack(pool.map(func, df_split)) — Neeti
– Neeti, Commented Sep 27, 2019 at 16:52

Massifox · Accepted Answer · 2019-09-27 17:54:56Z

1

Try this code, using multiprocessing:

import multiprocessing

def f(x):
    return x*x

def chunks(l, n):
    """Yield successive n-sized chunks from l."""
    for i in range(0, len(l), n):
        yield l[i:i + n]

if __name__ == '__main__':
    n_core = multiprocessing.cpu_count()
    p = multiprocessing.Pool(processes= n_core)

    data = range(0, 8)
    subsets = chunks(data, n_core)
    subset_results = []
    for subset in subsets:
        subset_results.append(p.map(f, subset))

    print(subset_results)

In your case, a chunks function that could do for you is the following:

def chunks_series(s):
    subsets = []
    for i in range(s.max() + 1):
        subset = s[s == i]
        subsets.append(subset.values)
    return subsets

subsets = chunks_series(df['col1'])

Or you can do everything in the same loop:

n_core = multiprocessing.cpu_count()
p = multiprocessing.Pool(processes=n_core)   
s = df['col1']
subset_results = []

for i in range(s.max() + 1):
    subset = s[s == i]
    subset_results.append(p.map(f, subset))

I had preferred to introduce a chunk function, even if for your case it does not introduce advantages, to make the code more clear and generalizable.

edited Sep 27, 2019 at 17:54

answered Sep 27, 2019 at 17:10

Massifox

4,5071 gold badge13 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Neeti Over a year ago

Thank you for the answer. I want to subset on the basis of col1. chunks will give equal subset of the data which is not I'm looking forward to.

Collectives™ on Stack Overflow

Parallel processing for loop in dataframe

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related