Run multiple function in sequence within a process using multiprocessing

Question

i'm a newbie on multiprocessing.

What I'm trying to achieve is to run multiple function in sequence within a process using multiprocessing.

Example:

I have 5 dataframes that needs to be inserted in a database in 5 separate tables. Before Inserting I'd like to check that data format is correct, that the column order is correct, a brief check of data quality. Let's suppose that I've wrote 4 functions, namely correct_data_format(), column_order() and data_quality(). Add also an insert_db() functions.

I'd like to run those 4 functions for each dataframe.

In my mind I should have 5 different process for each dataframe and each process should run in sequence the 4 functions mentioned above ( correct_data_format() -> column_order -> data_quality -> insert_db)

How to achieve that using multiprocessing package?

You could use multiprocessing.Pool, define a function that works on a dataframe and does all the checks you need and returns True/False, and use the Pool.map method — Tomer
– Tomer, Commented Aug 5, 2020 at 14:15
And If i din't want to wrap all functions into one? Is it possible to achieve what I described? — Marco Fumagalli
– Marco Fumagalli, Commented Aug 5, 2020 at 14:29
Pool.map doesn't work with lambda functions, so for that you'll have to define some function or class. Maybe there is a way I'm not aware of with some other api of multiprocessing — Tomer
– Tomer, Commented Aug 5, 2020 at 15:20

Shalom Rochman · Accepted Answer · 2020-08-05 14:41:31Z

2

Let's say you have 5 dataframes df1, df2, df3, df4 and df5. The following code will probably do what you want:

import multiprocessing

def func(df):
    correct_data_format(df)
    column_order(df)
    data_quality(df)
    insert_db(df)
    ...
    [some other stuff]

pool = multiprocessing.Pool()
pool.map(func, (df1, df2, df3, df4, df5))

answered Aug 5, 2020 at 14:41

Shalom Rochman

663 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Run multiple function in sequence within a process using multiprocessing

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related