Why pandas dataframe doesn't change when i used it as a input of a function with multiprocessing

Question

I have a code like this:

df1 = pd.DataFrame(
    {
        "A": ["A0", "A1", "A2", "A3"],
        "B": ["B0", "B1", "B2", "B3"],
        "C": ["C0", "C1", "C2", "C3"],
        "D": ["D0", "D1", "D2", "D3"],
    }
)
df2 = pd.DataFrame(
    {
        "A": ["A4", "A5", "A6", "A7"],
        "B": ["B4", "B5", "B6", "B7"],
        "C": ["C4", "C5", "C6", "C7"],
        "D": ["D4", "D5", "D6", "D7"],
    }
)

def changeDF(df):
    df['Signal'] = 0

changeDF(df1)
changeDF(df2)

when I run above, (changeDf) function add a column to df1 and df2 named 'Signal' with 0 values. but instead of run (changeDf) directly using multiprocessing like below it doesn't change any dfs.

s = [df1, df2]
with multiprocessing.Pool(processes=2) as pool:
    res = pool.map(changeDF, s)

What's wrong with my code?

Change your function to return the modified data frame

Shffl
– Shffl

2021-11-20 18:47:47 +00:00
Commented Nov 20, 2021 at 18:47 — Shffl
– Shffl, Commented Nov 20, 2021 at 18:47

CJR · Accepted Answer · 2021-11-20 18:57:25Z

2

Serializing df1 & df2 for multiprocessing means that you're making a copy.

Return your dataframe from the function and it'll work fine.

def changeDF(df):
    df['Signal'] = 0
    return(df)

with multiprocessing.Pool(processes=2) as pool:
    df1, df2 = pool.map(changeDF, [df1, df2])

I would warn you that the serialization costs of this will certainly be higher than the benefit you get from multiprocessing.

edited Nov 20, 2021 at 18:57

answered Nov 20, 2021 at 18:48

CJR

3,9872 gold badges13 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user17242583 Over a year ago

You can just do df1, df2 = pool.map(changeDF, [df1, df2]). No need for the brackets [df1, df2] at the beginning of the line

BinaryThinker Over a year ago

Thanks for your kindness. I'm looking for a way to parallelize my calculation without copy a new version of my dataframes because these are very big dataframes and with copy might have trouble with RAM. Do you have any idea to parallelize its calculation without copy of these?

CJR Over a year ago

You should look into libraries built for that purpose. I would suggest dask.

user17242583 · Accepted Answer · 2021-11-20 18:53:01Z

1

Change your function changeDF to be like this:

def changeDF(df):
    df['Signal'] = 0
    return df

answered Nov 20, 2021 at 18:53

user17242583

Collectives™ on Stack Overflow

Why pandas dataframe doesn't change when i used it as a input of a function with multiprocessing

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related