Unable to access dataframes in another function using multiprocessing

Question

import multiprocessing
from multiprocessing import Pool

import pandas as pd
globvar = 0
df1 = pd.DataFrame()


def set_globvar_to_one(n):
    global globvar  
    global df1
    globvar = 1
    df1 = n
    print('df1', df1)
    return df1

def print_globvar():
    print('globvar',globvar)    
    print('df1',df1)

if __name__ == "__main__":
    a ='Python Code'

    with Pool(1) as p:
        df2= p.map(set_globvar_to_one, [a])
        print ("df1----------------",df2)
    print_globvar()

Unable to access dataframes in another function using multiprocessing. Through multiprocessing I'm saving the content to dataframes df1 but when I try to access df1 in print_globvar function. I'm getting empty dataframes

wallisers · Accepted Answer · 2020-03-09 08:47:16Z

1

By using multiprocessing you are creating a new process with its own namespace and memory (etc.). So you cannot access the memory of your new process from within your main process and vice versa. Therefore you have to pass variables via Pipe or Queue or use a Manager.

edited Mar 9, 2020 at 8:47

answered Mar 9, 2020 at 7:54

wallisers

3883 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

supreeth k s Over a year ago

I see example using process with queue or process with pipe. can you share an example where Pool using queue or Pool using Pipe.

Collectives™ on Stack Overflow

Unable to access dataframes in another function using multiprocessing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related