python: problems with accessing variables while using multiprocessing

Question

I am new to multiprocessing concepts in python and I have problem accessing variables when I try to include multiprocessing in my code. Sorry if Iam sounding naive, but I just cant figure it out. Below is a simple version of my scenario.

class Data:
    def __init__(self):
        self.data = "data"
    def datameth(self):
        print self.data
        print mainvar

class First:
    def __init__(self):
        self.first = "first"
    def firstmeth(self):
        d = Data()
        d.datameth()
        print self.first

def mymethod():
    f = First()
    f.firstmeth()

if __name__ == '__main__':
    mainvar = "mainvar"
    mymethod()

When I run this, its running fine and gives the output:

data
mainvar
first

But when I try to run mymethod()as a process

from multiprocessing import Process
class Data:
    def __init__(self):
        self.data = "data"
    def datameth(self):
        print self.data
        print mainvar

class First:
    def __init__(self):
        self.first = "first"
    def firstmeth(self):
        d = Data()
        #print mainvar
        d.datameth()
        print self.first


def mymethod():
    f = First()
    f.firstmeth()

if __name__ == '__main__':
    mainvar = "mainvar"
    #mymethod()
    p = Process(target = mymethod)
    p.start()

I get an error like this:

NameError: global name 'mainvar' is not defined

The point is, Iam not able to access mainvar from inside First class or Data class. What am I missing here?

Edit: Actually in my real scenario, it is not just declaring mainvar, it is the return value of a method after some processing.

if __name__ == '__main__':
    ***some other stuff***
    mainvar = ***return value of some method**
    p = Process(target = mymethod)
    p.start()

Edit 2: As @dciriello mentioned in comments, It is working fine in Linux but not in Windows :(

I'm sorry I mismatched your question (i deleted the wrong comment) your code does works form me. the problem seems to be related to Windows, are you using python on windows? — dnlcrl
– dnlcrl, Commented Apr 22, 2015 at 10:18
@dciriello Yes, Iam using python2.7 on Win 7. I have checked the same code on linux and it works, but not on windows :( — Deepa
– Deepa, Commented Apr 22, 2015 at 10:57

dano · Accepted Answer · 2015-04-22 14:37:07Z

This is a limitation of Windows, because it doesn't support fork. When a child process is forked in Linux, it gets a copy-on-write replica of the parent's processes state, so the mainvar you defined inside if __name__ == "__main__": will be there. However, on Windows, the child process' state is created by re-importing the __main__ module of the program. This means that mainvar doesn't exist in the children, because it's only created inside the if __name__ == "__main__" guard. So, if you need to access mainvar inside a child process, your only option is to explicitly pass it to the child as an argument to mymethod in the Process constructor:

mainvar = "whatever"
p = Process(target=mymethod, args=(mainvar,))

This best-practice is mentioned in the multiprocessing docs:

Explicitly pass resources to child processes

On Unix a child process can make use of a shared resource created in a parent process using a global resource. However, it is better to pass the object as an argument to the constructor for the child process.

Apart from making the code (potentially) compatible with Windows this also ensures that as long as the child process is still alive the object will not be garbage collected in the parent process.

Notice the bold part - though it's not quite spelled out, the reason it helps with Windows compatibility is because it helps avoid the exact issue you're seeing.

This is also covered in the section of the docs that talks specifically about Windows limitations caused by the lack of fork:

Global variables

Bear in mind that if code run in a child process tries to access a global variable, then the value it sees (if any) may not be the same as the value in the parent process at the time that Process.start was called.

However, global variables which are just module level constants cause no problems.

Note the "if any". Because your global variable is declared inside the if __name__ == "__main__": guard, it doesn't even show up in the child.

In this case, how does one send constantly updating data (live data) from parent to child process?

Aaron Digulla · Accepted Answer · 2015-04-22 10:20:18Z

1

Operating systems don't allow processes to share variables easily. If they would, then each process could steal data from any other process and you never want that (like when you enter your credit card details in a web browser).

So when you use the multiprocessing module, you have to use special facilities to share variables (a.k.a "state") between the individual processes like Value and Array. See the documentation for details.

answered Apr 22, 2015 at 10:20

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

Comments

Zealous System · Accepted Answer · 2015-04-22 10:17:19Z

0

you are using 'mainvar' at wrong place,

Try below:

from multiprocessing import Process

mainvar = "mainvar"
class Data:
    def __init__(self):
        self.data = "data"
    def datameth(self):
        print self.data
        print mainvar

class First:
    def __init__(self):
        self.first = "first"
    def firstmeth(self):
        d = Data()
        #print mainvar
        d.datameth()
        print self.first


def mymethod():
    f = First()
    f.firstmeth()

if __name__ == '__main__':
    #mainvar = "mainvar"
    #mymethod()
    p = Process(target = mymethod)
    p.start()

answered Apr 22, 2015 at 10:17

Zealous System

2,33413 silver badges22 bronze badges

1 Comment

Deepa Over a year ago

Yes, I have tried that and this works fine. But, in the actual scenario, it is a lots of processing and objects and function calls and I get many return values that I need to use inside the classes. So, I was just wondering if i cud just use them without moving them out and need to make a consider amount of changes to the structure of the code.

Collectives™ on Stack Overflow

python: problems with accessing variables while using multiprocessing

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related