Python: can't set Process variable from outside Process

Question

I am trying to set a class variable of a Process class from outside, to indicate that the while loop inside the process should finish.

However, although the variable appears to get set just fine, the old value is retrieved from run(self). I have tried using threading.Thread where this works. However, the problem with Thread is that it doesn't always start when start() is called. Sometimes it waits for the scp process to be finished, which kills the point of using a Thread for this scenario.

from multiprocessing import Process
        class LogParserThread(Process):
            def __init__(self):
                super(LogParserThread, self).__init__()
                self.scp_done = False
                self.flights_to_add = ()

            def stop(self):
                self.scp_done = True
                print 'Stopping: ' + str(self.scp_done)


            def run(self):
                file_list = ()
                while self.scp_done is False or len(file_list) > 0:
                    print 'Status ' + str(self.scp_done) + ' ' + str(len(file_list))
                    if (len(file_list) > 1):
                        print ' Case 1 '
                        f = file_list[0]
                        os.system('rm -rf ' + f + '*')
                    if (len(file_list) is 1 and self.scp_done is True):
                        print ' Case 2 '
                        f = file_list[0]
                        os.system('rm -rf ' + f + '*')
                    update file_list

        files_to_copy = "file1 file2 file3"
        parser_thread = LogParserThread()
        parser_thread.start()
        print 'BEFORE COPY ' + str(parser_thread.scp_done)
        os.system('scp -C remote:' + files_to_copy + ' ' + '/destination/')
        print 'AFTER COPY ' + str(parser_thread.scp_done)
        parser_thread.stop()
        print 'AFTER STOP ' + str(parser_thread.scp_done)
        parser_thread.join()
        print 'AFTER JOIN ' + str(parser_thread.scp_done)

These are the prints from a test run:

BEFORE COPY: False
AFTER COPY False
Stopping: True
AFTER STOP True
Status False 1
Status False 1
Status False 1
Status False 1
Status False 1
Status False 1

Processes don't share memory by default - each process has its own memory "slice". You may want to read the multiprocessing documentation on sharing state — loopbackbee
– loopbackbee, Commented May 9, 2014 at 13:22
os.system('rm -rf ' + f + '*'): rm -rf and * in the same statement freaks me out. :) — unutbu
– unutbu, Commented May 9, 2014 at 13:25
@unutbu :) thanks, files aren't very important but the * wasn't really needed — liarspocker
– liarspocker, Commented May 9, 2014 at 13:29
@goncalopp but in that case how can I update the variable I initally send to the process? — liarspocker
– liarspocker, Commented May 9, 2014 at 13:38

unutbu · Accepted Answer · 2014-05-09 13:44:01Z

On Unix, subprocesses are spawned using os.fork. The fork copies globals (or uses copy-on-write) so each process (the parent and the child) have their own copy of the globals. On Windows, a second Python process is spawned and the calling module is imported. In either case (on Unix or Windows) when the parent modifies a global variable, the child's global variable of the same name does not change. Thus calling parser_thread.stop() in the main process does not affect the value of self.scp_done in the subprocess.

multiprocessing does provide certain objects which can facilitate sharing state between processes, such as mp.Value. For a simple boolean value, you could also use an mp.Event:

import multiprocessing as mp
import time

class LogParserProcess(mp.Process):
    def __init__(self):
        super(LogParserProcess, self).__init__()
        self.done = mp.Event()
        self.flights_to_add = ()

    def stop(self):
        self.done.set()
        print 'Stopping: ' + str(self.done.is_set())

    def run(self):
        file_list = ()
        while not self.done.is_set() or len(file_list) > 0:
            print 'Status ' + str(self.done.is_set()) + ' ' + str(len(file_list))

files_to_copy = "file1 file2 file3"
proc = LogParserProcess()
proc.start()
print 'BEFORE COPY ' + str(proc.done.is_set())
time.sleep(1)
print 'AFTER COPY ' + str(proc.done.is_set())
proc.stop()
print 'AFTER STOP ' + str(proc.done.is_set())
proc.join()
print 'AFTER JOIN ' + str(proc.done.is_set())

prints

BEFORE COPY False
Status False 0
...
Status False 0
AFTER COPY False
Status False 0
Status False 0
Status False 0
Stopping: True
AFTER STOP True
AFTER JOIN True

Collectives™ on Stack Overflow

Python: can't set Process variable from outside Process

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related