Why global var is not incremented when using threads in python?

Question

For my code, pytest_status['finished'] variable is incrementing very interesting way. Sometimes i can see '2' or '3' (I expect corrent incrementing for every run_test function call):

test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 1/53: success
test 2/53: success
test 1/53: success
test 1/53: success

and so on. I specially moved it out of lock, my code:

import subprocess
from multiprocessing import Pool, Lock

pytest_status = {
    'finished' : 0,
    'total'    : 0
}

print_lock = Lock()

def run_test(test):
    status = subprocess.call(...)

    global pytest_status
    pytest_status['finished'] += 1

    print_lock.acquire()
    print 'test ' + str(pytest_status['finished']) + '/' + str(pytest_status['total']) + ': ' + ('success' if status == 0 else 'failure')    
    print_lock.release()

def main():
    params = [...]

    global pytest_status    
    pytest_status['total'] = len(params)

    print 'will perform ' + str(pytest_status['total']) + ' tests'

    pool = Pool(30)
    pool.map(run_test, params)
    pool.close()
    pool.join()

if __name__ == '__main__':
    main()

Aside from the thread/process issue, you'll want to lock access to the object while updating as well as printing it. (Both in case the += 1 is not atomic as well as to ensure the print statement has the correct value.) — chepner
– chepner, Commented Oct 8, 2014 at 13:06
@chepner but when i use lock.. processes, that are not able to acquire lock just skip locked part of code?! — avasin
– avasin, Commented Oct 8, 2014 at 13:18
That sounds ... wrong. A lock should cause the process to block until they can acquire the lock and execute the protected code. Otherwise, you have no idea which processes might have executed the code, or whether or not an individual process has executed a particular block. — chepner
– chepner, Commented Oct 8, 2014 at 13:21

Daniel Roseman · Accepted Answer · 2014-10-08 12:58:32Z

3

You're not using threads. You're using processes. By definition, these have separate copies of all data. The only time you'd get 2 or 3 is when the same process happened to run more than one of the jobs.

answered Oct 8, 2014 at 12:58

Daniel Roseman

602k68 gold badges910 silver badges923 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

avasin Over a year ago

Oh.. god. Is there a way to share variable between them somehow?

Daniel Roseman Over a year ago

See the multiprocesing docs for examples of how to use Value or Array to do this.

sweber Over a year ago

Please also keep in mind this: If you have a global variable 'a' and use it inside a function, the latter one is just a copy of the global one. You have to write 'global a' once, if you really want to change the content of the global variable! This should not apply for your code, as pytest_status is a reference to variables you change, not a variable itself.

Darth Kotik · Accepted Answer · 2014-10-08 13:05:14Z

1

You can use Pipe or Query if you want to transfer data between processes.

answered Oct 8, 2014 at 13:05

Darth Kotik

2,3811 gold badge23 silver badges29 bronze badges

2 Comments

avasin Over a year ago

In fact yes, but i suppose it's overengineered solution there, in python should be standard solutions for this.

Darth Kotik Over a year ago

well Value and Array offerd by Daniel seems nice. Completly forget about it.

Collectives™ on Stack Overflow

Why global var is not incremented when using threads in python?

2 Answers 2

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related