multiprocessing map keeps running script not function

Question

hey im trying to speed up my code by using multiprocessing pool and in theory this:

    def render_frame(i):
        #os.system(f'python main.py video_imgs/out{i}.png {convo}')
        print(f'done {i}')
    
    if __name__ == '__main__':
        pic_list = list(range(1,num_of_pics))
        with Pool(10) as p:
            s = p.map(render_frame,pic_list)

should print done then the number i, but when I run the script it just gives me the input prompt I have at the beginning of the script 10 times, which is ->

    from multiprocessing import Pool
    import sys
    import os
    import time
    
    if(len(sys.argv) > 1):
        vidLoc = sys.argv[1]
        convo = sys.argv[2]
    else:
        vidLoc = input('video: ') #! <-- this is being repeated 10 times
        convo = input('convolute cfg file: ')

any help? thanks

The os.system() call is commented in the code you provided. Are you sure you provided the correct code snippet? — Thiago F. Pappacena
– Thiago F. Pappacena, Commented Nov 21, 2020 at 2:57
@ThiagoF.Pappacena yes it is the right snippet I was testing and seeing if that was the issue but to no avail — Potoo
– Potoo, Commented Nov 21, 2020 at 3:02
If you are hitting the else part in main.py, that means that there were no arguments passed to it likely from this line: #os.system(f'python main.py video_imgs/out{i}.png {convo}') <- what is {convo} in this? — Tim
– Tim, Commented Nov 21, 2020 at 3:16

ShadowRanger · Accepted Answer · 2020-11-21 03:19:06Z

On systems that use 'spawn' as the default multiprocessing start method (Windows, MacOS), you must guard all "main script-like" behaviors under the if __name__ == '__main__': guard. The 'spawn' method simulates a fork by importing the main module in each child process but with a non-__main__ name; if you do stuff like calling input outside the guard, it will get executed in your child processes. Generally, you should stick to simple imports and function/class/constant definitions outside the guard, never do anything "active" unless you're under the guard (e.g. avoid invoking functions with observable side-effects like input).

Guard the input code properly (you may just want to omit it as it's odd to allow optional command line arguments that you prompt for otherwise, and annoying to have them reflected in the child processes), and if you must use input, find other ways to initialize the variables in the child process (e.g. by means of an initializer function with initargs, both are optional arguments to Pool).

A rough example (your incomplete code makes it hard to match precisely):

from multiprocessing import Pool
import sys
import os
import time

def render_frame(i):
    #os.system(f'python main.py video_imgs/out{i}.png {convo}')
    print(f'done {i}')

def set_globals(v, c):
    global vidLoc, convo
    vidLoc = v
    convo = c

if __name__ == '__main__':
    if len(sys.argv) > 1:
        vidLoc = sys.argv[1]
        convo = sys.argv[2]
    else:
        vidLoc = input('video: ') #! <-- this is being repeated 10 times
        convo = input('convolute cfg file: ')

    pic_list = list(range(1,num_of_pics))
    with Pool(10, initializer=set_globals, initargs=(vidLoc, convo)) as p:
        s = p.map(render_frame,pic_list)

Collectives™ on Stack Overflow

multiprocessing map keeps running script not function

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related