Launch a single python script as different processes differing by command line arguments

Question

I have python script that takes command line arguments. The way I get the command line arguments is by reading a mongo database. I need to iterate over the mongo query and launch a different process for the single script with different command line arguments from the mongo query.

Key is, I need the launched processes to be:

separate processes share nothing
when killing the process, I need to be able to kill them all easily.

I think the command killall -9 script.py would work and satisfies the second constraint.

Edit 1

From the answer below, the launcher.py program looks like this

def main():


    symbolPreDict = initializeGetMongoAllSymbols()
    keys = sorted(symbolPreDict.keys())

    for symbol in keys:
            # Display key.
            print(symbol)
            command = ['python', 'mc.py', '-s', str(symbol)]
            print command
            subprocess.call(command)

if __name__ == '__main__':
     main()

The problem is that mc.py has a call that blocks

receiver = multicast.MulticastUDPReceiver ("192.168.0.2", symbolMCIPAddrStr, symbolMCPort ) 
while True: 
    try:
        b = MD()

        data = receiver.read() # This blocks
        ...
    except Exception, e:
        print str(e)

When I run the launcher, it just executes one of the mc.py (there are at least 39). How do I modify the launcher program to say "run the launched script in background" so that the script returns to the launcher to launch more scripts?

Edit 2

The problem is solved by replacing subprocess.call(command) with subprocess.Popen(command)

One thing I noticed though, if I say ps ax | grep mc.py, the PID seem to be all different. I don't think I care since I can kill them all pretty easily with killall.

[Correction] kill them with pkill -f xxx.py

You probably want multiprocessing not subprocess docs.python.org/3/library/multiprocessing.html — Padraic Cunningham
– Padraic Cunningham, Commented Apr 8, 2016 at 22:24
Yes but horses for courses, if you want to run and control multiple processes then use multiprocessing, hence the name. — Padraic Cunningham
– Padraic Cunningham, Commented Apr 8, 2016 at 22:26

MS-DDOS · Accepted Answer · 2016-04-08 20:21:27Z

1

There are several options for launching scripts from a script. The easiest are probably to use the subprocess or os modules.

I have done this several times to launch things to separate nodes on a cluster. Using os it might look something like this:

import os
for i in range(len(operations)):
     os.system("python myScript.py {:} {:} > out.log".format(arg1,arg2))

using killall you should have no problem terminating processes spawned this way.

Another option is to use subprocess which has got a wide range of features and is much more flexible than os.system. An example might look like:

import subprocess
for i in range(len(operations)):
    command = ['python','myScript.py','arg1','arg2']
    subprocess.call(command)

In both of these methods, the processes are independent and share nothing other than a parent PID.

answered Apr 8, 2016 at 20:21

MS-DDOS

5765 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Ivan Over a year ago

I gave you credit because you answered my question. Please See Edit 1 for an unanticipated problem.

Ivan Over a year ago

I figured it out Popen.

Padraic Cunningham Over a year ago

call blocks so you won't be running one process untll the previous process returns

Ivan Over a year ago

That does not appear to be the case. See Edit 2.

MS-DDOS Over a year ago

Popen will definitely work. But as a note to others reading this question, so will os.system() by simply adding an & to the end of the command. On UNIX/LINUX/UNIX-like systems the & runs a process in the background. For example try os.system('sleep 3') versus os.system('sleep 3 &')

Collectives™ on Stack Overflow

Launch a single python script as different processes differing by command line arguments

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related