Save output of four parallely running python program to different log files

Question

I have four different programs running which is started using a single python program.

import os                                                                       
from multiprocessing import Pool 
import sys   
processes = ('p1.py', 'p2.py', 'p3.py','p4.py')
def run_process(process):                                                             
    os.system('python3 {}'.format(process)) 
pool = Pool(processes=4)                                                        
pool.map(run_process, processes)

currently iam getting the log of all programs in a single file using nohup pmain.py>test.log But how can i get four different logs of p1,p2,p3 and p4 respectively in different log files.

A simple fix would be to processes = ('p1.py > p1.log', 'p2.py > p2.log', 'p3.py > p3.log', 'p4.py > p4.log'). But don't do that. And don't do os.system as well — han solo
– han solo, Commented Dec 31, 2019 at 4:28
Does this answer your question? Log output of multiprocessing.Process — abhilb
– abhilb, Commented Dec 31, 2019 at 4:31
docs.python.org/3/howto/… ,,, docs.python.org/3/library/… — wwii
– wwii, Commented Dec 31, 2019 at 4:49
Or this?..How should I log while using multiprocessing in Python? — wwii
– wwii, Commented Dec 31, 2019 at 4:53

han solo · Accepted Answer · 2019-12-31 04:35:51Z

4

A simple fix will be to do,

import os                                                                       
from multiprocessing import Pool 
import sys   
processes = ('p1.py > p1.log', 'p2.py > p2.log', 'p3.py > p3.log','p4.py > p4.log')
def run_process(process):                                                             
    os.system('python3 {}'.format(process)) 
pool = Pool(processes=4)                                                        
pool.map(run_process, processes)

But don't do the above.

The right way to do that would be using subprocess with something like,

import subprocess
processes = ('p1.py', 'p2.py', 'p3.py','p4.py')
procs = []
for pname in processes:
  logfile = os.path.splitext(pname)[0] + '.log'
  with open(logfile, 'w') as f:
    proc = subprocess.Popen(['python3', pname], stdout=f)
    procs.append(proc)

for proc in procs:
  proc.wait()

edited Dec 31, 2019 at 4:35

answered Dec 31, 2019 at 4:29

han solo

6,6501 gold badge20 silver badges22 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

sharath Over a year ago

why do you prefer the second way instead of the first method.

han solo Over a year ago

Because it is frowned upon to directly use os.system, and the second method is very readable, and does the same thing

han solo Over a year ago

subprocess.Popen is a non-blocking call. One would want to wait for all the processes to exit ?

han solo Over a year ago

No problem. I am glad, i was able to help

Collectives™ on Stack Overflow

Save output of four parallely running python program to different log files

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related