writing output to a file with subprocess in python

Question

I have a code which spawns at the max 4 processes at a go. It looks for any news jobs submitted and if it exists it runs the python code

for index,row in enumerate(rows):
    if index < 4:
        dirs=row[0]
        dirName=os.path.join(homeFolder,dirs)
        logFile=os.path.join(dirName,(dirs+".log"))
        proc=subprocess.Popen(["python","test.py",dirs],stdout=open(logFile,'w'))

I have few questions:

When I try to write the output or errors in log file it does not write into the file until the process finishes.Is it possible to write the output in the file as the process runs as this will help to know at what stage it is running.
When one process finishes, I want the next job in the queue to be run rather than waiting for all child processes to finish and then the daemon starts any new.

Any help will be appreciated.Thanks!

Please take a bit of care with your formatting. It's not hard to do and you've got a preview right there before you post. — Chris Morgan
– Chris Morgan, Commented Aug 25, 2011 at 10:45
I am sorry, I will be careful in future. Thanks for correcting me. — shash
– shash, Commented Aug 25, 2011 at 10:56

user647772 · Accepted Answer · 2011-08-25 10:49:25Z

1

For 2. you can take a look at http://docs.python.org/library/multiprocessing.html

answered Aug 25, 2011 at 10:49

user647772

Sign up to request clarification or add additional context in comments.

Comments

maazza · Accepted Answer · 2016-03-07 13:41:15Z

1

Concerning point 1, try to adjust the buffering used for the log file:

open(logFile,'w', 1) # line-buffered (writes to the file after each logged line)
open(logFile,'w', 0) # unbuffered (should immediately write to the file)

If it suits your need, you should choose line-buffered instead of unbuffered.

Concerning your general problem, as @Tichodroma suggests, you should have a try with Python's multiprocessing module.

edited Mar 7, 2016 at 13:41

maazza

7,25116 gold badges67 silver badges98 bronze badges

answered Aug 25, 2011 at 11:40

Oben Sonne

9,9732 gold badges42 silver badges61 bronze badges

1 Comment

shash Over a year ago

I tried both the options but it does not work. It buffers everything and then write it to the file at the end. proc=subprocess.Popen(["python","test.py",dirs],stdout=open(logFile,’w’,0))proc=subprocess.Popen(["python","test.py",dirs],stdout=open(logFile,’w’,1))

Collectives™ on Stack Overflow

writing output to a file with subprocess in python

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related