python multiprocessing for loop is not running for all loop arguments

Question

Here is the code I am trying to do. The output text file should contain 500 data. But it is always less than 500 (450 or 476 or 429 when I run). Any idea why this is happening and what should I do to get 500 data in output. It will be very helpful if I get output in order.

def foo(j):        
        output=[j]        
        f=open('output.txt','a')
        f.write('\n')
        
        np.savetxt(f,output)
        f.close()



if __name__=='__main__':
    pool = Pool(processes=4)
    pool.map(foo,range(500))

Shag · Accepted Answer · 2021-01-25 05:43:35Z

1

Try creating chunks before hand. For example

def f_amp(inputs):
    chunks = [inputs for inputs in range(500)]
 
    pool = Pool(processes=4)
 
    result = pool.map(f, chunks)

Also you can refer here for solutions.

edited Jan 25, 2021 at 5:43

answered Jan 23, 2021 at 13:34

Shag

4302 silver badges6 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python multiprocessing for loop is not running for all loop arguments

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related