Python multiprocessing not working while for loop works

Question

I am running multiprocessing to shorten my optimization time for multi periods.

Before I was using for loop, which is fast when my data is not large.

for date in DatesOpt:
  x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
  f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
  d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
  r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
  optimize(date,x,f,d,r)

The optimize function outputs the optimized result to a csv file.

However, when I changed it to

if __name__=='__main__':
  pool = mp.Pool(mp.cpu_count()-1)
  for date in DatesOpt:
    x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
    f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
    d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
    r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
    pool.apply_async(optimize,args=(date,x,f,d,r,))
  print('Waiting for all subprocesses done')
  pool.close()
  print('Pool Closed')
  pool.join()
  print('All subprocess done.')

everything just stops at the "Pool Closed" output and the optimizer never ends.

Is there any problem with this code?

Netwave · Accepted Answer · 2017-06-27 08:55:56Z

2

You are closing the pool before waiting, you better do it reversed:

if __name__=='__main__':
  pool = mp.Pool(mp.cpu_count()-1)
  for date in DatesOpt:
    x = X.loc[X['Date'] == np.int(date)].drop('Date',1) 
    f = F.loc[F['Date'] == np.int(date)].drop('Date',1)
    d = D.loc[D['Date'] == np.int(date)].drop('Date',1)
    r = R.loc[R['Date'] == np.int(date)].drop('Date',1)
    pool.apply_async(optimize,args=(date,x,f,d,r,))
  print('Waiting for all subprocesses done')
  pool.join()
  pool.close()
  print('Pool Closed')
  print('All subprocess done.')

answered Jun 27, 2017 at 8:55

Netwave

43.2k6 gold badges69 silver badges110 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Elaine Over a year ago

Thanks Daniel. I tried the method and an error says: pool.join() File "D:\Anaconda\lib\multiprocessing\pool.py", line 509, in join assert self._state in (CLOSE, TERMINATE) AssertionError

Collectives™ on Stack Overflow

Python multiprocessing not working while for loop works

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related