Python Multiprocessing - Object has no attribute

Question

I'm trying to use multiprocessing on my script. But it doesn't work. what am I doing wrong? I searched too much but I didn't find the solution. Can you help me guys?

It seems HistogramMerger working with multiprocessing. I saw some print-out when I run the script but I don't get any result file which I normally getting with for loop.

I'm getting this error message:

AttributeError: 'module' object has no attribute 'histogramAdd'

ps: This histogram merger script merging multiple files to one single file. And, I'm trying to run this script faster than normal. If you know better solution, please let me know.

without multiprocessing (working)

from histogram_merger import HistogramMerger

var1=697
var2=722
with HistogramMerger("results/resMergedHistograms_"+str(var1)+"_"+str(var1)+".root") as hm:
    for i in xrange(var1,var2+1):
        print "Run Number : " +str(i)
        hm.addHistogramFile("../results/run"+str(i)+"/run"+str(i)+"_histo.root")

with MultiProcessing

from histogram_merger import HistogramMerger
from multiprocessing import Pool

var1=697
var2=722

##################################################

arrayOfNumbers = [xx for xx in range(var1, var2+1)]
print(arrayOfNumbers)
pool = Pool(8) 

def histogramAdd(run):
    print("Run Number : "+str(run))
    hm.addHistogramFile("../results/run"+str(run)+"/run"+str(run)+"_histo.root")

if __name__ == '__main__':
    with HistogramMerger("results/resMergedHistograms_"+str(var1)+"_"+str(var2)+".root") as hm: 
        pool.map(histogramAdd, arrayOfNumbers)
        pool.join()

user2201041 · Accepted Answer · 2018-11-23 16:40:48Z

1

The error message is odd. hm is not in scope within the function histogramAdd. I would expect something like NameError: name 'hm' is not defined. Perhaps there is some hm import you are not showing.

Regardless, you need to pass the object to the function. You can use functools.partial for this. E.g.

from functools import partial

# ...

def histogramAdd(run, hm):  # <- extra parameter!
    print("Run Number : "+str(run))
    hm.addHistogramFile("../results/run"+str(run)+"/run"+str(run)+"_histo.root")

if __name__ == '__main__':
    with HistogramMerger("results/resMergedHistograms_"+str(var1)+"_"+str(var2)+".root") as hm: 
        pool.map(partial(histogramAdd, hm), arrayOfNumbers)
        pool.join()

answered Nov 23, 2018 at 16:40

user2201041

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Multiprocessing - Object has no attribute

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related