Python multiprocessing error

Question

I am new to multiprocessing in Python. I have this below code which is very simple. It does not work though! It just prints the Start and End. It never prints the "Into Run". What am I missing? I am on Windows, Python 2.6

import logging, sys
from multiprocessing import Process

class ZincDataExtract(Process):
    def __init__(self, sources=None):
        Process.__init__(self)
        if sources is None:
            self.src = 'Everything'
        else:
            self.src = sources

    def run(self):
        print "Into Run: " + str(self.src)
        sys.stdout.flush()

def main():

    from datetime import datetime
    cobDate = datetime.strptime('2013-11-29', '%Y-%m-%d').date()

    logging.info("Start: " + datetime.now().strftime('%Y-%m-%d %H:%M:%S'))

    procs = []
    sources='CRE,Mortgage,OGRE,RV2,TDR,Tiger,CIDW'
    p1 = Process(target=ZincDataExtract, args=(sources))
    procs.append(p1)
    p1.start()

    sources='RAM'
    p2 = Process(target=ZincDataExtract, args=(sources))
    procs.append(p2)
    p2.start()

    for p in procs:
        p.join()

    logging.info("End: " + datetime.now().strftime('%Y-%m-%d %H:%M:%S'))

Have you tried to reproduce the problem using only a couple of lines? — loopbackbee
– loopbackbee, Commented Dec 27, 2013 at 16:12
My understanding was that p.start() will call the run() method! — SreeniH
– SreeniH, Commented Dec 27, 2013 at 16:16

alexoneill · Accepted Answer · 2013-12-27 17:13:46Z

2

As you are extending the multiprocessing.Process class, you should treat ZincDataExtract as if it were that class itself. You are already calling the __init__() method of the super-class (line 5), so you shouldn't need to instantiate two Process classes.

To make the code work, change the lines:

p1 = Process(target=ZincDataExtract, args=(sources))
...
p2 = Process(target=ZincDataExtract, args=(sources))

To:

p1 = ZincDataExtract(sources)
...
p2 = ZincDataExtract(sources)

This will create two instances of ZincDataExtract instead of the two instances of Process, and the code will work!

answered Dec 27, 2013 at 17:13

alexoneill

5383 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Hannes Ovrén · Accepted Answer · 2013-12-27 17:21:53Z

Alex O'Neill described what you should do to fix the problem, but did not give an explanation, so here goes...

When you run this line

 p1 = Process(target=ZincDataExtract, args=(sources))

I think you expected it to create a new process of your sub class, and then run as that. But this is not what happens. What happens is that a Process object is created, which uses the function pointed to by target= as the the function to run in that process. In this case, it is a class reference. That means it will run ZincDataExtract.__init__ and then exit. The ZincDataExtract object is created, but is never run, and as the __init__ function finishes, the Process instance created also exits.

That is why you should instead change the above line to

p1 = ZincDataExtract(sources)

since this will create the kind of process you actually expected.

Collectives™ on Stack Overflow

Python multiprocessing error

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related