Multi Processing in Python for loop

Question

I have written a python code with multi-processing. I did not see much of a performance improvement. I will share the code below. Please help me to understand if there is something missing?

if len(aspackets) != len(uepackets):
    sys.exit("Packet number miss match between UE and AS")
else:
    for i in range (1,100):
        p = multiprocessing.Process(target=packet,args=(i,))
        p.start()
        #p.join()
        E2E(packet(i),i)
        Airinterface(packet(i))
        core(packet(i))
        switch(packet(i))
        inet(packet(i))
        bridge(packet(i))

The packet function is as

def packet(i):
    return aspackets[i]

In my logic, I want to start the processing of multiple packets from a list of 100 packets. The processing of one packet does not affect the other packet. That is, it is okay to start my computation on packet 10 and 33 at the same time. In the normal flow(without multi-processing), I was going sequentiall from 1st to the last packet and had to wait for 45 mins to get the whole results.

I am running this code on Ubuntu machine which is running inside Virtual Box.

If the code is working fine and you would just like help on improving it, then it is more of a code review question: codereview.stackexchange.com — Shaun Barney
– Shaun Barney, Commented Aug 4, 2017 at 8:43
There is no improvement because there is only one process at a time fetching the packet(i). Since this is done over an iterable i suggest you try to use Pool — bergerg
– bergerg, Commented Aug 4, 2017 at 8:46
@nutmeg64: So if its an iterable i, multi-processing won work? — Ashish Kurian
– Ashish Kurian, Commented Aug 4, 2017 at 8:49
It will work just not in the way you do it. Please read about Pool and correct your code accordingly. — bergerg
– bergerg, Commented Aug 4, 2017 at 8:59

bergerg · Accepted Answer · 2017-08-04 09:27:50Z

1

A solution using Pool:

from multiprocessing import Pool

def handle_packet(index):
    pac = aspackets[index]
    E2E(pac, index)
    Airinterface(pac)
    core(pac)
    switch(pac)
    inet(pac)
    bridge(pac)

with Pool(processes=10) as p:
    p.map(handle_packet, range(100))

answered Aug 4, 2017 at 9:27

bergerg

9959 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ashish Kurian Over a year ago

Thanks for your help. It worked. I can see that the print statements are juggled suggesting that the pools are processing paralelly. But is it possible to output the content again in sequential order?

bergerg Over a year ago

Yes. Just write print(p.map(handle_packet, range(100))) and make sure that handle_packet returns the result you wish to print.

Ashish Kurian Over a year ago

It is actually printing, but what I want is that the final results be printed from loop 1 to 100 and not just random as per which thread starts first. But if it is not possible, I can just do some text file processing afterwards also.

Collectives™ on Stack Overflow

Multi Processing in Python for loop

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related