Python parallel processing from client-server application

Question

I have a web application (django) that stores in mysql database PID numbers of processes from remote Linux machine. I designed a simple server-client application that talking to remote server and getting me some data about given PID number (cpu%, mem%) ... this data is from 5s interval.

But there is a performance problem .... I have 200 pids to check and every of them takes ~5 sec and they are processing in the for loop. So I have situation where I`m waiting 200*5 sec minimum

Can somebody advise me how to make it parallel processing? So my application will be able to fetch for example 50 pids at one time ... I believe python client - server library can handle multiple requests coming to the server.

I want to archive something like:

for pid in my_200_pid_list:
  // Some parallel magic to not wait and pass another 49...
  result[pid] = askforprocess(pid)

My client code:

def askforprocess(processpid):
    #Create TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect on host and port provided in command line arguments
    server_address = ('172.16.1.105', int('5055'))
    sock.connect(server_address)

    # Send the data
    try:
            message = processpid
            sock.sendall(message)

            data = sock.recv(2048)

    finally:
            sock.close()


    return data

notorious.no · Accepted Answer · 2015-04-09 14:10:13Z

In general, it's best to do stuff like this using a single thread when possible. You just have to make sure your functions don't block other functions. The builtin lib that comes to mind is select. Unfortunately, it's a bit difficult to explain and I haven't used it in quite some time. Hopefully this link will help you understand it http://pymotw.com/2/select/.

You can also use the multiprocessing lib and poll each pid in a separate thread. This can be very difficult to manage if you plan to scale out further! Use threads only as a last resort (this is my usual rule of thumb when it comes to threads). https://docs.python.org/2/library/multiprocessing.html#module-multiprocessing

from multiprocessing import Process

def askforprocess(processpid):
    #Create TCP/IP socket
    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

    # Connect on host and port provided in command line arguments
    server_address = ('172.16.1.105', int('5055'))
    sock.connect(server_address)

    # Send the data
    try:
            message = processpid
            sock.sendall(message)
            data = sock.recv(2048)
    finally:
            sock.close()
    return data

if __name__ == '__main__':
    info('main line')
    p = Process(target=askforprocess, args=(processpid,))
    p.start()

Lastly, there's Twisted library which is probably the most difficult to understand, but defiantly makes concurrent (not necessarily parallel) functions easy to write. Only bad thing is you'd probably have to rewrite your entire app in order to use Twisted. Don't be put off by this fact, try to use it if you can.

Hope that helps.

Mykhaylo Kopytonenko · Accepted Answer · 2015-04-09 13:50:02Z

0

Use threads to process your requests in parallel: https://docs.python.org/2/library/threading.html

answered Apr 9, 2015 at 13:50

Mykhaylo Kopytonenko

9837 silver badges15 bronze badges

1 Comment

notorious.no Over a year ago

actually, using the basic threading lib will not work because the sock.recv() function is a blocking function (even when using threading). If you don't want to block, then use multiprocessing docs.python.org/2/library/….

Collectives™ on Stack Overflow

Python parallel processing from client-server application

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related