How to limit the number of concurrent processes using subprocess module in asyncio python

Question

import asyncio
import asyncio.subprocess
args="blah blah argument "     
create=asyncio.create_subprocess_shell(args,stdout=asyncio.subprocess.PIPE)
proc = await create
output= await proc.stdout.read( )

This is a part of my server code , which gets 1000s of parallel hits from clients.Now how should i limit the maximum number of subprocesses created by the server to run the argument blah blah . As this is code is using 100% of my cpu. I need to deploy other servers on smae cpu

Is it worth considering something like Celery and RabbitMQ? It might give you more scaleability and finer control in the long run. Just a thought. — Andrew Guy
– Andrew Guy, Commented Sep 4, 2017 at 6:58
The client connects through websocket. And the process i run is cpu bound .so i needed asynchronus support ,that made me to prefer asyncio . Does Celery and RMQ support this? I never used them :/ — Aravind
– Aravind, Commented Sep 4, 2017 at 7:13
Celery is an asynchronous task queue, whereas RabbitMQ is the task broker. This article gives a nice example - suzannewang.com/celery-rabbitmq-tutorial — Andrew Guy
– Andrew Guy, Commented Sep 4, 2017 at 7:27
It might not be suitable for your application, but worth exploring if you haven't come across it before. — Andrew Guy
– Andrew Guy, Commented Sep 4, 2017 at 7:30

Ogaga Uzoh · Accepted Answer · 2018-03-06 17:59:12Z

5

asyncio.Semaphore is a way of limiting internal counter of simultaneous jobs:

sem = asyncio.Semaphore(10)

async def do_job(args):
    async with sem:  # Don't run more than 10 simultaneous jobs below
        proc = await asyncio.create_subprocess_shell(args, stdout=PIPE)
        output = await proc.stdout.read()
        return output

Note, you should be sure count of jobs doesn't increase much faster then you can actually do them. Otherwise, you'll need something more complex than that.

edited Mar 6, 2018 at 17:59

Ogaga Uzoh

2,2671 gold badge12 silver badges12 bronze badges

answered Sep 4, 2017 at 9:28

Mikhail Gerasimov

40.1k20 gold badges126 silver badges172 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to limit the number of concurrent processes using subprocess module in asyncio python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related