asynchronous HTTP POST requests in Python

Question

I have a code which posts HTTP request sequentially for all the tuples in a list and append the response to another list.

import requests

url = 'https://www....'
input_list = [(1,2),(6,4),(7,8)]
output_list = []

for i in input_list:
   resp = (requests.post('url',data = i[1]).text,i[0])
   output_list.append(resp)

print(output_list)

Can someone please help me with the directions to make the HTTP requests in parallel ?

You can do it in two ways, either using multiprocessing (all at once) or asynchronously (one by one, but overlapping while executing), You have to first decide what is the most time consuming part, is that you have to wait a lot after you make the request, then I can help you write a solution with async programming or does the processing of the whole loop take the most time, then I can help you with multiprocessing, but first you have to decide which concurrency approach gives the best solution for your problem. — SAK
– SAK, Commented Jun 6, 2021 at 6:52
@SAK, There is not a big wait time i'll say. So async programming will be a good suit for my problem. — mba026
– mba026, Commented Jun 6, 2021 at 6:56
If there is no wait time after sending the request then async programming will actually make it slower, for example if you make the request and it takes 500 ms to get the response then while the system waits for the response it will execute the next iteration in the loop, instead if it only takes like 5ms to complete then async will actually add more overhead and make it slower. — SAK
– SAK, Commented Jun 6, 2021 at 7:01
@SAK, Each request is taking ~60 ms. Which approach do you think will be efficient ? — mba026
– mba026, Commented Jun 6, 2021 at 7:04
60ms is actually way to less for multiprocessing and not suitable for async, but that said if you really want to save time in this case it would be setting up a multiprocessing pool executor which is the most easiest and standard way of doing in python. I would post an answer with the multiprocessing solution in a shortwhile. — SAK
– SAK, Commented Jun 6, 2021 at 7:06

Kryštof Vosyka · Accepted Answer · 2021-06-06 07:14:01Z

2

Since requests library doesn't support asyncio natively, I'd use multiprocessing.pool.ThreadPool, assuming the most of the time is spent waiting for IO. Otherwise it might be beneficial to use multiprocessing.Pool

from multiprocessing.pool import ThreadPool
import requests

url = 'https://www....'
input_list = [(1,2),(6,4),(7,8)]

def get_url(i):
  return (requests.post('url',data = i[1]).text,i[0])

with ThreadPool(10) as pool: #ten requests to run in paralel
  output_list = list(pool.map(get_url, input_list))

edited Jun 6, 2021 at 7:14

answered Jun 6, 2021 at 7:11

Kryštof Vosyka

5853 silver badges16 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Yu Da Chi Over a year ago

threadpool is now min(32, your_cpu+4) so you no need to define workers, moreover asyncio.to_thread looks much more neat

Collectives™ on Stack Overflow

asynchronous HTTP POST requests in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related