Multiprocessing and lists python

Question

I am here with another simple question in which I really need some help. I have a list of websites and I want to go through them all with requests. However to make this faster, I want to use multi processing. How can I execute this?

Example:

import requests
import threading
from threading import Thread

list_ex = ["www.google.com","www.bing.com","www.yahoo.com"]
def list_request():
  for item in list_ex:
    ex = request.get(list_ex)
    print(ex.text)

How can I do this but with multi processing due to me having 100+ so sites :)

Mukti threading in Python wont give you better performance....even could be bad performance because python not really open multi threading it just make pause to one thread run the other and come back to the first one. — jossefaz
– jossefaz, Commented May 15, 2020 at 2:58
Well with this it would if I could run this process on multiple threads it would provide better performance than on one thread due to it iterating more list values at one time. — xnarf
– xnarf, Commented May 15, 2020 at 2:59
Did you check this out? stackoverflow.com/questions/38280094/… — JenilDave
– JenilDave, Commented May 15, 2020 at 2:59

M_x · Accepted Answer · 2020-10-26 11:34:14Z

1

you can use DASK for multiprocessing ..

It will uses all your system cores .. while traditional python uses only one core

Dask Official ..... Dask Wikipedia

answered Oct 26, 2020 at 11:34

M_x

8982 gold badges9 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Niklas Mertsch Over a year ago

Using dask probably is overkill in this case, though. For simple parallelization, the standard library provides multithreading and multiprocessing facilities.

Niklas Mertsch · Accepted Answer · 2020-10-26 11:55:21Z

0

Multithreading is an option here, because fetching a url is not CPU-bound, but I/O-bound. Thus, a single process can have multiple threads running requests.get in parallel.

import requests
from multiprocessing.pool import ThreadPool

def read_url(url):
    return requests.get(url).text


urls = ["www.google.com","www.bing.com","www.yahoo.com"]
with ThreadPool() as pool:
    texts = pool.map(read_url, urls)

You can use multiprocessing by replacing ThreadPool with Pool. For the three provided URLs, I get similar runtimes with Pool and ThreadPool, and both are faster than running read_url sequentially in a loop.

answered Oct 26, 2020 at 11:55

Niklas Mertsch

1,49915 silver badges24 bronze badges

Collectives™ on Stack Overflow

Multiprocessing and lists python

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related