python - multi-threading in a for loop

Question

This code runs ok for a little bit, then it gives me this error:

thread.error: can't start new thread

What am I doing wrong? The names file is about 10,000 names long, the email file is about 5 emails long.

for x in open(names):
    name = x.strip()


    def check(q):
        while True:
            email = q.get()
            lock.acquire()
            print email, name, threading.active_count()
            lock.release()

            #Do things in 
            #the internet

            q.task_done()
        return

    for i in range(threads):
            t = threading.Thread(target=check, args=(q,))
            t.setDaemon(True)
            t.start()

    for word in open(emails):
            q.put(word.strip())

    q.join()

I only specify 2 threads, but it ends up creating hundreds then crashes when the active_count is around 890. How can I fix this?

It seams you are creating threads for every execution of for x in open(names):. This is what you actually wanted? — Marcin
– Marcin, Commented Feb 16, 2015 at 2:17
Yes but it should finish before going on to the next name so that it doesn't create too many threads — user3893623
– user3893623, Commented Feb 16, 2015 at 2:19
It'd be easier to show you how to fix this if it was a real self-contained example, i.e. with some appropriate definitions of "names", "q", "emails", "threads", etc. — Josh Kupershmidt
– Josh Kupershmidt, Commented Feb 16, 2015 at 3:40
your check function never returns.so, those threads are still alive. — Himal
– Himal, Commented Feb 16, 2015 at 3:45
Josh, q = Queue.Queue(maxsize=0), @Himal, shouldn't it return when the Queue is empty? — user3893623
– user3893623, Commented Feb 16, 2015 at 4:40

Himal · Accepted Answer · 2015-02-16 10:55:36Z

Here is a slightly modified version using a semaphore object

import threading
import Queue

NUM_THREADS = 2 # you can change this if you want

semaphore = threading.Semaphore(NUM_THREADS)

threads = NUM_THREADS

running_threads = []

lock = threading.Lock()

q = Queue.Queue()

# moved the check function out of the loop
def check(name, q, s):
    # acquire the semaphore
    with s:
        not_empty = True

        while not_empty:

            try:
                email = q.get(False) # we are passing false so it won't block.
            except Queue.Empty, e:
                not_empty = False
                break

            lock.acquire()

            print email, name, threading.active_count()

            lock.release()

            # additional work ...

            q.task_done()

for x in open(names):
    name = x.strip()

    for word in open(emails):
        q.put(word.strip())

    for i in range(threads):
            t = threading.Thread(target=check, args=(name, q, semaphore))
            # t.setDaemon(True) # we are not setting the damenon flag
            t.start()

            running_threads.append(t)

    # joining threads (we need this if the daemon flag is false)
    for t in running_threads:
        t.join()

    # joining queue (Probably won't need this if the daemon flag is false)
    q.join()

jfs · Accepted Answer · 2015-02-16 03:43:39Z

0

You could simplify your code using a thread pool:

from contextlib import closing
from itertools import product
from multiprocessing.dummy import Pool # thread pool

def foo(arg):
    name, email = map(str.strip, arg)
    try:
        # "do things in the internet"
    except Exception as e:
        return (name, email), None, str(e)
    else:
        return (name, email), result, None

with open(names_filename) as names_file, \
     open(emails_filename) as emails_file, \
     closing(Pool(max_threads_count)) as pool:
    args = product(names_file, emails_file)
    it = pool.imap_unordered(foo, args, chunksize=100)
    for (name, email), result, error in it:
        if error is not None:
            print("Failed to foo {} {}: {}".format(name, email, error))

answered Feb 16, 2015 at 3:43

jfs

417k210 gold badges1k silver badges1.7k bronze badges

2 Comments

user3893623 Over a year ago

It doesn't continue, but only the first email/name is used?

jfs Over a year ago

@user3893623: update your question with the exact code that you use

Collectives™ on Stack Overflow

python - multi-threading in a for loop

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related