Python PyQT/PySide QThread limiting

Question

I have problem with thread limiting. I want to do it using QThread. So SpiderThread is QThread object crawling some urls. But I want to limit working threads to X threads at once. I have done it earlier with threadpool and QRunnable but it's buggy in pyside when numbers of urls are big. So I have this simple code:

 self.threads = []
    for url in self.urls:
        th = SpiderThread(url)
        th.updateresultsSignal.connect(self.update_results)
        self.threads.append(th)
        th.start()

Anyone have working example of limiting threads using QThread ?

Oliver · Accepted Answer · 2014-02-24 21:52:02Z

2

So you want to have at most X threads running at any given time? So how about a URL queue shared by 10 threads:

self.threads = []
queueu = Queue(self.urls) # replace with a sync queue
for i in xrange(1,10):
    th = SpiderThread(queue)
    th.updateresultsSignal.connect(self.update_results)
    self.threads.append(th)
    th.start()

Then in the run of each thread, the thread gets a URL off the queue (so removes it from queue), and when it is done processing the URL, it gets a new one. In pseudocode:

class SpiderThread(Thread):
    def __init__(self, queue):
        self.queue = queue
    def run(self):
        while not self.queue.empty():
            maxWait = 100 # miliseconds
            try: 
                url = self.queue.get(true, maxWait)
                process(url)
            except Queue.Empty:
                break # no more URLs, work completed!

edited Feb 24, 2014 at 21:52

answered Feb 24, 2014 at 15:07

Oliver

30k12 gold badges81 silver badges114 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nuncjo Over a year ago

Sorry the example is too messy for me. Did You mean adding all urls to Queue and then url = queue.get() (not pop) or did you mean using list and then pop. I think it will couse that urls will be processed 10 times - once by each thread, not 1 time by 10 threads.

Nuncjo Over a year ago

Ok nvm, i managed this work. I will test it for a while thanks.

Oliver Over a year ago

You have to pop each url off the queue otherwise other threads will process same URL. I've cleaned up the example a bit for proper method names etc.

Collectives™ on Stack Overflow

Python PyQT/PySide QThread limiting

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related