Queue threading status progress

Question

I'm trying to create a web extractor, I have this code for multithreads, and I need print the status/progress of the scanner :

import time
import threading
import Queue       
import sys

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

class WorkerThread(threading.Thread) :

    def __init__(self, queue) :
        threading.Thread.__init__(self)
        self.queue = queue

    def run(self) :
        while True :
            counter = self.queue.get()
            sys.stdout.write("line nr : \r")
            self.queue.task_done()      

queue = Queue.Queue()

for i in range(50) :
    worker = WorkerThread(queue)
    worker.setDaemon(True)
    worker.start()

for line in Lista:
    queue.put(line)

queue.join()

print "All task over!"

How can I print the status/progress when the scanner working, I'm tried len(queue) but it doesn't work?

@OfirIsrael: Queues don't have lengths, so you get a TypeError or AttributeError depending on your Python version, and presumably he doesn't consider that a correct way to get information for a progress bar… — abarnert
– abarnert, Commented Sep 9, 2013 at 20:15

abarnert · Accepted Answer · 2013-09-09 20:24:12Z

Queue objects do not have a len because, by their very nature, they're being shared across threads, and it would be inaccurate and misleading.

However, they have a qsize method that gives you the approximate size, for exactly this kind of purpose.

If you want exact values, you need a second Queue for that, where each task puts something on the out-queue, and some extra thread (or possibly the main thread) loops over it and counts up the tasks done so far. Or, alternatively, something simpler, like a global int counter with a global Lock protecting it.

However, I think it would be a lot simpler to write this in terms of a pool or executor. That would take care of queueing up tasks for you, and returning a value for each one to the main thread, without you needing to manage anything. For example, using futures, the backport of the Python 3.x concurrent.futures module for 2.x, here's your whole program, with progress added:

import sys
import futures 

try:
    Lista = open(sys.argv[1], "r").readlines()
except(IOError): 
    print "Error: Check your ip list path\n"
    sys.exit(1)

def task(line):
    # Do something

with futures.ThreadPoolExecutor(50) as executor:
    fs = [executor.submit(task, line) for line in Lista]
    for i, f in enumerate(futures.as_completed(fs)):
        sys.stdout.write("line nr: {} / {} \r".format(i, len(Lista)))

print "All task over!"

Collectives™ on Stack Overflow

Queue threading status progress

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related