22

I am running a number of threads and collecting there result on a queue. I would like to dump it into array or list so that I can do indexing and retrieve those results. Each of the elements in the queue is a array of dimension n. I would like to access those arrays. Would you please let me know, how would i do it?

 def dump_queue(model_queue):
 queue_list = []
 for i in iter(model_queue.get,'STOP'):
         queue_list.append(i)
  return queue_list




aux_model=train_svm(np.array(trainExample),np.array(trainLabel))
model_queue.put(aux_model.coef_)

Thus the arrays are the learned model parameters of svm. model_queue is shared among the threads. I want to access each of the model parameters vectors not each of the entries of a model parameters.

Improve this question
0

4 Answers 4

Reset to default
56

You're done with the parallel part and just want to get the results in a list, is that it? Then try:

list(my_queue.queue)

Example:

from queue import Queue

q = Queue()
for i in range(5):
    q.put(i)

l = list(q.queue)
print(l)

Output:

[0, 1, 2, 3, 4]
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

@AkshayLAradhya I've just tested in Python 3.6.5 and it works fine. Are you using the same import?
Sorry my bad. I thought this was for multiprocessing.Queue
In the interest of full disclosure I think it should noted that the Queue.queue attribute isn't documented although it doesn't appear to be private since it has no leading _ character, which makes its status at least a little bit nebulous in my opinion — in other words it seems like it could simply be an "implementation detail"…
Does this leave the items on the queue or removes them?
This wouldn't actually empty the queue though so it kind of defeats the purpose of using a queue, doesn't it? If I just wanted to collect items and end up with a list, wouldn't I have just used a list from the start?
7

Not sure whether it's the same problem, but I needed to do the same thing and ended up writing this. I am using threading.Thread and the Queue object in python 2.7. and just wanted to dump out a queue to a list.

def queue_to_list(q):
    """ Dump a Queue to a list """

    # A new list
    l = []

    while q.qsize() > 0:
        l.append(q.get())
    return l
Improve this answer

Comments

1

I would go with the following solution:

from collections import deque
q = deque()
for i in range(5):
    q.append([i,i+10])
listed_q = list(q)

Now you can easily access the values by indexing or slicing:

print listed_q[1]        # [1, 11]
print listed_q[1][1]     # 11

Since list() creates a copy of q, you should keep an eye on the dimensionality of your queue. The time needed to do this operation grows the longer your queue is. An alternative approach can be found by using itertools.islice where you first slice the queue before you store the outcome in a list. Check out the following links (performance measures are also given there):

Use slice notation with collections.deque

How to slice a deque? [duplicate]

Improve this answer

Comments

1

Like Gabriel said, you should specifiy your questions.

When you speak of arrays, I think you refer to lists. Python Arrays are used to store numerical values.

However, what you could do to transform your queue to an nested list:# output list:

collectedData = list()
# # # 
# for every thread, call:
threadBuffer = [item for item in q.queue]
collectedData.append(threadBuffer)

This will collect your data in nested lists, which you can easily access, like in this example:

l = list()
l.append([1,2,3,4,5,6,7,9,10])
l.append([6,7,8,9,0,11,22,33])

Now you can access freely:

l[1][2] -> 8
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.