Python threading using a wrapper on thread-safe objects

Question

I am starting to learn about using threading in python, and I wanted to see if a thread safe object would still be safe if wrapped in a class or other object.

I wrote a simple wrapper for a queue:

queueWrapper.py

import queue
import time

class queueWrapper(object):

def __init__(self):
    self.queue = queue.Queue()

def addItem(self, id, item):
    try:
        self.queue.put(f'Item {id}:{item}')
    except:
        print('Add Error')

def getItem(self):
    try:
        item = self.queue.get()
    except:
        print('Get Error')
    else:
        self.queue.task_done()

    return item

I then wrote this to use it:

main.py

import threading
import time
from queueWrapper import *

def producer(id, q, lock):
    for i in range(10):
         # time.sleep(1)
         with lock:
             q.addItem(id, i)

         print(f'Worker {id}: Adding {i}')

def consumer(id, q, lock):
    for i in range(20):
        # time.sleep(0.1)
        with lock:
            item = q.getItem()

        if item:
            print(f'Get: Got {item}')


lock = threading.Lock()
q = queueWrapper()
producer1 = threading.Thread(target = producer, name = 'Add 1', args = {1, q, lock})
producer2 = threading.Thread(target = producer, name = 'Add 2', args = {2, q, lock})
consumer1 = threading.Thread(target = consumer, name = 'Get', args ={1, q, lock})

producer1.start()
producer2.start()
consumer1.start()

producer1.join()
producer2.join()
consumer1.join()

Before I added the locks i was getting the error below. After I added in the locks, on some runs it will run to completion but on others it will still give the error

Exception in thread Add 2:
Traceback (most recent call last):
    File "/usr/lib/python3.7/threading.py", line 917, in _bootstrap_inner
        self.run()
    File "/usr/lib/python3.7/threading.py", line 865, in run
        self._target(*self._args, **self._kwargs)
    File "/home/fleetscut/programming/python/threads/queue_wrapper/main.py", line 9, in producer
        q.addItem(id, i)
AttributeError: 'int' object has no attribute 'addItem'

I thought it might be due to trying to add/get items too quickly so i added the time.sleep calls in each thread method, however the error started happening on all runs.

Side-note: Your wrapper is pointless from a threading perspective, all it does is format inputs into strings before inserting them. You made an unbounded queue, so put and get will always succeed (barring weird circumstances like a user hitting Ctrl-C to cancel the program). get might block, but as long as something is eventually put, it will succeed. — ShadowRanger
– ShadowRanger, Commented Jul 28, 2019 at 14:15

Carcigenicate · Accepted Answer · 2019-07-28 14:26:29Z

1

Don't use a set to pass the arguments. Sets are, in many versions of Python, unordered. You can't guarantee that when iterated they'll be in the same order as the literal was in. It seems the 1 and q are swapping places while being iterated. I can semi-reproduce this in my version of Python:

q = queueWrapper()
lock = threading.Lock()

print(*{1, q, lock}) 

1 <unlocked _thread.lock object at 0xe7560830> <__main__.queueWrapper object at 0xe7552f50>

Note how the queue and lock switched places.

Use a list or a Tuple instead. Both properly maintain order:

producer1 = threading.Thread(target = producer, name = 'Add 1', args = (1, q, lock))

edited Jul 28, 2019 at 14:26

answered Jul 28, 2019 at 14:11

Carcigenicate

46.5k12 gold badges81 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Fleetscut Over a year ago

This was the issue. I didnt realize i was using the wrong type of brackets.

Collectives™ on Stack Overflow

Python threading using a wrapper on thread-safe objects

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related