Implement a push/pull (or queue) with redis in Python

Question

I'm currently working on a project where I have a lot of tasks that can be parallelized. I'm using redis to store data, and I'd also like to use to handle my jobs queue to.

At first, I've used the PubSub pattern described in Redis doc. The issue here is that it sends every message to every worker where I'd like one message per worker. I have, for the moment, circumvented the problem by adding status (in a DB) for each of the messages but that's clearly not optimal.

Is there a way to easily replace this pubsub pattern with a suitable pattern ?

A few useful references to this question: (implementation details for implementing queue blocking) stackoverflow.com/a/76057700/4790871 (existing implementation of job queues over redis) python-rq.org — David Parks
– David Parks, Commented Apr 19, 2023 at 20:15

RobinFrcd · Accepted Answer · 2020-03-19 20:51:05Z

8

Not sure it that's the most efficient way of implementing a queue with redis but it's definitely a simple way of doing it!

import time
from redis import Redis

REDIS_CLIENT = Redis()

def push(*values):
    REDIS_CLIENT.rpush('QUEUE', *values)

def pull():
    while True:
        msg = REDIS_CLIENT.rpop('QUEUE')
        if msg is None:
            time.sleep(0.1)
            continue

        work(msg)

answered Mar 19, 2020 at 20:51

RobinFrcd

5,7047 gold badges32 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Implement a push/pull (or queue) with redis in Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related