Python Redis Queue

Question

I would like to implement a queue in redis + Flask and Python. I already implemented a query like this with RQ and it works fine if you have the Flask app and the task working on the same server. I am wondering if it is possible to create a queue (multi-consumer) where the worker(s) are actually on an other server. For example:

Client post data to Flask -> Flask create an item in the Redis queue -> Redis queue is picked up by some workers on an other server (backend).

Since the code in Flask is something like:

redis_conn = Redis()
q = Queue('my_queue', connection=redis_conn)
job = q.enqueue_call(func='myqueue.myfunc', args=(json,), result_ttl=5000)

Obviously 'myqueue.myfunc' needs to stay on the Flask server but I would like to be able to push the data and have a worker on another server. Do you know if this is feasible or what are other things that can be used to solve this problem?

Thanks.

And by the way, I thought about Celery, but I wonder if there other possible solutions. — Rob
– Rob, Commented Nov 30, 2019 at 20:06
I see no one problem to connect many workers to your redis server. Can you describe you problem clearly? — Peter
– Peter, Commented Dec 1, 2019 at 10:38
Also there are lots of different solution of distributed queue. Just a quick googling shows: dramatiq.io, python-rq.org, rabbitmq.com — Peter
– Peter, Commented Dec 1, 2019 at 10:41
Hi Peter, what I am trying to do is building a redis queue (I want to use Redis and no other technologies like RabbitMQ - although it is actually good). Basically I want to have the possibility to push data into Redis (JSON data) and have multiple workers (on multiple servers) parsing the data, perform some heavy memory intensive tasks and then save the data into the DB. I would like to have for example 2/3 servers in the backend (not accessible from Public IP) and have 5/10 workers on each server to handle the Redis queue. I thought to use "rpush", but I wonder if there are better solutions. — Rob
– Rob, Commented Dec 1, 2019 at 15:41
I am not sure how to implement properly Python-RQ on a distributed environment as from Flask when I do the enqueue I need to provide the class name/function name and the class name/function name is on another server. — Rob
– Rob, Commented Dec 1, 2019 at 15:44

namizaru · Accepted Answer · 2022-01-06 17:12:12Z

4

+25

The pattern of having queue workers on other nodes is common.

So in your flask app node configure a remote redis server

r = redis.Redis(host='myredis.example.com', port=6379)

Then on one or more worker nodes

from redis import Redis
from rq import Queue, Worker

# Returns all workers registered in this connection
redis = Redis(host='myredis.example.com', port=6379)
workers = Worker.all(connection=redis)

# Returns all workers in this queue (new in version 0.10.0)
queue = Queue('queue_name')
workers = Worker.all(queue=queue)
worker = workers[0]
print(worker.name)

answered Jan 6, 2022 at 17:12

namizaru

8525 silver badges5 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Redis Queue

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related