How to use shared queues with python flask Restful web services

Question

i'm new to python flask REST web services. I'm trying to develop a rest web service which will have a shared queue, multiple threads will constantly write to that queue on the server side and finally when a user calls a GET methods, the service should return first item in the shared queue.

I was trying getting start to develop this by first implementing a shared variable, following is the code I used,

from flask import Flask
app = Flask(__name__)

count= 0 #Shared Variable

@app.route("/")
def counter():
    count = count+1
    return {'count':count}

if __name__ == "__main__":

    app.run()

But even above code is not working. Then I though of using cache for the shared variable, but it will not the correct way to implement a shared queue (my ultimate goal). Please give me your advises

Daniel Fairhead · Accepted Answer · 2013-11-24 17:10:44Z

7

The thing you want to do is a little bit more complex than that, I'm afraid.

Flask (and other WSGI python systems) don't work in a single thread - they will normally need to spawn multiple threads and instances to cope with requests coming in without blocking, or without multiple requests accessing the same 'first task' at the same time. Thus global variables don't work as they might in other simple single-threaded python scripts.

You need some way for the different processes to all access the same single queue of data.

Usually, this means outsourcing the data queue to an external database. One popular option is Redis. There's a good intro to flask and redis for exactly this:

http://flask.pocoo.org/snippets/73/

I hope this helps you in the right direction!

answered Nov 24, 2013 at 17:10

Daniel Fairhead

1,1819 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Miguel Grinberg · Accepted Answer · 2013-11-24 20:00:04Z

1

You have a couple of bugs in your example. Here is a version that works:

from flask import Flask, jsonify
app = Flask(__name__)

count= 0 #Shared Variable

@app.route("/")
def counter():
    global count
    count = count+1
    return jsonify({'count':count})

if __name__ == "__main__":
    app.run()

The two problems you have in your version are:

You missed to declare count as global in your view function. Without the global declaration the view function creates a local variable of the same name.
The response returned by the view function cannot be a dictionary, it needs to be a string or a Response object. I corrected this using jsonify() to convert the dict to a JSON string.

But note that this way of creating a shared value is not robust. In particular note that if you run this application under a web server that creates multiple processes then each process will have its own copy of the count value.

If you need to do this on a production server my recommendation is that you use a database to store your shared value(s).

answered Nov 24, 2013 at 20:00

Miguel Grinberg

68.1k14 gold badges143 silver badges160 bronze badges

2 Comments

Proceso Over a year ago

So is the procedure same, if I want to implement a shared queue (not a integer variable) ?. For now I'm thinking of using a pickle to store the shared queue after every get request, and give access to it through a synchronized block. But if I give the access to queue through a synchronized block, it will not function well with concurrent multiple requests ( every request will have to wait until it get a chance to enter to synchronized block ). what do u think ? (my ultimate goal is to implement a shared queue)

Miguel Grinberg Over a year ago

Read my answer again, this is not a good solution. Use a database instead, a lightweight one like Redis would be a good one to use for this.

Collectives™ on Stack Overflow

How to use shared queues with python flask Restful web services

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related