2

We have a python script that is kicked off after a user enters some data in a web form. The entire script takes 10-20 minutes to complete. A user can technically kick off 10 of these within 10 seconds if they so chose. If this happens, we'll have 10 of the same python scripts running at once and causing each other to fail due to various things the script is processing.

What is the go-to way to code an overarching queueing system so that these scripts know of each other and will wait in line to execute? We are people who usually write one off scripts but need to have this constant queueing system in place for the web form... sadly we aren't developers or have that background.

We're also open to different ways of architecting the solution in general. We went into this hacking it together. We've never made a process/service broker/worker but would that might make sense here?

How do people normally do this stuff?

Improve this question

1 Answer 1

Reset to default
3

Welcome to the wild world of distributing your computation!

Implementing your own queuing system (even in Python) can lead to a world of hurt. A very popular, enterprise-grade and open source message queuing application is RabbitMQ. They have a great starter tutorial that talks about ways you can begin configuring it and examples of its uses.

Additionally there is a Python task queue library called Celery that consequently uses RabbitMQ under the hood. It is a bit smaller in focus and capability but offers easy of use and a faster start up time as a tradeoff. One thing that it does not trade-off is RabbitMQs consistency which as you delve deeper into queuing and distributed systems, you will learn is extremely important. There getting started docs can be found here

Links:

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks for the info Matt. One of our interns surprised me and implemented rqueue to solve this. Out of curiosities sake, did I even ask the question correctly? I'll be checking your links out later.
Your question dropped the right amount of keywords/breadcrumbs to figure out what you were trying to get at. The problem of "something on a webpage that takes minutes and is spammable" is unfortunately not uncommon; putting a queuing system behind it and the "worker" is a common solution.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.