Using Django in a multithreading code

Question

I am running a python script using django ORM to handle calls to the database.

I have the following code:

with ThreadPoolExecutor(max_workers=2) as executor:
    executor.submit(process, 2)
    executor.submit(process, 1)

@transaction.atomic
def process(counter)
    MyModel.objects.filter(user_id=counter).delete()
    users = <create User object)
    MyModel.objects.bulk_create(users)

I am using django default behavior which sets the auto-commit to True.

When I debug the code, it looks that django uses the same connection among the application, and therefore, the last process that will exit process method, will commit all the transactions that are using the connection.

How can I prevent it to happen? I would like django to close (and commit) the connection at the end of each method so each thread could be handled separately.

Do you need these users in the code after? Or could you use celery? — Bojan Kogoj
– Bojan Kogoj, Commented Dec 3, 2017 at 21:01
As a rule of thumb, whenever you want to do something concurrently inside Django, you should use celery project — Mehdi Pourfar
– Mehdi Pourfar, Commented Dec 3, 2017 at 22:00

Bojan Kogoj · Accepted Answer · 2017-12-03 22:04:55Z

1

I suggest using Celery. You will need Redis or RabbitMQ, to send messages between Django and Celery.

@app.task
def process(counter):
    MyModel.objects.filter(user_id=counter).delete()
    users = <create User object)
    MyModel.objects.bulk_create(users)

After that, use apply_async

process.apply_async((2,))
process.apply_async((1,))

Setting this up is a bit more complicated, but works well.

answered Dec 3, 2017 at 22:04

Bojan Kogoj

5,6984 gold badges39 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using Django in a multithreading code

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related