Python - How to run parallel processing queries from a list [duplicate]

Question

I've the following list of queries:

queries = ["SELECT * FROM db.trans", "SELECT * FROM db.order", "SELECT * FROM db.account UNION ALL SELECT * FROM db.account2", "SELECT * FROM db.trans UNION ALL SELECT * FROM db.trans2"]

And using my database connection I am trying to run all this queries in parallel instead of:

for query in queries:
  cursor.execute(query)

I'm trying to use multiprocessing library and my current code is:

import multiprocessing
def work():
   for query in queries:
      cursor.execute(query)

if __name__ == '__main__':
    _p = multiprocessing.Process(target=work)
    _p.start()
    _p.join()

However, it is taking the same time as the sequential mode... anyone knows how to run a parallel processing?

Thanks!

You might find that multithreading would be the more efficient approach over multiprocessing. — Booboo
– Booboo, Commented Jan 14, 2021 at 20:00

Arseniy · Accepted Answer · 2021-01-14 19:43:19Z

0

It could be something like this:

if __name__ == '__main__':
    _processes = []
    for query in queries:
        _p = multiprocessing.Process(target=cursor.execute, args=(query,))
        _p.start()
        _processes.append(_p)

    for _p in _processes:
        _p.join()

answered Jan 14, 2021 at 19:43

Arseniy

6905 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python - How to run parallel processing queries from a list [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related