Parallel SQLite queries in Node.js

Question

Node.js has some very good bindings for SQLite thanks to the SQLite3 package, but unfortunately, since Node runs single-threaded, all queries are done in the same connection. SQLite runs all queries per connection in series, meaning that it is impossible to actually do parallel queries in SQLite under Node. For more details check this https://github.com/mapbox/node-sqlite3/issues/394

This is true whether you wrap your query-logic in async.each/async.parallel, or any of the other helper packages for parallelising/serialising database IO. At the end of the day, to benefit from Node's async IO architecture under SQLite3, you are going to need more than one processing thread.

How can this be done? :)

Why you need parallel queries? What problem do you want to solve? — MarcoL
– MarcoL, Commented Jan 19, 2015 at 23:05
Running more than 1 query at once, where the result of each do not rely on one another, and using UNION ALL would get really really messy :) — J.J
– J.J, Commented Jan 19, 2015 at 23:11

РАВИ · Accepted Answer · 2018-08-08 07:21:40Z

2

You can use cluster api to create child processes and inside of those you can run different actions, in your case db queries. https://nodejs.org/api/cluster.html.

Note: How many clusters you want to create depends on your hardware.

const cluster = require('cluster');
const db = new sqlite3.Database(':memory:')

if (cluster.isMaster) {
    console.log(`Master ${process.pid} is running`);
    cluster.fork()
        .on('exit', (worker, code, signal) => {
            console.log(`worker ${worker.process.pid} died`);
        });
    cluster.fork()
        .on('exit', (worker, code, signal) => {
            console.log(`worker ${worker.process.pid} died`);
        });

} else if (cluster.worker.id === 1){
  db.run(QUERY1);
} else {
 db.run(QUERY2);
}

answered Aug 8, 2018 at 7:21

РАВИ

12.3k7 gold badges36 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

J.J Over a year ago

Very cool! Many years later, this gets an answer :) Thanks Ravi!

Collectives™ on Stack Overflow

Parallel SQLite queries in Node.js

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related