How to continuously query a mongodb database efficiently

Question

I'm new to Node and MongoDB and I have a seemingly simple request. I've managed to connect to my database, and use a query to get my desired results. Now, I want to have this query continue indefinitely, since the end goal for my project is to plot data real time.

I would have thought a simple 'while (true)' loop would suffice, but that doesn't seem to be the case.

const MongoClient = require('mongodb').MongoClient;

// Connection URL
const url = 'mongodb://<username>:<password>@ds157614.mlab.com:57614/flight_data';


// Use connect method to connect to the Server
MongoClient.connect(url, { useNewUrlParser: true }, function(err, db) {

  if (err) throw err;

  var dbo = db.db("flight_data").collection("data");

while(true)
{

    dbo.find().sort({_id: 1}).limit(1).toArray(function(err, result) {

    if (err) throw err;

    console.log("Temperature: " + result[0].data.temperature);

  });

}

db.close();

});

I have found that the while loop is indeed running, but for some reason, the query just doesn't happen when inside the while loop. If you remove the while loop, the code functions fine. I just want it to continually print the results of the query being repeated.

Instead of continuously hitting your database with queries in a while loop, I'd suggest looking into Change Streams to efficiently watch for updates on the collection(s) of interest. — Stennie
– Stennie, Commented Apr 15, 2019 at 3:11

PrivateOmega · Accepted Answer · 2019-04-15 04:43:23Z

1

Querying a DB continuously is inefficient and resource wasting, instead use change streams. It watches collection for any changes and will make the db call then only. Works only for Mongo 3.6+.

const MongoClient = require("mongodb").MongoClient;

// Connection URL
const url =
  "mongodb://<username>:<password>@ds157614.mlab.com:57614/flight_data";

// Use connect method to connect to the Server
MongoClient.connect(url, { useNewUrlParser: true }, function(err, db) {
  if (err) throw err;

  const collection = db.collection("data");
  const changeStream = collection.watch();
  changeStream.on("change", next => {
    // process next document
    collection
      .find()
      .sort({ _id: 1 })
      .limit(1)
      .toArray(function(err, result) {
        if (err) throw err;

        console.log("Temperature: " + result[0].data.temperature);
      });
  });
  db.close();
});

edited Apr 15, 2019 at 4:43

answered Apr 15, 2019 at 3:24

PrivateOmega

2,9301 gold badge19 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

Jessie L Smith Over a year ago

I tried it and I'm getting some weird errors now. One of them is MongoError: Majority read concern requested, but it is not supported by the storage engine.

PrivateOmega Over a year ago

@JessieLSmith Which version of mongo are you using? Change Streams require MongoDB 3.6+.

Jessie L Smith Over a year ago

It looks like I have 3.2.2 . The newest version available for the node.js driver is 3.2.3. Will I need to figure out a different solution since I'm not using the mongo shell?

PrivateOmega Over a year ago

The newest version available for the node.js driver is 3.2.3 What did you mean by that? You must be confusing node.js driver version and mongodb server version. Go to mongo shell and type db.version(), the result is your mongodb version.

PrivateOmega Over a year ago

I am not sure why you don't have mongo shell installed, since it comes by default on mongo server installation. And also why do you think watching the db is not an option for you? I feel it is the best option according to your requirement compared to continously querying a db which is bad approach in so many ways.

|

Anish Jain · Accepted Answer · 2024-07-23 11:51:37Z

0

you can create a stream which will watch for any changes made in your db. Detailed Article - article

answered Jul 23, 2024 at 11:51

Anish Jain

5607 silver badges14 bronze badges

Collectives™ on Stack Overflow

How to continuously query a mongodb database efficiently

2 Answers 2

10 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related