Node.js code loading inconsistently to mongodb

Question

The following code tries to load from a csv file to a mongodb instance running on localhost.

Problem - It loads a different number of documents on every run (always less than total number of records in csv).

var csv = require('csv');

var server = new Server('localhost', 27017, {auto_reconnect: true, poolSize: 1});
var db = new Db('test', server);

db.open(function(err, db, onemore) {
  if(!err) {
    //Database connection is established.
    db.collection('teststocks', function(err, collection) {
      if(!err) {
        // Stocks collection is connected, open the file and insert the doc
        console.log("Trying to load from " + process.argv[2]);
        csv()
          .fromPath(process.argv[2], {
            columns: true
          })
          .on('data', function(data, index) {
            //data.stock = process.argv[2].substring(process.argv[2].lastIndexOf('/') + 1, process.argv[2].lastIndexOf('.'));
            collection.insert(data, {safe: true}, function(error, collection){
                    if ( error ) { console.log("Error inserting record : " + error); }
            });
            console.log("Inserted data for " + index);
          })
          .on('error', function(error) {
            db.close();
            console.log("Error: " + error);
          })
          .on('end', function(count) {
            console.log("Finished all writing.");
            db.close();
          });
      }
  });
  }
});

P.S: I am able to load the data using the mongoimport utility, but being a newcomer to node.js and mongodb; I'd like to understand the mistake I am making in the code above.

Are you getting any error messages?

Joachim Isaksson
– Joachim Isaksson

2012-09-24 17:27:25 +00:00
Commented Sep 24, 2012 at 17:27 — Joachim Isaksson
– Joachim Isaksson, Commented Sep 24, 2012 at 17:27
Nope, no error messages.

Rishabh Sagar
– Rishabh Sagar

2012-09-25 05:08:50 +00:00
Commented Sep 25, 2012 at 5:08 — Rishabh Sagar
– Rishabh Sagar, Commented Sep 25, 2012 at 5:08

Jonathan Lonowski · Accepted Answer · 2012-09-24 18:21:20Z

2

It's likely because you're closing the database connection after the csv() finishes reading. But, as collection.insert is asynchronous, there's no guarantee each call has finished before the csv is done and db.close() is called.

One option is to collect the csv data into an Array, then insert all of them at once:

var docs = [];

csv()
  // ...
  .on('data', function (data, index) {
    docs.push(data); // or possibly: docs[index] = data;
  })
  // ...
  .on('end', function () {
    console.log("Finished reading CSV.");    

    collection.insert(docs, { safe: true }, function (error, inserted) {
      console.log("Finished all writing.");
      db.close();
    });
  });

edited Sep 24, 2012 at 18:21

answered Sep 24, 2012 at 18:01

Jonathan Lonowski

124k35 gold badges203 silver badges202 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Rishabh Sagar Over a year ago

Thanks for the response, but I thought safe:true forces statement to wait for db's confirmation on insert?

Jonathan Lonowski Over a year ago

@RishabhSagar The safe option doesn't force the connection to stay open, though. It only affects the callback function passed to insert, by ensuring that it's called "after the record is saved to the database."

Rishabh Sagar Over a year ago

Yes, Looks like the problem was with early execution of db.close(), guess I still haven't wrapped my head around asynch programming yet. Thanks for your help.

Collectives™ on Stack Overflow

Node.js code loading inconsistently to mongodb

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related