Insert an array of documents into a model

Question

Here's the relevant code:

var Results = mongoose.model('Results', resultsSchema);
var results_array = [];

_.each(matches, function(match) {
    var results = new Results({
        id: match.match_id,
        ... // more attributes
    });
    results_array.push(results);
});
callback(results_array);
});
   }
], function(results_array) {
   results_array.insert(function(err) {
       // error handling

Naturally, I get a No method found for the results_array. However I'm not sure what else to call the method on.

In other functions I'm passing through the equivalent of the results variable here, which is a mongoose object and has the insert method available.

How can I insert an array of documents here?

** Edit **

function(results_array) {
            async.eachLimit(results_array, 20, function(result, callback) {
                result.save(function(err) {
                    callback(err);
                });
            }, function(err) {
                if (err) {
                    if (err.code == 11000) {
                        return res.status(409);
                    }
                    return next(err);
                }
                res.status(200).end();
                });
            });

So what's happening:

When I clear the collection, this works fine. However when I resend this request I never get a response.

This is happening because I have my schema to not allow duplicates that are coming in from the JSON response. So when I resend the request, it gets the same data as the first request, and thus responds with an error. This is what I believe status code 409 deals with.

Is there a typo somewhere in my implementation?

Edit 2

Error code coming out:

{ [MongoError: insertDocument :: caused by :: 11000 E11000 duplicate key error index:       
test.results.$_id_  dup key: { : 1931559 }]
    name: 'MongoError',
    code: 11000,
    err: 'insertDocument :: caused by :: 11000 E11000 duplicate key error index:    
test.results.$_id_  dup key: { : 1931559 }' }

So this is as expected. Mongo is responding with a 11000 error, complaining that this is a duplicate key.

Edit 3

if (err.code == 11000) {
    return res.status(409).end();
}

This seems to have fixed the problem. Is this a band-aid fix though?

What are you actually trying to do? Is there another model that has documents with an array of Results? Or are you trying to insert a whole bunch of "Results" documents at once? — Neil Lunn
– Neil Lunn, Commented Sep 4, 2014 at 4:48
You can't mix return with callbacks. console.log(err) to see what is coming out. Same with err.code and other things that may not be matching as you expect. But certainly duplicate keys here where this works on an empty collection. — Neil Lunn
– Neil Lunn, Commented Sep 4, 2014 at 6:18
Edited in the specifics but the error code is as I'm expecting. I'm just not sure what the alternative is to return in this case. I want to say that when the error code is 11000 respond with a 409 code or something appropriate and end the request/response cycle. This isn't happening at the moment and I'm not sure why exactly. — goralph
– goralph, Commented Sep 4, 2014 at 6:37
The hint here is you are asking more than one question. One part is answered, which is how to insert your array of documents. Now you are asking how to use express which really is another question. But basically you should just want res.status(409).end() and you don't prefix these things with return — Neil Lunn
– Neil Lunn, Commented Sep 4, 2014 at 6:42
Ah my mistake. I'm new around here and just trying to learn the ropes. That's no excuse though, I'll try to pick up my game. And yes that fixed my problem, thank's Neil. — goralph
– goralph, Commented Sep 4, 2014 at 6:45

Neil Lunn · Accepted Answer · 2014-09-04 05:38:21Z

You seem to be trying to insert various documents at once here. So you actually have a few options.

Firstly, there is no .insert() method in mongoose as this is replaced with other wrappers such as .save() and .create(). The most basic process here is to just call "save" on each document you have just created. Also employing the async library here to implement some flow control so everything just doesn't queue up:

async.eachLimit(results_array,20,function(result,callback) {
    result.save(function(err) {
        callback(err)
    });
},function(err) {
   // process when complete or on error
});

Another thing here is that .create() can just take a list of objects as it's arguments and simply inserts each one as the document is created:

Results.create(results_array,function(err) {

});

That would actually be with "raw" objects though as they are essentially all cast as a mongooose document first. You can ask for the documents back as additional arguments in the callback signature, but constructing that is likely overkill.

Either way those shake, the "async" form will process those in parallel and the "create" form will be in sequence, but they are both effectively issuing one "insert" to the database for each document that is created.

For true Bulk functionality you presently need to address the underlying driver methods, and the best place is with the Bulk Operations API:

mongoose.connection.on("open",function(err,conn) {

    var bulk = Results.collection.initializeUnorderedBulkOp();
    var count = 0;

    async.eachSeries(results_array,function(result,callback) {
        bulk.insert(result);
        count++;

        if ( count % 1000 == 0 ) {
            bulk.execute(function(err,response) {
               // maybe check response
               bulk = Results.collection.initializeUnorderedBulkOp();
               callback(err);
            });
        } else {
            callback();
        }

    },function(err) {
        // called when done
        // Check if there are still writes queued
        if ( count % 1000 != 0 )
            bulk.execute(function(err,response) {
               // maybe check response
            });
    });

});

Again the array here is raw objects rather than those cast as a mongoose document. There is no validation or other mongoose schema logic implemented here as this is just a basic driver method and does not know about such things.

While the array is processed in series, the above shows that a write operation will only actually be sent to the server once every 1000 entries processed or when the end is reached. So this truly does send everything to the server at once.

Unordered operations means that the err would normally not be set but rather the "response" document would contain any errors that might have occurred. If you want this to fail on the first error then it would be .initializeOrderedBulkOp() instead.

The care to take here is that you must be sure a connection is open before accessing these methods in this way. Mongoose looks after the connection with it's own methods so where a method such as .save() is reached in your code before the actual connection is made to the database it is "queued" in a sense awaiting this event.

So either make sure that some other "mongoose" operation has completed first or otherwise ensure that your application logic works within such a case where the connection is sure to be made. Simulated in this example by placing within the "connection open" event.

It depends on what you really want to do. Each case has it's uses, with of course the last being the fastest possible way to do this as there are limited "write" and "return result" conversations going back and forth with the server.

Hi Neil thanks for the response. I've included an edit to show my attempt at your first suggestion and an issue I'm having with it. Cheers

Collectives™ on Stack Overflow

Insert an array of documents into a model

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related