Infinite scroll with Angularjs and Mongoose - performance

Question

I have a list of images that I get with Angular using $http.get() from MongoDB using Mongoose and Expressjs.

What I did works fine but I have a doubt about performance.

So far I found two ways to do it:

using skip
using $nin

The queries looks like this:

    // Using $nin:
    var skip = req.query.skip || [];

    User.find({ _id : { $nin: skip }})
    .sort({ _id: -1 })
    .limit(15)
    .exec(function(err, users) {
    if (err) res.json({ 'msg': 'Error loading users' });
    res.json({
      users: users
    });
  });

and:

    // Using skip
    User.find({})
    .sort({ _id: -1 })
    .skip(15)
    .limit(15)
    .exec(function(err, users) {
    if (err) res.json({ 'msg': 'Error loading users' });
    res.json({
      users: users
    });
  });

Googleling around it seems like using skip lack of performance after a while...

But looking at the $nin option I found after scrolling and scrolling a very long query... with plenty of _id...

Which of the 2 solution should be better to use?

Or there is a third way much better than those, maybe I'm doing something wrong?

Thanks

You can also use lean() option to reduce time that it takes mongoose to build the model with all of it's methods etc. IMO the skip option is better here (that's how I implement my paging solutions and never ran into any problems) — Vsevolod Goloviznin
– Vsevolod Goloviznin, Commented Jan 9, 2015 at 14:03
Can you please tell me more about lean() and write a working example please? The documentation for lean() lack of every kind of information. Thanks. — Ayeye Brazo
– Ayeye Brazo, Commented Jan 9, 2015 at 14:11
lean option tells mongoose to not include it's methods to the results, so you won't be able to use mongoose on the results you will get. You can use it like this: User.find({}).sort({ _id: -1 }).skip(15).limit(15).lean().exec(...). It will reduce the time it takes to get the results from db and will act almost as fast as the native mongo client — Vsevolod Goloviznin
– Vsevolod Goloviznin, Commented Jan 9, 2015 at 14:15
@VsevolodGoloviznin .lean() has nothing to do with this. Please do not make misleading comments. — Neil Lunn
– Neil Lunn, Commented Jan 9, 2015 at 14:20
Using .skip(15).limit(15) seems that it doesn't work properly... I see always duplications. — Ayeye Brazo
– Ayeye Brazo, Commented Jan 9, 2015 at 14:22

Neil Lunn · Accepted Answer · 2015-01-09 14:16:29Z

4

Ideally you want to use $nin with a list of previously seen _id values but combine that with a $gte or $lte (depending on order) operator on something you are sorting on.

This is generally the case for most "otherwise sorted queries", but in the case where the _id field is the one you are sorting on ( and in decending order ) it just becomes a matter of working with $lt to find values that are less than the "last seen value" from the previous page.

So when iterating, store the "last seen value" from the last item in your "page limit" results, then use the $lt operator in subsequent queries:

ie:

var lastSeen = null; // declare in a global or session or something

User.find({})
    .sort({ "_id": -1 })
    .limit(15)
    .exec(function(err,docs) {
        lastSeen = docs.slice(-1).id;
    });

And then:

User.find({ "_id": { "$lt": lastSeen })
    .sort({ "_id": -1 })
    .limit(15)
    .exec(function(err,docs) {
        lastSeen = docs.slice(-1).id;
    });

answered Jan 9, 2015 at 14:16

Neil Lunn

151k36 gold badges356 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Ayeye Brazo Over a year ago

What about the infinite query Angular send to the server?

Neil Lunn Over a year ago

What is an "infinite query"? Simple parameter here in a "single" lastSeen value, being the "last" in the list that was displayed. 1-5 -> then( > 5 ) = 6 - 10 -> then ( > 10 ) = 10 - 15 and so on. Not hard.

Ayeye Brazo Over a year ago

not hard but I'm a noob trying to learn... Implementing this code I receive no data at the beginning...

Neil Lunn Over a year ago

@AyeyeBrazo Slow down a little. "No data from the beginning"? The query object is blank at the begining. So you retrieve everything and just limit to 15 results after sorting in reverse. Somewhere, even in the server side or the client, you want to pay attention to the last _id value seen in those results. Then in your "getMoreResults" call, you want to pass that value into the query. Since you are sorting in reverse you use the $lt ( or "less than" ) operator as shown. This is basic concept of "forward paging". Where you are retrieving based on the next highest value from the last seen.

Ayeye Brazo Over a year ago

Got it. Thanks it worked fine and I hope it will be fine with a big amount of data in terms of performance.

|

Collectives™ on Stack Overflow

Infinite scroll with Angularjs and Mongoose - performance

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related