How to stream data in a Node JS + Mongo DB REST API?

Question

I am developing a Rest API in Node JS + Mongo DB, handled with Mongoose's middleware, in which one of the methods allows the recovery of contents asociated to a certain user.

So far I've been retrieving all of the user's content, but the amount of data is starting to grow, and now I need to stream the data somehow.

The behaviour I want to implement would be for the server to answer the request with a stream of 10-20 items, and then, if the client needs more data, it would need to send another request, which would be answered with the following 10-20 items.

All I can come up with would be to answer with those first 10-20 items, and then, in case the client needs more data, to provide a new (optional) parameter for my method, which would allow the client to send the last item's id, so the server can send back the following 10-20 items.

I know that this approach will work, but I feel like it's too raw; there's gotta be a cleaner way to implement this behaviour, since it's the kind of behaviour a lot of web applications must implement.

So, my question would be: Do you know of any better way to solve this problem?

Thanks in advance.

Brant · Accepted Answer · 2015-07-21 17:03:28Z

1

Provide the ability to read an offset and a limit from the request, then do something like:

db.collection.find().skip(20).limit(10)

Also, set defaults on APIs you build so that someone can't request a million records at once. Maybe max results is always 200, and if the request doesn't have the above params set, return up to the first 200 results.

answered Jul 21, 2015 at 17:03

Brant

1,78811 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Alberto Gimeno Over a year ago

Yeah, but what if new items are created after the first request?

Brant Over a year ago

They'll appear in the result set when conditions of skip and limit grab the very last results in the DB. Order is determined by when they were inserted in the DB so they'll appear there as such.

Collectives™ on Stack Overflow

How to stream data in a Node JS + Mongo DB REST API?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related