Stream from a mongodb cursor to Express response in node.js

Question

I am toying around with all the fancy node.js/mongodb/express platforms, and stumbled across a problem:

app.get('/tag/:tag', function(req, res){
  var tag=req.params.tag;
  console.log('got tag ' + tag + '.');
  catalog.byTag(tag,function(err,cursor) {
     if(err) {
       console.dir(err);
       res.end(err);
     } else {
       res.writeHead(200, { 'Content-Type': 'application/json'});

       //this crashes
       cursor.stream().pipe(res);

     }
  });
});

As you probably guessed, catalog.byTag(tag, callback) does a find() query to Mongodb and returns the cursor

This leads to an error:

TypeError: first argument must be a string or Buffer

According to mongodb driver doc, I tried to pass this converter to stream():

function(obj) {return JSON.stringify(obj);}

but that does not help.

Can anybody tell me how to correctly stream something to a response?

Or is the only solution a boilerplate to manually pump the data using the 'data' and 'end' events?

does it work if you replace catalog.byTag with the direct call to MongoDB native driver? — Gal Ben-Haim
– Gal Ben-Haim, Commented Nov 18, 2013 at 23:13
No, same error. Also, I can call toArray() and send that via res.json(), which works, but I would prefer streaming instead of buffering on the server and sending it out afterwards. — h0ru5
– h0ru5, Commented Nov 19, 2013 at 20:12

robertklep · Accepted Answer · 2014-01-02 20:24:52Z

23

Use the cursor stream in combination with JSONStream to pipe it to your response object.

cursor.stream().pipe(JSONStream.stringify()).pipe(res);

answered Jan 2, 2014 at 20:24

robertklep

205k38 gold badges421 silver badges413 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jorge Aranda Over a year ago

(And make sure you res.set('Content-Type', 'application/json'); before, as your response will otherwise be interpreted as text.)

sshow · Accepted Answer · 2017-08-17 15:20:08Z

13

A working combination of other answers here

app.get('/comments', (req, res) => {
  Comment.find()
    .cursor()
    .pipe(JSONStream.stringify())
    .pipe(res.type('json'))
})

http://mongoosejs.com/docs/api.html#query_Query-cursor

cursor() returns a Node streams3 compatible stream and is preferred over the deprecated query.stream() interface.
Piping to JSONStream.stringify() to combine documents into an array instead of single objects
Piping to res.type('json') which sets the HTTP Content-Type header to application/json and returns itself (the response stream) again.

edited Aug 17, 2017 at 15:20

answered Jul 16, 2017 at 2:45

sshow

9,1745 gold badges54 silver badges83 bronze badges

9 Comments

robertklep Over a year ago

This doesn't produce valid JSON, though, it just concatenates the stringified objects together instead of wrapping it properly in an array.

sshow Over a year ago

@robertklep you are right! I did not know that. Thanks!

joniba Over a year ago

since we are dealing with express, it seems some error handling is required here. I would like to see some recommended pattern.

Tiago Over a year ago

@robertklep what's the alternative, for a valid JSON output?

robertklep Over a year ago

@Tiago the current answer will work (my comment referred to a previous version of the answer that has since been editted and fixed).

|

sshow · Accepted Answer · 2017-07-16 02:43:22Z

9

Simple. .stream({transform: JSON.stringify});

edited Jul 16, 2017 at 2:43

sshow

9,1745 gold badges54 silver badges83 bronze badges

answered Nov 25, 2014 at 17:09

Darron Smith

1011 silver badge2 bronze badges

Comments

bhurlow · Accepted Answer · 2014-01-02 19:08:14Z

Your mongo stream is dumping objects into the res stream which can only handle strings or buffers (hence the error).

Luckily, streams are easy to pipe together so its not too hard to make a transform stream to stringify your data.

in node v0.10.21:

var util = require('util')
var stream = require('stream')
var Transform = stream.Transform

util.inherits(Stringer, Transform)

function Stringer() {
  Transform.call(this, { objectMode: true } )
  // 'object mode allows us to consume one object at a time

}

Stringer.prototype._transform = function(chunk, encoding, cb) {
  var pretty = JSON.stringify(chunk, null, 2) 
  this.push(pretty) // 'push' method sends data down the pike.
  cb() // callback tells the incoming stream we're done processing 
}

var ss = new Stringer()

db.createObjectStreamSomehow()
  .pipe(ss)
  .pipe(res)

hope that helps

Julio Marins · Accepted Answer · 2015-09-22 15:30:46Z

2

Using mongoose and express:

function(req, res){
    var stream = database.tracks.find({}).stream();
    stream.on('data', function (doc) {
        res.write(JSON.stringify(doc));
    });
    stream.on('end', function() {
        res.end();
    });
}

answered Sep 22, 2015 at 15:30

Julio Marins

10.7k8 gold badges52 silver badges54 bronze badges

Comments

Ahsan Ehtesham · Accepted Answer · 2022-08-04 18:17:23Z

0

A better and recent solution for this problem:

const { JsonStreamStringify } = require("json-stream-stringify");

app.get("/", (req, res) => {
  const cursor = db.collection("hello").find({}).limit(10).stream();
  x = new JsonStreamStringify(cursor).pipe(res);
  x.on("data", (doc) => {
    res.write(doc);
  });
});

The above solution is using https://www.npmjs.com/package/json-stream-stringify package.

answered Aug 4, 2022 at 18:17

Ahsan Ehtesham

1561 silver badge6 bronze badges

Collectives™ on Stack Overflow

Stream from a mongodb cursor to Express response in node.js

6 Answers 6

1 Comment

9 Comments

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

1 Comment

9 Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related