Streaming / Piping JSON.stringify output in Node.js / Express

Question

I have a scenario where I need to return a very large object, converted to a JSON string, from my Node.js/Express RESTful API.

res.end(JSON.stringify(obj));

However, this does not appear to scale well. Specifically, it works great on my testing machine with 1-2 clients connecting, but I suspect that this operation may be killing the CPU & memory usage when many clients are requesting large JSON objects simultaneously.

I've poked around looking for an async JSON library, but the only one I found seems to have an issue (specifically, I get a [RangeError]). Not only that, but it returns the string in one big chunk (eg, the callback is called once with the entire string, meaning memory footprint is not decreased).

What I really want is a completely asynchronous piping/streaming version of the JSON.stringify function, such that it writes the data as it is packed directly into the stream... thus saving me both memory footprint, and also from consuming the CPU in a synchronous fashion.

you create a stream, write the object as string into the steam, and finally just pipe the stream to res. — wayne
– wayne, Commented Nov 21, 2012 at 23:50

Andrey Sidorov · Accepted Answer · 2013-07-13 05:00:50Z

10

Ideally, you should stream your data as you have it and not buffer everything into one large object. If you cant't change this, then you need to break stringify into smaller units and allow main event loop to process other events using setImmediate. Example code (I'll assume main object has lots of top level properties and use them to split work):

function sendObject(obj, stream) {
    var keys = Object.keys(obj);
    function sendSubObj() {
       setImmediate(function(){
          var key = keys.shift();
          stream.write('"' + key + '":' + JSON.stringify(obj[key]));
          if (keys.length > 0) {
            stream.write(',');
            sendSubObj();
          } else {
            stream.write('}');
          }
       });
    })
    stream.write('{');
    sendSubObj();
}

edited Jul 13, 2013 at 5:00

answered Nov 22, 2012 at 0:09

Andrey Sidorov

25.6k5 gold badges65 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Paul Over a year ago

Nice, but doesn't this omit the key names and object detail from the top level of obj?

Gates VP · Accepted Answer · 2012-11-21 23:59:40Z

5

It sounds like you want Dominic Tarr's JSONStream. Obviously, there is some assembly required to merge this with express.

However, if you are maxing out the CPU attempting to serialize (Stringify) an object, then splitting that work into chunks may not really solve the problem. Streaming may reduce the memory footprint, but won't reduce the total amount of "work" required.

answered Nov 21, 2012 at 23:59

Gates VP

45.3k11 gold badges108 silver badges108 bronze badges

2 Comments

Zane Claes Over a year ago

It may not reduce the total amount of work, but distributing it over time (separating via process.nextTick) means that the CPU won't get pegged.

root-aj Over a year ago

Note: process.nextTick now executes before yielding to the event loop. Use setImmediate.

Collectives™ on Stack Overflow

Streaming / Piping JSON.stringify output in Node.js / Express

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related