Using streams in Node.js to buffer HTTP communications

Question

I'm trying to use streams in Node.js to basically build a running buffer of HTTP data until some processing is done, but I'm struggling with the specifics of streams. Some pseudocode will probably help:

var server = http.createServer(function(request, response) {

    // Create a buffer stream to hold data generated by the asynchronous process
    // to be piped to the response after headers and other obvious response data
    var buffer = new http.ServerResponse();

    // Start the computation of the full response as soon as possible, passing
    // in the buffer stream to hold returned data until headers are written
    beginAsyncProcess(request, buffer);

    // Send headers and other static data while waiting for the full response
    // to be generated by 'beginAsyncProcess'
    sendObviousData(response, function() {

        // Once obvious data is written (unfortunately HTTP and Node.js have
        // certain requirements for the order data must be written in) then pipe
        // the stream with the data from 'beginAsyncProcess' into the response
        buffer.pipe(response);
    });
});

Most of this is almost legitimate code, but it doesn't work. The basic issue is figuring out a way to take advantage of the asynchronous nature of Node.js when there are certain order requirements associated with HTTP requests, namely that headers must always be written first.

While I would definitely appreciate any answers with little hacks to get around the order problem without directly addressing streams, I wanted to use the opportunity to get to know them better. There are plenty of similar situations, but this scenario is more to open the can of worms than anything else.

In this case your actual code would be better than a pseudo code :) — Piotr Dajlido
– Piotr Dajlido, Commented Mar 15, 2015 at 7:46
@Urahara That's basically it, sendObviousData sends headers, stylesheets, etc. These are static. The actual content is generated by beginAsyncProcess, and takes time to generate. The idea is to send the headers, stylesheets, etc. while waiting for the content to be generated, and then send that when it's ready. The problem is that there's always the chance that the content will outrun the headers and cause an error that would be really annoying to catch and try to handle. I'm trying to take advantage of asynchronous code while avoiding race conditions. — TheEnvironmentalist
– TheEnvironmentalist, Commented Mar 15, 2015 at 7:50
looks to me you can simply sendObviousData() before you even start beginAsyncProcess(), but it's hard to tell with make-believe parts... i do this a lot; write the headers right away then use an async process to pipe into response.end(). works well unless you want to change headers based on the slow async response. — dandavis
– dandavis, Commented Mar 15, 2015 at 7:52
@TheEnvironmentalist Have you heard of marko github.com/raptorjs/marko ? this allows you to pipe the incoming html template and inject data asynchronously, while writting to the response buffer. — Piotr Dajlido
– Piotr Dajlido, Commented Mar 15, 2015 at 7:53
2 thoughts: 1. node will catch it for you: github.com/joyent/node/blob/master/lib/_http_outgoing.js#L131 2. it appears writeHead() is all sync, so that can't happen. github.com/joyent/node/blob/master/lib/_http_server.js — dandavis
– dandavis, Commented Mar 15, 2015 at 8:23

Piotr Dajlido · Accepted Answer · 2015-03-15 08:45:00Z

1

Let's make a use of callbacks and streams in Node.js and .pause() / .resume() stream functions:

var server = http.createServer(function(request, response) {

    // Handle the request first, then..

    var body = new Stream(); // <-- you can implement stream.Duplex for read / write operations
        body.on('open', function(){
            body.pause();
            // API generate data
            // body.write( generated data ) <-- write to the stream
            body.resume();
        });

    var firstPartOfThePage = getHTMLSomeHow();

    response.writeHead(200, { 'Content-Type': 'text/html'});

    response.write(firstPartOfThePage, function(){ // <-- callback after sending first part, our body already being processed
        body.pipe( response ); // <-- This should fire after being resumed
        body.on('end', function(){
            response.end(); // <-- end the response
        });
    });
});

Check this: http://codewinds.com/blog/2013-08-31-nodejs-duplex-streams.html for costum duplex stream creation.

Note: it's still a pseudo code

edited Mar 15, 2015 at 8:45

answered Mar 15, 2015 at 8:27

Piotr Dajlido

2,04017 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

TheEnvironmentalist Over a year ago

The process isn't bad, but as I wrote in the question (or rather as I didn't write in the question) there was never any use of the filesystem. All of the data is computed using API's, none of which conform to the streams consumption pattern in Node.js.

Piotr Dajlido Over a year ago

I have updated my answer, can you comment on that, is it useful to you?

TheEnvironmentalist Over a year ago

I'd say it's definitely worth keeping. As long as it works (I haven't yet tested it) it could come in handy to someone in the future.

Piotr Dajlido Over a year ago

Then +1, I'm trying to gain some respect up in here ;)

Collectives™ on Stack Overflow

Using streams in Node.js to buffer HTTP communications

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related