Struggling with async synchronisation in node.js

Question

So I started a little project in Node.js to learn a bit about it. It's a simple caching proxy for arch linux's package system as node provides most of the heavy lifting.

This has two "main" phases, server setup and serving. Then serving has two main phases, response setup and response.

The "main" setup involves checking some files, loading some config from files. loading some json from a web address. Then launching the http server and proxy instance with this info.

setup logger/options - read config - read mirrors - read webmirror
start serving

Serving involves checking the request to see if the file exists, creating directories if needed, then providing a response.

check request - check dir - check file
proxy request or serve file

I keep referring to them as synchronisation points but searches don't lead to many results. Points where a set of async tasks have to be finished before the process can complete a next step. Perl's AnyEvent has conditional variables which I guess is what I'm trying to do, without the blocking.

To start with I found I was "cheating" and using the synchronous versions of any functions where provided but that had to stop with the web requests, so I started restructuring things. Immediately most search's led to using async or step to control the flow. To start with I was trying lots of series/parallel setups but running into issues if there were any async calls underneath the functions would "complete" straight away and the series would finish.

After much wailing and gnashing of teeth, I ended up with a "waiter" function using async.until that tests for some program state to be set by all the tasks finishing before launching the next function.

// wait for "test" to be true, execute "run", 
// bail after "count" tries, waiting "sleep" ms between tries;
function waiter( test, run, count, sleep, message ) {
    var i=0;
    async.until( 
        function () {
            if ( i > count ) { return true; }
            logger.debug('waiting for',message, test() );
            return test();  
        },
        function (callback) {
            i++;
            setTimeout(callback, sleep );
        },
        function (err) {
            if ( i > count ) {
                logger.error('timeout for', message, count*sleep );
                return;
            }
            run()
        }
    );
}

It struck me as being rather large and ugly and requiring a module to implement for something that I thought was standard, so I am wondering what's a better way. Am I still thinking in a non-async way? Is there something simple in Node I have overlooked? Is there a standard way of doing this?

I imagine with this setup, if the program get's complex there's going to be a lot of nesting functions to describe the flow of the program and I'm struggling to see a good way to lay it all out.

any tips would be appreciated.

Amith Koujalgi · Accepted Answer · 2013-02-27 12:09:45Z

3

You can't really make everything to be synchronous. Nodejs is designed to perform asynchronously (which may of course torment you at times). But there are a few ways techniques to make it work in a synchronous way (provided the pseudo-code is well-thought and code is designed carefully):

Using callbacks
Using events
Using promises

Callbacks and events are easy to use and understand. But with these, sometimes the code can get real messy and hard to debug.

But with promises, you can avoid all that. You can make dependency chains, called 'promises' (for instance, perform Promise B only when Promise A is complete).

Earlier versions of node.js had implementation of promises. They promised to do some work and then had separate callbacks that would be executed for success and failure as well as handling timeouts.

But in later versions, that was removed. By removing them from the core node.js, it created possibility of building up modules with different implementations of promises that can sit on top of the core. Some of these are node-promise, futures, and promises.

See these links for more info:

answered Feb 27, 2013 at 12:09

Amith Koujalgi

11k2 gold badges20 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Matt Over a year ago

ok, promises, now i have something to search for. Any idea why node removed them?

00dani Over a year ago

@mindthemonkey The promises that once were available in Node.js core were EventEmitter-based and didn't really work anything like modern Promises/A+ ones do. They were removed for being overly complex without adding much benefit, since unlike Promises/A+ ones I believe the ones Node had were non-chainable, didn't propagate errors, and had a much more verbose way of registering callbacks compared to the current Node style. None of these problems exist in Promises/A+ promises, however, which are in my opinion a much better async basis than Node's callbacks.

Matt Over a year ago

Bluebird.js, ES6 Promises and the ES6 Promise polyfill are where I ended up 3 years later. Bluebird mainly.

Collectives™ on Stack Overflow

Struggling with async synchronisation in node.js

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related