145

Suppose you maintain a library that exposes a function getData. Your users call it to get actual data:
var output = getData();
Under the hood data is saved in a file so you implemented getData using Node.js built-in fs.readFileSync. It's obvious both getData and fs.readFileSync are sync functions. One day you were told to switch the underlying data source to a repo such as MongoDB which can only be accessed asynchronously. You were also told to avoid pissing off your users, getData API cannot be changed to return merely a promise or demand a callback parameter. How do you meet both requirements?

Asynchronous function using callback/promise is the DNA of JavasSript and Node.js. Any non-trivial JS app is probably permeated with this coding style. But this practice can easily lead to so called callback pyramid of doom. Even worse, if any code in any caller in the call chain depends on the result of the async function, those code has to be wrapped in callback function as well, imposing a coding style constraint on caller. From time to time I find the need to encapsulate an async function (often provided in a 3rd party library) into a sync function in order to avoid massive global re-factoring. Searching for a solution on this subject usually ended up with Node Fibers or npm packages derived from it. But Fibers just cannot solve the problem I am facing. Even the example provided by Fibers' author illustrated the deficiency:

...
Fiber(function() {
    console.log('wait... ' + new Date);
    sleep(1000);
    console.log('ok... ' + new Date);
}).run();
console.log('back in main');

Actual output:

wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
back in main
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)

If function Fiber really turns async function sleep into sync, the output should be:

wait... Fri Jan 21 2011 22:42:04 GMT+0900 (JST)
ok... Fri Jan 21 2011 22:42:05 GMT+0900 (JST)
back in main

I have created another simple example in JSFiddle and looking for code to yield expected output. I'll accept a solution that only works in Node.js so you are free to require any npm package despite not working in JSFiddle.

12
  • 2
    Async functions can never be made synchronous in Node, and even if they could, you shouldn't. The problem is such that in the fs module you can see completely separate functions for synchronous and asynchronous access to the file system. The best you can do is mask the appearance of async with promises or coroutines (generators in ES6). For managing callback pyramids, give them names instead of defining in a function call, and use something like the async library. Commented Feb 17, 2014 at 2:52
  • 8
    To dandavis, async bubbles up implementation detail to the call chain , sometimes forcing global refactoring. This is detrimental and even disastrous for a complex application where modularization and containment is important. Commented Feb 17, 2014 at 3:28
  • 5
    "Callback pyramid of doom" is only the representation of the problem. Promise can hide or disguise it but cannot address the true challenge: If the caller of an async function depends on the results of async function, it has to use callback, and so does its caller etc. This is a classical example of imposing constraints to caller simply because of implementation details. Commented Feb 17, 2014 at 6:17
  • 1
    @abbr: Thanks for the deasync module, the description of your problem is exactly what I have been looking for, and could not find any workable solutions. I messed around with generators and iterables, but came to the same conclusions as you. Commented Sep 14, 2015 at 0:24
  • 2
    It's worth noting that it's almost never a good idea to force an async function into being sync. You almost always have a better solution that keeps the async-ness of the function intact, while still achieving the same effect (like sequencing, variable setting, etc). Commented Nov 26, 2015 at 19:41

9 Answers 9

121

deasync turns async function into sync, implemented with a blocking mechanism by calling Node.js event loop at JavaScript layer. As a result, deasync only blocks subsequent code from running without blocking entire thread, nor incuring busy wait. With this module, here is the answer to the jsFiddle challenge:

function AnticipatedSyncFunction(){
  var ret;
  setTimeout(function(){
      ret = "hello";
  },3000);
  while(ret === undefined) {
    require('deasync').runLoopOnce();
  }
  return ret;    
}


var output = AnticipatedSyncFunction();
//expected: output=hello (after waiting for 3 sec)
console.log("output="+output);
//actual: output=hello (after waiting for 3 sec)

(disclaimer: I am the co-author of deasync. The module was created after posting this question and found no workable proposal.)

Sign up to request clarification or add additional context in comments.

8 Comments

Anybody else had luck with this? I can't make it work.
I can't make it work properly. you should improve your documentation for this module, if you wish that it gets used more. I doubt the authors know exactly what the ramifications are for using the module, and if they do, they certainly don't document them.
So far there is one confirmed problem documented in github issue tracker. The problem has been fixed in Node v0.12. The rest I know of are merely groundless speculations that are not worth to document. If you believe your problem is caused by deasync, post a self-contained, duplicatable scenario and I will look into.
I tried to use it and I get some improvements in my script but still I had no luck with the date. I modified the code as follow: function AnticipatedSyncFunction(){ var ret; setTimeout(function(){ var startdate = new Date() //console.log(startdate) ret = "hello" + startdate; },3000); while(ret === undefined) { require('deasync').runLoopOnce(); } return ret; } var output = AnticipatedSyncFunction(); var startdate = new Date() console.log(startdate) console.log("output="+output); and I expect to see 3 secs of different in the date output!
@abbr can this be browserified and used without node dependency>
|
5

You've got to use promises:

const asyncOperation = () => {
    return new Promise((resolve, reject) => {
        setTimeout(()=>{resolve("hi")}, 3000)
    })
}

const asyncFunction = async () => {
    return await asyncOperation();
}

const topDog = () => {
    asyncFunction().then((res) => {
        console.log(res);
    });
}

I like arrow function definitions more. But any string of the form "() => {...}" could also be written as "function () {...}"

So topDog is not async despite calling an async function.

enter image description here

EDIT: I realize a lot of the times you need to wrap an async function inside a sync function is inside a controller. For those situations, here's a party trick:

const getDemSweetDataz = (req, res) => {
    (async () => {
        try{
            res.status(200).json(
                await asyncOperation()
            );
        }
        catch(e){
            res.status(500).json(serviceResponse); //or whatever
        }
    })() //So we defined and immediately called this async function.
}

Utilizing this with callbacks, you can do a wrap that doesn't use promises:

const asyncOperation = () => {
    return new Promise((resolve, reject) => {
        setTimeout(()=>{resolve("hi")}, 3000)
    })
}

const asyncFunction = async (callback) => {
    let res = await asyncOperation();
    callback(res);
}

const topDog = () => {
    let callback = (res) => {
        console.log(res);
    };

    (async () => {
        await asyncFunction(callback)
    })()
}

By applying this trick to an EventEmitter, you can get the same results. Define the EventEmitter's listener where I've defined the callback, and emit the event where I called the callback.

1 Comment

This still ends up running a callback. I'm in a huge sync framework. I need to make a single call to aws sts get-caller-identity. I'd like to use the aws-sdk framework to do that... how do I write a function which encapsulates all the async stuff? For the life of me, I can't figure it out. :)
4

Nowadays this generator pattern can be a solution in many situations.

Here an example of sequential console prompts in nodejs using async readline.question function:

var main = (function* () {

  // just import and initialize 'readline' in nodejs
  var r = require('readline')
  var rl = r.createInterface({input: process.stdin, output: process.stdout })

  // magic here, the callback is the iterator.next
  var answerA = yield rl.question('do you want this? ', r=>main.next(r))    

  // and again, in a sync fashion
  var answerB = yield rl.question('are you sure? ', r=>main.next(r))        

  // readline boilerplate
  rl.close()

  console.log(answerA, answerB)

})()  // <-- executed: iterator created from generator
main.next()     // kick off the iterator, 
                // runs until the first 'yield', including rightmost code
                // and waits until another main.next() happens

Comments

4

There is a npm sync module also. which is used for synchronize the process of executing the query.

When you want to run parallel queries in synchronous way then node restrict to do that because it never wait for response. and sync module is much perfect for that kind of solution.

Sample code

/*require sync module*/
var Sync = require('sync');
    app.get('/',function(req,res,next){
      story.find().exec(function(err,data){
        var sync_function_data = find_user.sync(null, {name: "sanjeev"});
          res.send({story:data,user:sync_function_data});
        });
    });


    /*****sync function defined here *******/
    function find_user(req_json, callback) {
        process.nextTick(function () {

            users.find(req_json,function (err,data)
            {
                if (!err) {
                    callback(null, data);
                } else {
                    callback(null, err);
                }
            });
        });
    }

reference link: https://www.npmjs.com/package/sync

Comments

3

If function Fiber really turns async function sleep into sync

Yes. Inside the fiber, the function waits before logging ok. Fibers do not make async functions synchronous, but allow to write synchronous-looking code that uses async functions and then will run asynchronously inside a Fiber.

From time to time I find the need to encapsulate an async function into a sync function in order to avoid massive global re-factoring.

You cannot. It is impossible to make asynchronous code synchronous. You will need to anticipate that in your global code, and write it in async style from the beginning. Whether you wrap the global code in a fiber, use promises, promise generators, or simple callbacks depends on your preferences.

My objective is to minimize impact on the caller when data acquisition method is changed from sync to async

Both promises and fibers can do that.

5 Comments

this is the ABSOLUTE worst thing you can with Node.js: "synchronous-looking code that uses async functions and then will run asynchronously." if your API does that, you will ruin lives. if it is asynchronous, it should require a callback, and throw an error if no callback is provided. that is the best way to create an API, unless your goal is to trick people.
@AlexMills: Yes, that would be horrible indeed. However, luckily this is nothing that an API can do. An asynchronous API always needs to accept a callback / return a promise / expect to be run inside a fiber - it doesn't work without. Afaik, fibers were mostly used in quick'n'dirty scripts that were blocking and don't have any concurrency, but want to use async APIs; just like in node there sometimes are cases where you'd use the synchronous fs methods.
I generally like node. Especially if I can use typescript instead of pure js. But this whole async nonsense that permeates everything you do and literally infects every function in the call chain as soon as you decide to make a single async call is something I really... really hate. Async api is like an infectious disease, one call infects your entire code base forcing you to rewrite all the code you have. I really don't understand how anyone can possibly argue this is a good thing.
@Kris Node uses an asynchronous model for IO tasks because it's fast and simple. You can do many things synchronously as well, but blocking is slow as you can't do anything concurrently - unless you go for threads, which make everything complicated.
@Bergi I read the manifesto so I know the arguments. But changing your existing code into async the moment you hit that first api call that has no sync equivalent is not simple. Everything breaks and every single line of code has to be scrutinised. Unless your code is trivial I guarantee... it will take a while to convert and get it working again after converting the whole thing to async idiom.
1

I can't find a scenario that cannot be solved using node-fibers. The example you provided using node-fibers behaves as expected. The key is to run all the relevant code inside a fiber, so you don't have to start a new fiber in random positions.

Lets see an example: Say you use some framework, which is the entry point of your application (you cannot modify this framework). This framework loads nodejs modules as plugins, and calls some methods on the plugins. Lets say this framework only accepts synchronous functions, and does not use fibers by itself.

There is a library that you want to use in one of your plugins, but this library is async, and you don't want to modify it either.

The main thread cannot be yielded when no fiber is running, but you still can create plugins using fibers! Just create a wrapper entry that starts the whole framework inside a fiber, so you can yield the execution from the plugins.

Downside: If the framework uses setTimeout or Promises internally, then it will escape the fiber context. This can be worked around by mocking setTimeout, Promise.then, and all event handlers.

So this is how you can yield a fiber until a Promise is resolved. This code takes an async (Promise returning) function and resumes the fiber when the promise is resolved:

framework-entry.js

console.log(require("./my-plugin").run());

async-lib.js

exports.getValueAsync = () => {
  return new Promise(resolve => {
    setTimeout(() => {
      resolve("Async Value");
    }, 100);
  });
};

my-plugin.js

const Fiber = require("fibers");

function fiberWaitFor(promiseOrValue) {
  var fiber = Fiber.current, error, value;
  Promise.resolve(promiseOrValue).then(v => {
    error = false;
    value = v;
    fiber.run();
  }, e => {
    error = true;
    value = e;
    fiber.run();
  });
  Fiber.yield();
  if (error) {
    throw value;
  } else {
    return value;
  }
}

const asyncLib = require("./async-lib");

exports.run = () => {
  return fiberWaitFor(asyncLib.getValueAsync());
};

my-entry.js

require("fibers")(() => {
  require("./framework-entry");
}).run();

When you run node framework-entry.js it will throw an error: Error: yield() called with no fiber running. If you run node my-entry.js it works as expected.

Comments

-1

You shouldn't be looking at what happens around the call that creates the fiber but rather at what happens inside the fiber. Once you are inside the fiber you can program in sync style. For example:

function f1() {
    console.log('wait... ' + new Date);
    sleep(1000);
    console.log('ok... ' + new Date);   
}

function f2() {
    f1();
    f1();
}

Fiber(function() {
    f2();
}).run();

Inside the fiber you call f1, f2 and sleep as if they were sync.

In a typical web application, you will create the Fiber in your HTTP request dispatcher. Once you've done that you can write all your request handling logic in sync style, even if it calls async functions (fs, databases, etc.).

5 Comments

Thanks Bruno. But what if I need sync style in bootstrap code that needs to get executed before server binds to tcp port - such as config or data that have to be read from db which is opened async? I may ended up with wrapping entire server.js in Fiber, and I suspect that will kill concurrency at the entire process level. Nonetheless it's a suggestion that worth to verify. To me the ideal solution should be able to wrap an async function to provide a sync call syntax and only blocks next lines of code in the caller chain without sacrificing concurrency at process level.
You can wrap your entire bootstrap code inside one big Fiber call. Concurrency should not be a problem because the bootstrap code usually needs to run to completion before you start serving requests. Also, a fiber does not prevent other fibers to run: every time you hit a yield call you give other fibers (and the main thread) a chance to run.
I have wrapped Express bootstrap file server.js with fiber. The execution sequence is what I am looking for, but that wrap doesn't have any effect on request handler. So I guess have to apply the same wrapper to EACH dispatcher. I gave up at this point because it doesn't seem to do any better to help avoid global re-factoring. My objective is to minimize impact on the caller when data acquisition method is changed from sync to async in DAO layer and Fiber still falls a little short to the challenge.
@fred: It doesn't make much sense to "synchronize" event streams like the request handler - you'd need to have a while(true) handleNextRequest() loop. Wrapping each request handler in a fiber would.
@fred: fibers won't help you much with Express because Express' callback is not a continuation callback (a callback that is always called exactly once, either with an error or with a result). But fibers will solve the pyramid of doom when you have lots of code written on top of async APIs with continuation callbacks (like fs, mongodb, and lots of others).
-1

I struggled with this at first with node.js and async.js is the best library I have found to help you deal with this. If you want to write synchronous code with node, approach is this way.

var async = require('async');

console.log('in main');

doABunchOfThings(function() {
  console.log('back in main');
});

function doABunchOfThings(fnCallback) {
  async.series([
    function(callback) {
      console.log('step 1');
      callback();
    },
    function(callback) {
      setTimeout(callback, 1000);
    },
    function(callback) {
      console.log('step 2');
      callback();
    },
    function(callback) {
      setTimeout(callback, 2000);
    },
    function(callback) {
      console.log('step 3');
      callback();
    },
  ], function(err, results) {
    console.log('done with things');
    fnCallback();
  });
}

this program will ALWAYS produce the following...

in main
step 1
step 2
step 3
done with things
back in main

1 Comment

async works in your example b/c it's main, which doesn't care about caller. Imagine all your code is wrapped in a function which is supposed to return the result of one of your async function calls. It can be easily proofed not working by adding console.log('return'); at the end of your code. In such case the output of return will happen after in main but before step 1.
-1

Making Node.js code sync is essential in few aspects such as database. But actual advantage of Node.js lies in async code. As it is single thread non-blocking.

we can sync it using important functionality Fiber() Use await() and defer () we call all methods using await(). then replace the callback functions with defer().

Normal Async code.This uses CallBack functions.

function add (var a, var b, function(err,res){
       console.log(res);
});

 function sub (var res2, var b, function(err,res1){
           console.log(res);
    });

 function div (var res2, var b, function(err,res3){
           console.log(res3);
    });

Sync the above code using Fiber(), await() and defer()

fiber(function(){
     var obj1 = await(function add(var a, var b,defer()));
     var obj2 = await(function sub(var obj1, var b, defer()));
     var obj3 = await(function sub(var obj2, var b, defer()));

});

I hope this will help. Thank You

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.