Function iteration with Async waterfall in Node.js

Question

I am trying to get resource from the some url, that resource is separated as multiple pages, the number of page is over 500, but the order of resource have to be guaranteed, so I decided to use Async module.

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // get data from the url ...
    // when success, 
    callback();
};

So, above function is get resource from some url, I have to iterate this function to many times, but I don't know how can I iterate this with async waterfall. What point I should push the for iteration?

async.waterfall([

    // page 1 
    function(callback){
        getData(1, function(){
            callback(null);
        });
    },

    // page 2
    function(callback){
        getData(2, function(){
            callback(null);
        });
    },

    // page 3, 4, 5..... 100, ... 500


],function(err, result){
    if(err) return next();

    console.log('finish !');
});

I wouldn't even use for, use recursion.

Ben Fortune
– Ben Fortune

2016-08-02 08:52:05 +00:00
Commented Aug 2, 2016 at 8:52 — Ben Fortune
– Ben Fortune, Commented Aug 2, 2016 at 8:52

Nicholas Robinson · Accepted Answer · 2016-08-04 07:49:12Z

2

Why don't you use Promises:

function getData(page, callback){
    var url = "http://someurl.com/document?page="+page;
    // var request = call to get data from the url ... 
    return request;
};

var promises = [];

for (var page = 1; page < totalPages; page++) {
    promises.push(getData(page));
}

Promise.all(promises).then(function (listOfResults) {
    // Do callback here
});

Just make sure that your method of sending the get request returns a promise. If not you can use a function like this:

function usesCallback(data, callback) {
    // Do something with data
    callback();
}

function makePromise(data) {
    return new Promise(function (resolve, reject) {
        usesCallback(data, resolve);
    });
}

Here is some more info on Promises

edited Aug 4, 2016 at 7:49

answered Aug 2, 2016 at 8:59

Nicholas Robinson

1,4291 gold badge11 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

slebetman Over a year ago

Promises is not good enough for this. Promises only implement promise.all as a flow control method which is equivalent to async.parallel. And only that. The async.js library on the other hand provides many other algorithms to process async functions: parallel, series, waterfall, whilst (while), during, until etc. If you prefer the promise design pattern there is a promisified version of async.js called async-q

lujjjh · Accepted Answer · 2016-08-02 11:57:14Z

If you do want to use async, async.map() and async.mapLimit() are better choices since they apply the iteratee to each item in parallel and guarantee the results array to be in the same order as the original collection.

async.mapLimit(_.range(1, 500), 10, getData, function (err, results) {
    // do something with results
});

The code above means to get data from page 1 to page 500, with no more than 10 async operations at a time.

_.range(1, 500) is a function from underscore to generate an array [1, 2, 3, ..., 500]. If you do not like to use underscore in your project, you can simply rewrite it such as:

function range(lower, upper) {
    return Array.apply(null, Array(upper - lower + 1))
        .map(function (_, i) { return lower + i; });
}

Harekam Singh · Accepted Answer · 2016-08-02 11:31:53Z

0

You can use async's whilst for this check here.

answered Aug 2, 2016 at 11:31

Harekam Singh

669 bronze badges

Collectives™ on Stack Overflow

Function iteration with Async waterfall in Node.js

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related