issue using async module in Node.js

Question

Having an issue, appreciate any help.

I'm trying to gather all my async functions together. Tried async.parallel, async.each, gather-gm. Nothing makes the final callback work. Here's the updated code(but yet not working properly):

var calls = [];

async.each(parser.allHrefs,function (href,callback) {

    getHtml(href,function(err, add){
        console.log("Passing data: " + href);

        if(err){
            return callback(err);
        };

        if(add){
            calls.push(href);
        };

        return callback();
    });
}, function (err) {
    if(err){
        console.log('something went wrong');
    }else{
        console.log('finished');
    };
});

And the first function:

function getHtml(link, callback) {    
  httpreq.get(link, function(err, res) {

    if(err) {            
      return callback(err);      
    }

    if(res.statusCode >= 300) {      
      return callback(null, false);       
    } else {
      //cut parsing code here...
      return callback(null, true);          
    }
  });
}

p.s.:I've updated the code couple times. In this example i've tried to use async.parallel. And the thing is when i even get no errors, i still dont get to the "getLocations" function. Code looks much more better than first versions, but still refuses to work correctly.

There's a possibility that you could call gethtml and it not call callback. You should make sure it ALWAYS executes callback regardless of outcome. If it's an error, pass the error to the callback. — user400654
– user400654, Commented Mar 24, 2015 at 18:17
parallel is different from each because the first executes an array of functions and the second moves through an array and executes the same function for each element. The other thing is that gethtml is not returning a value so you aren't pushing anything into the calls array. — Osukaa
– Osukaa, Commented Mar 24, 2015 at 20:16
i've updated the code, according to your advice, still having an issue... — Lazyexpert
– Lazyexpert, Commented Mar 24, 2015 at 21:35

Reto · Accepted Answer · 2015-03-24 18:21:51Z

1

There are several issues in your gethtml function you need to fix.

Make sure that everywhere you have now return, you call the callback, e.g.

return callback(err)

when you want to communicate an error or

return callback(null,  result)

when you want to communicate success and return a result. Never return without calling the callback.

answered Mar 24, 2015 at 18:21

Reto

3,1631 gold badge24 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Lazyexpert Over a year ago

Thanks a lot. This makes sense. And i've managed to move further. The error was in 404 || 500. As for this, i probably shouldn't return it as error, its a common situation having bad url, i suppose?

Reto Over a year ago

Well, I don't know your requirements or the use case you are implementing. It really depends on the context of what you are trying to do whether a 404 or a 500 for one URL is acceptable and just produces an empty result or whether this is a condition that should stop the whole processing.

Osukaa · Accepted Answer · 2015-03-24 20:34:57Z

0

Instead of deleting the parser.allHrefs while you go through it. You should refactor de getHtml just to know if you can add the link or not. Like this:

function getHtml (link, callback) {    
    httpreq.get(link, function(err, res) {
        //the request has already finished executing here
        if(err) {      
            return callback(err);      
        }

        if(res.statusCode >= 300) {
            return callback(null,false);
        } else {
            //cut parsing code here...
            return callback(null,true);
        }
    });
};

This way when you call the gatherSecondLevelData function you check if you add the link or not like this:

function gatherSecondLevelData (err) {
    var calls = [];

    async.each(parser.allHrefs,function (href,callback) {

        getHtml(href,function(err, add){

            if(err){
                return callback(err);
            };

            if(add){
                calls.push(href);
            };

            return callback();
        });
    }, function (err) {
        if(err){
            console.log('something went wrong');
        }else{
            console.log('finished');
        };
    });
};

answered Mar 24, 2015 at 20:34

Osukaa

7083 gold badges11 silver badges23 bronze badges

3 Comments

Lazyexpert Over a year ago

Thanks for advice. Unfortunatelly, still have the same issue. I get neither error, nor success. The data is passed to the getHtml successfully and before i've checked the parser, it also does exactly what i expect. Can't find any idea why this isn't working.

Osukaa Over a year ago

if you put a console.log in the getHtml callback what are you getting in the err, add parameters?

Lazyexpert Over a year ago

Well, i did have this (err) in third function and it didn't work. Actually, i've managed to reach "finished" today just commenting out all the parsing logic. That's already a victory ;) Thanks all for help.

Collectives™ on Stack Overflow

issue using async module in Node.js

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related