Node.js Recursive Loop fail

Question

Below is the Node.js Script. It downloads the images contained in a div. The loop works fine for 9.86% that is upto id = 36. When id > 36 it exits the loop. I am using the node 0.12 version. The loop needs to run 365 times before its completion. I am usign the method of recursive callback.

Code:

//Required modules
var fs = require('fs'),
         cheerio = require('cheerio'),
         request = require('request');

//Default Variables
var baseURI =  'http://www.website/';
var year = 2013;
var id = 1;
var savePath = process.argv[2]; 

//Download Function
var download = function(uri, filename, callback){
    request({ uri: uri }, function(err, res, body){    
    var $ = cheerio.load(body);
    var imgDiv = $('#img-wallpaper').children()['0'];
    if(err)
        console.err(err);
    if(typeof imgDiv !== 'undefined') {
    request(imgDiv.attribs.src).pipe(fs.createWriteStream(filename)).on('close', callback);}
    });
};



//Main Function
console.log("Downloading . . .");

// Loop function to create a recursive effect
(function loop(){
    download(baseURI+year+'/'+id+'/wallpaper/', savePath+id+'.jpg', 
    function(){
        console.log(((id/365)*100).toFixed(2)+'% completed');
        if(id == 330) 
            year = "2014";
        if(((id/365)*100) != 100){
            id=id+1;
            loop();}
        });
})(1)

Why are you recursing when there's no dependency between the loops? Why not dispatch all the download calls and gather up the results? — martypdx
– martypdx, Commented Jun 13, 2015 at 18:43
Image Write will be 'partial'. The callstack buffer gets filled up resulting in socked hang up error!. — kingspp
– kingspp, Commented Jun 13, 2015 at 18:48

stdob-- · Accepted Answer · 2015-06-13 19:05:07Z

1

Do I understand correctly that if you set the starting value for the id more than 35 (36?) the script is not downloaded any images?

Test the script on the fixed uri and on the fixed image by changing only the variables. Script is expected to work out? If this is the case:

Or not called callback for body request
Or a false condition typeof imgDiv !== 'undefined'
Or not called callback for request image

Thus, when an error in one of these points the script stops working. It is necessary to change the severity of conditions.

answered Jun 13, 2015 at 19:05

stdob--

29.2k6 gold badges62 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

kingspp Over a year ago

If i change the condition to id <= 365 then if id>36, the script works, but if id = 1, the script stops at 36. I think the second option is causing the error. Thanks for the reply!!

kingspp · Accepted Answer · 2015-06-13 19:41:46Z

0

As @stdob said, The error was caused due to

Or a false condition typeof imgDiv !== 'undefined'

Though the answer is not the right way to overcome the error, it is more of an hack type. It ignores the error and continues the script!

if(typeof imgDiv !== 'undefined') {
    request(imgDiv.attribs.src).pipe(fs.createWriteStream(filename)).on('close', callback);
}
else{
    id++;
    request(uri).pipe(fs.createWriteStream(filename)).on('close', callback);
}

answered Jun 13, 2015 at 19:41

kingspp

3111 gold badge3 silver badges12 bronze badges

Collectives™ on Stack Overflow

Node.js Recursive Loop fail

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related