Testing URLs with Node.js

Question

Assuming I have an array of URLs and I want to ensure each URL is working I have created the following code. However only the last URL in the array is getting tested. How can I ensure each url returns a 200 response code? To be clear these are all remote addresses I am testing and they point to decently sized PDFs.

Updated based on @lukas.pukenis's response. The results are similar, only a few files are actually checked.

function check(l) {
    console.log(l);

    http.get(l, function(res) {
        if (res.statusCode != 200) {
            console.log(res.statusCode + ' on '+l);
        } else {
            console.log('success on ' + l);
        }
    });
}

for (link in fileLinks) {
  check(fileLinks[link]);
}

This code outputs:

http://somesite.com/somefile1.pdf
http://somesite.com/somefile2.pdf
http://somesite.com/somefile3.pdf
...
all the rest of them
...
http://somesite.com/somefile99.pdf
success on http://somesite.com/somefile1.pdf
success on http://somesite.com/somefile2.pdf
404 on http://somesite.com/somefile5.pdf
success on http://somesite.com/somefile7.pdf

If fileLinks is an Array, try using the .forEach(function(item){}) function instead of for, which is used to loop through the various keys in a javascript object. It's not meant to be used with arrays. — Jondlm
– Jondlm, Commented Oct 24, 2013 at 21:17

lukas.pukenis · Accepted Answer · 2013-10-24 21:08:57Z

1

This is because your loop rewrites l variable each time with var l = fileLinks[link];

so l has a value of the last value of array. In order to preserve unique l value you need to store it somewhere. Better - function. Like this:

function check(l) {
  var req = http.get(l, function(res) {
    if (res.statusCode != 200) {
      console.log(res.statusCode + ' on '+l);
    } else {
      console.log('success on ' + l);
    }
  }

  req.on('close', function() {
    console.log('Request done');
  });

for (link in fileLinks) {
  var l = fileLinks[link];
  check(l);
}

Having a function is no magic here. It just preserves your local values in memory for each function call, so l is unique each time it needs to be.

edited Oct 24, 2013 at 21:08

answered Oct 24, 2013 at 20:25

lukas.pukenis

13.7k13 gold badges50 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jjr2527 Over a year ago

Thanks Lukas. This helped but still didn't fix the issue. I updated the question to reflect your suggested edits.

lukas.pukenis Over a year ago

Are you sure your array gets iterated correctly? for .. in .. loop is used for object iteraction. Please show your dataset or just a structure of it :)

jjr2527 Over a year ago

It's just an array of strings. I think it's looping because the log entry before the .get call is still being written for every record. I'm guessing that the .get call is being executed in the background and nodemon isn't waiting for execution to finish before closing the app.

lukas.pukenis Over a year ago

Can you tell me what happens if you modify your code according to my updated code? I have read that node waits until all callback pools are empty and you can add your event with .on() so give it a try :)

Jondlm · Accepted Answer · 2013-10-24 22:15:20Z

1

The for expression shouldn't be used with arrays. Replace the for loop for something like this:

fileLinks.forEach(function(item){
  check(item);
});

When doing this many outgoing requests, you may want to increase the maxSockets to something greater than 5, the default, otherwise you might get unexpected behavior. Do this after you require('http'):

http.globalAgent.maxSockets = 150;

Also, when you stick your console.log outside the callback function, it's not going to be displayed at the same time the response comes back from the server. It's redundant anyway. Here is a complete working example:

var http = require('http');
var url = require('url');

function check(l) {
  var u = url.parse(l);

  var opts = {
    host: u.host,
    path: u.path,
    agent: false // prevents pooling behavior
  };

    http.get(opts, function(res) {
        if (res.statusCode != 200) {
            console.log(res.statusCode + ' on '+l);
        } else {
            console.log('success on ' + l);
        }
    });
}

fileLinks = ['http://www.google.com','http://www.google.com'];

fileLinks.forEach(function(item){
  check(item);
});

edited Oct 24, 2013 at 22:15

answered Oct 24, 2013 at 21:20

Jondlm

8,8923 gold badges26 silver badges30 bronze badges

3 Comments

jjr2527 Over a year ago

Your example works as does Lukas's. The maxSockets setting helps but still the app closes before all requests complete. In your example 150 are processed then another 12 or so then the app just closes.

Jondlm Over a year ago

Try setting agent to false to disable the pooling behavior.

Jondlm Over a year ago

Err, I was wrong about that false option. I just edited my answer. Try that.

Collectives™ on Stack Overflow

Testing URLs with Node.js

2 Answers 2

4 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related