My web-crawler with node.js didn't display the expected output?

Question

Here is my web-crawler with node.js using cheerio library:

var request = require('request');
var cheerio = require('cheerio');
var fs = require('fs');
var urls = [];

request('http://www.reddit.com', function(err, resp, body){
    if(!err && resp.statusCode == 200){
        var $ = cheerio.load(body);
        $('a.title may-blank').each(function(){
            var url = this.attr('href');
            urls.push(url);
        });
        console.log(urls);
    }
});

But when I run it I get the the following output:

[]

Instead of 25 links in the array.

What have I done wrong?

How can I fix that?

T.J. Crowder · Accepted Answer · 2014-08-25 12:25:08Z

4

I'm guessing may-blank is a class, so you need a . in front of it:

$('a.title .may-blank').each(...
// Here ---^

...although at present, a.title .may-blank doesn't match any elements on the reddit front page for me; there are no .may-blank elements that are descendants of a.title.

If you wanted a elements that have both the class title and has the class may-blank, remove the space before .may-blank; for me there are currently 36 of those:

$('a.title.may-blank').each(...
//        ^-- no space

Or just .may-blank matches 167.

edited Aug 25, 2014 at 12:25

answered Aug 25, 2014 at 12:19

T.J. Crowder

1.1m201 gold badges2k silver badges2k bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Michael Over a year ago

now I am getting this error: TypeError: Object #<Object> has no method 'attr'

T.J. Crowder Over a year ago

@MichaelVayvala: That would be a different question. (And are you sure you didn't want $(this).attr(...)? That would be what it would be with jQuery; last time I used Cheerio, though, it was slightly different from jQuery in terms of what it did with this...)

e-nouri · Accepted Answer · 2014-08-25 12:34:16Z

-1

There is a little tpyo if I am not mistaking, the tag selector should be 'title may-blank ', notice the space after the blank, or you should change the selector to '^⁼' starts with to be more forgiven, hope that will help.

answered Aug 25, 2014 at 12:34

e-nouri

2,6262 gold badges23 silver badges38 bronze badges

1 Comment

e-nouri Over a year ago

my bad, didn't pay attention the selector type, it was a class !

Collectives™ on Stack Overflow

My web-crawler with node.js didn't display the expected output?

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related