Nodejs scraping website after javascript has loaded the values

Question

Probably a newbie question on nodejs/jsdom

I am trying to scrape a website using node.js. I am using jsdom and jquery to get the html and parse the required things. But, somehow the values i am getting are not the ones shown on the website. Basically the values are dynamically changed by javascript and i want those values. The whole reason i was using nodejs/jsdom for scraping was that js would be executed and I get the values after that event.

Is there some way to tell jsdom to wait until the javascript executes? or have i got this all wrong? I have googled a lot on this matter.

tapan · Accepted Answer · 2012-05-04 07:34:50Z

You would be better of using something like casperjs http://casperjs.org/. It is a testing utility based on phantomjs. It is basically exactly like opening the page in a webkit browser, just without the GUI. You could write something like. I dont think it works with node, but it should be easy enough to run a casper script and pipe the output back to node.:

var casper = require('casper').create({
    loadImages: true,
    loadPlugins: true,
    verbose: true,
    //logLevel: 'info',
    clientScripts: [
        'jquery-1.7.1.min.js',
    ],
    viewportSize: {
        width: 1366,
        height: 768,
    },
    pageSettings: {
        javascriptEnabled: true,
        userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.9 Safari/536.5',
    },
});

casper.start(url);

casper.thenEvaluate(function () {
    //javascript code to run in the scope of the page
});

Linus Thiel · Accepted Answer · 2012-02-21 09:39:51Z

0

First off, how are you using jsdom? Apparently, jsdom.env does not execute scripts in the DOM, only the scripts that you add in the call to jsdom.env. If you want to execute scripts, I think you should use jsdom.jsdom.

Second, you need to specify an onload handler. This should execute after the document is ready, and hopefully any scripts will have changed the DOM to your liking.

Something like this:

var jsdom = require('jsdom').jsdom
  , document = jsdom(html)
  , window = document.createWindow();

document.onload = function() {
  // Do your stuff
}

edited Feb 21, 2012 at 9:39

answered Feb 21, 2012 at 9:34

Linus Thiel

39.3k9 gold badges112 silver badges104 bronze badges

Collectives™ on Stack Overflow

Nodejs scraping website after javascript has loaded the values

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related