How can I handle Javascript in a Perl web crawler?

Question

I would like to crawl a website, the problem is, that its full of JavaScript things, such as buttons and such that when they are pressed, they do not change the URL, but the data on the page is changed.

Usually I use LWP / Mechanize etc to crawl sites, but neither support JavaScript. any idea?

erickb · Accepted Answer · 2010-09-22 12:55:19Z

8

Another option might be Selenium with WWW::Selenium module

answered Sep 22, 2010 at 12:55

erickb

6,3094 gold badges27 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ishnid · Accepted Answer · 2010-09-22 12:35:48Z

7

The WWW::Scripter module has a JavaScript plugin that may be useful. Can't say I've used it myself, however.

answered Sep 22, 2010 at 12:35

ishnid

5392 silver badges5 bronze badges

Comments

Eric Strom · Accepted Answer · 2010-09-22 15:40:30Z

5

WWW::Mechanize::Firefox might be of use. that way you can have Firefox handle the complex JavaScript issues and then extract the resultant html.

answered Sep 22, 2010 at 15:40

Eric Strom

40.2k2 gold badges83 silver badges152 bronze badges

Comments

Minh Le · Accepted Answer · 2013-11-18 19:47:33Z

1

I would suggest HtmlUnit and Perl wrapper: WWW::HtmlUnit.

answered Nov 18, 2013 at 19:47

Minh Le

1,1651 gold badge12 silver badges21 bronze badges

Collectives™ on Stack Overflow

How can I handle Javascript in a Perl web crawler?

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related