Perl - Submit Javascript action to host

Question

I am building a Spider in Perl and have a problem:

The Site I want to spider uses a JavaScript for Age-Verification and I don't know how to get past this in Perl...?

The Script looks like this:

<script type = "text/javascript">

function set_age_verified(){

    new Request({

        method: "post",

        url: "/user/set_age_verified"

    }).send();

    $('age_verification').setStyles({visibility: 'hidden', display: 'none'});

    $('page_after_verification').setStyles({visibility: 'visible', display: 'block'});

    return false;

}

</script>

And here the OnClick Event :

<a href="#" onclick="return set_age_verified();"><img src="http://example.com/age-verification-enter.gif" alt="ENTER"></a>

Ben · Accepted Answer · 2011-03-17 17:23:42Z

2

The function has two effects. One is to POST a request to the URL "/user/set_age_verified" and the other is to alter the display visibility of some HTML.

Your spider can easily ignore the second effect, but presumably the first effect, by going to the server, sets some cookie or server variable which the server will require.

You do not have to actually run the javascript, so long as the server sees the same POST data.

The answer is for your Perl script to detect pages which have this javascript, and to call a Perl function to POST the data to the age verification URL.

Any cookie or similar which is returned will have to be recorded by you - your HTTP library may take care of this for you though.

answered Mar 17, 2011 at 17:23

Ben

35.8k6 gold badges81 silver badges117 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

smcphill · Accepted Answer · 2011-03-17 17:29:09Z

1

What Perl modules are you using? WWW::Mechanize has an AJAX plugin, although it hasn't been updated in a while. I guess you could also look at something like WWW::Selenium.

But I bet that AJAX request is going to inject some HTML that requires the user to input some data, then submit a form. Pretty tricky to cover all bases for that general case...

answered Mar 17, 2011 at 17:29

smcphill

8265 silver badges13 bronze badges

Comments

manatwork · Accepted Answer · 2011-03-17 17:35:58Z

1

Take a look at the WWW::Mechanize::Firefox module. It allows you handle some JavaScript.

answered Mar 17, 2011 at 17:35

manatwork

1,7501 gold badge30 silver badges33 bronze badges

Comments

Allen Holman · Accepted Answer · 2011-03-17 17:53:06Z

1

Also, in Firefox HTTPHeaders is your best friend.

Turn it on, manually click what ever you need to in order for the Javascript to run and submit to the server, then go back to the HTTPHeaders window. It will show you exactly what that Javascript event sent to the server (GET or POST + the data, even if it is HTTPS) - as well as the server response.

answered Mar 17, 2011 at 17:53

Allen Holman

1257 bronze badges

1 Comment

tadmc Over a year ago

an alternative to HTTPHeaders that I like is the "Web Scraping Proxy" from AT&T (google it). You set it up as a proxy in your browser, then navigate to the info you want to scrape. It logs all HTTP traffic in the form of Perl code that generates the identical HTTP request/response(s).

Collectives™ on Stack Overflow

Perl - Submit Javascript action to host

4 Answers 4

Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related