Phantomjs/Casperjs get url from JS script inside page

Question

I'm building a scraper with phantom/casper.

At this point, I need to extract a URL that appears in the page only inside a js script.

Example of the page source code :

<script>
    queueRequest('URL.aspx?var1='+VAR1+'&var2='+VAR2, getPageMenu');
</script>

I have no problem evaluating VAR1 and VAR2, as they are in the page context, but I need URL, which is hardcoded and has no reference to it. URL is of course different according to the page I'm on and I have no way of guessing it. Any ideas?

My ideas :

As the URL is called on page load to fill a div wih AJAX, I was thinking of maybe capturing the XHR request, but I don't know how.
I managed to get the script elem I need, using document.getElementsByTagName('script'). That may be one way to go, but how do I get only the line I need out of 200+ lines? (the one starting with queueRequest)

SO to make my question clear :

Which idea is better, 1 or 2?

if 1 : How do I capture the request URL with casper?

if 2 : How do I get the right line in my script?

struthersneil · Accepted Answer · 2013-10-19 19:20:38Z

2

If you want to search your script blocks, you can try something like this:

found = null;
scripts = document.getElementsByTagName('script');

for (i = 0; i < scripts.length; i++)
{
  matches = /queueRequest\('(.+)\?/.exec(scripts[i].innerText)

  if (matches) 
  {
    found = matches[1];
    break;
  }
}

alert(found);

There might be tighter ways to implement the same thing but the regex is roughly what you're after. Note that this will only get you the URL part of the first appearance of queueRequest('something.something?...) in embedded script blocks.

answered Oct 19, 2013 at 19:20

struthersneil

2,76013 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Phantomjs/Casperjs get url from JS script inside page

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related