How to get html source code after javascript transformation?

Question

for a project at school I am trying to make a website that can show your grades in a prettier way than it's being done now. I have been able to log in to the site using cURL and now I want to get the grades in a string so I can edit it with PHP. The only problem is that cURL gets the html source code when it hasn't been edited by the javascript that gets the grades.

So basically I want the code that you get when you open firebug or inspector in a string so I can edit it with php.

Does anyone have an idea on how to do this? I have seen several posts that say that you have to wait till the page has loaded, but I have no clue on how to make my site wait for another third-party site to be loaded.

The code that I am waiting to be executed and of which I want the result is this:

<script type="text/javascript">
    var widgetWrapper = $("#objectWrapper325");
    if (widgetWrapper[0].timer !== undefined) {
        clearTimeout( jQuery('#objectWrapper325')[0].timer );
    }
    widgetWrapper[0].timer = setTimeout( function() {
        if (widgetWrapper[0].xhr !== undefined) {
            widgetWrapper[0].xhr.abort();
        }
        widgetWrapper[0].xhr = jQuery.ajax({
            type: 'GET',
            url: "",
            data: {
                "wis_ajax": 1,
                "ajax_object": 325,
                'llnr': '105629'
            },
            success: function(d) {
                var goodWidth = widgetWrapper.width();
                widgetWrapper.html(d);
                /* update width, needed for bug with standard template */
                $("#objectWrapper325 .result__overview").css('width',goodWidth-$("#objectWrapper325         .result__subjectlabels").width());
            }
        });
    }, 500+(Math.random()*1000));
</script>

You cannot get content generating in javascript from PHP - this is why it called server side and client side - server side can take only what the server serves and client side handling the client data - what you can do is sniff that page and call to the page that the ajax js code is calling - and then get the response and process it in the server — Adidi
– Adidi, Commented Apr 2, 2013 at 22:10
If I understand correctly, The process is opposite of what you want to do. The PHP executes before anything client-side. You would have to pull the information on another page and use something like an ajax call to collect the information and display it. Or pull the information from one side, then place it in a database, then push it to the page you want to view it on. — ntgCleaner
– ntgCleaner, Commented Apr 2, 2013 at 22:11
This is where something like phantomjs or zombiejs is useful — Mark Baker
– Mark Baker, Commented Apr 2, 2013 at 22:15
You won't be able to edit it with PHP because PHP is executed in the server (away from your computer) and HTML/Javascript is nothing but code until it reaches your browser (at this time PHP has already resolved and there is no way you can execute it) — Ryoku
– Ryoku, Commented Apr 2, 2013 at 22:15
@David: Haha, I am not trying to intercept my grades, I am just trying to display them. No need to worry there ;) — Blarfor
– Blarfor, Commented Apr 3, 2013 at 10:56

Mohamed Ennahdi El Idrissi · Accepted Answer · 2016-12-01 17:10:54Z

6

First you have to understand a subtle but very important difference between using cURL to get a webpage, and using your browser visiting that same page.

1. Loading a page with a browser

When you enter the address on the location bar, the browser converts the url into an ip address . Then it tries to reach the web server with that address asking for a web page. From now on the browser will only speak HTTP with the web server. HTTP is a protocol made for carrying documents over network. The browser is actually asking for an html document (A bunch of text) from the web server. The web server answers by sending the web page to the browser. If the web page is a static page, the web server is just picking an html file and sending it over network. If it's a dynamic page, the web server use some high level code (like php) to generate to the web page then send it over.

Once the web page has been downloaded, the browser will then parse the page and interprets the html inside which produces the actual web page on the browser. During the parsing process, when the browser finds script tags it will interpret their content as javascript, which is a language used in browser to manipulate the look of the web page and do stuff inside the browser.

Remember, the web server only sent a web page containing html content he has no clue of what's javascript.

So when you load a web page on a browser the javascript is ONLY interpreted once it is downloaded on the browser.

2. What is cURL

If you take a look at curl man page, you'll learn that curl is a tool to transfer data from/to servers which can speak some supported protocols and HTTP is one of them. When you download a page with curl, it will try to download the page the same way your browser does it but will not parse or interpret anything. cURL does not understand javascript or html, all it knows about is how to speak to web servers.

3. Solution

So what you need in your case is to download the page like cURL does it and also somehow make the javascript to be interpreted as if it was inside a browser.

If you had follwed me up to here then you're ready to take a look at CasperJS.

edited Dec 1, 2016 at 17:10

Mohamed Ennahdi El Idrissi

2,9872 gold badges21 silver badges35 bronze badges

answered Apr 2, 2013 at 22:39

blob42

5492 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Blarfor Over a year ago

Ok I understand everything you said, that cURL doesn't have a java interpreter as it is not a browser. I have read some posts which mentioned phantomJS and other related utilities, but I have no idea on how to use them. If I download the library is there just a function like interpret.javascript or something that makes the javascirpt work?

Blarfor Over a year ago

Also, I am not the only one that should be able to see my grades. Are PhantomJs and CasperJS installed on your computer, or is it just a repository that you can include in a directory on your server?

Daedalus Over a year ago

Javascript.. not java. They are two different languages.

Collectives™ on Stack Overflow

How to get html source code after javascript transformation?

1 Answer 1

1. Loading a page with a browser

2. What is cURL

3. Solution

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1. Loading a page with a browser

2. What is cURL

3. Solution

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related