Get the content (text) of an URL after Javascript has run with PHP

Question

Is it possible to get the content of a URL with PHP (using some sort of function like file_get_contents or header) but only after the execution of some JavaScript code?

Example:

mysite.com has a script that does loadUrlAfterJavascriptExec('http://exampletogetcontent.com/') and prints/echoes the content. imagine that some jQuery runs on http://exampletogetcontent.com/ that changes DOM, and loadUrlAfterJavascriptExec will get the resulting HTML

Can we do that?

Just to be clear, what I want is to get the content of a page through a URL, but only after JavaScript runs on the target page (the one PHP is getting its content).

I am aware PHP runs before the page is sent to the client, and JS only after that, but thought that maybe there was an expert workaround.

no :-) you want browser to get the page and run all js files for that page and get the page after execution? but you don't have control of that page? then the answer is NO, you can't — Alex
– Alex, Commented Feb 13, 2015 at 17:47

Community · Accepted Answer · 2020-06-20 09:12:55Z

18

Update 2 Adds more details on how to use phantomjs from PHP.

Update 1 (after clarification that javascript on target page need to run first)

Method 1:Use phantomjs(will execute javascript);

1. Download phantomjs and place the executable in a path that your PHP binary can reach.

2. Place the following 2 files in the same directory:

get-website.php

<?php
    
    $phantom_script= dirname(__FILE__). '/get-website.js'; 


    $response =  exec ('phantomjs ' . $phantom_script);

    echo  htmlspecialchars($response);
    ?>

get-website.js

var webPage = require('webpage');
var page = webPage.create();

page.open('http://google.com/', function(status) {
 console.log(page.content);
  phantom.exit();
});

3. Browse to get-website.php and the target site, http://google.com contents will return after executing inline javascript. You can also call this from a command line using php /path/to/get-website.php.

Method 2:Use Ajax with PHP (No phantomjs so won't run javascript);

/get-website.php

<?php
    
    $html=file_get_contents('http://google.com');
    echo $html;
    ?>

test.html

<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>on demo</title>
<style>
p {
color: red;
}
span {
color: blue;
}
</style>
<script src="https://code.jquery.com/jquery-1.10.2.js"></script>
</head>
<body>
<button id='click_me'>Click me</button>
<span style="display:none;"></span>
<script>

$( "#click_me" ).click(function () {
    $.get("/get-website.php", function(data) {
        var json = {
            html: JSON.stringify(data),
            delay: 1
        };
        alert(json.html);
        });
});
</script>
</body>
</html>

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Feb 13, 2015 at 18:44

AndrewD

5,2583 gold badges33 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

AndrewD Over a year ago

@victor-ferreira Did you have a chance to look at this solution?

Adamantus Over a year ago

This is out of date and PhantomJS is no longer in production.

Adamantus · Accepted Answer · 2019-01-15 21:26:00Z

3

I found a fantastic page on this, it's an entire tutorial on how to process the DOM of a page in PHP which is entirely created using javascript.

https://www.jacobward.co.uk/using-php-to-scrape-javascript-jquery-json-websites/ "PhantomJS development is suspended until further notice" so that option isn't a good one.

answered Jan 15, 2019 at 21:26

Adamantus

8211 gold badge12 silver badges30 bronze badges

1 Comment

valepu Over a year ago

The article does not seem to be available anymore but it's available on waybackmachine

Mahdi mehrabi · Accepted Answer · 2020-09-24 16:10:45Z

2

I think the easiest and best way is using this package https://github.com/spatie/browsershot just install it completely and use the below code

Browsershot::url('https://example.com')->bodyHtml()

edited Sep 24, 2020 at 16:10

answered Sep 24, 2020 at 11:19

Mahdi mehrabi

1,7522 gold badges22 silver badges23 bronze badges

Comments

The E · Accepted Answer · 2015-02-13 18:18:49Z

All the PHP runs before the information is sent to the client. All the JavaScript runs after the information is sent to the client.

To do something with PHP after the page loads, the page will need to either

reload, saving the JavaScript generated info in a cookie or as POST data (not ideal) OR
make an Ajax call to another PHP file to get the data. (much better)

Since the data appears to be in different file than your PHP anyway, this is a pretty good solution. Since you tagged it jQuery, I assume you're using it.

jQuery has a set of pages about how it implements Ajax

But the easiest way to use jQuery for this is .post

ex:

$.post( "http://example.com/myDataFile.txt", function( data ) {
    //do more JavaScript stuff with the data you just retrieved
});

$.post(), as the name implies, can send data along with the request for the data file, so if that request is to, say, a PHP file, the PHP file can use that data.

ex:

$.post( "http://example.com/myDataFile.txt",
    { foo: "bar"; yabba: "dabba" },
    function( data ) {
       //do more JavaScript stuff with the data you just retrieved
});

the data should be in JSON format in key/value pairs.

Collectives™ on Stack Overflow

Get the content (text) of an URL after Javascript has run with PHP

4 Answers 4

Method 1:Use phantomjs(will execute javascript);

Method 2:Use Ajax with PHP (No phantomjs so won't run javascript);

2 Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Method 1:Use phantomjs(will execute javascript);

Method 2:Use Ajax with PHP (No phantomjs so won't run javascript);

2 Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related