Simple PHP web crawler returning simple HTML DOM error

Question

I have a PHP script that returns links on a webpage. I am getting 500 internal error and this is what my server logs say. I let my friend try the same code on his server and it seems to run correctly. Can someone help me debug my problem? The warning says something about the wrapper is disabled. I checked line 1081 but I do not see allow_url_fopen.

PHP Warning: file_get_contents(): http:// wrapper is disabled in the server configuration by allow_url_fopen=0 in /hermes/bosweb/web066/b669/ipg.streamversetv/simple_html_dom.php on line 1081

PHP Warning: file_get_contents(http://www.dota2lounge.com/): failed to open stream: no suitable wrapper could be found in /hermes/bosweb/web066/b669/ipg.streamversetv/simple_html_dom.php on line 1081

PHP Fatal error: Call to a member function find() on a non-object in /hermes/bosweb/web066/b669/ipg.streamversetv/sim

<?php
 include_once('simple_html_dom.php');
 $target_url = 'http://www.dota2lounge.com/';
 $html = new simple_html_dom();
 $html->load_file($target_url);
  foreach($html->find(a) as $link){
    echo $link->href.'<br />';
  }
?>

Server disabled the option to use file_get_contents() with remote files. — Vlad
– Vlad, Commented Sep 27, 2013 at 16:50
The errors message are VERY clear. Line 1081 is where you're ATTEMPTING to use a url in an fopen context. You need to look in your php.ini to enable the setting. — Marc B
– Marc B, Commented Sep 27, 2013 at 16:51

DB5 · Accepted Answer · 2013-12-20 14:45:26Z

6

Download latest simple_html_dom.php: LINK TO DOWNLOAD

Open simple_html_dom.php in your favourite editor and add this code to the first couple of line(can be added right after <?php):

function file_get_contents_curl($url) {

    $ch = curl_init();
    curl_setopt($ch, CURLOPT_AUTOREFERER, TRUE);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_URL, $url);     

    $data = curl_exec($ch);
    curl_close($ch);

    return $data; }

Find line starting with function file_get_html($url..... for me it is line 71, but you can use search in your editor as well. (search for file_get_html)
Edit this line(some lines below after function file_get_html):

$contents = file_get_contents($url, $use_include_path, $context, $offset);

to this:

$contents = file_get_contents_curl($url);
Instead of load_file, use file_get_html an it will work for you, without editing php.ini

edited Dec 20, 2013 at 14:45

DB5

14.1k7 gold badges69 silver badges73 bronze badges

answered Dec 20, 2013 at 14:04

p.tamas

611 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Sikandar Ali Chishty Over a year ago

Please clarify 5th point. where to use "file_get_html" instead of "load_file" ? @DB5

DB5 Over a year ago

@SikandarAliChishty I only edited this answer to make the formatting clearer. p.tamas is the one who wrote the answer, you’ll need to direct your question to him.

Sikandar Ali Chishty Over a year ago

Please clarify 5th point. where to use "file_get_html" instead of "load_file" ? @p.tamas

Aak Over a year ago

I had the same error in log, changing to the curl version it works now fine.

Chris James Champeau Over a year ago

I wish I could upvote this more, really good solution

Jason · Accepted Answer · 2013-09-27 17:27:43Z

1

You need to set the allow_url_fopen php setting to 1 to allow using fopen() with urls.

Reference: PHP: Runtime Configuration

Edit:
Also tracked down another thing, have you tried loading this way?

<?php
    include_once('simple_html_dom.php');

    $html = file_get_html('http://www.dota2lounge.com/');

    foreach($html->find('a') as $link)
    {
        echo $link->href.'<br />';
    }
?>

edited Sep 27, 2013 at 17:27

answered Sep 27, 2013 at 16:52

Jason

1,9621 gold badge14 silver badges16 bronze badges

7 Comments

Instinct Over a year ago

I just changed, allow_url_fopen = 1. Still getting 500 error, should I allow some time for my server to change?

Jason Over a year ago

What happens when you test it now? If you still get the same message, it's probably because you need to change it in your php.ini settings and restart the server.

Instinct Over a year ago

I'm still getting the 500 error, I'm not sure what the actual error is because my log isn't updated yet. How would I go about to resetting a server? I have my website hosted on ipage.com

Jason Over a year ago

You can try adding this line to your .htaccess instead: php_value allow_url_fopen On it may not work, depending on how your server is configured. By the way, 500 is a server error, so if you're trying to connect to someone else's site, their .htaccess might be preventing you from accessing the file.

Instinct Over a year ago

Here is the new error im getting. 20130927T131614: stre/crawler.php PHP Fatal error: Call to a member function find() on a non-object in ipg.htfhft/simple_html_dom.php on line 1113

|

Collectives™ on Stack Overflow

Simple PHP web crawler returning simple HTML DOM error

2 Answers 2

5 Comments

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related