PHP Simple HTML Dom Parser Memory Leak / Usage

Question

I'm trying to use PHP Simple HTML Dom Parser to parse some information from some sites. Does not matter what and where. But it seems, that there is some HUGE memory problem with it. I managed to cut the html code to only 6kB, but script that finds some elements and saves them to database takes even 700MB of ram and over 1GB of virtual memory! I read somewhere that I should use ->clear() to free up some memory, but seems that this is not the case.

I use str_get_html() once and 5 times using ->find() assigning the result to variable.

$main_html = str_get_html($main_site);
$x = $main_html->find(...);
$y = $main_html->find(...);

etc.

I tried to use for example $y->clear() after using $y but I get an error PHP Fatal error: Call to a member function clear() on a non-object even tho $y does exist and if($y) is true. Even foreach($y) echo $y->plaintext does return plaintext of $y.

From htop:

PID USER     PRI  NI  VIRT   RES   SHR S CPU% MEM%   TIME+  Command
8839 username    20   0 1068M  638M   268 R 23.0  8.0  0:08.41 php myscript.php

What is wrong?

Simple test:

echo "(MEM:".memory_get_usage()."->";
$product = $p->find('a',0)->href;
echo memory_get_usage()."->";
unset($product);
$p->clear();
unset($p);
echo memory_get_usage().")";

The result is:

(MEM:11865648->11866192->11865936)

More readable form:

11865648->
11866192-> (+544 in total)
11865936 (+288 in total)

Of course I can't use $product->clear() as it says that PHP Fatal error: Call to a member function clear() on a non-object

I am clearing every object that was created using str_get_html(), and that does not help. — Flash Thunder
– Flash Thunder, Commented Aug 6, 2013 at 22:01

Flash Thunder · Accepted Answer · 2018-02-18 13:20:23Z

9

Seems there are some memory problems when using str_html_get or similar function that creates simple_html_dom object few times without clearing and destroying the previous one. Especially when using ->find that creates array of simple_html_dom_node objects. Even FAQ on authors site says to clear and destroy previous simple_html_dom object before creating new one, but sometimes it can't be done without additional code and memory.

That's why I created this function, to remove all PHP Simple HTML Dom Parser traces from memory:

function clean_all(&$items,$leave = ''){
    foreach($items as $id => $item){
        if($leave && ((!is_array($leave) && $id == $leave) || (is_array($leave) && in_array($id,$leave)))) continue;
        if($id != 'GLOBALS'){
            if(is_object($item) && ((get_class($item) == 'simple_html_dom') || (get_class($item) == 'simple_html_dom_node'))){
                $items[$id]->clear();
                unset($items[$id]);
            }else if(is_array($item)){
                $first = array_shift($item);
                if(is_object($first) && ((get_class($first) == 'simple_html_dom') || (get_class($first) == 'simple_html_dom_node'))){
                    unset($items[$id]);
                }
                unset($first);
            }
        }
    }
}

Usage:

Clean ALL traces of PHP Simple HTML Dom Parser from memory: clean_all($GLOBALS);

Clean all traces of PHP Simple HTML Dom Parser from memory, except $myobj: clean_all($GLOBALS,'myobj');

Clean all traces of PHP Simple HTML Dom Parser from memory, except list of objects ($myobj1,$myobj2...): clean_all($GLOBALS,array('myobj1','myobj2'));

Hope it will help others too.

Generally I use it when I use str_to_html() two times like:

$site=file_get_contents('http://google.com');
$site_html=str_get_html($site);
foreach($site->find('a') as $a){
   $site2=file_get_contents($a->href);
   $site2_html=str_get_html($site2);
   echo $site2->find('p',0)->plaintext;
}
clean_all($_GLOBALS);

In this example I can't $site_html->clear() before foreach{}, because foreach then will fail. And because calling multiple str_get_html() without clearing previous ones, the redundant dependencies are being broken and clearing it after all leaves memory leaks. That's why my function has to search the defined variables for simple_html_dom objects and clear them manually.

In my case I forked inside foreach and after few steps main php script used like 100MB of memory. And when forked few times, it have been increasing and increasing and finally killing my server to death. Well almost. Of course when PHP script ends, it does free up memory. But when using 8GB of memory, it took like ages to end.

edited Feb 18, 2018 at 13:20

answered Aug 7, 2013 at 5:29

Flash Thunder

12.1k10 gold badges55 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Enissay Over a year ago

So, we must call this function everytime we finish using an object $x = $main_html->find(...); ? can you give a call example please with $x since it seems there's an error at the 1st line in the Usage part ?

Enissay Over a year ago

Clearing all at the end of the script wont help me that much... Isn't it better to clear $site2 and $site2_html at the end of each loop ?

Flash Thunder Over a year ago

First use of $site_html2 will still leave memory leaks because it will cross redundant dependencies of $site_html. You can't fully clear $site_html after evrything is done. No matter if you clear $site2_html every loop or not. Please note it affects only PHP >= 5.3 as I remember.

Enissay Over a year ago

I see now, thanks <3... However, you must correct the first line in Usage section to clean_all($_GLOBALS);

Flash Thunder Over a year ago

@Enissay ah... after few years I finally understand what you meant :D fixed :D

|

ata · Accepted Answer · 2019-09-27 16:02:18Z

3

I believe you need to call clear() on $main_html

From the docs...

Q: This script is leaking memory seriously... After it finished running, it's not cleaning up dom object properly from memory..

A: Due to PHP5 circular references memory leak, after creating DOM object, you must call $dom->clear() to free memory if call file_get_dom() more than once.

Example:

$html = file_get_html(...); 
// do something... 
$html->clear(); 
unset($html);

edited Sep 27, 2019 at 16:02

ata

3,6935 gold badges22 silver badges32 bronze badges

answered Aug 6, 2013 at 20:52

Paul Dessert

6,3879 gold badges50 silver badges74 bronze badges

1 Comment

Flash Thunder Over a year ago

Does not help at all, still the same... why can't I clear $y? even when I clear $main_html, $y is still available.

Collectives™ on Stack Overflow

PHP Simple HTML Dom Parser Memory Leak / Usage

2 Answers 2

6 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related