how to fix or optimize code to ignore memory size error in PHP

Question

I wrote a image crawler with simple_html_dom library, I used this code to fetch all images in a web site ;

include 'simple_html_dom.php';
$img_array = array();
if (isset($_POST['url'])) {
    $url = $_POST['url'];
    $html = file_get_html($url);
    echo $html->getElementByTagName('title')->innertext();
    foreach ($html->find('a') as $a) {

        if (strpos($a->href, $url) !== FALSE) // only images from this site
        {
            //
            //    find_images($a->href);
            $imgs = file_get_html($a->href);
            foreach ($imgs->find('img') as $img) {
                if(!in_array($img->src, $img_array))
                {
                    echo '<img src="' .$img->src. '" class="thumb">';
                    $img_array[] = $img->src;
                }
            }
            echo '<hr>';
        }
    }
}

but whene i execute this code i get Fatal error: Allowed memory size of 209715200 bytes exhausted (tried to allocate 71 bytes) in /home/iphotosh/public_html/test/simple_html_dom.php on line 122

test and demo : test.iphotoshop.ir

how to fix this error or how to opti,ize this code for fetch all images from a web site?

i used ini_set('memory_limit', '500M'); but when images in web site to many again get this error — Saeed M.
– Saeed M., Commented Jun 21, 2012 at 12:56
Increasing the memory is prob the right thing to do BUT if you can't for w/e reason (permission, shared hosting etc) you could write the items to a file or database instead of stashing them all in memory. — EmmanuelG
– EmmanuelG, Commented Jun 21, 2012 at 12:57

Sliq · Accepted Answer · 2012-06-21 13:22:12Z

1

You should do two things at the same time: Set your memory limit very high:

in php.ini:

memory_limit = 512M

or/and in php file:

ini_set("memory_limit","512M");

At the same time you should delete big variables to free up some memory, usually via:

unset($var);

By the way you can check the amount of used memory via

echo memory_get_usage();

I would try a demo run and check for memory usage in EACH line of your code, so you can see what happening here, and where to start your fixing

answered Jun 21, 2012 at 13:22

Sliq

16.6k28 gold badges118 silver badges146 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

amburnside · Accepted Answer · 2012-06-21 12:56:16Z

1

Have you tried increasing the memory using ini_set():

ini_set("memory_limit","256M");

answered Jun 21, 2012 at 12:56

amburnside

1,9035 gold badges24 silver badges45 bronze badges

Comments

Aleks G · Accepted Answer · 2012-06-21 13:01:40Z

Seems like you're trying to allocate way too much memory. You could try increasing available memory in your php.ini (look for memory_limit= directive). Yet, you could still exceed it if you are allocating A LOT. You can check dynamically what's available and how much is used:

function get_available_memory() {
    $ini_mem = ini_get('memory_limit');
    $m = substr($ini_mem, strlen($ini_mem) - 1;
    if($m == 'k' || $m == 'K') {
        $max_mem = 1024 * substr($ini_mem, 0, strlen($ini_mem) - 1);
    }
    elseif($m == 'm' || $m == 'M') {
        $max_mem = 1024 * 1024 * substr($ini_mem, 0, strlen($ini_mem) - 1);
    }
    elseif($m == 'g' || $m == 'M') {
        $max_mem = 1024 * 1024 * 1024 * substr($ini_mem, 0, strlen($ini_mem) - 1);
    }
    else {
        $max_mem = $ini_mem;
    }

    $used_mem = memory_get_usage(true);

    return $max_mem - $used_mem;
}

Now you can do

$available_memory = get_available_memory();

and, if not enough is available, do not try to allocate it and gracefully close your process.

Collectives™ on Stack Overflow

how to fix or optimize code to ignore memory size error in PHP

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related