Scraping much data from different URLs with simple_html_dom.php

Question

i basically want to do exactly something like this: Simple Html DOM Caching

i got everything to work so far, but now i'm getting the following error because i scrape many sites (6 at the moment, i want up to 25 sites):

Fatal error: Allowed memory size of 33554432 bytes exhausted (tried to allocate 39 bytes)

i'm a php newbie =/...so, how can i "serialize" the scraping process step by step that my memory don't give up? :-)

code sample:

// Include the library
include('simple_html_dom.php');

// retrieve and find contents
$html0 = file_get_html('http://www.site.com/');
foreach($html0->find('#id') as $aktuelle_spiele);

file_put_contents("cache/cache0.html",$aktuelle_spiele);

thank you very much in advance for your help!

Up teh memory limit....it seems you have only 32 mb of memory limit — Hackerman
– Hackerman, Commented May 17, 2013 at 15:59

Hackerman · Accepted Answer · 2013-05-17 17:00:21Z

3

In your php.ini, change this line:

memory_limit = 32M

With this one:

memory_limit = 256M //or another greater value

Or add this piece of code at the start of every php script that use simple_html_dom:

ini_set('memory_limit', '128M'); //or a greater value

edited May 17, 2013 at 17:00

answered May 17, 2013 at 16:00

Hackerman

12.3k2 gold badges37 silver badges47 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Luigi Siri Over a year ago

@ThomasVeit, you can run a memory increase at the start of your script. Like this: ini_set('memory_limit', '128M')

Thomas Veit Over a year ago

mhhh, my hoster does not allow more than 32M memory... someone has another idea to reduce my script memory?

Hackerman Over a year ago

I thought you have your own server...you can add the line proposed by @Luigi and the start of every script that use simple html dom php

Luigi Siri Over a year ago

@ThomasVeit. I have pasted my comment as an answer. Sorry Robert. +1

Luigi Siri · Accepted Answer · 2013-05-17 18:01:54Z

0

you can run a memory increase at the start of your script.

Like this:

ini_set('memory_limit', '128M');

answered May 17, 2013 at 18:01

Luigi Siri

2,0182 gold badges19 silver badges29 bronze badges

Collectives™ on Stack Overflow

Scraping much data from different URLs with simple_html_dom.php

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related