php memory overflow multi curl

Question

I have a PHP routine that I call shell_exec to traverse my own website and build an index.

However since I do this multi-threaded I have been running into problems with memory storage...server memory spikes to 250MB then back down to 100MB randomly.

I am constantly accessing the mysql database during this time; and the php slows down.

Is this possible using php or should I use jsp/java.

You are not telling us what exactly you are doing in the shell_exec call, so this is impossible to answer really. What tool are you using to do this? Is the memory usage giving you problems? — Pekka
– Pekka, Commented Oct 23, 2010 at 21:29
I use the rolling_curl library and my code is php-cli /home/ex.php shell_exec I pull via curl_multi and get the collection of data then parse it in my callback function to get the next set of urls to pull via php curl multi - I can't use a cronjob to do this unfortunately. — user477512
– user477512, Commented Oct 23, 2010 at 21:38

rook · Accepted Answer · 2010-11-02 07:49:21Z

1

PHP uses a reference counting for memory management. As long as there is a variable declared (could be an array or global) that can be used to access that variable then it is never freed.

The solution is to call unset() on anything that has already been used. To make it use much less memory the parts of the index that have been created should be stored in a sql database. This allows much more data can be unset.

In short this can be done in PHP or JSP and use much less memory by using a database.

answered Nov 2, 2010 at 7:49

rook

67.1k38 gold badges171 silver badges248 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

php memory overflow multi curl

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related