8

I have a few time-consuming and (potentially) memory-intensive functions in my LAMP web application. Most of these functions will be executed every minute via cron (in some cases, the cron job will execute multiple instances of these functions).

Since memory is finite, I don't want to run into issues where I am trying to execute a function the environment can no longer handle. What is a good approach at dealing with potential memory problems?

I'm guessing that I need to determine how much memory is available to me, how much memory each function requires before executing it, determine what other functions are being executed by the cron AND their memory usage, etc.

Also, I don't want to run into the issue where a certain function somehow gets execution priority over other functions. If any priority is given, I'd like to have control over that somehow.

Improve this question
2
  • 1
    More importantly, why are you doing a Cronjob every minute? Part of he problem may be the cron is simpyl trying to do too much at once. If you could explain the problem and your proposed solution we'd be able to give better made answers to your specific memory problem. There are different approaches for different problems. Is it a lot of data from a database? A flat file? Simply a ton of mathematical calculations? Each one would have their own sets of optimization steps. Please flesh out what your issue is so we can give a proper response :). Commented Jul 26, 2010 at 18:54
  • I think you need to be more descriptive and state what you're doing in this cron job. Obvious savings would be not loading huge files into memory for processing, use the streaming functions, same with pulling a large DB result set into memory, use a cursor to operate on the record set instead of pulling all the data into an array etc. Commented Jul 28, 2010 at 10:06

5 Answers 5

Reset to default
1

you could look into caching technologies like APC which lets you write stuff right into the RAM so that you can access it fast which of use if you dont want to do expensive tasks like mysql queries repeatedly.

an example for caching i could think of would be that you could cache emails rather than retreiving them again and again from the email server. basicaly ram caching is a very useful technique if you have things in your script that you want to preserve for the next time of script execution but if your script does unique things every time it is executed it would be useless. also as for contoll you could call memory_get_usage() on each script execution and write that value into the apc cache so that every cron could retreive that value and look whether enough memory is free for it to complete.

as for average usage you could write an array with the last lets say 100 function executions and when you call that function again it could apc_fetch that from the ram and calculate the average memory usage for that function then compare it to how much ram is being used right now and then decide wheter to start. furthermore it could write that estimate into the current memory usage variable to prevent other scripts from being run. at the end of that function you subtract that amount from the variable again. tl;dr: look into the apc_fetch, apc_store and memory_get_usage functions

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

The memory taxing tasks are not related to MySQL. Most of has something to do with image processing, PDF processing, downloading and uploading files from FTP servers, sending and retrieving emails, etc.
0

Part of your problem may be the fact you are doing a cron every minute? Why not set some flags so only one instance of that cron is running before another executes the full logic? i.e. create a flat file thats deleted at the end of the cron to act as a 'lock'. This will make sure one cron process fully completes before any others go forward. However, I urge you to refer to my comment on your post so that I and others can give you more solid advice.

Improve this answer

1 Comment

Asssume there is a legitimate reason for me to run cron job every minute.
0

Try optimizing your algorithms. Like...

  • Once you're finished with a variable you should destroy it if you no longer need it.
  • Close MySQL connections after you've finished with them.
  • Use recursion.

Also as Jauzsika said change your memory limit in your php.ini, although don't make it too high. If you need more than 256MB RAM then I would suggest changing to a different language instead of PHP.

Improve this answer

2 Comments

I don't think you can specify an arbitrary limit of 256MBs given he hasn't described what his code is doing.
All variables are destroyed, all DB connections are closed, recursion is not an option. 256mb is arbitrary. There is no reason to switch languages. I just need a way to make sure that I don't inadvertently use too much memory.
0

In your position, I'd consider writing a daemon instead of relying on cron. The daemon could monitor a queue and be aware of the number of child processes it has running. Managing multiple processes definitely isn't php's biggest strength, but you can do it. Pear even includes a System_Daemon package.

Your daemon could use memory_get_usage and call out out free, uptime, and friends to throttle the number of workers to match system conditions.

I don't have any direct experience with this, and I wouldn't be too too surprised if a daemon written in PHP gradually leaked memory. But if it's acceptably slow, cron could cycle the daemon every so often...

Improve this answer

1 Comment

...I guess the child processes would probably have to call memory_get_usage and communicate the result to the parent... or maybe the parent could call out to ps or top...
0

You can find out how much memory is currently in use by your script using memory_get_usage But you can not determine how much your next function will need, before executing it. You can only see after execuiting, using memory_get_usage. You can however store the memory your function used the last times in a database and calculate with the average memory amount.

Regarding the eecution priority, I don't think it is posible to determine with PHP. Apache (or whatever webserver you are using) spawns multiple processes and the operating system schedules which one will be executed in which order.

Improve this answer

1 Comment

memory_limit defines how much memory each php process can eat. It's in your php.ini file. By default it's 8 or 16 MB. You can change that anytime to a bigger value, until you have enough system memory (+swap).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.