23

The following PHP code does return me a runtime of about 3.5 seconds (measured multiple times and averaged):

$starttime = microtime(true);
exec('/usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');
$endtime = microtime(true);
$time_taken = $endtime-$starttime;

When i run the same command over a ssh terminal, the runtime is reduced to about 0.6 seconds (measured with the command line tool time).

The version of the imagemagick library is

Version: ImageMagick 6.7.0-10 2012-12-18 Q16 http://www.imagemagick.org
Copyright: Copyright (C) 1999-2011 ImageMagick Studio LLC
Features: OpenMP

What could be the reason for this time difference?

One answer to a similar question here on stackoverflow was that the overhead comes from the Webserver having to start a thread/shell. Could this be really the reason? I thought threads are leightweight and don't take long at all to start/terminate.

Prior to calling exec i set the number of threads used by imagemagick (because this was/is a bug in OpenMP?, Reference) to 1 with exec('env MAGICK_THREAD_LIMIT=1');. The runtime from PHP does not change much, no matter what value i set for MAGICK_THREAD_LIMIT. Anyway there does not seem to be a bug in OpenMP on in this version because the runtime of the command line execution is ok.

Any suggestions of how i could improve the runtime of the above command would be greatly appreciated.

Thank you very much for your help.

Improve this question
4
  • 3
    I don't know if you know this but your PHP version most likely has a ImageMagick extension: php.net/imagick Commented Jan 11, 2013 at 12:16
  • The reason i run this via command line is because of php.ini memory limitations. Is using the ImageMagick extension better considering the runtime? Commented Jan 11, 2013 at 12:19
  • @nkr PHP::GD is more commonplace than PHP::imagick as default web server configurations. Commented Jan 11, 2013 at 12:43
  • @MihaiStancu: I heard that GD provides bad image quality and since @Philipp said that he tries to use ImageMagick via exec, he should try using it with the PHP extension. But in general you are right, GD is available more often. Commented Jan 11, 2013 at 12:48

6 Answers 6

Reset to default
15

When you log in to a Unix machine, either at the keyboard, or over ssh, you create a new instance of a shell. The shell is usually something like /bin/sh or /bin/bash. The shell allows you to execute commands.

When you use exec(), it also creates a new instance of a shell. That instance executes the commands you sent to it, and then exits.

When you create a new instance of a shell command, it has it's own environment variables. So if you do this:

exec('env MAGICK_THREAD_LIMIT=1');
exec('/usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');

Then you create two shells, and the setting in the first shell never gets to the second shell. To get the environment variable into in the second shell, you need something like this:

exec('env MAGICK_THREAD_LIMIT=1; /usr/local/bin/convert 1.pdf -density 200 -quality 85% 1.jpg');

Now, if you think that the shell itself may be the problem, because it takes too long to make a shell, test it with something that you know takes almost no time:

$starttime = microtime(true);
exec('echo hi');
$endtime = microtime(true);
$time_taken = $endtime-$starttime;

At that point you know to try and find some way to make the shell instantiate faster.

Hope this helps!

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

the echo example executes in 0.005 seconds on my machine
14

I've been programming computers for over 56 years, but this is the first time I've encountered a bug like this. So I spent nearly a week trying to understand the 7X worse execution speed when executing a perl program from php via exec versus executing the perl program directly at the command line. As part of this effort, I also pored over all the times this issue was raised on the web. Here's what I found:

(1) This is a bug that was first reported in 2002 and has not been fixed in the subsequent 11 years.

(2) The bug is related to the way apache interacts with php, so both of those organizations pass the buck to the other.

(3) The bug is the same on exec, system, or any of the alternatives.

(4) The bug doesn't depend on whether the exec-ed program is perl, exe, or whatever.

(5) The bug is the same on UNIX and Windows.

(6) The bug has nothing to do with imagemagick or with images in general. I encountered the bug in a completely different setting.

(7) The bug has nothing to do with startup times for fork, shell, bash, whatever.

(8) The bug is not fixed by changing the owner of the apache service.

(9) I'm not sure, but I think it relates to vastly increased overhead in calling subroutines.

When I encountered this problem I had a perl program that would execute in 40 seconds but through exec took 304 seconds. My ultimate solution was to figure out how to optimize my program so that it would execute in 0.5 seconds directly or in 3.5 seconds through exec. So I never did solve the problem.

Improve this answer

2 Comments

At last, someone!!... It's so annoying when everyone keeps explaining this phenomenon with "having to start a shell" when the times are clearly proportional, not just a "0.5 sec plus the same", as it would be in the shell-starting-overhead case. Now, using file redirections I managed to gain some speed (like dir > something.txt and then reading the file from php) but it's still not good enough. Please if you find the REAL answer let me know!
@dkellner take a look at this answer: stackoverflow.com/a/48505455/1098534. It works for my case (running a git command from php), but I'm looking for another way other than sudo to get the process prioritized and thus executed faster.
6

@Philipp since you have SSH and since your server allows access to the exec() I will assume that you also have full root access to the machine.

Recommended for single file processing

Having root access to the machine means that you can change the /etc/php5/php.ini memory limit settings.

Even without direct access to /etc/php5/php.ini you could check if your server supports overriding php.ini directives by creating a new php.ini file in you projects directory.

Even if the overrides are not permitted you can change your memory settings from .htaccess if AllowOverride is All.

Yet another means of changing the memory limit is by setting it during the PHP runtime using ini_set('memory_limit', 256);.

Recommended for batch file processing

The only good thing about running the convert via exec() is if you don't plan on getting a result back from exec() and allowing it to run asynchronously:

exec('convert --your-convert-options > /dev/null 2>/dev/null &');

The above approach is usually helpful if you're trying to batch process many files, you don't want to wait for them to finish processing and don't need confirmation regarding each having been processed.

Performance notes

Using the code above to make exec run async for processing a single file will cost more processor time and more memory than using GD/Imagick within PHP. The time/memory will be used by a different process that does not affect the PHP process (making the visitors feel the site moving faster), but the memory consumption exists and when it comes to handling many connections that will count.

Improve this answer

Comments

5

This is not a PHP bug and has nothing to do with Apache/Nginx or any webserver.

I had the same issue lately and had a look at the PHP source code to check the exec() implementation.

Essentially, PHP's exec() calls Libc's popen() function.

The offender here is C's popen() which seems to be very slow. A quick google search on "c popen slow" will show you lots of questions such as yours.

I also found that someone implemented a function called popen_noshell() in C to overcome this performance problem:

https://blog.famzah.net/2009/11/20/a-much-faster-popen-and-system-implementation-for-linux/

Here is a screenshot showing the speed difference vs popen() and popen_noshell():

Speed difference

PHP's exec() uses the regular popen() - the one at the right of the screenshot above. The CPU used by the system when executing C's popen() is very high as you can see.

I see 2 solutions to this problem:

  1. Create a PHP extension that implements popen_noshell
  2. Request from the PHP team to create a new set of functions popen_noshell(), exec_noshell(), etc... which is unlikely to happen I guess...

Additional note:

While searching for this, I discovered the PHP function of the same name as the C's one: popen()

It is interesting because one can execute an external command asynchronously with: pclose(popen('your command', 'r'));

Which essentially has the same effect as exec('your command &');

Improve this answer

1 Comment

This seems unlikely to be the issue because the execution time with PHP's exec() grows proportionally with the execution time from a normal shell.
3

I have experienced this problem, a graphic processing command that when run via the command line would take about .025 seconds was taking about .3 seconds when called via exec() in PHP. Upon much research, it seems that most people believe this to be a problem with apache or PHP. I then tried running the command via a CGI script, bypassing PHP altogether, and i got the same result.

It seemed therefore the problem must be apache, so i installed lighttpd, and got the same result!

After some thought and experimentation i realized this must be a problem with processor priority. So if you want your commands to run with a similar speed as the command line it must be executed as follows.

exec('echo "password" | sudo -S nice -n -20 command')

PLEASE NOTE: I know there will be all sorts of security objections to this. I just wanted to focus simply in the answer that all you have to do is add nice before your command.

Improve this answer

Comments

2

When you call exec php does not create a thread, It creats a new child process. Creating a new process is big overhead.

However when you connect with ssh you are just passing a command to execute. You are not owner of that program so it executes as the user whom you connected with. For exec its the user who runs PHP.

Improve this answer

6 Comments

And when you type a command in the shell, the shell also creates a new child process. So that cannot be the reason in itself.
Running a program as apache and a regular user is not same. I have added some more information about why ssh technique performs faster.
I also figure it doesn't take 3 seconds to create a process regardless
@shiplu.mokadd.im: Actually it's exactly the same (apart from effective permissions, environment vars, etc etc). The extra text does not really add anything.
This is not the real reason. It's a constant slowdown, not just when the command is starting. Something that runs 1s from shell will run 7s from php, and if it's 2s in the shell it will be 14 thru php so it's not just a starting delay. It's rather like friction, keeps pulling back the whole time. I'd be glad to find out what's really going on.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.