PHP How can a run a .sh script that calls a python program from a php webpage

Question

I have a unix .sh script below which runs a python program, that can take 10 mins to complete some data processing, it works fine. I would like to run this script from a php web page.

run_virt_da_py.sh

#! /bin/bash

# activate virtual environment
cd /Users/data/myproject
source bin/activate

# virtualenv is now active 

python iterate_dir_analyse_files_1.py

echo "data analysis completed ok"

The script above runs "perfectly" when using the command line to run it manually using the following commands from terminal on an apple mac running osx 10.6. Below shows the prompt followed by the commands entered.

[~]% cd myproject
[myproject]% source bin/activate
(myproject)[myproject]% sh run_virt_da_py.sh

However when I try and run the .sh script from php it fails the script below returns the first echo statement of the python program and nothing else.

<html lang="en">
    <head>
        <title>Data_analysis</title>
    </head>
    <body>
    <?php 
     echo " data analysis script started ";
     echo shell_exec('sh   /Users/Data/myproject/run_virt_da_py.sh');
     echo "data analysis script may have completed";
     ?>
    </body>
</html>

Research I have tried using, system, passthru, exec, shell_exec on scripts that do not use virtualenv and don't have a problem.

Any help, solution, method or pointers welcome. Thanks in advance.

Maybe problem is in user, under which the script is run. When You run it from command-line, under what user do You run it? When You run it from PHP, it will probably be running under Your httpd user, which may not have sufficient rights? Try "whoami" in Your shell script to see, what user You actually run and try running the script under the same user from command-line to see what happens. — Roman Hocke
– Roman Hocke, Commented Dec 15, 2013 at 17:28

Ignacio Ocampo · Accepted Answer · 2013-12-15 17:22:47Z

3

Some hostings like GoDaddy (on Shared Hosting) have restrictions in exec function whithin PHP.

Another point is that you need to configure your php.ini to increase your timeout max_execution_time = 1200, by default, php.ini have 60 seconds for timeout and your script will die.

I think that you are reaching your script, but you aren't waiting to finish. Try adding some code at top of your php to write a "Hello world" file to test it.

answered Dec 15, 2013 at 17:22

Ignacio Ocampo

2,7231 gold badge22 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Zarathuztra · Accepted Answer · 2013-12-15 17:25:14Z

2

First, make sure to modify php.ini (if possible) to allow the extended execution time. If your script takes too long your script may just terminate it.

Then, make sure you're actually giving the script enough time to get to shell_exec()

Third, make sure safemode is disabled.

Fourth, make sure your hosting provider allows shell_exec() or backticks.

answered Dec 15, 2013 at 17:25

Zarathuztra

3,2411 gold badge23 silver badges34 bronze badges

Comments

Iguananaut · Accepted Answer · 2013-12-15 21:32:01Z

2

The accepted answer is correct, but it should be pointed out that virtualenv comes with a script called activate_this.py that exists for the purpose of setting up a virtualenv-like envirnoment for use in a web server. See https://pypi.python.org/pypi/virtualenv and search the page for activate_this.py. Then you can just this without having to go about launching a subshell from the webserver to modify the environment to run the virtualenv's Python interpreter.

answered Dec 15, 2013 at 21:32

Iguananaut

23.8k6 gold badges54 silver badges65 bronze badges

Comments

Carlos Robles · Accepted Answer · 2013-12-15 18:50:49Z

Why you would want to have a web request during for 10 minutes? PHP is not meant for that at all. I've seen some backoffice process, for example, generating files to be downloaded, taking that long or even longer to complete, and users can wait for it, but for sure it wont be trustable, the process could die for many reasons (hardware limitations, config limitations, network failure, etc) and even more, in a shared the server will explicitely kill your process no matter what the configuration is.

In a dedicated or VPS, for sure you can modify directives and allow your script to run forever, but the question is not how to do it, but if that is the approach that you need. Probably it is, i dont say thats an absolute forbidden approach, but probably it is not. Alternatives could be:

Have a cronjob running your sh on a regular basis. When there is a web request, you can serve precalculated data.
If you dont want to have it generated too often, you can have it running often, but checking if it has to actually continue running or not, probably checking database or a file, and then when there is a request, from php you change that value of that database field or that file, so the cron knows that it has to run.
Have the php launching the sh, but in an asynchronous thread, so the php executiom ends inmediately, informs the user that the data is being processing, and the actual process is running in background

This is a really common approach, you can see it in many webs, for example in your server if you request some heavy stuff, they generate it and notify you when it is ready. Or in facebook when you want a backup of your data, the do the same. Also business inteligence systems, and those analyzing data, use to have background process than generate precalculated data so user processes can be lightweight.

At enterprise level application, what i use to do with this kind of heavy process is to have a link for acessing the last generated data (informing about its date), a process running frequently, and a link to force a new process to be launched right now.

With this, you get a more reliable system, and also, minimize the chances of have two users analyzing the same data in different computers at the same time, which means two process of 10 minutes to get the same data.

UPDATE

If you dont really need to let the user see the result of the process in real time, then its even easier. i have to say that a like crons since the help a lot to keep your process in order and under control when the system grows. But if you dont want it at this moment, you can try to launch a background process directly from php. Try this code (i used it many years ago, but i guess it will work). Is good for both unix and windows servers.

/**
 * execute $cmd in the background without PHP waiting for it to finish
*/
function execInBackground($cmd) { 
    if (substr(php_uname(), 0, 7) == "Windows"){ 
        pclose(popen("start /B ". $cmd, "r"));  
    } 
    else { 
        exec($cmd . " > /dev/null &");   
    } 
}

so, to start your command, you have to do:

<html lang="en">
<head>
    <title>Data_analysis</title>
</head>
<body>
<?php 
 echo " data analysis script started ";
 echo execInBackground('sh   /Users/Data/myproject/run_virt_da_py.sh');
 echo "data analysis script has been started";
 ?>
</body>
</html>

Of course, since it is running in a background thread, you dont directly know when the process is completed, so you have to include some adicional actions at the end of the sh script, such as launching another php with thenotification process you could need, such as sending an email, updating a database, or so.

Your point 3. is what I am trying to do. Users would not see this page. This script is part of larger the data analysis program. The python program outputs a large number of files that have to be passed to database. I am just looking for a way to initiate the script from php I can check if the process is completed elsewhere. Could you explain or point me in a direction of asynchronous threads, or how I can do what you said and have php just launch the shell in an asynchronous thread.
@GeorgeThompson Is there a reason you need an actual page to do this? PHP supports CLI mode if you install it/compile it as such, so you can use a cron job to execute your script directly instead of having to hit a webpage like that. I'm also not positive what your analysis script does, but doing it with python may not be the way to go. Have you thought about ETL or some other framework/language/platform that is meant for data analysis? Just my two cents.

Collectives™ on Stack Overflow

PHP How can a run a .sh script that calls a python program from a php webpage

4 Answers 4

Comments

Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related