I would like to run a php script that collects data from an other site (something like a crawler) for the time the user has specified. Ex. If user choose to crawl for 5 minutes, the php will start and run for 5 minutes, even if the user leaves the page. He then comes back and can see the results.
Add a comment
|
2 Answers
ignore_user_abort — Set whether a client disconnect should abort script execution
http://php.net/manual/en/function.ignore-user-abort.php
There are more things to consider like script execution timeouts. A more 'enterprisey' way to do this is to offload the work to a job or message queue.
2 Comments
user1798707
Thanks for the answer. I've searched for job queues but I can't understand them. If 5 users want me to crawl a site, will the crawling start "simultaneously" or will they start one after the other?
Mike B
It depends how you have apache setup. Usually it will throttle the requests before it even gets to php.
I think you should look at crontab /cronjob (if you use a linux webserver).
You can then make a database of what should be crawled and for whom.
And if you do not have a login system, you should link the crawl result to a session or cookie from the users browser.
When the script and database is made, you can make the php script that collects data to run every minute with this command in SSH/Telnet:
* * * * * php /path/to/script.php
If you do not have access to the SSH/Telnet shell, most control panels has a crontab function.
1 Comment
user1798707
Hi. Thanks for the answer. I've thought exactly the same about the database with cookies and the cron jobs. But till now I'm not sure is it possible to call a cron job from inside an php. Do you know how this is done? I'm a bit confused with the tutorials I've read because most of them use cron jobs from command line.