1

I have a python webscraping program which needs to be scrapped continuously after the program is terminated. my technique is as follows

crontab -e (settings)

* * * * * /home/ahmed/Desktop/run.sh

run.sh

    TMP_FILE=/tmp/i_am_running
    [ -f $TMP_FILE ] && exit
    touch $TMP_FILE
    /usr/bin/python /home/ahmed/Desktop/python.py
    rm $TMP_FILE

The bash code must have some problem or may be my command in the crontab is wrong. the program is not running. Please guide

After Mark suggestions I modified the script like this

#!/bin/bash
PATH=$PATH:/bin:/usr/bin

date +'%H:%M:%S Started' >> /home/ahmed/Desktop/log.txt

TMP_FILE=/tmp/i_am_running
[ -f $TMP_FILE ] && exit
touch $TMP_FILE

date +'%H:%M:%S Starting Python' >> /home/ahmed/Desktop/log.txt
/usr/bin/python /home/ahmed/Desktop/python.py
rm $TMP_FILE

date +'%H:%M:%S Ended' >> /home/ahmed/Desktop/log.txt

The cron command i am using is * * * * * /home/ahmed/Desktop/run.sh

the log file which is created is this

15:21:01 Started
15:21:02 Starting Python
15:22:02 Started
15:23:01 Started
15:24:01 Started
15:24:30 Ended
15:25:01 Started
15:25:01 Starting Python
15:26:01 Started
15:27:18 Started
15:28:01 Started
15:29:01 Started
15:30:01 Started
15:31:01 Started
15:31:16 Ended
15:32:01 Started
15:32:01 Starting Python
15:33:01 Started
15:34:01 Started

It seems like the program is restarted before its ended. the log file should have starting program, started, ended, starting program, started, ended and so on.

Can someone guide me please?

Improve this question
4
  • What is not running? The crontab? The script itself? Provide more details. Commented Mar 24, 2014 at 9:26
  • Use logger inside run.sh; perhaps start it with #!/bin/bah -vx for debugging purposes. Commented Mar 24, 2014 at 9:36
  • Typo I meant #!/bin/bash -vx Commented Mar 24, 2014 at 9:44
  • If you want to keep the Python script running even if it is crashed then you could use something like supervisord or create an upstart job Commented Mar 24, 2014 at 10:35

1 Answer 1

Reset to default
2

Have you made your script executable?

chmod +x /home/ahmed/Desktop/run.sh

Put a proper shebang and PATH in your script so it starts like this:

 #!/bin/bash
 PATH=$PATH:/bin:/usr/bin

Try your script on its own from the command line

/home/ahmed/Desktop/run.sh

If that doesn't work, change the shebang line to add -xv at the end

#!/bin/bash -xv

Check to see if /tmp/i_am_running exists

Check your cron log

grep CRON /var/log/syslog

Consider changing your script so you can see when it started and/or if it actually ran your python:

#!/bin/bash
PATH=$PATH:/bin:/usr/bin

date +'%H:%M:%S Started' >> /home/ahmed/Desktop/log.txt

TMP_FILE=/tmp/i_am_running
[ -f $TMP_FILE ] && exit
touch $TMP_FILE

date +'%H:%M:%S Starting Python' >> /home/ahmed/Desktop/log.txt
/usr/bin/python /home/ahmed/Desktop/python.py
rm $TMP_FILE

date +'%H:%M:%S Ended' >> /home/ahmed/Desktop/log.txt

By the way, I am not sure how running once at 18:01 constitutes "continuous scraping"?

Improve this answer
Sign up to request clarification or add additional context in comments.

9 Comments

i will change the timing to * * * * * . the timing was just for testing purpose. let me follow the steps you mentioned. thanks
I changed the code as u mentioned. and when i type grep CRON /var/log/syslog in the command line, it says binary file matches
"cd /var/log" and see if there is a cron.log or other logfile that your cron has been configured to use.
and i think the cron is working too because in the python its suppose to save data in the database and its saving. to be honest, i have no idea how the bash program is running and accomplishing the task. can you help me finding out how can i see log files. i want to learn the method please
Get the script running correctly first by checking the log.txt on your Desktop and seeing if the time is in there. Once the script works, then introduce it to cron.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.