Running python script with cron only if not running

Question

I need to run a python script (job.py) every minute. This script must not be started if it is already running. Its execution time can be between 10 seconds and several hours.

So I put into my crontab:

* * * * * root cd /home/lorenzo/cron && python -u job.py 1>> /var/log/job/log 2>> /var/log/job/err

To avoid starting the script when it is already running, I use flock().

This is the script (job.py):

import fcntl
import time
import sys

def doIncrediblyImportantThings ():
    for i in range (100):
        sys.stdout.write ('[%s] %d.\n' % (time.strftime ('%c'), i) )
        time.sleep (1)

if __name__ == '__main__':
    f = open ('lock', 'w')
    try: fcntl.lockf (f, fcntl.LOCK_EX | fcntl.LOCK_NB)
    except:
        sys.stderr.write ('[%s] Script already running.\n' % time.strftime ('%c') )
        sys.exit (-1)
    doIncrediblyImportantThings ()

This approach seems to work.

Is there anything I am missing? Are there any troubles I can run into using this approach?

Are there more advised or "proper" ways of achieving this behaviour?

I thank you for any suggestion.

Jean-Paul Calderone · Accepted Answer · 2011-05-27 01:28:19Z

15

The only suggestion I would make is to make your exception handling a little more specific. You don't want to accidentally delete the fcntl import one day and hide the NameError that results. Always try to catch the most specific exception you want to handle. In this case, I suggest something like:

import errno

try:
    fcntl.lock(...)
except IOError, e:
    if e.errno == errno.EAGAIN:
        sys.stderr.write(...)
        sys.exit(-1)
    raise

This way, any other cause of the lock being unobtainable shows up (probably in your email since you're using cron) and you can decide if it's something for an administrator to look at, another case for the program to handle, or something else.

answered May 27, 2011 at 1:28

Jean-Paul Calderone

48.5k6 gold badges101 silver badges126 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

R.M. Over a year ago

For those using Python 3: Since Python 3.3, fcntl uses OSError instead of IOError

Mel · Accepted Answer · 2011-05-27 00:30:12Z

3

You're in trouble when the machine reboots or freezes with the script running (and thus an active lock). Simple way to counter this is to use the @reboot cron timestamp to run rm /path/to/lock.

answered May 27, 2011 at 0:30

Mel

6,1971 gold badge18 silver badges12 bronze badges

3 Comments

Hyperboreus Over a year ago

Thank you very much. I will take this into consideration. Are file locks persistent through reboots? Depends this on the file system I am using (actually ext4)?

Jean-Paul Calderone Over a year ago

File locks are not persistent across reboots. They aren't even persistent across processes restarting, which is why you don't have to release the lock in your code - it is released when the process terminates.

Hyperboreus Over a year ago

@Jean-Paul This means I do not have worry about reboots and freezes as Mel states?

Nate Ferrero · Accepted Answer · 2015-01-28 06:48:04Z

I ran into this exact problem last week, and although I did find some good solutions, I decided to make a very simple and clean python package and uploaded it to PyPI.

Install with: pip install quicklock

Using it is extremely simple:

[nate@Nates-MacBook-Pro-3 ~/live] python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from quicklock import singleton
>>> # Let's create a lock so that only one instance of a script will run
...
>>> singleton('hello world')
>>>
>>> # Let's try to do that again, this should fail
...
>>> singleton('hello world')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/nate/live/gallery/env/lib/python2.7/site-packages/quicklock/quicklock.py", line 47, in singleton
    raise RuntimeError('Resource <{}> is currently locked by <Process {}: "{}">'.format(resource, other_process.pid, other_process.name()))
RuntimeError: Resource <hello world> is currently locked by <Process 24801: "python">
>>>
>>> # But if we quit this process, we release the lock automatically
...
>>> ^D
[nate@Nates-MacBook-Pro-3 ~/live] python
Python 2.7.6 (default, Sep  9 2014, 15:04:36)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> from quicklock import singleton
>>> singleton('hello world')
>>>
>>> # No exception was thrown, we own 'hello world'!

Take a look: https://pypi.python.org/pypi/quicklock

Collectives™ on Stack Overflow

Running python script with cron only if not running

3 Answers 3

1 Comment

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related