Execute Python scripts with Selenium via Crontab

Question

I have several python scripts that use selenium webdriver on a Debian server. If I run them manually from the terminal (usually as root) everything is ok but every time I tried to run them via crontab I have an exception like this:

WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.

Try for example this script:

from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from pyvirtualdisplay import Display
from selenium import webdriver
import datetime
import logging

FIREFOX_PATH = '/usr/bin/firefox'

if __name__ == '__main__':
    cur_date = datetime.datetime.now().strftime('%Y-%m-%d')
    logging.basicConfig(filename="./logs/download_{0}.log".format(cur_date),
                        filemode='w',
                        level=logging.DEBUG,
                        format='%(asctime)s - %(levelname)s - %(message)s')
    try:
        display = Display(visible=0, size=(800, 600))
        display.start()
        print 'start'
        logging.info('start')
        binary = FirefoxBinary(FIREFOX_PATH,
                               log_file='/home/egor/dev/test/logs/firefox_binary_log.log')
        driver = webdriver.Firefox()
        driver.get("http://google.com")
        logging.info('title: ' + driver.title)
        driver.quit()
        display.stop()
    except:
        logging.exception('')
    logging.info('finish')
    print 'finish'

The crontab command for it:

0 13 * * * cd "/home/egor/dev/test" && python test.py

The log file for this script looks like this:

2016-09-27 16:30:01,742 - DEBUG - param: "['Xvfb', '-help']" 
2016-09-27 16:30:01,743 - DEBUG - command: ['Xvfb', '-help']
2016-09-27 16:30:01,743 - DEBUG - joined command: Xvfb -help
2016-09-27 16:30:01,745 - DEBUG - process was started (pid=23042)
2016-09-27 16:30:01,747 - DEBUG - process has ended
2016-09-27 16:30:01,748 - DEBUG - return code=0
2016-09-27 16:30:01,748 - DEBUG - stdout=
2016-09-27 16:30:01,751 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']" 
2016-09-27 16:30:01,751 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1724']
2016-09-27 16:30:01,751 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1724
2016-09-27 16:30:01,753 - DEBUG - param: "['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']" 
2016-09-27 16:30:01,753 - DEBUG - command: ['Xvfb', '-br', '-nolisten', 'tcp', '-screen', '0', '800x600x24', ':1725']
2016-09-27 16:30:01,753 - DEBUG - joined command: Xvfb -br -nolisten tcp -screen 0 800x600x24 :1725
2016-09-27 16:30:01,755 - DEBUG - process was started (pid=23043)
2016-09-27 16:30:01,755 - DEBUG - DISPLAY=:1725
2016-09-27 16:30:01,855 - INFO - start
2016-09-27 16:30:31,965 - ERROR - 
Traceback (most recent call last):
  File "test.py", line 31, in <module>
    driver = webdriver.Firefox()
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/webdriver.py", line 103, in __init__
    self.binary, timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 51, in __init__
    self.binary.launch_browser(self.profile, timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
    self._wait_until_connectable(timeout=timeout)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 106, in _wait_until_connectable
    % (self.profile.path))
WebDriverException: Message: Can't load the profile. Profile Dir: /tmp/tmpQ4vStP If you specified a log_file in the FirefoxBinary constructor, check it for details.

2016-09-27 16:30:31,966 - INFO - finish

What I have tried:

Ensure that the script file is owned by root
Use export DISPLAY=:0; or export DISPLAY=:99; in crontab command
set the HOME variable in the crontab to the path of the home directory of the user that the cronjob was being run as

I'm really stuck with this problem.

I have python 2.7.10, selenium 2.53.6 with Xvbf and Firefox 47.0.1 on Debian 7.7

just_flare · Accepted Answer · 2016-10-06 22:16:56Z

1

The problem concerns environment variables: cron is started by the system and knows nothing about user environments.

So, the solution to a problem is to make cron run a shell script that sets the needed environment variables first and then runs the script. In my case I needed to set the PATH varibable like this: PATH=/root/anaconda3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin Also, it may be useful to set HOME or DISPLAY varibles in some cases.

answered Oct 6, 2016 at 22:16

just_flare

316 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Maximilian Peters · Accepted Answer · 2016-09-27 18:25:04Z

0

Try a hardcoded Firefox binary https://seleniumhq.github.io/selenium/docs/api/py/webdriver_firefox/selenium.webdriver.firefox.firefox_binary.html

selenium.webdriver.firefox.firefox_binary.FirefoxBinary("/your/binary/location/firefox")
driver = webdriver.Firefox(firefox_binary=binary)

answered Sep 27, 2016 at 18:25

Maximilian Peters

31.8k12 gold badges95 silver badges102 bronze badges

Collectives™ on Stack Overflow

Execute Python scripts with Selenium via Crontab

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related