Selenium crontab script

Question

I have a simple Selenium Python script (I know this particular task can be done without Selenium, but it is just as an example)

from selenium import webdriver
driver = webdriver.Chrome("./lib/chromedriver")
options = webdriver.ChromeOptions()
options.add_argument("headless")
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + currency.upper())
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

when I run it simply in my terminal, as a user I see this output:

Exchange rate updated:  0.00724094

My crontab looks like this:

# m h  dom mon dow   command
SHELL=/bin/bash
PATH=/usr/local/bin/:/usr/bin:/usr/sbin

*/5 * * * *  DISPLAY=:0 python3 /path/to/my_script.py >  /path/to/logs.txt

I added DISPLAY=:0 per this answer

The log file is updated every 5 minutes, but is always empty, which leads me to believe that my Selenium script is not running correctly

EDIT: I made the following changes: to my_script:

options = webdriver.ChromeOptions()
options.add_argument("--no-sandbox")
options.add_argument("--headless")
options.add_argument("--disable-dev-shm-usage")
driver = webdriver.Chrome(PATH + "/lib/chromedriver", chrome_options=options)
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + currency.upper())
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

to the crontab:

SHELL=/bin/bash
PATH=/usr/local/bin/:/usr/bin:/usr/sbin
*/5 * * * *  DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt 2>&1

and now I have this error in the logs.txt file

Traceback (most recent call last):
  File "path/to/my_script.py", line 90, in <module>
    update_exchange_rate()
  File "path/to/my_script.py", line 69, in update_exchange_rate
    body = {"values": [[get_exchange_rate(currency=currency)]]}
  File "path/to/my_script.py", line 82, in get_exchange_rate
    driver = webdriver.Chrome("/path/to/lib/chromedriver", options=options)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Setting an environment variable in crontab is an awkward approach - add full path to python3 like /usr/bin/python3. — ipaleka
– ipaleka, Commented Aug 14, 2019 at 15:31
@ipaleka I tried */5 * * * * DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt but it still doesn't work :/ — Jessica Chambers
– Jessica Chambers, Commented Aug 14, 2019 at 15:51
In script, options.add_argument("--headless") and another line options.add_argument("--disable-extensions"). — ipaleka
– ipaleka, Commented Aug 15, 2019 at 20:30

ipaleka · Accepted Answer · 2019-08-16 09:33:26Z

1

I changed that currency.upper() to "USD" for testing purposes. Also changed your code as options should be passed to the driver during initialization.

from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless")
driver = webdriver.Chrome("/path/to/chromedriver", options=options)
driver.get("https://www.xe.com/currencyconverter/convert/?Amount=1&From=ISK&To=" + "USD")
exchange_rate = driver.find_element_by_xpath('//*[@class="converterresult-toAmount"]').text
driver.close()
print("Exchange rate updated: ", exchange_rate)

crontab:

*/5 * * * *  DISPLAY=:0 /usr/bin/python3 /path/to/my_script.py > /path/to/logs.txt 2>&1

The output:

$ cat /path/to/logs.txt
Exchange rate updated:  0.00806602

edited Aug 16, 2019 at 9:33

answered Aug 15, 2019 at 21:05

ipaleka

3,9772 gold badges16 silver badges39 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jessica Chambers Over a year ago

Somehow it still doesn't work - I've tried on two different computers too, with no success

ipaleka Over a year ago

From where you import that currency? Use the changed cron line (added " 2>&1") so any error will be saved to log file. Post the error here.

Jessica Chambers Over a year ago

the currency is stored in a global variable (and it works when I run it outside of cron) CURRENCIES = ["eur", "usd", "gbp"] -> currency = CURRENCIES[0]

Jessica Chambers Over a year ago

Tldr; I think something I'm doing wrong is causing an error here: chrome/webdriver.py", line 81, in __init__ desired_capabilities=desired_capabilities)

ipaleka Over a year ago

Dunno, really, example code provided in this answer works flawlessly on my computer.

Pritish · Accepted Answer · 2020-07-01 06:30:03Z

0

For me after adding this options.binary_location = "/opt/google/chrome/chrome" as part of options object it worked

options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument('--headless')
options.add_argument("--no-sandbox")
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--disable-setuid-sandbox")
options.add_argument('--remote-debugging-port=9222')
options.binary_location = "/opt/google/chrome/chrome"

answered Jul 1, 2020 at 6:30

Pritish

1,3902 gold badges22 silver badges47 bronze badges

Collectives™ on Stack Overflow

Selenium crontab script

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related