2

Got this cron, which run some scrapy code

* * * * * cd /var/projects/project-1/scrappers && python3 runner.py

It return this error on the logs

Continue with error: [Errno 2] No such file or directory: 'scrapy'

Scrapy is correctly installed and working fine as if I run the same command from the cli, with the same user, it works.

I being looking for a solution to this triying few aproaches from the last days and can't understand why is not working, any idea?

Debian version:

Distributor ID: Debian
Description:    Debian GNU/Linux 12 (bookworm)
Release:    12
Codename:   bookworm

Update 1

More details related to the commens:

Cron is running as the same user as the one I use on the CLI

This server was configured using cloud-config and scrapy installation looks like this

 - wget https://bootstrap.pypa.io/get-pip.py
 - rm /usr/lib/python3.11/EXTERNALLY-MANAGED
 - python3 get-pip.py
 - pip3 install scrapy

On the CLI I got this when doing type scrapy

type scrapy
scrapy is /usr/local/bin/scrapy

But if I add type scrapy as part of the cron it returns scrapy not found which make sense based on the error

Improve this question
4
  • At a guess you installed scrapy in a virtual environment or under conda for your user, and are running the cron job as root? Commented Mar 22 at 17:27
  • Is scrapy an executable file? If so: What is the output of type scrapy / which scrapy? Commented Mar 22 at 17:50
  • 2
    Do the answers in Where is cron's PATH set? help? According to the referenced question '/usr/local/bin' isn't in the default PATH for cron which would explain the issue. This is a comment, and not an answer, since I'm not sure what a suitable fix is. Commented Mar 22 at 18:51
  • 1
    Hey @ChesterGillon you where right! I add PATH to the crontab config file and now is working, as it explains on this answer: stackoverflow.com/a/14694543/4886775 If you want to make an answer I mark it as the solution Commented Mar 22 at 19:43

1 Answer 1

Reset to default
2

The question shows the absolute path of the executable to be run from cron is /usr/local/bin/scrapy

On a search found the existing question Where is cron's PATH set?, where the question notes that /usr/local/bin isn't in the default PATH used by cron. The accepted answer recommended adding PATH= into the cronjob.

Also, the Stack Overflow crontab PATH and USER answer says:

According to "man 5 crontab" you can set environment variables in your crontab, by writing them before your cron lines. Julien Palard

Therefore, suggest try editing crontab to include:

PATH=/usr/local/bin
Improve this answer
0

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.