Running a python script (which crawl scrapy spiders) from another python script

Question

I have a scrapy project with two spiders. Also I created test.py (in this scrapy project) to crawl spiders

code :

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())

process.crawl('nameofspider1', domain='domain')
process.crawl('nameofspider2', domain='domain')

process.start()

and it works fine.

The problem is when I want to run this script (test.py) from another python script, then I got

KeyError: 'Spider not found: nameofspider1'

The code of the second python script:

import os

os.system('python C:\Users\Adam\nameofproject\test.py')

Thanks for all replies.

Adam K. · Accepted Answer · 2016-03-16 14:45:38Z

1

The problem was with path. I had to add:

import os
os.chdir(r'project_path')

answered Mar 16, 2016 at 14:45

Adam K.

512 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ji-ruh · Accepted Answer · 2017-07-20 06:00:56Z

1

I do this to run my spider.

import os
import subprocess

os.chdir('projectname')
subprocess.call(
    ['scrapy', 'crawl', 'spidername']
)

answered Jul 20, 2017 at 6:00

ji-ruh

7011 gold badge7 silver badges24 bronze badges

Collectives™ on Stack Overflow

Running a python script (which crawl scrapy spiders) from another python script

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related