Running spiders from a separate python script

Question

I was wondering what is the best way to run spiders from another python script. My scrapy project consist of 4 different spiders, all of them create files that help the other spiders work and some of them have to read some files to work. That part is already done but individually (running the spiders separate from the console).

How can I, for example, do something like this

if (productToSearchIsBlue):

    #Make one spider crawl

else:

    #Make another spider crawl

My final plan is to upload the full program to the cloud and make it run automatically, can this be done?

I found some answers to this question but they were pretty old, probably for another version of scrapy.

malberts · Accepted Answer · 2019-02-06 09:53:05Z

2

Assuming you have everything else set up correctly, here is a trivial example as per the documentation.

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings


process = CrawlerProcess(get_project_settings())

productToSearchIsBlue = False

if productToSearchIsBlue:
    # Make one spider crawl
    process.crawl('spider-one')
else:
    # Make another spider crawl
    process.crawl('spider-two')

process.start()

You can then run this on a cloud server. But I cannot answer whether this is the optimal solution for the problem you are trying to solve.

edited Feb 6, 2019 at 9:53

answered Feb 6, 2019 at 9:02

malberts

2,5361 gold badge14 silver badges18 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running spiders from a separate python script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related