Running a Scrapy spider from another python script

Question

I'm trying to run a scrapy spider from a simple PyQt4 GUI I built, the user has to fill his email and password to run the spider, the spider works well if I call it from the command prompt like so

scrapy crawl my_spider -a email -a password

So once the user has filled his email and password and I saved them in my script, and after reading the documentation and some examples I found of google I still can't figure out how to run it

    self.BumpPushButton.clicked.connect(self.BumpListings)

    def BumpListings(self):
        email = self.emailTextEdit.toPlainText()
        password = self.passwordTextEdit.toPlainText()
        bumpCycleInMinutes = self.MinutesTextEdit.toPlainText()

is there a simple way to call the spider at this point?

costrouc · Accepted Answer · 2018-01-20 15:19:59Z

1

There are two approaches to this problem. You could call the spider directly by importing the spider or you could use python subprocesses. I would recommend python subprocesses because you don't want to block your pyqt process.

 import subprocess
 process = subprocess.Popen(['scrapy', 'crawl', 'my_spider', 
                                       '-a', email, '-a', password]
                             shell=True, 
                             stdout=subprocess.PIPE, 
                             stderr=subprocess.PIPE)

You can then check on the process using wait, communicate, poll etc. See docs to see what actions you might want to perform.

answered Jan 20, 2018 at 15:19

costrouc

3,1753 gold badges21 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Running a Scrapy spider from another python script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related