Calling Scrapy Spider from python script?

Question

I have created a spider with name aqaq it is in the file name image.py. the contents of image.py is as follows:

from scrapy.spider import BaseSpider
from scrapy.selector import HtmlXPathSelector
from scrapy.http import Request
a=[]
from aqaq.items import aqaq
import os
class aqaqspider(BaseSpider):
    name = "aqaq"
    allowed_domains = ["aqaq.com"]
    start_urls = [
                        "http://www.aqaq.com/list/female/view-all?limit=all"
    ]

    def parse(self, response):
        hxs = HtmlXPathSelector(response)
        sites=hxs.select('//ul[@class="list"]/li')
        for site in sites:
                name=site.select('a[@class="product-name"]/@href').extract()
                a.append(name)
        f=open("url","w+")
        for i in a:
                if str(i)=='[]':
                        pass;
                else:
                        f.write(str(i)[3:-2]+os.linesep)
                        yield Request(str(i)[3:-2].rstrip('\n'),callback=self.parsed)

        f.close()
    def parsed(self,response):
        hxs = HtmlXPathSelector(response)
        sites=hxs.select('//div[@class="form"]')
        items=[]
        for site in sites:
                item=aqaq()
                item['title']=site.select('h1/text()').extract()
                item['cost']=site.select('div[@class="price-container"]/span[@class="regular-price"]/span[@class="price"]/text()').extract()
                item['desc']=site.select('div[@class="row-block"]/p/text()').extract()
                item['color']=site.select('div[@id="colours"]/ul/li/a/img/@src').extract()
                items.append(item)
                return items

i am trying to run this spider with my python script which is as follows:

from twisted.internet import reactor
from scrapy.crawler import Crawler
from scrapy.settings import Settings
from scrapy import log, signals
from spiders.image import aqaqspider
from scrapy.xlib.pydispatch import dispatcher
def stop_reactor():
    reactor.stop()
dispatcher.connect(stop_reactor, signal=signals.spider_closed)
spider = aqaqspider(domain='aqaq.com')
crawler = Crawler(Settings())
crawler.configure()
crawler.crawl(spider)
crawler.start()a
log.start(loglevel=log.DEBUG)
log.msg("------------>Running reactor")
result = reactor.run()
print result
log.msg("------------>Running stoped")

while running the above script i am getting the following error:

2013-09-27 19:21:06+0530 [aqaq] ERROR: Error downloading <GET http://www.aqaq.com/list/female/view-all?limit=all>: 'Settings' object has no attribute 'overrides'

I am a beginner and in need of help ???

R. Max · Accepted Answer · 2013-09-28 17:09:31Z

4

You must use CrawlerSettings instead of Settings.

Change this line:

    from scrapy.settings import Settings

by:

    from scrapy.settings import CrawlerSettings

And this line:

    crawler = Crawler(Settings())

by:

    crawler = Crawler(CrawlerSettings())

answered Sep 28, 2013 at 17:09

R. Max

6,7501 gold badge29 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

praxmon Over a year ago

What if my python script is in another directory? Could you also look at: stackoverflow.com/questions/28020360/…

Kiran Kyle Over a year ago

Can we use scrapy downloader as library from custom python script?

R. Max Over a year ago

@KiranKyle you could use the downloader component but as far I remember it's tied to twisted, so you need to setup a twisted reactor and so on.

Collectives™ on Stack Overflow

Calling Scrapy Spider from python script?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related