Using python scrapy based crawler but getting error

Question

Hi folks I have written a crawler in python for scraping......

import scrapy

from c2.items import C2Item

try:

    class C2(scrapy.Spider):
            name = 'cn'
            allowed_domains = ['priceraja.com']
            start_urls = ['https://www.priceraja.com']



            def parse_item(self, response):

               Item = []
               Item['url']=response.xpath('//a/@href/text()').extract()
               yield Item

except Exception:
logging.exception("message")

I keep on getting NotImplemented error

2017-08-05 01:12:28 [scrapy.core.scraper] ERROR: Spider error processing 
<GET 
https://www.killerfeatures.com> (referer: None)
Traceback (most recent call last):
File "D:\Ana\lib\site-packages\twisted\internet\defer.py", line 653, in _ 
runCallbacks
current.result = callback(current.result, *args, **kw)
File "D:\Ana\lib\site-packages\scrapy\spiders\__init__.py", line 90, in 
parse raise NotImplementedError
NotImplementedError
2017-08-05 01:12:28 [scrapy.core.engine] INFO: Closing spider (finished)
2017-08-05 01:12:28 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 435,
 'downloader/request_count': 2,
 'downloader/request_method_count/GET': 2,

'downloader/response_bytes': 9282, 'downloader/response_count': 2, 'downloader/response_status_count/200': 1, 'downloader/response_status_count/301': 1, 'finish_reason': 'finished', 'finish_time': datetime.datetime(2017, 8, 4, 19, 42, 28, 837000), 'log_count/DEBUG': 3, 'log_count/ERROR': 1, 'log_count/INFO': 7, 'response_received_count': 1, 'scheduler/dequeued': 2, 'scheduler/dequeued/memory': 2, 'scheduler/enqueued': 2, 'scheduler/enqueued/memory': 2, 'spider_exceptions/NotImplementedError': 1, 'start_time': datetime.datetime(2017, 8, 4, 19, 42, 25, 976000)} 2017-08-05 01:12:28 [scrapy.core.engine] INFO: Spider closed (finished)

File "D:\Ana\lib\site-packages\scrapy\spiders_init_.py", line 90, in parse raise NotImplementedError NotImplementedError 2017-07-05 23:54:45 [scrapy.core.engine] INFO: Closing spider (finished) is the error I get — msnegi
– msnegi, Commented Jul 5, 2017 at 18:41
Could you please edit your post to include the error message, instead of posting it as a comment? Thank you! — Andrew Myers
– Andrew Myers, Commented Jul 5, 2017 at 22:17

Salmaan P · Accepted Answer · 2017-07-05 18:52:41Z

1

Scrapy is looking for the parse function while you have implement the parse_item function. Changing parse_item to parse might work, or you can override the parse function.

Another solution from here is to use CrawlSpider

answered Jul 5, 2017 at 18:52

Salmaan P

8672 gold badges13 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

msnegi Over a year ago

Changed code as per suggestion still not working . Can someone add a working example and also scrapy version

Collectives™ on Stack Overflow

Using python scrapy based crawler but getting error

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related