scrape information from multiple urls using a for loop in scrapy

Question

I want to scrape information from multiple urls. I use the following code but it doesn't work. May someone please points me to where I have gone wrong?

import scrapy

class spider1(scrapy.Spider):
    name = "spider1"
    domain = "http://www.amazon.com/dp/"
    ASIN = ['B01LA6171I', 'B00OUKHTLO','B00B7LUVZK']

    def start_request(self):
        for i in ASIN:
            yield scrapy.Request(url=domain+i,callback = self.parse)

    def parse(self, response):
       title =response.css("span#productTitle::text").extract_first().strip()
       ASIN_ext = response.xpath("//input[@name='ASIN']/@value").extract_first()
       data = {"ASIN":ASIN_ext,"title":title,}
       yield data

The log didn't show any errors. But just says 0 pages are crawled. — user45857
– user45857, Commented Apr 21, 2017 at 6:52

Joe D · Accepted Answer · 2017-04-21 07:18:59Z

6

You just need to add an 's' to the first function

def start_requests(self):

Subtle difference, but Scrapy looks for that specific function so it has to match perfectly.

answered Apr 21, 2017 at 7:18

Joe D

3981 silver badge10 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

user45857 Over a year ago

Thanks! It works if i move the 'domain' and 'ASIN' variable into the start_requests() block. I am a newbie to python and scrapy. May I how to make these two variable accessible for all the functions defined in the class?

Joe D Over a year ago

In your case, just prefix them with the class name when you reference them. The way you defined them is fine, but when you go to reference it would be spider1.domain and spider1.ASIN. Take a look here to learn more about the difference between class and object variables: ibiblio.org/g2swap/byteofpython/read/class-and-object-vars.html

paul trmbrth Over a year ago

@user45857, class attributes are accessible using the self object that is passed as first argument in class methods, e.g. def start_request(self): for i in self.ASIN:

Collectives™ on Stack Overflow

scrape information from multiple urls using a for loop in scrapy

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related