Using Scrapy's FormRequest.from_response method automate scraping of dropdown menu wise data

Question

I have been struggling with this for the past two days. I need to scrape data from this website for all "Cadres" or categories. Unfortunately, the website allows access to this data via a dropdown menu "Select Cadre" which doesn't have an "All Categories" option. To circumvent this, I am using Scrapy's FormRequest.from_response method but the spider is returning a blank file with no data in it. Any help is appreciated. Here's the code:

import scrapy

class IASWinnerSpider(scrapy.Spider):

    name = 'iaswinner_list'
    allowed_domains = ['http://civillist.ias.nic.in']

    def start_requests(self):
        urls = [ 'http://civillist.ias.nic.in/UpdateCL/DraftCL.asp' ]
        for url in urls:
            yield scrapy.Request(url=url, callback=self.parse)

    def parse(self, response):
        return scrapy.FormRequest.from_response(response, method='POST',
                    formdata={'cboCadre': 'UT'}, dont_click=True, callback=self.after_post)

    def after_post(self, response):

        table      = response.xpath('/html/body/div/table//tr')

        for t in table:

            yield {
                'serial': t.xpath('td[1]/text()').extract(),
                'name': t.xpath('td[2]/text()').extract(),
                'qual': t.xpath('td[3]/text()').extract(),
                'dob': t.xpath('td[4]/text()').extract(),
                'post': t.xpath('td[5]/text()').extract(),
                'rem': t.xpath('td[6]/text()').extract(),
            }

The code given is not yet "complete" (cf. minimal reproducible example). It might be kindly advised to add a __main__ section exhibiting the issue. — ederag
– ederag, Commented Aug 19, 2017 at 13:28
If Linhart's answer does what you needed please don't forget to mark it 'accepted'. — Bill Bell
– Bill Bell, Commented Aug 19, 2017 at 16:12

Tomáš Linhart · Accepted Answer · 2017-08-19 13:55:01Z

1

When I run your code, I see this in the log:

2017-08-19 15:52:20 [scrapy.spidermiddlewares.offsite] DEBUG: Filtered offsite request to 'civillist.ias.nic.in': <POST http://civillist.ias.nic.in/UpdateCL/DraftCL.asp>

Just change allowed_domains to this:

allowed_domains = ['civillist.ias.nic.in']

and it works.

answered Aug 19, 2017 at 13:55

Tomáš Linhart

10.2k1 gold badge30 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Using Scrapy's FormRequest.from_response method automate scraping of dropdown menu wise data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related