How to get values from conditional drop down fields with Scrapy/Python?

Question

This tutorial helped me in order to gather the first drop down list entries on a website. But I am not able to get the data of the second conditional/dependent drop down box with Scrapy.

Assuming the following procedure is the way to go, how does step 2 work? The values never appear so far...

Get list entries of first drop down box
Select 1st value of first drop down box.
get all dependent drop down values.
iterate though the first drop down list...

I think the code below needs to be adjusted as the code is for a submit form:

    for i in range(len(brandCategoryList)):

        # Generate new request for each brand category's page
        yield FormRequest("http://www.xxxxxxxxxx.com",
                    method='POST',                         
                    formdata={'BrandName':'','CatBrand':brandCategoryList[i],'submit1':'Find+Product'},
                    callback=self.parseBrandPage,
                    meta={'brandCategoryId':i,'brandCategoryName':brandCategoryList[i]})

Thanks

Community · Accepted Answer · 2017-05-23 11:48:09Z

2

Scrapy won't run any javascript code in the page you are scraping, and dependent drop-downs often rely on javascript to populate their options based on the choice in their controlling drop-down.

If you need to automate/scrape content which requires javascript, you may want to consider another library. Start here for some pointers.

edited May 23, 2017 at 11:48

CommunityBot

11 silver badge

answered Oct 10, 2012 at 22:54

abeyer

7454 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to get values from conditional drop down fields with Scrapy/Python?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related