Selecting every options from a drop-down box using selenium python

Question

I am trying to extract some data from a website. However, the site has a hierarchical structure. It has a dropdown menu on the top, whose option valueare URLs. Thus, my approaches are:

find the dropdown box,
select an option,
extract some data, and
repeat Steps 2 to 4 for all available options.

Below is my code, I am able to extract data under the default selected option (the first one). But I got error Message: Element not found in the cache - perhaps the page has changed since it was looked up. It seems like my browser was not switched to the new page. I tried time.sleep() or driver.refresh(), but failed... Any suggestions are appreciated!

<select class="form-control"> 
    <option value="/en/url1">001 Key</option>
    <option value="/en/url2">002 Key</option>
</select>

# select the dropdown menu
select_box = Select(driver.find_element_by_xpath("//select[@class='form-control']"))
# get all options
options = select_box.options

for ele_index, element in enumerate(options):
    # select a url
    select_box.select_by_index(ele_index)
    time.sleep(5)
    print element.text

    # extract page data
    id_comp_html = driver.find_elements_by_class_name('HorDL')
    for q in id_comp_html:
        print q.get_attribute("innerHTML")
        print "============="

update 1 (based on alecxe's solution)

# dropdown menu
select_box = Select(driver.find_element_by_xpath("//select[@class='form-control']"))
options = select_box.options

for ele_index in range(len(options)):
    # select a url
    select_box = Select(driver.find_element_by_xpath("//select[@class='form-control']"))
    print select_box.options[ele_index].text
    select_box.select_by_index(ele_index)
    # print element.text
    # print "======"
    driver.implicitly_wait(5)
    id_comp_html = driver.find_elements_by_class_name('HorDL')
    for q in id_comp_html:
        print q.get_attribute("innerHTML")
        print "============="

alecxe · Accepted Answer · 2017-05-03 05:58:35Z

1

Your select_box and element references got stale, you have to "re-find" the select element while operating the option indexes inside the loop:

# select the dropdown menu
select_box = Select(driver.find_element_by_xpath("//select[@class='form-control']"))
# get all options
options = select_box.options

for ele_index in range(len(options)):
    # select a url
    select_box = Select(driver.find_element_by_xpath("//select[@class='form-control']"))
    select_box.select_by_index(ele_index)

    # ...
    element = select_box.options[ele_index]

You might also need to navigate back after selecting an option and extracting the desired data. This can be done via driver.back().

answered May 3, 2017 at 5:58

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

TTT Over a year ago

Thanks for your comment! I added element = select_box.options[ele_index] to the end of this loop, but still got the same error...

alecxe Over a year ago

@tao.hong wait, but have you copied the parts of the presented code including the redefining the select_box?

TTT Over a year ago

correct, see my update 1 in the post. Maybe the driver is still not refreshed?

alecxe Over a year ago

@tao.hong right. Let's try the following. What if you add, just for testing purposes, time.sleep(5) at the beginning of the loop, firs line? (don't forget to import time)

TTT Over a year ago

Thx for staying with me. I moved time.sleep(5) to the beginning of the code, but still have the same error...

|

Ken Wei · Accepted Answer · 2017-05-03 06:05:22Z

1

From your description of the site and your code, it looks like selecting an option from the dropdown sends you to another page; so after your first iteration in the for loop, you have moved to a different page, while your options variable is pointing to elements that are in the previous page.

One solution, specific to your situation (and likely the best in this case), would be to store the option values (i.e. the urls) instead, and directly navigate to those urls instead via the .get() method.

Otherwise, you would either need to keep a counter and get the contents of the dropdown with every iteration, or navigate backwards after every iteration, both choices being unnecessary in this case.

edited May 3, 2017 at 6:05

answered May 3, 2017 at 5:58

Ken Wei

3,1381 gold badge12 silver badges31 bronze badges

1 Comment

TTT Over a year ago

Thanks for the suggestion! Storing URLs seems like a solution.

Collectives™ on Stack Overflow

Selecting every options from a drop-down box using selenium python

update 1 (based on alecxe's solution)

2 Answers 2

9 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

update 1 (based on alecxe's solution)

2 Answers 2

9 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related