Python Selenium Webpage Scraping Selecting Drop Down and entering text in html form

Question

I have a problem scraping data from the following site: https://arcc.sdcounty.ca.gov/Pages/Assessors-Roll-Tax.aspx.

I have to do these steps in order:

Select a drop down option "Street Address'
Enter a street address into a text field (ie 43 Hadar Dr)
Click the 'Submit' button.

After clicking submit, I should be directed to a page that has the APN number for a given address.

The problem: I am able to do the above steps. However, when I select a drop down option and input address in the textbox, it fails as the textbox input address for some reason is cleared before clicking 'submit' ONLY when I have selected a drop down option.

I have tried using Selenium's Expected Conditions to trigger the input in the text box after a drop down option has been selected, but did nothing. I am looking for any help on identifying the why there is this problem as well as any advice on solutions.

Thanks.Much appreciated.

My code:

    driver = webdriver.Chrome()
    driver.get('https://arcc.sdcounty.ca.gov/Pages/Assessors-Roll-Tax.aspx')
    #Selects drop down option ('Street Address')
    mySelect =        Select(driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25  ea12_ctl00_ddlSearch"))
    my=mySelect.select_by_value('0')  
    wait = WebDriverWait(driver,300)
    #Enter address in text box to left of drop down
   driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ct    l00_txtSearch").send_keys("11493 hadar dr")
    #Click 'Submit' button to return API numbers associated with address
    driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_btnSearch").click()
    driver.quit()

Also, it may be important than when I manually input an address to the text box, THEN select a drop down, the text box is automatically cleared, although it is not cleared when I manually select a drop down value and THEN input address in text box manually...however, my code does select a drop down value first, and then inputs address in text box. — Jacob Neumann
– Jacob Neumann, Commented Oct 10, 2016 at 23:34

A.Sherif · Accepted Answer · 2016-10-11 01:11:43Z

1

Just changed a few things in your code to make it work.

mySelect = Select(driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25  ea12_ctl00_ddlSearch"))

To find_element_by_name(...):

mySelect = Select(driver.find_element_by_name("ctl00$ctl43$g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12$ctl00$ddlSearch"))

And

my=mySelect.select_by_value('0')

To select_by_visible_text('...'):

my = mySelect.select_by_visible_text("Street Address")

And

driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ct l00_txtSearch").send_keys("11493 hadar dr")

To find_element_by_xpath(...), since I usually get better results when finding elements by xpath.

driver.find_element_by_xpath('//*[@id="ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_txtSearch"]').send_keys("11493 hadar dr")

This is how it all looks like:

from selenium.webdriver.support.ui import WebDriverWait
from selenium import webdriver
from selenium.webdriver.support.ui import Select

driver = webdriver.Chrome()
driver.get('https://arcc.sdcounty.ca.gov/Pages/Assessors-Roll-Tax.aspx')

#Selects drop down option ('Street Address')
mySelect = Select(driver.find_element_by_name("ctl00$ctl43$g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12$ctl00$ddlSearch"))
my = mySelect.select_by_visible_text("Street Address")

wait = WebDriverWait(driver,300)

#Enter address in text box to left of drop down
driver.find_element_by_xpath('//*[@id="ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_txtSearch"]').send_keys("11493 hadar dr")

#Click 'Submit' button to return API numbers associated with address
driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_btnSearch").click()

driver.quit()

answered Oct 11, 2016 at 1:11

A.Sherif

1442 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jacob Neumann Over a year ago

Did this work for you? Using your code, I get the same result, and no fixes seem to be made. If this worked on your computer, do you know of any reason it would not work on my computer? Thanks so much for your help!

Jacob Neumann Over a year ago

I just made this youtube video of my screen while running your code so you can see what your code does on my computer and how it fails. youtube.com/watch?v=MVTZwOKhXeE&feature=youtu.be

A.Sherif Over a year ago

Its just the timing, the driver clicks search before the text is entered in the text field. If you can add the following to make sure its all about the timing. raw_input("Press Enter to Continue") add this line after you select the drop down and wait for a few seconds then click enter, and add it after the send_keys() aswell. and wait until you see the text entered in the text field then press enter to continue it should work without any problems.

Jacob Neumann Over a year ago

Thanks so much! So it worked ONLY when I use the additional code 'input("Press Enter To Continue"). Do you know how I can automate this process so that'll work without me having to press 'Enter' ? I tried using Expected Conditions (of Selenium) but it didn't work. Perhaps, I used ECs incorrectly though. Very much appreciate you continued help if possible!

A.Sherif Over a year ago

Glad it worked for you! Yea, you can replace the input('.....') with a sleep. import time; time.sleep(5) or increase the wait time used by WebDriverWait(driver, 300)

DVOPS · Accepted Answer · 2016-10-11 00:01:37Z

0

Not sure if this is your situation. But one thing that jumped out from your question is the text box input... Often, when filling in a website text box, even though the text is clearly visible, the text is not actually read by the text-box method until after the focus (cursor) is clicked or tabbed out and away from the text box.

Tabbing the text cursor out of the text entry box first, before 'clicking submit', will often solve this issue.

answered Oct 11, 2016 at 0:01

DVOPS

564 bronze badges

2 Comments

Jacob Neumann Over a year ago

OK, applying your advice, I adjusted my code like the following: driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_txtSearch").send_keys("11493 hadar dr") driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_txtSearch").send_keys(Keys.TAB) driver.find_element_by_id("ctl00_ctl43_g_d30f33ca_a5a7_4f69_bb21_cd4abc25ea12_ctl00_btnSearch").click()

Jacob Neumann Over a year ago

However, this did not resolve the issue unfortunately.

Collectives™ on Stack Overflow

Python Selenium Webpage Scraping Selecting Drop Down and entering text in html form

2 Answers 2

5 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related