Python Selenium issue with submit button, not working consistently

Question

I am trying to build a web scraper for redfin to pull the redfin estimate. I have a function that does this and sometimes it works and sometimes it does not work. I noticed that the reason it does not work is because of the submit function. Sometimes the chrome page wont press the search (submit) button and go to the property page.

I am not sure how to fix this issue and make it more consistent.

Here is my code:

from selenium import webdriver
from selenium.webdriver.remote import webelement
import pandas as pd
import time
from bs4 import BeautifulSoup

driver = webdriver.Chrome('chromedriver.exe')
driver.get('https://www.redfin.com/')
time.sleep(3)

def get_redfin_estimate(address):
    search_box = driver.find_element_by_name('searchInputBox')
    search_box.send_keys(address)
    search_box.submit()
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    try:
        price1 = soup.find('div', {'class', 'avm'}).div.text
        return(price1)
    except AttributeError:
        try:
            time.sleep(10)
            price2 = soup.find('span',class_='avmLabel').find_next('span', class_='value').text
            return(price2)
        except:
            return('N/A')


print(get_redfin_estimate('687 Catalina Laguna Beach, CA 92651'))
print(get_redfin_estimate('693 Bluebird Canyon Drive, Laguna Beach, CA 92651'))



driver.quit()

Have you tried not using .submit() and actually clicking on the magnifying glass element to submit the search? — JeffC
– JeffC, Commented Jun 19, 2019 at 21:29
@JeffC I am totally new to this stuff, if you have an answer please provide one. — Wolfy
– Wolfy, Commented Jun 19, 2019 at 21:30
Delete the search_box.submit() line, find the element by the CSS selector button[title='Search'], and click it. — JeffC
– JeffC, Commented Jun 19, 2019 at 21:32
@JeffC I am sure how to do that can you provide the line I need to replace for search_box.submit() please — Wolfy
– Wolfy, Commented Jun 19, 2019 at 21:34

JeffC · Accepted Answer · 2019-06-19 21:42:17Z

1

It may be an issue with using .submit() on that element. One alternative is to just click on the magnifying glass to initiate the search.

def get_redfin_estimate(address):
    driver.find_element_by_name('searchInputBox').send_keys(address)
    driver.find_element_by_css_selector("button[title='Search']").click()
    time.sleep(3)
    soup = BeautifulSoup(driver.page_source, 'html.parser')
    ... and so on

edited Jun 19, 2019 at 21:42

answered Jun 19, 2019 at 21:40

JeffC

26.4k5 gold badges35 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Wolfy Over a year ago

driver.find_element_by_css_selector('button[title='Search']').click() ^ SyntaxError: invalid syntax

JeffC Over a year ago

Sorry, fixed typo

JeffC Over a year ago

You didn't get the final update. I hit save too quickly and only changed one typo. You really need to spend some time reading some python tutorials so you can fix these basic items on your own. It was just a case of mismatched quotes.

Wolfy Over a year ago

Yea I know I was asked to build this at work and I normally do research in ML and use C++ to write algorithms so I am totally new to this

undetected Selenium · Accepted Answer · 2019-06-19 21:54:31Z

1

The reason submit button is not working consistently because the desired element is a JavaScript enabled element and the Locator Strategy you have used doesn't identifies the search box with placeholder as City, Address, School, Agent, ZIP uniquely and identifies 3 elements.

To send a character sequence to the desired field you have to induce WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:

Using CSS_SELECTOR:

def get_redfin_estimate(address):
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.searchInputNode input.search-input-box#search-box-input"))).send_keys(address)
    driver.find_element_by_css_selector("div.searchInputNode button.inline-block.SearchButton.clickable").click()

Using XPATH:

def get_redfin_estimate(address):
    WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[@class='searchInputNode']//input[@class='search-input-box' and @id='search-box-input']"))).send_keys(address)
    driver.find_element_by_xpath("//div[@class='searchInputNode']//button[@class='inline-block SearchButton clickable float-right']").click()

Note : You have to add the following imports :

from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

edited Jun 19, 2019 at 21:54

answered Jun 19, 2019 at 21:07

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

10 Comments

Wolfy Over a year ago

Thank you, could you update the code I have in your answer?

undetected Selenium Over a year ago

@Wolfy I didn't get you. How can I help you?

Wolfy Over a year ago

I understand your answer, but could you update to include my code with the appropriate changes that you suggested?

Wolfy Over a year ago

Does that make sense? I just want to make sure my code is perfectly corrected given the changes you suggested...

undetected Selenium Over a year ago

@Wolfy Check out the answer update and let me know if you have got the expected answer.

|

Collectives™ on Stack Overflow

Python Selenium issue with submit button, not working consistently

2 Answers 2

4 Comments

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related