How to click on an element that is not a button using Selenium in Python

Question

I am new to web scraping and am trying to scrape data from this real estate website to get only the places that have recently been rented. To do this I need to click "Leased Listing" from this dropdown menu. Picture of what I need to click

The issue I am having is this is not a button class so using the selenium .click() function is giving me an error. There are also multiple objects with the same class name as the "Leased Listing" section.

Here is my code:

 for page in range(0, total_pages + 1):
  chrome_options = webdriver.ChromeOptions()
  chrome_options.add_argument('--headless')
  chrome_options.add_argument('--no-sandbox')
  chrome_options.add_argument('--disable-dev-shm-usage')
  driver = webdriver.Chrome('chromedriver',chrome_options=chrome_options)
  headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0 ; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.75 Safari/537.36'}
  url = 'https://www.zoocasa.com/toronto-on-real-estate-for-rent?page=' + str(page)
  driver.get(url)
  elements = driver.find_elements(By.CLASS_NAME, "style_component__DR_Bs")
  elements[6].click() #Leased listing is the 7th appearance of this class name

And here is the site's html (whatever is clicked has the "style_active__eGbvT"):

<div class="style_component__DR_Bs">
    ::before
    Active Listing
    ::after
</div>
<div class="style_component__DR_Bs style_active__eGbvT">
    ::before
    Leased Listing
    ::after
</div>
<div class="style_component__DR_Bs">
    ::before
    Expired Listing
    ::after
</div>

If anyone has any suggestions I would really appreciate it, Thanks.

should clicking this on its page redirect you somewhere? I dont see anything on it like a href or and id for javascript to reference. go to the page, right click + inpect, then right click the element and click inspect again, should take you directly to the links element in the html code — Christian Trujillo
– Christian Trujillo, Commented Aug 10, 2022 at 16:27
@ChristianTrujillo no you are not redirected anywhere after clicking, the URL stays the same but the contents of the page get updated. Inspecting the element gives the html code provided above — James McCarron
– James McCarron, Commented Aug 10, 2022 at 16:31

halfer · Accepted Answer · 2022-09-07 18:18:49Z

1

I think Selenium allows to click div.

However, you need to identify the correct div and id before proceeding.

The easiest way: can you driver.find_elements_by_xpath, and it will return a list then you need to check text for each item

Code:

list_subcontent = driver.find_elements_by_xpath(".//div[@class='style_component__lT4sh style_theme-dark__rZF3s']//div[@class='style_component__DR_Bs ']")

for item in list_subcontent:
    if item.text == 'Leased Listing':
        item.click()

edited Sep 7, 2022 at 18:18

halfer

20.2k20 gold badges110 silver badges207 bronze badges

answered Aug 10, 2022 at 16:35

Amzar

787 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

James McCarron Over a year ago

Thanks for the response, however I am getting the same error of the element not being interactable as the method I tried above. None of the elements also seem to have a ".text()" string associated with them.

Amzar Over a year ago

@JamesMcCarron can you print all item.text? for item in list_subcontent: print(item.text)

James McCarron Over a year ago

This does not print anything

Prophet · Accepted Answer · 2022-08-10 17:17:33Z

0

There are 2 problems here:

You are trying to access the elements before the page is completely loaded. The best way to resolve this issue is to use expected conditions explicit waits to wait for some element visibility etc.
The elements matching your locator are not clickable. You have to open the drop down menu to make that element clickable.
This should work better:

from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver.get(url)
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Active Listings')]"))).click()
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(@class,'style_theme-dark')]//div[contains(text(),'Leased Listing')]"))).click()

answered Aug 10, 2022 at 17:17

Prophet

33.5k28 gold badges58 silver badges90 bronze badges

8 Comments

James McCarron Over a year ago

Thanks for the response, I am getting a Timeout Exception when I run this code at the last line. The first wait.until() line seems to execute but it gets stopped at the last line looking for "Leased Listing"

Prophet Over a year ago

So, you can confirm the drop-down menu is opened by the first command. Right? Are you sure you copied the second command correctly?

James McCarron Over a year ago

The line to open the dropdown does not result in any errors, but I am not sure if it actually opened the dropdown. Is there a selenium function that could confirm this? (sorry very inexperienced with this library)

Prophet Over a year ago

In case expected condition fails i.e. the condition not matched it throws a Timeout Exception as you saw. If the first line doesn't throw exception it should work. We need to debug your code, but I have no Selenium with Python on my PC, I'm sorry..

James McCarron Over a year ago

Ok thank you for the help. Should the second line have the same "//div[contains(@class,'style_theme-dark')]" as the first line? Why is this the case?

|

Collectives™ on Stack Overflow

How to click on an element that is not a button using Selenium in Python

2 Answers 2

3 Comments

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related