Python Selenium driver by CSS selector inside of a loop

Question

I am scraping a page heavy with JS, where I select a js button

driver.find_element_by_css_selector('.u-textColorDarker').click()

which displays a from. Inside that form, some info is displayed, but to get all the content there is a button ("show more") that needs to be clicked n times to display all the content. When there is nothing else to show, the button disappears.

In this particular page, the button needs to be clicked 3 times. So a quick solution for my problem in:

driver.find_element_by_css_selector('.js-showMoreRecommends').click()
time.sleep(2)
driver.find_element_by_css_selector('.js-showMoreRecommends').click()
time.sleep(2)
driver.find_element_by_css_selector('.js-showMoreRecommends').click()
time.sleep(2)

Is there a way to put this in a loop, so the action is done inside a loop until the selector '.js-showMoreRecommends' is no longer present?

Sure, use while True: and break out of it when your condition is met — Mike Scotty
– Mike Scotty, Commented Jun 18, 2019 at 12:30

Raven King · Accepted Answer · 2019-06-18 12:42:16Z

2

Yes, you can using expected conditions.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Firefox()
driver.get("http://somedomain/url")

while not EC.invisibility_of_element_located((By.CSS_SELECTOR, ".js-showMoreRecommends")):
    driver.find_element_by_css_selector('.js-showMoreRecommends').click()

I am writing this from memory, but it should point you in the general direction. The meat of this is the expected conditions method EC.invisibility_of_element_located. It will give you a boolean value to determine if it is gone yet.

answered Jun 18, 2019 at 12:42

Raven King

1761 silver badge13 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Raven King Over a year ago

Yeah we were writing posts about the same time.

Luis Miguel Over a year ago

Thank you Raven, this solution executed, but the buttons did not get clicked. Jose Martinez solution worked well. Not really sure why yours did not work. Thanks again, anyway

Raven King Over a year ago

Probably just needed a different expected conditions selector. Glad you found a solution. Just know that one isn't deterministic, so if anything on the website changes your program will break.

undetected Selenium · Accepted Answer · 2019-06-18 14:25:42Z

2

As @mike-scotty comented: "Use while"

It is recommended to use WebDriverWait and not time.sleep().

Should look something like this:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait

while WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".js-showMoreRecommends"))):
    show_more = WebDriverWait(driver, 30).until(EC.visibility_of_element_located((By.CSS_SELECTOR, ".js-showMoreRecommends")))
    show_more.click()

edited Jun 18, 2019 at 14:25

undetected Selenium

194k44 gold badges304 silver badges387 bronze badges

answered Jun 18, 2019 at 12:37

Moshe Slavin

5,2445 gold badges27 silver badges40 bronze badges

2 Comments

Luis Miguel Over a year ago

Thank you Moshe, this solution got stuck in a loop. Jose Martinez version (below) worked well, although it does not useWebDriverWait, but time.sleep.

Moshe Slavin Over a year ago

@LuisMiguel Glad to hear! you Should try using WebDriverWait more... ;)

Jose Martínez Poquet · Accepted Answer · 2019-06-18 12:36:31Z

1

Insted of a while true loop you can try to while loop when the button is displayed

while driver.find_element_by_css_selector('.js-showMoreRecommends').is_displayed():
    driver.find_element_by_css_selector('.js-showMoreRecommends').click()
    time.sleep(2)

answered Jun 18, 2019 at 12:36

Jose Martínez Poquet

3351 silver badge5 bronze badges

1 Comment

Moshe Slavin Over a year ago

this will work fine! yet it's recommended to use WebDriverWait not time.sleep...

Collectives™ on Stack Overflow

Python Selenium driver by CSS selector inside of a loop

3 Answers 3

3 Comments

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related