Find nth element by classname using selenium python

Question

I just started using selenium yesterday to help scrape some data and I'm having a difficult time wrapping my head around the selector engine. I know lxml, BeautifulSoup, jQuery and Sizzle have similar engines. But what I'm trying to do is:

Wait 10 seconds for page to completely load
Make sure there are the presence of ten or more span.eN elements (two load on intitial page load and more after)
Then start processing the data with beautifulsoup

I am struggling with the selenium conditions of either finding the nth element or locating the specific text that only exists in an nth element. I keep getting errors (timeout, NoSuchElement, etc)

    url = "http://someajaxiandomain.com/that-injects-html-after-pageload.aspx"
    wd = webdriver.Chrome()
    wd.implicitly_wait(10)
    wd.get(url)
    # what I've tried
    # .find_element_by_xpath("//span[@class='eN'][10]"))
    # .until(EC.text_to_be_present_in_element(By.CSS_SELECTOR, "css=span[class='eN']:contains('foo')"))

It's hard to provide any solution without knowing the html ! Provide some html if possible — Saifur
– Saifur, Commented May 11, 2015 at 21:11
Here is example of the prettified HTML: paste.ee/p/hR3f6 - I am after span.eN or tbody.EventBody being greater than 10 OR for a span.eN to contain "Triple Jump" (usually the last to load). It's really just the tabular data I'm interested in. Initially only 4 or 5 tbod[ies] load and then the rest is injected after the initial pageload. — user1645914
– user1645914, Commented May 11, 2015 at 21:37

Community · Accepted Answer · 2017-05-23 11:51:02Z

4

You need to understand the concept of Explicit Waits and Expected Conditions to wait for.

In your case, you can write a custom Expected Condition to wait for elements count found by a locator being equal to n:

from selenium.webdriver.support import expected_conditions as EC

class wait_for_n_elements_to_be_present(object):
    def __init__(self, locator, count):
        self.locator = locator
        self.count = count

    def __call__(self, driver):
        try:
            elements = EC._find_elements(driver, self.locator)
            return len(elements) >= self.count
        except StaleElementReferenceException:
            return False

Usage:

n = 10  # specify how many elements to wait for

wait = WebDriverWait(driver, 10)
wait.until(wait_for_n_elements_to_be_present((By.CSS_SELECTOR, 'span.eN'), n))

Probably, you could have also just used a built-in Expected Condition such as presence_of_element_located or visibility_of_element_located and wait for a single span.eN element to be present or visible, example:

wait = WebDriverWait(driver, 10)
wait.until(presence_of_element_located((By.CSS_SELECTOR, 'span.eN')))

edited May 23, 2017 at 11:51

CommunityBot

11 silver badge

answered May 11, 2015 at 21:21

alecxe

476k127 gold badges1.1k silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

user1645914 Over a year ago

@Saifur I really hope you are not leveraging your self appointed prowess or calling someone a "wrong person" to be a troll -- it's an honest question and I am grateful for those who are trying to help.

alecxe Over a year ago

@user1645914 nono, we've just exchanged a couple of jokes and removed off-topic comments - the only one left so it's out of the context. Saifur is definitely here on SO to help.

Saifur Over a year ago

@user1645914 My apology. I personally respect alecxe and all his efforts and of course ANYONE who asks questions on SO. It's the best place on the earth to get help when you are ALONE in dark.

user1645914 Over a year ago

@alecxe Thank you for your help. I am putting this into my code and still running into selenium.common.exceptions.TimeoutException: Message: '' -- it's probably on my end still but I will accept your detailed answer once I get it to work on my end. Thank you.

alecxe Over a year ago

@Saifur thank you! Sorry for leaving your comment alone. Though, it was a funny coincidence. The comment totally changed it's sense without a context :)

|

Collectives™ on Stack Overflow

Find nth element by classname using selenium python

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related