python selenium get text from element

Question

how would I get the text "Premier League (ENG 1)" extracted from this HTML tree? (marked part)

I treid ti get the text with xpath, css selector, class... but I seem to cant get this text extracted.

Basically I want to create a list and go over all "class=with icon" elements that include a text (League) and append the text to that list.

This was my last attempt:

def scrape_test():
    alleligen = []

    #click the dropdown menue to open the folder with all the leagues
    league_dropdown_menue = driver.find_element_by_xpath('/html/body/main/section/section/div[2]/div/div[2]/div/div[1]/div[1]/div[7]/div')
    liga_dropdown_menue.click()
    time.sleep(1)
    
    #get text form all elements that conain a league as text
    leagues = driver.find_elements_by_css_selector('body > main > section > section > div.ut-navigation-container-view--content > div > div.ut-pinned-list-container.ut-content-container > div > div.ut-pinned-list > div.ut-item-search-view > div.inline-list-select.ut-search-filter-control.has-default.has-image.is-open.active > div > ul > li:nth-child(3)')
    
    #append to list
    alleligen.append(leagues)
    
    print(alleligen)

But I dont get any output.

What am I missing here?

(I am new to coding)

Sashaank · Accepted Answer · 2020-09-08 16:37:37Z

2

try this

path = "//ul[@class='inline-list']//li[first()+1]"
element = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.XPATH, path))).text

print(element)

the path specifies the element you want to target. the first // in the path means that the element you want to find is not the first element in the page and exists somewhere in the page. li[first()+1] states that you are interested in the li tag after the first li.

The WebDriverWait waits for the webpage to load completely for a specified number of seconds (in this case, 5). You might want to put the WebdriverWait inside a try block.

The .text in the end parses the text from the tag. In this case it is the text you want Premier League (ENG 1)

answered Sep 8, 2020 at 16:37

Sashaank

9723 gold badges23 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

exec85 Over a year ago

Thanks. But I get an error that the selenium timed out. It seems it can not find the element / path.

Sashaank Over a year ago

@exec85 can you share the link from which you want to scrape the data?

rahul rai · Accepted Answer · 2020-09-08 15:46:17Z

1

Can you try :

leagues = driver.find_elements_by_xpath(“//li[@class=‘with-icon’ and contains(text(), ‘League’)]”)
For league in leagues:
     alleligen.append(league.text)
print(alleligen)

answered Sep 8, 2020 at 15:46

rahul rai

2,3361 gold badge9 silver badges18 bronze badges

6 Comments

exec85 Over a year ago

Changed it bit (removed 'League', but now it works: leagues = driver.find_elements_by_xpath("//li[@class='with-icon' and contains(text(), '')]")

rahul rai Over a year ago

What do you mean by extracting xpath?? Do you mean any attribute value ?? Like along with inner text you need class value as well? If so then you can element.get_attribute(“class”), it will return you value of class for the element.

exec85 Over a year ago

ok what I mean is, I would like to get the text (league which i have now) and the corresponding xpath of each element containing league text. Target is, that I can later choose the league in my GUI (dropdown e. g.) and then the tool will select the league via xpath in the browser. Therefore I would need the corresponding xpath for each league. Then I will try to get league + xpath in a list like csv or something.

exec85 Over a year ago

something like this:

scrape_leagues = driver.find_elements_by_xpath("//li[@class='with-icon' and contains(text(), '')]")     for league in scrape_leagues:         leagues.append(league.text)          scrape_leagues_xpath = driver.find_elements_by_xpath("//li[@class='with-icon')]")     for xpath in scrape_leagues_xpath:         leagues.append(xpath.xpath)      print(leagues)

Sorry dont know how to format the code here in comment.

rahul rai Over a year ago

As xpath is a locator strategy you are proving to inspect element, so there is no nothing we can get returned from element as xpath. However if I understand correctly you can always store your xpath based on your return something like, xpath=“//li[@class=‘with-icon’ and text()=“+LeagueName+”’]”). Where league name is variable in which you are storing your read text for each league in above code. But please note this xpath has nothing to done with element , we are simply using earlier returned inner text only. I hope I clear some what :)

|

Razvan · Accepted Answer · 2020-09-08 21:01:39Z

1

If you know that your locator will remain on the same position in that list tree, you can use the following where the li element is taken based on its index:

locator= "//ul[@class='inline-list']//li[2]"
element = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, locator))).text

answered Sep 8, 2020 at 21:01

Razvan

4053 silver badges14 bronze badges

Collectives™ on Stack Overflow

python selenium get text from element

3 Answers 3

2 Comments

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related