Access the content of the tags with -xpath -selenium -python

Question

I try access to content of some tags in a page.

But they are impossible to access.

Here are the first lines of code:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()
url = "https://www.flashscore.com/match/hzciHz2I/#h2h;overall"
driver.get(url)

Here is the middle code (the one that pose a problem):

I did two version, here is the first one

data = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH,
"//table[contains(@class,'h2h_mutual')]/tbody/tr[contains(@class,'highlight')]")))

and here is the second one

data = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH,
"//table[@class='h2h_mutual']/tbody/tr[@class='highlight']")))

but it did not find tags and returned an exception 'TimeoutException:'

I not understund why this does not work, someone can help me?

here is the end of the code, that serves to display the content :

for elem in data:
     print(elem.text)

here is first version in full :

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException

driver = webdriver.Chrome()
url = "https://www.flashscore.com/match/hzciHz2I/#h2h;overall"
driver.get(url)

data = WebDriverWait(driver, 10).until(EC.visibility_of_all_elements_located((By.XPATH,
"//table[contains(@class,'h2h_mutual')]/tbody/tr[contains(@class,'highlight')]")))

for elem in data:
     print(elem.text)

I specify that this page is a window open after click on a link in this page and that I did good the switch between the two page and even perform some actions on it before this one. (The firsts lines are so there just for that you can reproduce the error quickly)

this is how I proceeded :

driver.find_element(By.CSS_SELECTOR, "#li2").click()
WebDriverWait(driver, timeout=5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#li2")))
window_first = driver.window_handles[0]
link_page = WebDriverWait(driver, timeout=5).until(EC.element_to_be_clickable((By.XPATH,"//div[text()='Nantes']")))
driver.execute_script("arguments[0].click();", link_page)
Window_second = driver.window_handles[1]
driver.switch_to.window(Window_second)
driver.find_element(By.CSS_SELECTOR, "#a-match-head-2-head").click()
WebDriverWait(driver, timeout=5).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "#a-match-head-2-head")))

the below the page, from that point 'Head-to-head matches: NANTES - PARIS SG' — the star
– the star, Commented Oct 25, 2020 at 22:16
Most of the elements you are finding with your XPATH expression are hidden and only show if you click on 'show more matches' and even then there are more in a hidden div. You could try to wait only for the first element. — Maximilian Peters
– Maximilian Peters, Commented Oct 25, 2020 at 22:19

Maximilian Peters · Accepted Answer · 2020-10-25 22:42:12Z

1

As written in the comment, most of the elements returned by your XPATH expression are not visible, either because the user needs to click on 'show more results' or because they are in a div with visibility: none. If you change your wait to presence_of_all_elements_located, it should work.

data = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH,
"//table[contains(@class,'h2h_mutual')]/tbody/tr[contains(@class,'highlight')]")))


print(data[0].text)

>>> '04.02.20 L1 Nantes Paris SG 1 : 2'

Some background: presence_of_all_elements_located check if the elements are located, i.e. existing, on the web page but doesn't need them to be visible. visibility_of_all_elements_located checks for presence and visibility, i.e. one could say it's a more stringent version of presence.

See Selenium Documentation

edited Oct 25, 2020 at 22:42

answered Oct 25, 2020 at 22:25

Maximilian Peters

31.8k12 gold badges95 silver badges102 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

the star Over a year ago

Thank you, it worked. I already click on "show more results" , I just didn't put it here

Maximilian Peters Over a year ago

@thestar: added some explanation.

Maximilian Peters Over a year ago

There is a still a hidden div, even after clicking on 'show more results', that's the reason why the wait still times out.

Collectives™ on Stack Overflow

Access the content of the tags with -xpath -selenium -python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related