Cannot get dynamical element with python selenium

Question

There is site, that streams youtube videos. I want to get playlist with them. So I use selenium webdriver to get the needed element div with class-name ytp-title-text where youtube link is located.

It is located here for example, when I use browser console to find element:

<div class="ytp-title-text"><a class="ytp-title-link yt-uix-sessionlink" target="_blank" data-sessionlink="feature=player-title" href="https://www.youtube.com/watch?v=VyCY62ElJ3g">Fears - Jono McCleery</a><div class="ytp-title-subtext"><a class="ytp-title-channel-name" target="_blank" href=""></a></div></div>

I wrote simple script for testing:

from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.common.exceptions import TimeoutException

driver = webdriver.Firefox()
driver.get('http://awsmtv.com')

try:
    element = WebDriverWait(driver, 10).until(
        EC.visibility_of_element_located((By.CLASS_NAME, "ytp-title-text"))
    )
finally:
    driver.quit()

But no element is found and timeout exception is thrown. I cannot understand, what actions selenium needs to perform to get the full page source.

you are locating by ID but I don't see any id in your html. You can either locate by class name or css — theGuy
– theGuy, Commented Oct 5, 2018 at 14:59
@theGuy yes, I edited my question with By.CLASS_NAME. But result is the same — approximatenumber
– approximatenumber, Commented Oct 5, 2018 at 15:03
idk if its me or the site..the site is not loading on my browser, used proxy also same situation.. — iamklaus
– iamklaus, Commented Oct 5, 2018 at 15:04

Andersson · Accepted Answer · 2018-10-05 15:29:45Z

2

Required link is hidden and also located inside an iframe. Try below to locate it:

WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it("tvPlayer_1"))
try:
    element = WebDriverWait(driver, 10).until(
        EC.presence_of_element_located((By.CLASS_NAME, "ytp-title-link")))
    print(element.get_attribute('href'))
finally:
    driver.quit()

answered Oct 5, 2018 at 15:29

Andersson

52.8k18 gold badges83 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

approximatenumber Over a year ago

merci, it works. i`ve got no experience with iframes

Rain9333 · Accepted Answer · 2018-10-05 15:33:44Z

1

Just saw this element is inside iframe... You need to switch to the iframe first -> find it by ClassName -> ifame = ...(By.CLASS_NAME, "player") then switch to it driver.switch_to_frame(iframe) and you should be able now to get the wanted element :)

The XPath locator like this one will work (or your locator) -> "//a[@class='ytp-title-link yt-uix-sessionlink']".

You then need via the element to get the property href for the youtube video url or the text of the element for the song title.

If still not working I can suggest to get the page source - html = driver.page_source which will give you the source of the page and via some regex to get the info you want eventually.

edited Oct 5, 2018 at 15:33

answered Oct 5, 2018 at 15:22

Rain9333

5704 silver badges22 bronze badges

Collectives™ on Stack Overflow

Cannot get dynamical element with python selenium

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related