Get text from a tag name using Python and Selenium

Question

This is the Python Selenium code I am trying to use to get the title of the articles:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time

PATH = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(PATH)
driver.get("https://techwithtim.net")

search = driver.find_element_by_name("s")
search.send_keys("test")
search.send_keys(Keys.RETURN)

try:
    main = WebDriverWait(driver, 20).until(
             EC.presence_of_element_located((By.ID, "main"))
           )
    articles = main.find_elements_by_tag_name("article")
    for article in articles:
        header = article.find_elements_by_tag_name("a")[0]
        #print(header.get_attribute('href'))
        print(header.text)

finally:
    time.sleep(5)
    driver.quit()

The code is working well when extracting the href attribute, but it didn't work for the .text as I got empty lines instead of the headers of the articles

How can I fix that?

I have updated the post and added a snapshot of the first desired title HTTP Methods - GET & POST — YasserKhalil
– YasserKhalil, Commented Apr 28, 2020 at 23:49
Please do not share information as images unless absolutely necessary. See: meta.stackoverflow.com/questions/303812/…, idownvotedbecau.se/imageofcode, idownvotedbecau.se/imageofanexception. — AMC
– AMC, Commented Apr 29, 2020 at 2:33

Peter Mortensen · Accepted Answer · 2022-11-10 18:13:16Z

2

You may mean

print(header.get_attribute('innerHTML'))

To replace the ampersand sign, try

print(header.get_attribute('innerHTML').replace('&amp;', '&'))

Or just use the innerText property:

print(header.get_attribute('innerText'))

Or the textContent Property:

print(header.get_attribute('textContent'))

edited Nov 10, 2022 at 18:13

Peter Mortensen

31.4k22 gold badges110 silver badges134 bronze badges

answered Apr 28, 2020 at 23:49

0m3r

12.5k15 gold badges40 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

YasserKhalil Over a year ago

Amazing. Thanks a lot. How can I get the ampersand instead of GET & POST

YasserKhalil Over a year ago

I tried print(header.get_attribute('innerText')) and this works well. Thanks a lot for great help.

YasserKhalil Over a year ago

Thanks a lot. But what is the problem of using innerText as there may be other special characters.

YasserKhalil Over a year ago

Thank you very much for your awesome help. Best Regards

Collectives™ on Stack Overflow

Get text from a tag name using Python and Selenium

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related