Python Selenium - fetching titles and links in a loop

Question

I have trying to do this using Selenium in Python:

title1

link1

title2

link2 ...

Currently I have this code:

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
import time

PATH = r"C:\Users\Desktop\py\msedgedriver.exe"
driver = webdriver.Edge(PATH)

driver.maximize_window()
driver.get('https://www.google.com/')
searchbar = driver.find_element(by=By.CLASS_NAME, value='gLFyf')
searchbar.send_keys('selenium')
searchbar.send_keys(Keys.RETURN)
titles = driver.find_elements(by=By.CLASS_NAME, value='LC20lb')
links = driver.find_elements(by=By.TAG_NAME, value='a')
for link in links:
    href = link.get_attribute('href')
    print(href)
for title in titles:
    print(title.text)

time.sleep(5)
driver.quit()

However, the links printed out are Google search links, instead of the links of the websites themselves. Also, all the links are printed out before the titles (I understand why this happens but don't know how to fix that)

May I ask what are the ways to solve the 2 problems? Thank you in advance.

@Zero yes, like title1 (enter) link1 (enter) title2 (enter) link2... but not title1234 then link1234 — Questions
– Questions, Commented May 19, 2022 at 7:33
There aren't equal numbers or titles and links so for the links without titles it should be blank? — Zero
– Zero, Commented May 19, 2022 at 7:38

Zero · Accepted Answer · 2022-05-19 07:43:46Z

1

Replace the for loop in your code with,

for i, link in enumerate(links):
    try:
        print(titles[i].text)
    except:
        pass
    print(link.get_attribute("href"));print()

Output -

Selenium Tutorial for Beginners: Learn WebDriver & Testing
https://www.google.com/search?q=selenium&source=lnms&tbm=bks&sa=X&ved=2ahUKEwiHuNKqiOv3AhXITWwGHZXxBlwQ_AUoAXoECAIQAw

Selenium: Definition, How it works and Why you need it
https://www.google.com/search?q=selenium&source=lnms&tbm=isch&sa=X&ved=2ahUKEwiHuNKqiOv3AhXITWwGHZXxBlwQ_AUoAnoECAIQBA

What Is Selenium � A Tutorial on How to Use ... - LambdaTest
https://www.google.com/search?q=selenium&source=lnms&tbm=vid&sa=X&ved=2ahUKEwiHuNKqiOv3AhXITWwGHZXxBlwQ_AUoA3oECAIQBQ

answered May 19, 2022 at 7:43

Zero

1,9091 gold badge10 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Selenium - fetching titles and links in a loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related