python selenium loop through some links

Question

I have an array of links that I am trying to access to every link and print something from it, then return to the main page and access the second link, then do the same until I finish all links in the array.

What happens is that the first link is the only one that works, like if all the links in the array are gone. I get the error:

File "e:\work\MY CODE\scraping\learn.py", line 25, in theprint link.click()

    from selenium import webdriver
from selenium.webdriver.common import keys
#it make us able to use keybored keys like enter ,esc , etc....
from selenium.webdriver.common.keys import Keys
import time

#make us can wait for event to happen until run the next line of code
from selenium.webdriver.common.by import By
from selenium.webdriver.remote import command
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

#get the google chrome driver path
PATH="E:\work\crom\chromedriver.exe"
#pass the pass to selenium webdriver method
driver=webdriver.Chrome(PATH)
#get the link of the site we want
driver.get("https://app.dealroom.co/companies.startups/f/client_focus/anyof_business/company_status/not_closed/company_type/not_government%20nonprofit/employees/anyof_2-10_11-50_51-200/has_website_url/anyof_yes/slug_locations/anyof_france?sort=-revenue")

#wait for the page to load
time.sleep(5)
#get the links i want to get info from
the_links=driver.find_elements_by_class_name("table-list-item")

#function that go the link and print somethin and return to main page
links=[]
the_links=driver.find_elements_by_class_name("table-list-item")
for link in the_links:
      links.append(link.get_attribute('href'))

for link in links:
      driver.get(link)
      website=driver.find_element_by_class_name("item-details-info__url")
      print(website.text)
      driver.back()
      time.sleep(3)

Are you getting a stale element reference? You can't define an element, switch pages, then use that element again. It looks like that's what you are trying to do, which would cause a stale element error.' — C. Peck
– C. Peck, Commented Jun 2, 2021 at 23:37

Muhammad Suleman · Accepted Answer · 2021-06-03 04:20:04Z

1

Your code will throw a stale element reference error because when you navigate to the next page, the variable holding any elements of the previous page will become unusable.

So what you need to do is either store all elements in array and then loop through it like this:

links=[]
the_links=driver.find_elements_by_class_name("table-list-item")
for link in the_links:
    links.append(link.get_attribute('href'))

for link in links:
    driver.get(link)
    print("do something on this link")

Or you can use a while loop in your current and after driver.back() again populate the the_links variable.

answered Jun 3, 2021 at 4:20

Muhammad Suleman

745 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Karim Ahmed Over a year ago

okay, I tried this code but the get() method doesn't work do you know why? that what I added to it : for link in links: driver.get(link) website=driver.find_element_by_class_name("item-details-info__url") print(website.text) driver.back() time.sleep(3)

Muhammad Suleman Over a year ago

If get method not working then you can use JavaScript executor to load new URL. For example, driver.execute_script("window.location.href = {}".format(link)). This will take you to new url.

sahil goyal · Accepted Answer · 2021-06-03 07:04:56Z

0

Karim, is the element with class_name "item-details-info__url" present on all pages? Also, what error does get() method throwing?

answered Jun 3, 2021 at 7:04

sahil goyal

135 bronze badges

1 Comment

Karim Ahmed Over a year ago

yes, the class is present on all pages, the get() method simply doesn't work at all and I get an error in the line it exist in I edited the code now you can see it

Collectives™ on Stack Overflow

python selenium loop through some links

2 Answers 2

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related