Using selenium webdriver, how to click on multiple random links in webpage one after another continuously to detect broken links?

Question

I'm trying to write a test script that would essentially test all visible links randomly rather than explicitly specifying them, in a webpage upon login. Is this possible in Selenium IDE/Webdriver, and if so how can I do this?

links = driver.find_element_by_tag_name("a")
list = links[randint(0, len(links)-1)]

The above will fetch all links in the first page but how do I go about testing all or as many links possible without manually adding the above code for each link/page? I suppose what I'm trying to do is find broken links that would result in 500/404s. Any productive way of doing this? Thanks.

Maciej Gol · Accepted Answer · 2013-07-25 18:05:23Z

2

Currently, you can't get the status code legitimately from selenium. You could use selenium to crawl for urls, and other library like requests to check link's status like this (or use solution with title check proposed by @MrTi):

import requests

def find_broken_links(root, driver):
    visited = set()
    broken = set()
    # Use queue for BFS, list / stack for DFS.
    elements = [root]
    session = requests.session()

    while len(elements):
        el = elements.pop()
        if el in visited:
            continue

        visited.add(el)

        resp = session.get(el)
        if resp.status_code in [500, 404]:
            broken.add(el)
            continue

        driver.get(el)
        links = driver.find_element_by_tag_name("a")
        for link in links:
            elements.append(link.get_attribute('href'))

    return broken

answered Jul 25, 2013 at 18:05

Maciej Gol

15.9k4 gold badges35 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

14 Comments

L P Over a year ago

There is one valid url after hitting causes a 401 which would only be caused if there was a session time out or logout. I added a working condition to prevent both cases from happening but it still gives a 401. Any advice on dealing with this? thanks.

Maciej Gol Over a year ago

Are you sure the url should be allowed to be visited by a regular user? Maybe you're hitting a logout url in the meantime that logs you out.

L P Over a year ago

I created a list of urls that won't be hit; that list included logout and reproduced the same result. That particular url does not contain any ID so anyone logged in to the app can access it.

Maciej Gol Over a year ago

It is hard to tell what's the cause without knowledge of the url and the actual code.

L P Over a year ago

That's true. What could be an error possibility though? There's just one particular URL .../my_user/setting/edit/ that when called by selenium webdriver returns a 401 but if I open the same URL in the web browser it returns 200 giving proper html details. I have checked the session IDs as well. The only way I would receive the 401 would be if I was logged out and if I check the previously visited URL in the list, it wasn't a logout.

|

Nathan Merrill · Accepted Answer · 2013-07-25 16:27:35Z

1

When testing for a bad page, I usually test for the title/url. If you are testing a self-contained site, then you should find/create a link that is bad, and see what is unique in the title/URL, and then do something like:

assert(!driver.getTitle().contains("500 Error"));

If you don't know what the title/url will look like, you can check if the title contains "500"/"404"/"Error"/"Page not found" or if the page source contains those as well.

This will probably lead to a bunch of bad pages that aren't really bad (especially if you check for the page source), and will require you to go through each of them, and verify that they really are bad

edited Jul 25, 2013 at 16:27

answered Jul 25, 2013 at 16:15

Nathan Merrill

8,5065 gold badges41 silver badges60 bronze badges

Collectives™ on Stack Overflow

Using selenium webdriver, how to click on multiple random links in webpage one after another continuously to detect broken links?

2 Answers 2

14 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

14 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related