Getting href with Selenium and Python

Question

I am trying to get the href with selenium and python.

This is my page:

Some class information are changing depending on which elements. So I am trying basically to get all href for <a id="job____ .....

links.append(job.find_element_by_xpath('//a[@aria-live="polite"]//span').get_attribute(name="href"))

I tried couple of things but can't figure out how. How can i get all my href from the screenshot above?

This looks OK to me. Are you sure your xpath is correct? Have you tried this on other anchor tags? — Dimitar
– Dimitar, Commented Oct 29, 2021 at 8:57
It's not compleetly clear what you want. Do you want to get the href on that spesiffic id or on all <a>s on the page? — Bendik Knapstad
– Bendik Knapstad, Commented Oct 29, 2021 at 8:58
Hi @DimitarVeljanovski NoSuchElementException: no such element: Unable to locate element: {"method":"xpath","selector":"//a[@aria-live="polite"]//span"} — Simon GIS
– Simon GIS, Commented Oct 29, 2021 at 8:59
@BendikKnapstad I want to get the href from all the <a Id = "job_....> which have a href — Simon GIS
– Simon GIS, Commented Oct 29, 2021 at 9:17
@SimonGIS seems to me like the issue is with finding the element and not obtaining an href attribute. You are essentially trying to obtain the href of an element that doesnt exist. Try getting the anchor through its CSS selector or its ID. — Dimitar
– Dimitar, Commented Oct 29, 2021 at 9:18

Wonka · Accepted Answer · 2021-10-29 09:45:52Z

1

Try this, but take care your xpath

"//a[@aria-live="polite"]//span"

will get a span, and i dont see any span with href on your html. Maybe this xpath solve it

//a[./span[@aria-live="polite"]]

links.append(job.find_element_by_xpath('//a[./span[@aria-live="polite"]]').get_attribute("href"))

But it wont get all urls, this with find_elements (return a list), extend your url list with list comprehension

links.extend([x.get_attribute("href") for x in job.find_elements_by_xpath('//a[./span[@aria-live="polite"]]')])

edit 1, other xpath solution

links.extend(["website_base_url"+x.get_attribute("href") for x in job.find_elements_by_xpath('//a[contains(@id, "job_")]')])

edited Oct 29, 2021 at 9:45

answered Oct 29, 2021 at 8:59

Wonka

1,9012 gold badges15 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Simon GIS Over a year ago

with the second solution i am getting this error: links.extend([x.get_attribute("href") for x in job.find_elements_by_xpath('//a[./span[@aria-live="polite"]]')]) TypeError: 'NoneType' object is not iterable

Wonka Over a year ago

Ok, i will edit with other xpath and you let me know if works. I edited it, added too the option to concatenate base_url to get a good link

Simon GIS Over a year ago

cant make it work out. May I share you my code on chat?

Wonka Over a year ago

Sure, open it and we will check it

Kisuro · Accepted Answer · 2021-10-29 08:59:43Z

1

list_of_elements_with_href = wd.find_elements_by_xpath("//a[contains(@href,'')]")
for el_with_href in list_of_elements_with_href :
 links.append(el.with_href.get_attribute("href"))

or if you need more specify:

list_of_elements_with_href = wd.find_elements_by_xpath("//a[contains(@href,'') and contains(@id,'job_')]")

answered Oct 29, 2021 at 8:59

Kisuro

1249 bronze badges

Comments

fam · Accepted Answer · 2021-10-29 10:14:43Z

1

Based on your description and attached image, I think you have got the wrong xpath. Try the following code.

find_links = driver.find_elements_by_xpath("//a[starts-with(@id,'job_')]")

links = []
for link in find_links:
        links.append(link.get_attribute("href"))

Please note elements in find_elements_by_xpath instead of element.

I am unable to test this solution as you have not provided the website.

answered Oct 29, 2021 at 10:14

fam

7417 silver badges18 bronze badges

Collectives™ on Stack Overflow

Getting href with Selenium and Python

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related