Create a script to catch links on a webpage with python 3

Question

I have to catch all the links of the topics in this page: https://www.inforge.net/xi/forums/liste-proxy.1118/

I've tried with this script:

import urllib.request
from bs4 import BeautifulSoup

url = (urllib.request.urlopen("https://www.inforge.net/xi/forums/liste-proxy.1118/"))
soup = BeautifulSoup(url, "lxml")

for link in soup.find_all('a'):
    print(link.get('href'))

but it prints all the links of the page, and not just the links of the topics as I'd like to. could you suggest me the fast way to do it? I'm still a newbie, and i've started learning python recently.

Aran-Fey · Accepted Answer · 2016-07-18 15:19:02Z

2

You can use BeautifulSoup to parse the HTML:

from bs4 import BeautifulSoup
from urllib2 import urlopen

url= 'https://www.inforge.net/xi/forums/liste-proxy.1118/'
soup= BeautifulSoup(urlopen(url))

Then find the links with

soup.find_all('a', {'class':'PreviewTooltip'})

answered Jul 18, 2016 at 15:19

Aran-Fey

44k13 gold badges113 silver badges161 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Allexj Over a year ago

thanks for answer but if I follow your method, and I print (soup) it gives me the source of the page, and not the links of topics :\

Aran-Fey Over a year ago

This gives you tag objects. To get the urls as strings, use [tag.get('href') for tag in soup.find_all('a', {'class':'PreviewTooltip'})

Allexj Over a year ago

ok. now i got the links I wanted, but they are inside html codes.

<a class="PreviewTooltip" data-previewurl="threads/dichvusocks-us-23h10-pm-update-24-24-good-socks.455661/preview" href="threads/dichvusocks-us-23h10-pm-update-24-24-good-socks.455661/" title="">[DICHVUSOCKS.US] 23h10 PM UPDATE 24/24- Good Socks</a>

but it is a good step forward! :)

Allexj Over a year ago

man, thanks to you, I solved. this is the final code:

from bs4 import BeautifulSoup import urllib.request  url= 'https://www.inforge.net/xi/forums/liste-proxy.1118/' soup= BeautifulSoup(urllib.request.urlopen(url), "lxml")  for tag in soup.find_all('a', {'class':'PreviewTooltip'}):     print(tag.get('href'))

thanks again

Collectives™ on Stack Overflow

Create a script to catch links on a webpage with python 3

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related