python html parsing

Question

I have the following problem:

I would like to parse html files and get links from the html file. I can get links with the following code:

class MyHTMLParser(HTMLParser):
    links=[]
    def __init__(self,url):
        HTMLParser.__init__(self)
        self.url = url

    def handle_starttag(self, tag, attrs):
        try: 
            if tag == 'a':
                for name, value in attrs:
                    if name == 'href':
                        if value[:5]=="http:":
                            self.links.append(value)
        except: 
            pass

But I dont want to get audio files, video files, etc. I only want to get html links. How can I do that?

I can check link ending and if it is particular format I can avoid appending that link to the list. Is there other way? — Ali Ismayilov
– Ali Ismayilov, Commented Nov 17, 2012 at 14:02
stackoverflow.com/questions/717541/parsing-html-in-python?rq=1 — ppaulojr
– ppaulojr, Commented Nov 17, 2012 at 14:19

Community · Accepted Answer · 2017-05-23 12:27:44Z

3

I can check link ending and if it is particular format I can avoid appending that link to the list. Is there other way?

You could look at the 'Content-Type' header:

import urllib2
url = 'https://stackoverflow.com/questions/13431060/python-html-parsing'
req = urllib2.Request(url)
req.get_method = lambda : 'HEAD'    
response = urllib2.urlopen(req)
content_type = response.headers.getheader('Content-Type')
print(content_type)

yields

text/html; charset=utf-8

Many thanks to @JonClements for req.get_method = lambda : 'HEAD'. More info on this and alternate methods for sending a HEAD request can be found here.

edited May 23, 2017 at 12:27

CommunityBot

11 silver badge

answered Nov 17, 2012 at 14:20

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Jon Clements Over a year ago

Instead of using Range - I'd probably go for request = urllib2.Request(someurl); request.get_method = lambda : 'HEAD'; response = urllib2.urlopen(request) and continue from there...

unutbu Over a year ago

@JonClements: Thank you very much for the info. I didn't know you could do that.

unutbu Over a year ago

@JonClements: What does it mean for req.get_method() to return HEAD? The docs seem to say it always returns GET or POST...?

Jon Clements Over a year ago

If a payload is present in the request, then get_method is POST otherwise it's a GET - replacing the method is a very kludgly way of writing requests.head(url)...

Collectives™ on Stack Overflow

python html parsing

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related