Fetch HTML From URL in Python

Question

I am trying to read the HTML contents of a URL with Python. To fetch the HTML contents of a URL, would I use the module wget, urllib or a different module entirely?

After Answers: I will use the urllib module since that comes with the default Python 2.7 build, and I can't download external modules from this computer.

List of Modules That Fetch URL Contents:

Wget
Beautiful Soup
Urllib
Requests

Yes. There are multiple choices (add requests to the list) and you can use any of them. — tdelaney
– tdelaney, Commented Oct 4, 2016 at 16:50
Can you show us what you've tried? There are a lot of questions and answers on this site about this topic. Do you have a specific question? — MooingRawr
– MooingRawr, Commented Oct 4, 2016 at 16:51
@MooingRawr I will upload some code soon. I'll try my 1st attempt at fetching html elements from a URL with wget and post it once I'm done. — jerbear
– jerbear, Commented Oct 4, 2016 at 16:55
@345243lkj I have not tried BeautifulSoup, but I will look into it, and decide if it's the module I should use for this task. — jerbear
– jerbear, Commented Oct 4, 2016 at 16:56

Robᵩ · Accepted Answer · 2016-10-04 17:00:05Z

5

Here is a sample to get you started with requests:

import requests

resp = requests.get('http://httpbin.org/get')
if resp.ok:
    print (resp.text)
else:
    print ("Boo! {}".format(resp.status_code))
    print (resp.text)

edited Oct 4, 2016 at 17:00

answered Oct 4, 2016 at 16:57

Robᵩ

170k20 gold badges251 silver badges323 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Rafael Over a year ago

This is the simplest. Replace print resp.text with print(resp.text) if you're using Python3.x

Robᵩ Over a year ago

Aside: httpbin.org is an excellent resource for testing clients.

Robᵩ Over a year ago

Thanks, @345243lkj - I fixed the sample and the comment.

Collectives™ on Stack Overflow

Fetch HTML From URL in Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related