Python getting HTML content via 'requests' returns partial response

Question

I'm reading a web site content using following 3 liners. I used an example domain for sale which doesn't have many content.

url = "http://localbusiness.com/"
response = requests.get(url)
html = response.text

It returns following html content where the website contains more html when you check through view source. Am I doing something wrong here

Python version 2.7

<html><head></head><body><!-- vbe --></body></html>

JRodDynamite · Accepted Answer · 2015-11-17 12:12:15Z

6

Try setting a User-Agent:

import requests

url = "http://localbusiness.com/"

headers = {
    'User-Agent': 'Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36',
    'Content-Type': 'text/html',
}

response = requests.get(url, headers=headers)
html = response.text

The default User-Agent set by requests is 'User-Agent': 'python-requests/2.8.1'. Try to simulate that the request is coming from a browser and not a script.

answered Nov 17, 2015 at 12:12

JRodDynamite

12.7k5 gold badges47 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Satyapal Sharma · Accepted Answer · 2015-11-18 08:10:06Z

@jason answered it correctly so I am extending his answer for the reason

Why It happens

Some DOM elements code changed through the Ajax calls and JavaScript code so that will not be seen in the response of your call (Although it's not the case here as you are already using the view source (ctrl+u) to compare and not view element)
Some sites uses user-agent to know the nature of user (as of desktop or mobile user) and provide the response accordingly (as the probable case here)

Other alternatives

You can use the mechanize module of python to mimic a browser to fool a web site (come handy when the site is using some short of authentication cookies) A small tutorial
Use selenium to actually implement a browser

Collectives™ on Stack Overflow

Python getting HTML content via 'requests' returns partial response

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related