Curl and Python Requests (get) reporting different http status code

Question

I have written a python script to validate url connectivity from a host. What is reporting successful (http 200) in linux curl is reported as a 403 in the python (3.6) requests module.

I'm hoping someone can help me understand the differences here in reported http status codes?

Curl from the Linux command line....

$ curl -ILs https://www.h2o.ai|egrep ^HTTP
HTTP/1.1 200 OK

Python requests module.....

>>> import requests
>>> url = 'https://www.h2o.ai'
>>> r = requests.get(url, verify=True, timeout=3)
>>> r.status_code
403
>>> requests.packages.urllib3.disable_warnings()
>>> r = requests.get(url, verify=False, timeout=3)
>>> r.status_code
403

What headers are you sending? What headers are you receiving? The reason for the 403 is probably explained in more detail in the body. — tripleee
– tripleee, Commented Jul 10, 2018 at 14:59

heemayl · Accepted Answer · 2018-07-10 15:51:30Z

28

It seems the python-requests/<version> User-Agentis being served the 403 response from the site:

In [98]: requests.head('https://www.h2o.ai', headers={'User-Agent': 'Foo bar'})
Out[98]: <Response [200]>

In [99]: requests.head('https://www.h2o.ai')
Out[99]: <Response [403]>

You can contact the site owner if you want or just use a different user-agent via the User-Agent header (like i used above).

How did i debug this:

I have run curl with -v (--verbose) option to check the headers being sent, and then checked out the same with requests using response.request (assuming the response is saved as response).

I did not find any significant difference apart from the User-Agent header; hence, changing the User-Agent header worked as i expected.

edited Jul 10, 2018 at 15:51

answered Jul 10, 2018 at 15:00

heemayl

42.5k10 gold badges86 silver badges87 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user9074332 Over a year ago

This is helpful, thank you. Would you mind sharing a little background on how you reached the conclusion that: "User-Agent is being served the 403 response from the site". E.g. if I was troubleshooting, how would I know that this dummy header was the thing that h2o.ai was expecting?

user9074332 Over a year ago

This response and (especially) the update is a thing of beauty. Thank you!

Collectives™ on Stack Overflow

Curl and Python Requests (get) reporting different http status code

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related