How can I go about getting the JSON from a https link using Python?

Question

I am learning Python, and I want to start using Twitter's streaming API. I know how to build a very basic application that reads the contents of a page using urllib2 and have done this successfully in the past. However, the secure connection https:// that this particular Twitter streaming API requires doesn't appear to work with urllib2.

I have tried to fetch the JSON from the following link: https://stream.twitter.com/1.1/statuses/filter.json?locations=-122.75,36.8,-121.75,37.8

By writing the following:

import urllib2
url = "https://stream.twitter.com/1.1/statuses/filter.json?locations=-122.75,36.8,-121.75,37.8"
print urllib2.urlopen(url).read()

The traceback, though, says I am not authorized; I presumed this is because of the HTTPS SSL connection, and research supported this. So I guess my question is what is the best way to go about getting the JSON from a link requiring such SSL connection? I also tried curl, but only get error 401 again.

Below is the traceback in its entirety.

Traceback (most recent call last): File "", line 1, in File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 126, in urlopen File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 400, in open File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 513, in http_response File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 438, in error File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 372, in _call_chain File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 521, in http_error_default urllib2.HTTPError: HTTP Error 401: Unauthorized

Many thanks in advance.

Ah. It does say that it requires a Twitter user to be logged in. I suppose that is why the stream works when I type the url into my browser (because I am logged into Twitter). It does not require OAuth, so could I just add my login credentials to the header of my Python file? — zch
– zch, Commented Nov 13, 2012 at 5:02

Community · Accepted Answer · 2017-05-23 12:27:45Z

2

As others mention, you need to authorize. Here's how to do it:

See Twitter API with urllib2 in python

import urllib2
import base64

request = urllib2.Request( 'https://stream.twitter.com...' )
request.add_header( 'Authorization', 'Basic ' + base64.b64encode( username + ':' + password ) )
response = urllib2.urlopen( request )

edited May 23, 2017 at 12:27

CommunityBot

11 silver badge

answered Nov 13, 2012 at 5:06

Purrell

13k16 gold badges61 silver badges70 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

zch Over a year ago

Thank you! I just tried that answer from the question you linked to and it seemed to work (terminal seems to be processing, though I don't know how long this should take – print the response upon reading it)

Bruno Over a year ago

The problem with that approach is that it will work, but it won't be secure, since urllib2 doesn't make any verification of the certificate or the host name.

Purrell Over a year ago

@Bruno ... I guess it's insecure if you're paranoid that someone is impersonating twitter or sniffing your requests to the twitter api. thejosephturner.com/blog/2011/03/19/… has a workaround

Bruno Over a year ago

It's insecure if there's a man-in-the-middle, which is one of the things HTTPS aims to protect against (beside passive eavesdropping).

Purrell Over a year ago

@Bruno. Right, that's what I said ("someone impersonating twitter or sniffing your requests"). Anyway, follow the link if you want https.

Xymostech · Accepted Answer · 2012-11-13 05:00:53Z

1

It's not the SSL connection that's not authorizing, its that you haven't explicitly authorized with the system before making this call. You need to perform some sort of authorization with the server before making your request.

answered Nov 13, 2012 at 5:00

Xymostech

9,8803 gold badges36 silver badges45 bronze badges

Collectives™ on Stack Overflow

How can I go about getting the JSON from a https link using Python?

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related