How to access a https secured web page using python

Question

So basically the idea is to use python to login to a website and copy the content of a html page that can only be viewed after you have logged in. (under https)

Any suggestions on how to achieve this? Requests? http.client.HTTPSConnection?

I currently have

h1 = http.client.HTTPSConnection(URL)  #question: what exactly should this url page be?
                                  https://accounts.google.com/ServiceLoginhl=en&continue=https://www.google.ca/
                                   or https://google.ca
userAndPass = b64encode(b"usrname:pwd").decode("ascii")
headers = { 'Authorization' : 'Basic %s' %  userAndPass }
#then connect
h1.request('GET', '$THEPAGETHATIWANTTOACCESS', headers=headers)

Thanks a lot!

I've had much better success using Mechanize or requests than httplib. — jedwards
– jedwards, Commented Jun 19, 2013 at 21:46

brice · Accepted Answer · 2013-06-19 21:46:39Z

2

you can use requests

r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
>>> r.status_code
200
>>> r.headers['content-type']
'application/json; charset=utf8'
>>> r.encoding
'utf-8'
>>> r.text
u'{"type":"User"...'
>>> r.json()
{u'private_gists': 419, u'total_private_repos': 77, ...}

answered Jun 19, 2013 at 21:46

brice

25.2k7 gold badges82 silver badges97 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to access a https secured web page using python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related