Accessing secured website using SSL certificate in python

Question

I have listed two sets of codes - one that does not work (first one) and the other one that does (second one). Question is how can I make the first code work using the SSL cert, since it does not work without verification. I have a SSL cert that I use to access the web page. I exported to as .pfx (that is the only option). How would I use the cert credentials to access the main page (first code). This would immensely help with my progress! Thanks.

If I hit the main page (where the XMLS links are embedded) with request, I am getting a blank soup (soup.title is blank and so other soup functions). This is the code:

from bs4 import BeautifulSoup
import requests

url = 'https://www.oasis.oati.com/cgi-bin/webplus.dll?script=/woa/woa-planned-outages-report.html&Provider=MISO'
response = requests.get(url, verify=False)
soup = BeautifulSoup(response.content, "html.parser")

However, if I hit the page directly (with no verification) with the specific XML link, I am able to retrieve it using this code:

import requests
import xml.etree.ElementTree as ET
import pandas as pd

url = 'https://www.oasis.oati.com/woa/docs/MISO/MISODocs/CurrentData/2308_Planned_Outages_2017-09-19-18-50-00.xml'
response = requests.get(url, verify=False)
root=ET.fromstring(response.text)

all_records = [] #This is our record list which we will convert into a dataframe
for i, child in enumerate(root): #Begin looping through our root tree
    record = {} #Place holder for our record
    for subchild in child: #iterate through the subchildren to user-agent, Ex: ID, String, Description
        record[subchild.tag] = subchild.text #Extract the text create a new dictionary key, value pair
        all_records.append(record) #Append this record to all_records

df = pd.DataFrame(all_records).drop_duplicates().reset_index(drop=True)

Roman Mindlin · Accepted Answer · 2017-09-29 08:53:04Z

1

You should use cert option in request.get call:

requests.get(url, cert=('/path/client.cert', '/path/client.key'))

Besides you should extract your certificate from pfx archive (read this for example)

answered Sep 29, 2017 at 8:53

Roman Mindlin

8521 gold badge8 silver badges13 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Shyama Sonti Over a year ago

Roman - I was able to export .cer (not .cert) file but not the key. The example does not talk about .key file. Can you help?

Roman Mindlin Over a year ago

Try requests.get(url, verify='/path/to/certfile'). Probably I misunderstood what you want.

Collectives™ on Stack Overflow

Accessing secured website using SSL certificate in python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related