python - getting SSL error when trying to scrape a webpage

Question

I'm trying to scrape this webpage using Python: https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php

I've been using the requests package. I can "solve" the issue by setting verify=False, however I've read that that's not secure. In other threads, people said to point the requests.get() function to the filepath of the relevant certificate. I exported the certificate from my browser, and then tried that, but with no luck. This

requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',verify='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')

gives the SSL error still

SSLError: HTTPSConnectionPool(host='fftoolbox.scoutfantasysports.com', port=443): Max retries exceeded with url: /football/rankings/PrintVersion.php (Caused by SSLError(SSLError("bad handshake: Error([('SSL routines', 'ssl3_get_server_certificate', 'certificate verify failed')],)",),))

And this

requests.get('https://fftoolbox.scoutfantasysports.com/football/rankings/PrintVersion.php',cert='C:/Users/ericb/Desktop/fftoolboxscoutfantasysportscom.crt')

yields

Error: [('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]

I've done a decent amount of webscraping before, but I've never had to deal with certificates until now. How can I get around this? I should also note that I'd like to put my final Python script and any files it uses onto a public GitHub repo. But I don't want do do anything that would jeopardize my security, like uploading keys or something.

try adding headers to requests

Ravinder Baid
– Ravinder Baid

2018-07-24 15:29:08 +00:00
Commented Jul 24, 2018 at 15:29 — Ravinder Baid
– Ravinder Baid, Commented Jul 24, 2018 at 15:29
I've tried that, and I still get the SSL error

Eric Brenner
– Eric Brenner

2018-07-24 15:31:58 +00:00
Commented Jul 24, 2018 at 15:31 — Eric Brenner
– Eric Brenner, Commented Jul 24, 2018 at 15:31

Community · Accepted Answer · 2020-06-20 09:12:55Z

1

The server is misconfigured, it does not send the intermediate certificate it needs to send. See this report: https://www.ssllabs.com/ssltest/analyze.html?d=fftoolbox.scoutfantasysports.com&hideResults=on

Certificates provided 1 (1776 bytes)

Chain issues Incomplete

Or https://sslanalyzer.comodoca.com/?url=fftoolbox.scoutfantasysports.com

Trusted by Microsoft? No (unable to get local issuer certificate) UNTRUSTED

Trusted by Mozilla? No (unable to get local issuer certificate) UNTRUSTED

With openssl s_client -connect fftoolbox.scoutfantasysports.com:443 -showcerts you can see:

Certificate chain
 0 s:/OU=Domain Control Validated/CN=fftoolbox.scoutfantasysports.com
   i:/C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2

And the webserver should be configured to send the /C=US/ST=Arizona/L=Scottsdale/O=GoDaddy.com, Inc./OU=http://certs.godaddy.com/repository//CN=Go Daddy Secure Certificate Authority - G2 intermediary certificate but it does not.

So, you could contact the website and tells them they are misconfigured. You will not be the only one impacted by that, as the second link shows.

Alternatively, you could add the missing certificate locally as fully trusted, but this kind of lowers your security. You can also download the missing certificate (not the one of the website, the intermediary one) locally and add verify=/path/to/certificate in your requests.get call.

edited Jun 20, 2020 at 9:12

CommunityBot

11 silver badge

answered Jul 24, 2018 at 15:34

Patrick Mevzek

12.9k17 gold badges46 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Eric Brenner Over a year ago

Thanks! How exactly do I download the intermediate certificate? And do I have to process the directory "using the c_rehash utility supplied with OpenSSL" as the requests documentation (docs.python-requests.org/en/master/user/advanced) says?

Patrick Mevzek Over a year ago

You should be able to find the certificate on the CA website. And yes, if you store it in system location, you will need the c_rehash.

Collectives™ on Stack Overflow

python - getting SSL error when trying to scrape a webpage

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related