Python web scrape data request error

Question

I'm trying to retrieve the response json data by a web site that I call.

The site is this:

On this page are shown on map some data. With browser debugger I can see the end point

that sends response data json.

I have use this python to try scrape the json response data:

 import requests
 import json

 headers = {
'Host': 'api2.drive-now.com',
'X-Api-Key': 'adf51226795afbc4e7575ccc124face7'
 }
 r = requests.get('https://api2.drive-now.com/cities/42756?expand=full',    headers=headers)
 json_obj = json.loads(r.content)

but I get this error:

hostname doesn't match either of 'activityharvester.com'

How I can retrieve this data?

Thanks

I have tried to call the endpoint that show json response using Postam, and passing into Header only Host and Api-Key. The result is the json that i want. But i i try the same call into python i recive the error hostname doesn't match either of 'activityharvester.com'

Response shows "302 Found" page. You need to correct your script — Greg Eremeev
– Greg Eremeev, Commented Feb 27, 2017 at 14:20
I get a "bad request" with your new code, bit it works once again adding 'User-Agent': 'Mozilla/5.0' in the headers. And your problem is not related to the API key, otherwise you would get "code : 403," message ":" Permission denied. " — Ettore Rizza
– Ettore Rizza, Commented Feb 27, 2017 at 15:30

Ettore Rizza · Accepted Answer · 2017-02-27 14:15:47Z

1

I don't understand your script, nor your question. Why two requests and three headers ? Did you mean something like this ?

import requests
import json

headers = {
'User-Agent': 'Mozilla/5.0',
'X-Api-Key':'adf51226795afbc4e7575ccc124face7',
}

res = requests.get('https://api2.drive-now.com/cities/4604?expand=full', headers=headers, allow_redirects=False)

print(res.status_code, res.reason)

json_obj = json.loads(res.content)

print(json_obj)

edited Feb 27, 2017 at 14:15

answered Feb 27, 2017 at 13:29

Ettore Rizza

2,8402 gold badges15 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Craicerjack Over a year ago

Im curious, if you dont understand his question how can you post an answer?

Ettore Rizza Over a year ago

@Craicerjack I don't undestand what "activityharvester.com" is and I'm not sure if the OP want to retrieve a single Json file or to find a method for scraping the entire website. In any case, this script will be useful to him.

APPGIS Over a year ago

Hi Ettore, sorry but i call 2 different end point for this reason. The response data that i would like use are show by a website that execute a internal get request. So if i call the second end point without the first the response is null. I have try your code but i have my same error: hostname 'api2.drive-now.com' doesn't match either of 'activityharvester.com'

Ettore Rizza Over a year ago

What is activityharvester.com ?

Ettore Rizza Over a year ago

@ APPGIS I've re-tried changing the variable names and I get once again "200 OK" and a Json. I try to understand why you get this response and me another one. I'm pretty sure that requests doesn't use any cache.

|

Collectives™ on Stack Overflow

Python web scrape data request error

1 Answer 1

6 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

6 Comments

Your Answer

Sign up or log in

Post as a guest

Related