python requests randomly breaks with JSONDecodeError

Question

I have been debugging for hours why my code randomly breaks with this error: JSONDecodeError: Expecting value: line 1 column 1 (char 0)

This is the code I have:

while True:
    try:
        submissions = requests.get('http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since).json()['submission']['records']
        break
    except requests.exceptions.ConnectionError:
        time.sleep(100)

And I've been debugging by printing requests.get(url) and requests.get(url).text and I have encountered the following "special "cases:

requests.get(url) returns a successful 200 response and requests.get(url).text returns html. I have read online that this should fail when using requests.get(url).json(), because it won't be able to read the html, but somehow it doesn't break. Why is this?
requests.get(url) returns a successful 200 response and requests.get(url).text is in json format. I don't understand why when it goes to the requests.get(url).json() line it breaks with the JSONDecodeError?

The exact value of requests.get(url).text for case 2 is:

{
  "submission": {
    "columns": [
      "pk",
      "form",
      "date",
      "ip"
    ],
    "records": [
      [
        "21197",
        "mistico-form-contacto-form",
        "2018-09-21 09:04:41",
        "186.179.71.106"
      ]
    ]
  }
}

does it work when you try json.loads(requests.get(url).text)? — Danilo Souza Morães
– Danilo Souza Morães, Commented Sep 25, 2018 at 0:49

Henry · Accepted Answer · 2018-09-27 19:51:11Z

11

Looking at the documentation for this API it seems the only responses are in JSON format, so receiving HTML is strange. To increase the likelihood of receiving a JSON response, you can set the 'Accept' header to 'application/json'.

I tried querying this API many times with parameters and did not encounter a JSONDecodeError. This error is likely the result of another error on the server side. To handle it, except a json.decoder.JSONDecodeError in addition to the ConnectionError error you currently except and handle this error in the same way as the ConnectionError.

Here is an example with all that in mind:

import requests, json, time, random

def get_submission_records(client, since, try_number=1):
    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since
    headers = {'Accept': 'application/json'}
    try:
        response = requests.get(url, headers=headers).json()
    except (requests.exceptions.ConnectionError, json.decoder.JSONDecodeError):
        time.sleep(2**try_number + random.random()*0.01) #exponential backoff
        return get_submission_records(client, since, try_number=try_number+1)
    else:
        return response['submission']['records']

I've also wrapped this logic in a recursive function, rather than using while loop because I think it is semantically clearer. This function also waits before trying again using exponential backoff (waiting twice as long after each failure).

Edit: For Python 2.7, the error from trying to parse bad json is a ValueError, not a JSONDecodeError

import requests, time, random

def get_submission_records(client, since, try_number=1):
    url = 'http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since
    headers = {'Accept': 'application/json'}
    try:
        response = requests.get(url, headers=headers).json()
    except (requests.exceptions.ConnectionError, ValueError):
        time.sleep(2**try_number + random.random()*0.01) #exponential backoff
        return get_submission_records(client, since, try_number=try_number+1)
    else:
        return response['submission']['records']

so just change that except line to include a ValueError instead of json.decoder.JSONDecodeError.

edited Sep 27, 2018 at 19:51

answered Sep 25, 2018 at 0:54

Henry

15.9k7 gold badges51 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

mateocam Over a year ago

yes, I was very confused when I was receiving html output. thanks ! this example look great, I'll try it out. hope the header parameter fixes this.

mateocam Over a year ago

I have been testing this implementation and I am getting this error: AttributeError: 'module' object has no attribute 'JSONDecodeError'. I am using python 2.7, is it because of this?

Henry Over a year ago

@mateocam oh yes, in Python 2.7, the error is a ValueError, I'll update my answer, but basically you should except a ValueError instead of json.decoder.JSONDecodeError

mateocam Over a year ago

thanks !! this was so helpful, thank you very much!!

Edgar Over a year ago

I used your answer, but got the following error: "AttributeError: 'dict' object has no attribute 'decoder'". Should it be json.JSONDecodeError instead of json.decoder.JSONDecodeError? (docs.python.org/3/library/json.html#exceptions)

|

vlk · Accepted Answer · 2024-12-03 07:39:09Z

0

Catching ValueError is universal across python versions from py2 to py3

For py3 only is better to catch from already imported requests library: requests.JSONDecoderError, so you don't need to import json library.

answered Dec 3, 2024 at 7:39

vlk

2,8193 gold badges35 silver badges36 bronze badges

Comments

Tharindu Ekanayake · Accepted Answer · 2020-10-30 07:28:14Z

-1

Try this. it might work

while True:
        try:
            submissions = requests.get('http://reymisterio.net/data-dump/api.php/submission?filter[]=form,cs,'+client+'&filter[]=date,cs,'+since).json()['submission']['records']
            sub = json.loads(submissions.text)
            print(sub)
            break
        except requests.exceptions.ConnectionError:
            time.sleep(100)

answered Oct 30, 2020 at 7:28

Tharindu Ekanayake

311 silver badge1 bronze badge

2 Comments

Willy satrio nugroho Over a year ago

At least you can say that you add sub = json.loads(submissions.text) line.

Z4-tier Over a year ago

@Willysatrionugroho is that like a participation trophy?

Collectives™ on Stack Overflow

python requests randomly breaks with JSONDecodeError

3 Answers 3

8 Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

8 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related