Python script that runs GET request loop throws "KeyError", but not every time

Question

I have created a python script that allows me to run through a loop in order to process several GET requests. I am then pulling the value for each request for a field called "creative" and then printing the final counts of both "True" and "False" values at the end of the script. Here is what my code looks like:

import requests
import json


false_count = 0
true_count = 0

qa_creatives = open("numbers.txt")
arraylist = []
for line in qa_creatives.readlines():
arraylist.extend(line.split())
qa_creatives.close()

arraylist = map(int, arraylist)



authorization_code = raw_input("please enter your authorization code: ")
creatives = arraylist
print "Creative Check script is now running, please wait."

for creative in creatives:
    url = "http://api.wiki123.com/v1.11/creative?id="+str(creative) 

    header = {"Authorization": authorization_code}

    response = requests.get(url, headers=header)

    creative_check = json.loads(response.text)

final_creative_status = creative_check["response"]["creative"]["is_expired"]

#print str(creative) + " expired status: " + str(final_creative_status)

if final_creative_status == False:
    false_count += 1
else:
    true_count += 1

print "Creative Check Complete:"

print str(false_count) + " creatives are still valid"
print str(true_count) + " creatives have expired"

Here is a sample of the JSON data that is returned when making one of these GET requests:

{
'response': {
'count': 1,
'creative': {
  'prime_id': 1092343,
  'off_audit': None,
  'allow_audit': False,
  'allow_ssl_audit': False,
  'audit_feedback': None,
  'audit_status': 'no_audit',
  'backup_upload_status': None,
  'brand': {
    'category_id': 0,
    'id': 1,
    'name': 'Unknown'
  }
  }}}

The odd thing is that this script works some of the time but other times, I get the following error: KeyError: 'creative'. I am puzzled by this because my requests should be returning the same thing each time and as such, the key should never change. Can anyone tell what might be going on here? Any recommendations on how I can debug this issue? If a certain "creative" is breaking the loop, what would be the best way to exclude it? Thanks.

UPDATED Code for my loop:

for creative in creatives:
    url = "http://api.wiki123.com/v1.11/creative?id="+str(creative) 

    header = {"Authorization": authorization_code}

    api_response = requests.get(url, headers=header)

    creative_check = json.loads(api_response.text)

    creative_status = creative_check.get("response", 
    {}).get("creative{}).get("is_expired", None)


if creative_status is None:
    pass

elif creative_status == True:

    true_count += 1 
else : 

    false_count += 1

print "Creative Check Complete: "

print str(false_count) + " creatives are still valid"
print str(true_count) + " creatives have expired"

Note: Now, I don't receive a key error back, but both of my counts for false_count and true_count are always 0 after my script has finished.

Apparently, some times the response does not return a dictionary with creative in it. — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Apr 4, 2017 at 19:02
That's what I am figuring... if a particular "creative" GET request does not return data with the "creative" key in it, is there a good way to just spit those ones out? Or more specifically, put those in a separate list? — user7681184
– user7681184, Commented Apr 4, 2017 at 19:08
... you could catch the KeyError and process it accordingly... no? — juanpa.arrivillaga
– juanpa.arrivillaga, Commented Apr 4, 2017 at 19:09
You can use : if "creative" in creative_check["response"] : ( ..Your code here.. ) else : Separate_list += [ creative_check["response"] ] — t.m.adam
– t.m.adam, Commented Apr 4, 2017 at 19:13
Thanks for the responses. I am still new to Python: are you referring to using "try-except" to just weed out the error cases? — user7681184
– user7681184, Commented Apr 4, 2017 at 19:14

t.m.adam · Accepted Answer · 2017-04-04 22:47:12Z

1

Try this solution :

for creative in creatives:
    url = "http://api.wiki123.com/v1.11/creative?id="+str(creative) 
    header = {"Authorization": authorization_code}
    response = requests.get(url, headers=header)
    creative_check = json.loads(response.text)

    creative_status = creative_check.get("response", {}).get("creative", {}).get("is_expired", None)
    if creative_status is None : 
        ## That means that the key 'is_expired' does not exist. ##
        ## You can handle this case here or just pass ## 
        pass
    elif creative_status : 
        # 'is_expired' is True 
        true_count += 1 
    else : 
        # 'is_expired' is False
        false_count += 1

edited Apr 4, 2017 at 22:47

answered Apr 4, 2017 at 22:41

t.m.adam

15.4k3 gold badges34 silver badges54 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user7681184 Over a year ago

Thanks for all the help. I have added my updated code for my for loop in my original post. Now, I don't get any key error back, but both of my counts for "false_count" and "true_count" always come out to "0" as soon as my script is finished. I must be doing something wrong. Any suggestions? Thanks.

t.m.adam Over a year ago

1 . It seems like you have a syntax error on line 10 : get("creative{}) should be get("creative", {}) 2 . If both false_count and true_count are 0 , it means that key 'is_expired' does not exist. 3 . Could you post some json data to verify that 'is_expired' exists and is either True or False ?

t.m.adam Over a year ago

Also if you leave your if - else out of the for loop, it only applies to the last creative_status

user7681184 Over a year ago

That was it! It was an indentation problem on my part; after adding my if statement into the loop, the script works perfectly. Thanks again!

t.m.adam Over a year ago

I'm glad i could help.

Pentux · Accepted Answer · 2017-04-04 19:18:28Z

0

Your problem is here:

final_creative_status = creative_check["response"]["creative"]["is_expired"]

You want access in objects of json that doesn't exist. Put a try-except KeyError to catch the exception or put conditions like this:

if 'creative' in creative_check["response"].keys():
    if 'is_expired' in creative_check["response"]["creative"].keys():
        blablabla

I'm sure that in your json doesn't exist these objects.

answered Apr 4, 2017 at 19:18

Pentux

4064 silver badges13 bronze badges

Collectives™ on Stack Overflow

Python script that runs GET request loop throws "KeyError", but not every time

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related