Issues dynamically changing Python 'Requests' header to iterate through API URL endpoints

Question

My issue is as follows:

I am attempting to pull down a list of all email address entries from an API. The data set is so large that it spans multiple API 'pages' with unique URLs. The page number can be specified as a parameter in the API request URL. I wrote a loop to try and collect email information from an API page, add the email addresses to a list, add 1 to the page number, and repeat the process up to 30 pages. Unfortunately it seems like the loop is only querying the same page 30 times and producing duplicates. I feel like I'm missing something simple (beginner here) but please let me know if anyone can help. Code is below:

import requests
import json

number = 1
user_list = [] 
parameters = {'page': number, 'per_page':50} 
response = requests.get('https://api.com/profiles.json', headers=headers, params=parameters)

while number <=30:
    formatted_data = response.json()
    profiles = formatted_data['profiles']
    for dict in profiles:
        user_list.append(dict['email'])
    number = number + 1

print(sorted(user_list))

Seagullie · Accepted Answer · 2020-06-26 22:58:07Z

1

In Python, numbers and strings are passed by value, not by reference. That means you need to update dictionary after every iteration. You also need to place requests.get() inside your loop to get different results

import json

number = 1
user_list = [] 
parameters = {'page': number, 'per_page':50} 

while number <=30:
    response = requests.get('https://api.com/profiles.json', headers=headers, params=parameters)
    formatted_data = response.json()
    profiles = formatted_data['profiles']
    for dict_ in profiles: # try to avoid using keywords for your variables
        user_list.append(dict_['email'])
    number = number + 1
    parameters['page'] = number

print(sorted(user_list))

answered Jun 26, 2020 at 22:58

Seagullie

261 silver badge3 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Issues dynamically changing Python 'Requests' header to iterate through API URL endpoints

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related