python loop requests.get() only returns first loop

Question

Trying to scrape a table from multiple webpages and store in a list. The list prints out the results from the first webpage 3 times.

import pandas as pd
import requests
from bs4 import BeautifulSoup

dflist = []
for i in range(1,4):
    s = requests.Session()
    res = requests.get(r'http://www.ironman.com/triathlon/events/americas/ironman/world-championship/results.aspx?p=' + str(i) + 'race=worldchampionship&rd=20181013&agegroup=Pro&sex=M&y=2018&ps=20#axzz5VRWzxmt3')
    soup = BeautifulSoup(res.content,'lxml')
    table = soup.find_all('table')
    dfs = pd.read_html(str(table))
    dflist.append(dfs)
    s.close()

print(dflist)

jwodder · Accepted Answer · 2018-10-31 22:10:14Z

2

You left out the & after '?p=' + str(i), so your requests all have p set to ${NUMBER}race=worldchampionship, which ironman.com presumably can't make sense of and just ignores. Insert a & at the beginning of 'race=worldchampionship'.

To prevent this sort of mistake in the future, you can pass the URL's query parameters as a dict to the params keyword argument like so:

    params = {
        "p": i,
        "race": "worldchampionship",
        "rd": "20181013", 
        "agegroup": "Pro",
        "sex": "M",
        "y": "2018",
        "ps": "20",
    }

    res = requests.get(r'http://www.ironman.com/triathlon/events/americas/ironman/world-championship/results.aspx#axzz5VRWzxmt3', params=params)

answered Oct 31, 2018 at 22:10

jwodder

58.1k12 gold badges116 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python loop requests.get() only returns first loop

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related