Python pulling html table from webpage

Question

The table on this page needs to be scraped daily. We are trying to keep the scraping as simple (robust) as possible so there are no issues with the code running on our server. Would like to steer clear of Selenium:

import requests
import pandas as pd

page_list = pd.read_html('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
page_df = pd.DataFrame(page_list) 
# won't convert to df (ValueError: Must pass 2-d input. shape=(1, 356, 9)

r = requests.get('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
# not sure what to do with response

page_list is close but it is a 3-dimensional list. How can we get this into a 2-dimensional list, or into a pandas dataframe?

Corralien · Accepted Answer · 2022-05-12 02:28:46Z

3

pd.read_html doesn't return a DataFrame but a list of dataframes. Use page_list[0] to get the first dataframe:

page_df = pd.DataFrame(page_list[0])

From the documentation:

Read HTML tables into a list of DataFrame objects.

answered May 12, 2022 at 2:28

Corralien

121k8 gold badges43 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

chitown88 · Accepted Answer · 2022-05-12 15:12:28Z

1

No need to do page_df = pd.DataFrame(page_list[0]). Can actually simply this to page_df = page_list[0]:

page_list = pd.read_html('https://www.ncaa.com/rankings/basketball-women/d1/ncaa-womens-basketball-net-rankings')
page_df = page_list[0]

answered May 12, 2022 at 15:12

chitown88

29.1k6 gold badges34 silver badges67 bronze badges

Collectives™ on Stack Overflow

Python pulling html table from webpage

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related