Scraping onclick tables in Python

Question

I am trying to scrape Steams hard- and software survey for December 2020 (the table at the bottom of the page). The table is expandable by clicking on one of the parents (e.g. "OS Version"). My goal is to access the tables within these parents.

https://store.steampowered.com/hwsurvey#main_stats

So far I have tried to retrieve this information with requests and BeautifulSoup (with different parsers) but Beautifulsoup always returns an TypeError: 'NoneType' object is not callable. After searching unsuccessfully for an API, I tried Selenium in combination with pd.read_html(). With this approach I was able to access at least the y-labels from the charts above the table but not the desired table below:

import pandas as pd
from selenium import webdriver

url = "https://store.steampowered.com/hwsurvey#main_stats"
opt = webdriver.FirefoxOptions()
opt.add_argument('-headless')
driver = webdriver.Firefox(options=opt)
driver.get(url)

pd.read_html(driver.page_source)

I am appreciative of any suggestion that may help me to overcome this problem.

swillerd · Accepted Answer · 2021-01-21 22:16:06Z

2

You can try again from this code:

import requests
from bs4 import BeautifulSoup

url="https://store.steampowered.com/hwsurvey#main_stats"
headers = {"User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6),AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.98 Safari/537.36"}

response = requests.get(url, headers=headers).text
soup =  BeautifulSoup(response,"html.parser")
names=soup.find_all("div",{"class":"stats_col_left"})
os=soup.find_all("span",{"id":"osversion_val_1_on"})
#val=soup.find_all("div",{"class":"stats_col_mid"})
list_names=list()
for i in names:
    i=i.text
    i=i.strip("\xa0 ")
    list_names.append(i)
    list_names = [x for x in list_names if x]

answered Jan 21, 2021 at 22:16

swillerd

362 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

qualitaetsmuell Over a year ago

With your help I was able to resolve my problem. Thank you very much!

Collectives™ on Stack Overflow

Scraping onclick tables in Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related