Scraping data - attributes from a web page

Question

Hello i am new in web scraping and i have a problem. I want to scrape data from this html code:

I want to have the data that belongs inside the

<tr> .. </tr>

tag.

My code is shown as below:

from bs4 import BeautifulSoup
import requests

html_text = requests.get('https://www.basketball-reference.com/leagues/').text
soup = BeautifulSoup(html_text, 'lxml')
rows = soup.select('tr[data-row]')

print(rows)

I am inspired by this thread, but it's returning a empty array. Can anyone help me with this

Just looked at the html and it seems like the attribute data-row is being added at the client side. Which is why your select query returns an empty array — dumbPotato21
– dumbPotato21, Commented Mar 10, 2021 at 12:29
FYI 'to scrap' means to throw away like rubbish. The correct term is scrape — DisappointedByUnaccountableMod
– DisappointedByUnaccountableMod, Commented Mar 10, 2021 at 14:00

dumbPotato21 · Accepted Answer · 2021-03-10 12:37:48Z

1

Like I said in the comment, it looks as if the attribute data-row is being added at the client side - I couldn't find it in the HTML.

A quick and easy way to fix this would be to change your css selector. I came up with something like this

rows = soup.select('tr')
for row in rows:
    if row.th.attrs['data-stat']=='season' and 'scope' in row.th.attrs:
        print(row)

answered Mar 10, 2021 at 12:37

dumbPotato21

5,7035 gold badges25 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

The_elevator Over a year ago

This is helpful and the solution is correct thank you

baduker · Accepted Answer · 2021-03-10 12:22:57Z

0

How about using pandas to make your web-scraping life (a bit) easier?

Here's how:

import pandas as pd
import requests

df = pd.read_html(requests.get('https://www.basketball-reference.com/leagues/').text, flavor="bs4")
df = pd.concat(df)
df.to_csv("basketball_table.csv", index=False)

Output:

answered Mar 10, 2021 at 12:22

baduker

20.2k9 gold badges43 silver badges63 bronze badges

2 Comments

The_elevator Over a year ago

indeed, that's make my life easier thank you

chitown88 Over a year ago

why use requests here? Bit redundant. simply use the url and pandas will do that part: df = pd.read_html('https://www.basketball-reference.com/leagues/')

Collectives™ on Stack Overflow

Scraping data - attributes from a web page

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related