Scraping data- attributes from web page

Question

I am needing some assistance on using python to scrape some data- attributes form a site. I have tried using lxml and requests with no luck and have looked online and I found some articles about using beautiful soup. The only problem is I am not sure how.

Here is what I would like to scrape.

<div class="card-body ">

<div class="card-entry" data-var1="0" data-var2="1" data-var3="20" data-var4="3" data-var5="9">… </div>">
<div class="card-entry" data-var1="1" data-var2="2" data-var3="9" data-var4="2" data-var5="7">… </div>">
<div class="card-entry" data-var1="2" data-var2="3" data-var3="1" data-var4="3" data-var5="3">…</div>
<div class="card-entry" data-var1="3" data-var2="4" data-var3="5" data-var4="2" data-var5="9">…</div>

I am trying to get the data-var5 value out but I have no idea how. Hope someone can help.

Regards,

Hazza

Its returning a blank list s = tree.xpath('//div[@class="card-entry"]/@data-var5') print(s) — TheAmazingHAzza
– TheAmazingHAzza, Commented Jun 15, 2020 at 16:12

Humayun Ahmad Rajib · Accepted Answer · 2020-06-15 17:19:38Z

1

you can use select. you can try it:

from bs4 import BeautifulSoup
html = """
<div class="card-entry" data-var1="0" data-var2="1" data-var3="20" data-var4="3" data-var5="9">… </div>
<div class="card-entry" data-var1="1" data-var2="2" data-var3="9" data-var4="2" data-var5="7">… </div>
<div class="card-entry" data-var1="2" data-var2="3" data-var3="1" data-var4="3" data-var5="3">…</div>
<div class="card-entry" data-var1="3" data-var2="4" data-var3="5" data-var4="2" data-var5="9">…</div> 
"""

soup = BeautifulSoup(html, "lxml")
data_var = soup.select('div[data-var5]')

for data in data_var:
    print("data-var5: " + data['data-var5'])

Output will be:

data-var5: 9
data-var5: 7
data-var5: 3
data-var5: 9

edited Jun 15, 2020 at 17:19

answered Jun 15, 2020 at 17:06

Humayun Ahmad Rajib

1,5581 gold badge12 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

TheAmazingHAzza Over a year ago

Thank you for your help

Artyom Vancyan · Accepted Answer · 2020-06-15 16:31:48Z

1

from bs4 import BeautifulSoup

html = """
<div class="card-entry" data-var1="0" data-var2="1" data-var3="20" data-var4="3" data-var5="9">… </div>
<div class="card-entry" data-var1="1" data-var2="2" data-var3="9" data-var4="2" data-var5="7">… </div>
<div class="card-entry" data-var1="2" data-var2="3" data-var3="1" data-var4="3" data-var5="3">…</div>
<div class="card-entry" data-var1="3" data-var2="4" data-var3="5" data-var4="2" data-var5="9">…</div> 
"""

soup = BeautifulSoup(html, "html.parser")
divs = soup.find_all("div", "card-entry")
for div in divs:
    print(div["data-var5"])

answered Jun 15, 2020 at 16:31

Artyom Vancyan

5,4023 gold badges17 silver badges38 bronze badges

Collectives™ on Stack Overflow

Scraping data- attributes from web page

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related