Scraping "data" attributes in HTML with Python

Question

I'm currently learning basic web scraping and I'm curious about .find() and if it works with attributes that aren't class or id.

As a practice, I'm trying to get data from the nyc.gov website and export a list of the NYPD precincts + their addresses into a .csv file. In the HTML on the nyc.gov site, the precinct numbers and addresses are labeled as data-label="Precinct" and "Address" respectively, but the guide I'm using only shows how to use class_="etc" with .find(), and I don't know if data attributes are different.

This is my code so far:

import pandas as pd
import requests
import re
from bs4 import BeautifulSoup

url = "http://wgetsnaps.github.io/nyc.gov--nypd-videos/html/nypd/html/home/precincts.shtml.html"
response = requests.get(url)
soup = BeautifulSoup(response.content, "html.parser")
precincts = soup.find_all("td")

numbers = []
addresses = []

for precinct in precincts:
    number = precinct.find("td", data-label_="Precinct").text.strip()
    address = precinct.find("td", data-label_="Address").text.strip()
    numbers.append(number)
    address.append(address)

I'll compile it into a dataframe and add an export into .csv later. Jupyter Notebook says, for "data-label_": SyntaxError: expression cannot contain assignment, perhaps you meant "=="?

I did not mean "==", but I understand its confusion. Any tips?

Does this answer your question? Scrape content of element with data- attribute - Python BeautifulSoup — esqew
– esqew, Commented Aug 23, 2023 at 23:52
Read and try pandas.read_html : [link]("scrapehero.com/web-scraping-with-pandas/") — Parvez Alam
– Parvez Alam, Commented Aug 24, 2023 at 0:34

Parvez Alam · Accepted Answer · 2023-08-24 00:36:14Z

0

Try:

import pandas as pd

df_tables = pd.read_html("http://wgetsnaps.github.io/nyc.gov--nypd-videos/html/nypd/html/home/precincts.shtml.html")
print(df_tables[0])

Output:

answered Aug 24, 2023 at 0:36

Parvez Alam

1398 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ggorlen Over a year ago

Thanks for the answer. Please see Why should I not upload images of code/data/errors?. That DF could be easily shown as text, or dumped to CSV. An image eats up bandwidth and is less accessible, among other things.

Collectives™ on Stack Overflow

Scraping "data" attributes in HTML with Python

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related