Web scraping with Python BeautifulSoup returning different HTML than expected

Question

I am trying to do some web scraping on a surf report website using BeautifulSoup, but the returned html does not appear to match the html when viewed in a browser, meaning I can't scrape the data that I am looking for. I am trying to scrape from the following website on the "quiver-surf-height" class, which contains the local surf height estimate. https://www.surfline.com/surf-report/paradise-beach/584204214e65fad6a7709cc1

import requests
from bs4 import BeautifulSoup

url = "https://www.surfline.com/surf-report/paradise-beach/584204214e65fad6a7709cc1"
res = requests.get(url)
soup = BeautifulSoup(res.text,"lxml")

print(soup.select(".quiver-surf-height"))

The print statement returns an empty list. Reading through the returned html I found a statement "Please turn JavaScript on and reload the page." I'm following the steps laid out in a class, so I'm not sure how to handle this response. Any input is appreciated!

The page HTML is likely dynamically generated by the browser using JavaScript. To scrape the resulting HTML you will need something like selenium. — tromgy
– tromgy, Commented Sep 25, 2022 at 1:48

baduker · Accepted Answer · 2022-09-25 08:17:33Z

1

As mentioned in the comments, the data you're after is generated dynamically, however, there's an API you can query to get what you want.

All you need it the surf spot id and how much of days-worth data you want. By default it comes for the last 16 days in an 1-hour intervals. But you can change these params too.

For example, this gets last two days of surf height data served per every hour.

import datetime

import requests

surf_sopt_id = "584204214e65fad6a7709cc1"
days = "2"
api_url = f"https://services.surfline.com/kbyg/spots/forecasts/wave?spotId={surf_sopt_id}&days={days}&intervalHours=1"

headers = {
    "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.114 Safari/537.36",
}

data = requests.get(api_url, headers=headers).json()

for day in data["data"]["wave"]:
    _time = (
        datetime
        .datetime
        .fromtimestamp(day['timestamp'])
        .strftime('%Y-%m-%d %H:%M:%S')
    )
    print(f"{_time}")
    surf = day["surf"]
    print(f"Surf: {surf['min']} - {surf['max']}")
    print(f"{surf['humanRelation']}")

Output:

2022-09-25 06:00:00
Surf: 0.9 - 1.4
Waist to shoulder
2022-09-25 07:00:00
Surf: 0.9 - 1.4
Waist to shoulder
2022-09-25 08:00:00
Surf: 0.9 - 1.4
Waist to shoulder
2022-09-25 09:00:00
Surf: 0.9 - 1.2
Waist to chest
2022-09-25 10:00:00
Surf: 0.9 - 1.2
Waist to chest
2022-09-25 11:00:00
Surf: 0.9 - 1.2
Waist to chest
2022-09-25 12:00:00
Surf: 0.9 - 1.2
Waist to chest
2022-09-25 13:00:00
Surf: 0.9 - 1.2
Waist to chest
2022-09-25 14:00:00
Surf: 0.6 - 1.1
Thigh to stomach
2022-09-25 15:00:00
Surf: 0.6 - 1.1
Thigh to stomach
2022-09-25 16:00:00

and more ...

answered Sep 25, 2022 at 8:17

baduker

20.2k9 gold badges43 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Jujimufoo Over a year ago

This is great, thank you! Is this API created by Surfline to cater to web scraping? If not, where did this API come from, and is this something that I can find by myself in the future?

baduker Over a year ago

The API is created by Surfline to serve the content you see on the page. This is a very common practice. You can find the API endpoint in your browser. Go to Devloper Tools -> Network -> XHR.

Collectives™ on Stack Overflow

Web scraping with Python BeautifulSoup returning different HTML than expected

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related