Python Beautifulsoup not finding text of HTML data

Question

I'm trying to make a web request that gets data from a specific website and prints it out on the console. For that I'm using Beautifulsoup and requests. You can see the HTML Data from the website below but I'm not getting the output of the 2 coordinates that you can see below. I don't know if it is important to mention, that the 2 coordinates on the webiste are constantly changing. Thanks for helping me!

HTML:

<div class="cockpitItem">
     <h3>Bodenpunkt</h3>
   <p id="gpt">
    "42,67° Nord" #Coordinate 1 that I want to get
    <br>
    "170,38° Ost" #Coordinate 2 that I want to get
  </p>
</div>

Python Code:

url = "https://www.astroviewer.net/iss/de/"
response = requests.get(url)
soup = BeautifulSoup(response.text, "html.parser")

body = soup.find("p",{"id": "gpt"})

print(f"HTML: {body}")
print(f"Text: {body.text}")

Output:

HTML: <p id="gpt"> <br/> </p> #It finds the right part of the HTML data
Text:

I think the problem is that the page is constantly updating, and I don't think BeatifulSoup is suited for that. — norie
– norie, Commented Mar 7, 2021 at 13:31

QHarr · Accepted Answer · 2021-03-07 13:42:49Z

1

I think it is polling for that data from https://www.astroviewer.net/iss/ws/orbit.php. As specified in this file: https://www.astroviewer.net/iss/javascript/orbit2.js. See function requestOrbitData(). It returns json you can parse.

I would have expected there to be an API which provided this data for you. You could also set-up something to poll for the data but perhaps less frequently.

edited Mar 7, 2021 at 13:42

answered Mar 7, 2021 at 13:31

QHarr

84.5k14 gold badges58 silver badges105 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Matan Malka · Accepted Answer · 2021-03-07 13:32:35Z

0

your code is ok and should work. Ran your snippet on my machine and found out comes back empty, that's why you're not seeing anything.

if you want to view the request use:

print(soup.prettify())

check your GET request.

answered Mar 7, 2021 at 13:32

Matan Malka

363 bronze badges

Comments

PApostol · Accepted Answer · 2021-03-07 13:34:26Z

0

Your code is fine for the given html. Are you sure you're getting to the right parts of the html? See below for reproducible example. Also, as long as the html structure of the page doesn't change, it doesn't matter if the coordinates change.

from bs4 import BeautifulSoup

html_text = '''
<div class="cockpitItem">
     <h3>Bodenpunkt</h3>
   <p id="gpt">
    "42,67° Nord"
    <br>
    "170,38° Ost"
  </p>
</div>
'''

soup = BeautifulSoup(html_text, 'html.parser')
body = soup.find('p', {'id': 'gpt'})
print(f"Text: {body.text.strip()}")

Output:

Text: "42,67° Nord"

    "170,38° Ost"

answered Mar 7, 2021 at 13:34

PApostol

2,3122 gold badges17 silver badges24 bronze badges

Comments

bguernouti · Accepted Answer · 2021-03-07 13:35:54Z

0

This is because the default value of the "gpt div" is empty, and his content is updated by javascript.

Unfortunately, you can't use the requests library in this case, you have to use a library that contains a navigator like, I most known one is Selenium.

answered Mar 7, 2021 at 13:35

bguernouti

1708 bronze badges

Comments

baduker · Accepted Answer · 2021-03-07 14:43:22Z

You can easily simulate what's on that web-page with a simple call to an API.

Here's how:

from datetime import datetime
import time

import requests

while True:
    iss_position = requests.get(
        "http://api.open-notify.org/iss-now.json",
    ).json()["iss_position"]
    output = [
        round(float(iss_position['latitude']), 2),
        round(float(iss_position['longitude']), 2),
        datetime.utcnow().time().strftime('%H:%M:%S'),
    ]
    print("{:>5}° Nord {:>5}° Ost {:>5} UTC".format(*output))
    time.sleep(1)

Sample output:

49.58° Nord 167.23° Ost 14:42:54 UTC
49.61° Nord 167.37° Ost 14:42:56 UTC
49.63° Nord 167.51° Ost 14:42:57 UTC
49.66° Nord 167.64° Ost 14:42:59 UTC
49.69° Nord 167.78° Ost 14:43:00 UTC
49.72° Nord 167.92° Ost 14:43:02 UTC

Collectives™ on Stack Overflow

Python Beautifulsoup not finding text of HTML data

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

Comments

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related