Extract list from a Python script in web scraped html page

Question

I'm new to web scraping and ran into a small road block with the following code:

import requests
from bs4 import BeautifulSoup
url = "www.website.com"
page = requests.get(url)
soup = BeautifulSoup(page.content, "html.parser")
price_scripts = soup.find_all('script')[23]
print(price_scripts)

The scripts that are pulled all appear to be Python scripts. Here's what's printed from the above code:

<script>
        p.a = [0,"6.93","9.34","3.42","7.88"];
        p.output();
</script>

What I'm trying to do is pull the list from this script, but when I attempt it just returns "None".

What are you running that returns "None"? The print(price_scripts) is doing so or? There isn't enough information here. What are the contents of soup.find_all('script')[23]? Can you provide some form of output for what is being stored in price_scripts? Also, in HTML, these are not Python but are Javascript. If the <script> output is what your output is from print, what are you doing that is returning "None" when attempting to grab the list? — ScriptAutomate
– ScriptAutomate, Commented Nov 27, 2019 at 22:57
Is www.website.com the actual website you are trying to scrape? — Claire Furney
– Claire Furney, Commented Nov 27, 2019 at 22:57

Jack Fleeting · Accepted Answer · 2019-11-27 23:05:40Z

1

You should be able to extract the data this way:

target = price_scripts.text

which outputs:

p.a = [0,"6.93","9.34","3.42","7.88"];
    p.output();

At this point you need to resort to string manipulation, by stripping out everything between the brackets, like so:

print(target.text.split('[')[1].split(']')[0])

Note that each use of the split() method creates a list, so you have to choose the correct element from the list. output:

0,"6.93","9.34","3.42","7.88"

answered Nov 27, 2019 at 23:05

Jack Fleeting

25k6 gold badges27 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

ttammountain Over a year ago

that did it! thank you! Don't know why I didn't think about using a split method. cheers!

Collectives™ on Stack Overflow

Extract list from a Python script in web scraped html page

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related