Parsing Javascript In Python

Question

I usually use Beautiful Soup to parse html that I need, but I came across some Javascript that I would like to get from here.

 <script>
function Model(){
    this.players = [{".....data......:""}];...etc

I tried to load it like...

import json
scrape_url = "https://swishanalytics.com/optimus/nba/daily-fantasy-projections?date=2016-12-15"

result = json.loads(scrape_url)

But I get "No Json Can Be Decoded". Not sure how to go about this.

As your tag says, you know you should be using BeautifulSoup, but then why is your code using json.loads(URL)? Please check json document, it does not do what you think it does — Moinuddin Quadri
– Moinuddin Quadri, Commented Dec 16, 2016 at 20:23
so you want to extract data from within the script? You would first need to isolate it I.E. get just the string {".....data......:"} then use that with json.loads — Tadhg McDonald-Jensen
– Tadhg McDonald-Jensen, Commented Dec 16, 2016 at 20:25

Alex Hall · Accepted Answer · 2016-12-16 20:31:48Z

14

You can extract JSON from arbitrary text with the jsonfinder library:

from jsonfinder import jsonfinder
import requests

scrape_url = "https://swishanalytics.com/optimus/nba/daily-fantasy-projections?date=2016-12-15"
content = requests.get(scrape_url).text
for _, __, obj in jsonfinder(content, json_only=True):
    if (obj and
            isinstance(obj, list) and
            isinstance(obj[0], dict) and
            {'player_id', 'event_id', 'name'}.issubset(obj[0])
            ):
        break
else:
    raise ValueError('data not found')

# Now you can use obj
print(len(obj))
print(obj[0])

answered Dec 16, 2016 at 20:31

Alex Hall

36.2k5 gold badges63 silver badges98 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Nabin Over a year ago

Works pretty fine :)

Collectives™ on Stack Overflow

Parsing Javascript In Python

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related