How to scrape data from list inside <script> from website?

Question

This is my first time doing web scraping and I am not sure how to scrape the data from a list of dictionaries inside of script tag. Since the script tag does not have a class, I do not know how to access the content from that specific script tag.

The code so far is:

from bs4 import BeautifulSoup
import requests

source = requests.get('https://champion.gg/statistics/').text


soup = BeautifulSoup(source, 'lxml')

stats = soup.find('script')

This is a small portion of the data that I would to scrape from:

<script>
      matchupData.stats = [{"key":"Ezreal","role":"ADC","title":"Ezreal","general":{"winPercent":0.5046896283323604,"playPercent":0.17104628134933184,"banRate":0.03167301835610511,"experience":8.02309599159886,"kills":6.8780177725887,"deaths":5.2307193357981445,"assists":7.356567425569177,"totalDamageDealtToChampions":23163,"totalDamageTaken":18062,"totalHeal":2607,"largestKillingSpree":8,"minionsKilled":175.73481222027632,"neutralMinionsKilledTeamJungle":6.87918531491211,"neutralMinionsKilledEnemyJungle":1.9552831290134267,"goldEarned":11840,"overallPosition":1,"overallPositionChange":0}},{"key":"LeeSin","role":"Jungle","title":"Lee Sin","general":{"winPercent":0.47603732897085066,"playPercent":0.11936072603416044,"banRate":0.016735176155369534,"experience":11.93326860841424,"kills":6.229476502082094,"deaths":5.414578375966687,"assists":7.773758179654967,"totalDamageDealtToChampions":12340,"totalDamageTaken":26015,"totalHeal":7518,"largestKillingSpree":8,"minionsKilled":25.68465571088638,"neutralMinionsKilledTeamJungle":71.98670806067817,"neutralMinionsKilledEnemyJungle":8.807053093396787,"goldEarned":10255,"overallPosition":6,"overallPositionChange":0}},{"key":"Thresh","role":"Support","title":"Thresh","general":{"winPercent":0.4940108608284812,"playPercent":0.11318544159496746,"banRate":0.012075421458170381,"experience":10.618539868530172,"kills":1.975553333725421,"deaths":5.545197906251838,"assists":12.660628516536297,"totalDamageDealtToChampions":6957

Jack Fleeting · Accepted Answer · 2020-07-10 22:41:48Z

1

The data is in json format. You can get to it this way:

import json

stats = soup.find_all('script')
for s in stats:
    if s.string and "matchupData.stats" in s.string:
        target = s.string.strip().split(" = ")[1][:-1]        
json.loads(target)

Output:

[{'key': 'Ezreal',
  'role': 'ADC',
  'title': 'Ezreal',
  'general': {'winPercent': 0.5046896283323604,
   'playPercent': 0.17104628134933184,
   'banRate': 0.03167301835610511,

etc., etc....

answered Jul 10, 2020 at 22:41

Jack Fleeting

25k6 gold badges27 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

bigbounty Over a year ago

There's a possibility of target being None

Jack Fleeting Over a year ago

@bigbounty Not on this particular link; it's actually there.

Collectives™ on Stack Overflow

How to scrape data from list inside <script> from website?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related