Python Web Scraping Beautiful Soup Parse Javascript Element and process variable

Question

I am trying to retrieve the variable in json format to python.

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
source = requests.get(url).text

soup = BeautifulSoup(source, 'lxml')
print(soup.findAll('script')[10])

The code above produces:

<script>window.ShopifyAnalytics = window.ShopifyAnalytics || {};
window.ShopifyAnalytics.meta = window.ShopifyAnalytics.meta || {};
window.ShopifyAnalytics.meta.currency = 'SGD';
var meta = {"product":{"id":4738795503687,"gid":"gid:\/\/shopify\/Product\/4738795503687","vendor":"adidas Originals","type":"footwear","variants":[{"id":32432939204679,"price":18000,"name":"+ Jonah Hill Superstar - 7","public_title":"7","sku":"FW7577"},{"id":32432939237447,"price":18000,"name":"+ Jonah Hill Superstar - 7.5","public_title":"7.5","sku":"FW7577"},{"id":32432939270215,"price":18000,"name":"+ Jonah Hill Superstar - 8","public_title":"8","sku":"FW7577"},{"id":32432939302983,"price":18000,"name":"+ Jonah Hill Superstar - 8.5","public_title":"8.5","sku":"FW7577"},{"id":32432939335751,"price":18000,"name":"+ Jonah Hill Superstar - 9","public_title":"9","sku":"FW7577"},{"id":32432939368519,"price":18000,"name":"+ Jonah Hill Superstar - 9.5","public_title":"9.5","sku":"FW7577"},{"id":32432939401287,"price":18000,"name":"+ Jonah Hill Superstar - 10","public_title":"10","sku":"FW7577"},{"id":32432939434055,"price":18000,"name":"+ Jonah Hill Superstar - 10.5","public_title":"10.5","sku":"FW7577"},{"id":32432939466823,"price":18000,"name":"+ Jonah Hill Superstar - 11","public_title":"11","sku":"FW7577"},{"id":32432939499591,"price":18000,"name":"+ Jonah Hill Superstar - 11.5","public_title":"11.5","sku":"FW7577"}]},"page":{"pageType":"product","resourceType":"product","resourceId":4738795503687}};
for (var attr in meta) {
  window.ShopifyAnalytics.meta[attr] = meta[attr];
}</script>

Basically I want to retrieve the meta variable's content and access the key value pairs in python.

How do I do that?

Does this answer your question? Parsing variable data out of a javascript tag using python — metatoaster
– metatoaster, Commented Jul 13, 2020 at 6:52
Use this regex - meta = ({(.*)}) to get the json. regex101.com/r/CTaxPX/1 — bigbounty
– bigbounty, Commented Jul 13, 2020 at 6:57
@bawagoc25 the linked thread gave you the general idea on how you should approach the solution, but sometimes I guess nice people will freely hand out a complete solution specifically catered to your case. — metatoaster
– metatoaster, Commented Jul 13, 2020 at 7:12

bigbounty · Accepted Answer · 2020-07-13 07:00:39Z

import re, json

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
source = requests.get(url).text

soup = BeautifulSoup(source, 'lxml')
script = soup.findAll('script')[10].text.strip()

p = re.compile("meta = ({(.*)})")
m = p.search(script)
json_str = m[1]

json_data = json.loads(json_str)
print(json_data)

Output:

{'product': {'id': 4738795503687,
  'gid': 'gid://shopify/Product/4738795503687',
  'vendor': 'adidas Originals',
  'type': 'footwear',
  'variants': [{'id': 32432939204679,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 7',
    'public_title': '7',
    'sku': 'FW7577'},
   {'id': 32432939237447,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 7.5',
    'public_title': '7.5',
    'sku': 'FW7577'},
   {'id': 32432939270215,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 8',
    'public_title': '8',
    'sku': 'FW7577'},
   {'id': 32432939302983,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 8.5',
    'public_title': '8.5',
    'sku': 'FW7577'},
   {'id': 32432939335751,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 9',
    'public_title': '9',
    'sku': 'FW7577'},
   {'id': 32432939368519,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 9.5',
    'public_title': '9.5',
    'sku': 'FW7577'},
   {'id': 32432939401287,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 10',
    'public_title': '10',
    'sku': 'FW7577'},
   {'id': 32432939434055,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 10.5',
    'public_title': '10.5',
    'sku': 'FW7577'},
   {'id': 32432939466823,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 11',
    'public_title': '11',
    'sku': 'FW7577'},
   {'id': 32432939499591,
    'price': 18000,
    'name': '+ Jonah Hill Superstar - 11.5',
    'public_title': '11.5',
    'sku': 'FW7577'}]},
 'page': {'pageType': 'product',
  'resourceType': 'product',
  'resourceId': 4738795503687}}

Gregor · Accepted Answer · 2020-07-13 14:32:04Z

0

This works for me

import json
import requests

url = "https://limitededt.com/collections/footwear/products/adidas-originals-jonah-hill-superstar-fw7577"
json_data = session.get(url + '.json').json()

answered Jul 13, 2020 at 14:32

Gregor

6921 gold badge7 silver badges20 bronze badges

Collectives™ on Stack Overflow

Python Web Scraping Beautiful Soup Parse Javascript Element and process variable

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related