How can I extract value of variable from script element in Scrapy

Question

I need to extract some data from a website, I found that all I need is exist in <script> element, So I extracted them with this command:

script = response.css('[id="server-side-container"] script::text').get()

And this is the value of script:

    window.bottomBlockAreaHtml = '';
    ...
    window.searchQuery = '';
    window.searchResult = {
  "stats": {...},
  "products": {...},
  ...
  };
    window.routedFilter = '';
  ...
    window.searchContent = '';

What is the best way to get the value of "products" in my python code?

extracting the text with regular expressions and then using json.loads() — Alexander
– Alexander, Commented Jan 7, 2023 at 0:10
you can craft a regex pattern that specific data in the value of the products field. Post the url and I can show you an example — Alexander
– Alexander, Commented Jan 7, 2023 at 1:44

Alexander · Accepted Answer · 2023-01-07 07:07:48Z

1

In your example the best strategy would be to use regex to extract the value of the window.searchResults using regex. Then convert it to a dictionary using json.loads(), and then getting the value from the "products" key of the dictionary.

For example.

import json
import scrapy
import re

class LoplabbetSpider(scrapy.Spider):

    name = "loplabbet"
    start_urls = ["https://www.loplabbet.se/lopning/"]
    pattern = re.compile(r'window\.searchResult = (\{.*?\});', flags=re.DOTALL)

    def parse(self, response):
        for script in response.css("script").getall():
            matches = self.pattern.findall(script)
            if matches:
                results = json.loads(matches[0])
                product = results["products"]
                yield product

answered Jan 7, 2023 at 7:07

Alexander

17.5k5 gold badges15 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I extract value of variable from script element in Scrapy

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related