How to parse a JavaScript object from a HTML page I crawl?

Question

I'm trying to index a food recipes page, and the actual recipe is stored as an object within a JavaScript in the page.

One example URL: http://www.dagbladet.no/mat/oppskrift/bakt-potet-med-romme-og-blamuggostdressing

If I open the developer tool in the browser and type:

console.dir(food.recipeItem.title)

I get the title back:

"Bakt potet med rømme- og blåmuggostdressing"

All nice and dandy, and just what I need. But how can I get ahold of that script and parse it within a Node.js application? Cheerio will maybe help me find the script, but not do much more than that? Or maybe it will? I'm not sure how to do it, and not what is the most computation-effective answer. Or most solid.

Glorfindel · Accepted Answer · 2022-03-07 09:03:50Z

1

It's pretty easy, all you have to do is parse the returned HTML. If you inspect the returned HTML (view-source:http://www.dagbladet.no/mat/oppskrift/bakt-potet-med-romme-og-blamuggostdressing), you will find a script tag which contains all information you need in several javascript variables. These variables holds JSON data. Since the script is hardcoded directly into the HTML document, and not obtained by XHR or similar, parsing the HTML is the only way of doing this.

So basically you have these 3 steps:

1. send HTTP GET request to the link above

2. parse the HTML string to extract the script tag by using some library (check this link to decide which library to use).

3. parse the javascript string (extracted script from step 2) to extract JSON data. Check UglifyJS library for Node.js

edited Mar 7, 2022 at 9:03

Glorfindel

22.8k13 gold badges97 silver badges124 bronze badges

answered Nov 25, 2016 at 6:28

Borna

1,1992 gold badges12 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Espen Klem Over a year ago

Thanks @Borna ! Step 2 is the part I'm struggling with. I'll handle finding the script tag with cheerio, but do I then need to parse the content of the script? And how to do that?

Borna Over a year ago

Hi, I made some changes, hope it helps

Espen Klem Over a year ago

Thanks, I'll try that!

Collectives™ on Stack Overflow

How to parse a JavaScript object from a HTML page I crawl?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related