Return Javascript variable data from external page using NodeJS

Question

I am trying to send a request to a page and grab the entire DOM. Basically a crawl. On this website, there is a variable loaded directly into the HTML (not a script file) with some data. Using my NodeJS backend, which I use request with, how would I request this page and return the variable's data? Here's an example:

http://some-page.com/index.html

<html>
    <head>
        <script>
            var my_var = {
                title: "Good title",
                description: "Nice description",
                page: 5
            };
        </script>
    </head>
</html>

If I visit the website, open the console and type in my_var I can see the content in the console, so it's a global variable.

How can I do something like this? I can use another request library if that is needed.

Perhaps in the response callback, something to the effect of eval(body.match(/<script>.*<\/script>/).join('\n')) where body is the third parameter of the callback (per the example in the repo's readme). — wbadart
– wbadart, Commented Jul 13, 2017 at 18:34
@wilusdaman So executing all the JS on the page into the Node backend? Surely there's a better way. — MortenMoulder
– MortenMoulder, Commented Jul 13, 2017 at 18:36
@MortenMoulder there surely is. Just wanted to give what's probably the simplest solution (and ignore the horrendous security implications of that approach) — wbadart
– wbadart, Commented Jul 13, 2017 at 18:38

Hanzhao Deng · Accepted Answer · 2017-07-13 18:49:09Z

2

You are looking for jsdom: https://github.com/tmpvar/jsdom

const dom = new JSDOM(`<body>
  <script>document.body.appendChild(document.createElement("hr"));</script>
</body>`, { runScripts: "dangerously" });

// The script will be executed and modify the DOM:
dom.window.document.body.children.length === 2;

it also come with a Virtual Console

Virtual consoles

Like web browsers, jsdom has the concept of a "console". This records both information directly sent from the page, via scripts executing inside the document, as well as information from the jsdom implementation itself.

answered Jul 13, 2017 at 18:49

Hanzhao Deng

1186 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

MortenMoulder Over a year ago

In my example, how would I enter the URL of the website and extract the data from the known variable?

Hanzhao Deng Over a year ago

you get the content of the page and put it in the jsdom to execute it. maybe add code in there to console log to the virtual console and get the output.

MortenMoulder Over a year ago

Ah okay, it seems a bit "dangerous" (hence the runScripts: "dangerously") to run unknown code on my backend. I guess it works, though. Thanks!

Hanzhao Deng Over a year ago

There are security guidelines on the jsdom github page. I am not sure about your exact use case, but I recommended jsdom because say the script is manipulating DOM, putting the output in a div or something you can get the that in the jsdom.

Collectives™ on Stack Overflow

Return Javascript variable data from external page using NodeJS

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related