Node.js return html as a variable

Question

I'm setting up a web scraper using Node.js and want to grab some html from a url and save it as a variable. A stripped down version follows.

var request = require('request');
var get_html = function(){
    var url = "http://www.google.com";
    var html = '';
    request.get(url,function(error, response, body){
        html += body;
    });
    return html;
};
console.log(get_html());

It seems that the function returns before request can concatenate the html to the variable html. As far as I can see, request only allows me to manipulate the html within the callback function or pipe it to a file. Is there anyway to just return it as a variable?

Alongkorn · Accepted Answer · 2016-06-14 05:15:42Z

1

request.get is asynchronous and it will return result in the callback function.

You need to adapt your code a little bit like this

var request = require('request');

// get_html receive callback to process result
var get_html = function(callback) {
    var url = "http://www.google.com";
    var html = '';
    request.get(url,function(error, response, body){
        return callback(body); // call callback and parse result to it
    });
};

// call get_html function
// and log html result here
get_html(function (body) { console.log(body); });

Code with a lot of function callbacks looks not beautiful. I prefer promise than callback. If you wish to use promise, try 'request-promise' lib.

answered Jun 14, 2016 at 5:15

Alongkorn

4,2371 gold badge28 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mentoc3000 Over a year ago

So it looks like there's no way to actually return the html as a string, only to pass the rest of the code into request as a callback? I mean to use this small function as a utility that can be called by different modules. It sounds like this will get unwieldy quickly.

Milk Biscuit · Accepted Answer · 2016-06-13 23:30:33Z

0

It appears that request.get is async, so you have to put return html; in the callback. Otherwise it's returning instantly, before request.get can finish running.

answered Jun 13, 2016 at 23:30

Milk Biscuit

283 bronze badges

Collectives™ on Stack Overflow

Node.js return html as a variable

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related