how to save file with request in json nodejs?

Question

I'm trying to save the result to a json file but when I see it goes in half, something this wrong in my code but I do not understand that you part, thanks for your help.

var request = require("request");
var cheerio = require("cheerio");
var fs = require('fs');
var urls = ["http://www.fordencuotas.com.ar"]

var req = function(url){
    request({
        uri: url,
    }, function(error, response, body) {
        var $ = cheerio.load(body);
        $("a").each(function() {
        var link = $(this);
        var itri = {iti: new Array(link.attr("href"))}
        var data = JSON.stringify(itri);
        fs.writeFile("file.json", data, function(err){
            if(err){console.log(err);} else {console.log("archivo guardado..");}
        });
        });
    });
}

for (var i = 0; i < urls.length; i++){
    req(urls[i]);
}

console.log("cargando...");

this output

[opmeitle@localhost crawler1]$ node crawmod.js
cargando...
archivo guardado..
archivo guardado..
archivo guardado..
archivo guardado..
archivo guardado..
...
archivo guardado..
[opmeitle@localhost crawler1]$ cat file.json
{"iti":["productos/autos/nuevo-focus.html"]}us.html"]}
[opmeitle@localhost crawler1]$

robertklep · Accepted Answer · 2013-05-18 20:37:20Z

4

There's a couple of issues in your code.

First, you are trying to overwrite the same file (file.json) for each a element. I'm not sure if that's your intention, but it seems rather pointless.

Secondly, fs.writeFile is asynchronous. That means that Node doesn't wait until the file is written before it returns to your loop. In other words, for each a element you open the same file, while it might have already been opened by an earlier iteration of your loop. And each iteration is writing to the same file, so you're going to end up with unexpected results.

You can either use fs.writeFileSync to synchronously write the file, which would make Node wait until the data has been written to the file before continuing, or gather all the data that you want saved to the file in a variable, and — after the $("a").each(...) loop — write that variable to the file just once.

That last solution could look something like this:

var data = [];
$("a").each(function() { 
  var link = $(this);
  var itri = {iti: new Array(link.attr("href"))}
  data.push( itri );
});
fs.writeFile("file.json", JSON.stringify(data), function(err){
  if(err){console.log(err);} else {console.log("archivo guardado..");}
});

answered May 18, 2013 at 20:37

robertklep

205k38 gold badges421 silver badges413 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

opmeitle Over a year ago

First, thank you for your simple explanation, i could resolve correctly with your example - thank you for your help

Collectives™ on Stack Overflow

how to save file with request in json nodejs?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related