Node.js file write in loop fails randomly

Question

Here is my code :

function aCallbackInLoop(dataArray) {
        dataArray.forEach(function (item, index) {

            fs.appendFile(fileName, JSON.stringify(item) + "\r\n", function (err) {
                if (err) {
                    console.log('Error writing data ' + err);
                } else {
                    console.log('Data written');
                }
            });
        });
    }

I get random errors :

Data written
Data written
.
.
Error writing data Error: UNKNOWN, open 'output/mydata.json'
Error writing data Error: UNKNOWN, open 'output/mydata.json'
.
.
Data written
Error writing data Error: UNKNOWN, open 'output/mydata.json'

The function (aCallbackInLoop) is a callback for a web-service request, which returns chunks of data in dataArray. Multiple web-service requests are being made in a loop, so this callback is perhaps being called in parallel. I doubt it's some file lock issue, but I am not sure how to resolve.

PS: I have made sure it's not a data issue (I am logging all items in dataArray)

Edit : Code after trying write stream :

function writeDataToFile(fileName, data) {
    try {
        var wStream = fs.createWriteStream(fileName);
        wStream.write(JSON.stringify(data) + "\r\n");
        wStream.end();
    } catch (err) {
        console.log(err.message);
    }
}

function aCallbackInLoop(dataArray){
    dataArray.forEach(function(item, index){
        writeDataToFile(filename, item);   //filename is global var
    });
}

thefourtheye · Accepted Answer · 2015-03-29 05:10:14Z

8

As you have observed, multiple appendFile calls are not able to proceed because of the previous appendFile calls. In this particular case, it would be better to create a write stream.

var wstream = fs.createWriteStream(fileName);

dataArray.forEach(function (item) {
    wstream.write(JSON.stringify(item + "\r\n");
});

wstream.end();

If you want to know when all the data is written, then you can register a function with the finish event, like this

var wstream = fs.createWriteStream(fileName);

wstream.on("finish", function() {
    // Writing to the file is actually complete.
});

dataArray.forEach(function (item) {
    wstream.write(JSON.stringify(item + "\r\n");
});

wstream.end();

answered Mar 29, 2015 at 5:10

thefourtheye

241k53 gold badges466 silver badges505 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

dev Over a year ago

I see this is the right way to go, but I am still getting the error. If I use appendFileSync - then everything works. But I don't want to use the synchronous version. Still wondering what am I doing wrong !

thefourtheye Over a year ago

@SlowAndSteady Can you show the actual code which you tried?

thefourtheye Over a year ago

@SlowAndSteady Why are you creating a stream for every data? Better change it like the way I have shown (create stream only once) and try.

dev Over a year ago

Actually the writeDataToFile is in a seperate module/file and it's a utility to handle multiple functionalities depending on the fileName I pass. I am trying to avoid the clients of this utility to open any file streams etc. Why would multiple streams be an issue though ? Shouldn't the streams just wait in case someone else has lock on the resource ?

dev Over a year ago

It seems I would have to reengineer my design. Will try and update !

|

manojlds · Accepted Answer · 2015-03-29 05:04:01Z

2

Try using the synchronous version of appendFile - https://nodejs.org/api/fs.html#fs_fs_appendfilesync_filename_data_options

answered Mar 29, 2015 at 5:04

manojlds

303k66 gold badges482 silver badges426 bronze badges

4 Comments

dev Over a year ago

This solves the issue, but I am still hoping to get the async way to work. If not, I will mark this answer as the solution.

jfriend00 Over a year ago

Using a synchronous version is rarely the right way to go in a node.js server.

manojlds Over a year ago

@jfriend00 - True, but with the given context of writing to the same file again and again, the sync is the best approach. OP might have to redesign.

jfriend00 Over a year ago

@manojlds - no it is NOT the best approach. Using synchronous I/O anywhere in a node.js server (other than startup code) absolutely kills the scalability because the single threaded nature of node.js means no other request can be processed while the server is sitting there waiting for disk I/O to complete. There are plenty of other options that don't ruin the scalability. A write stream, nested async writes, chained promise writes are all much better options.

Collectives™ on Stack Overflow

Node.js file write in loop fails randomly

2 Answers 2

6 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related