How to read large binary files in node js without a blocking loop?

Question

I am trying to learn some basics of event driven programming. So for an exercise I am trying to write a program that reads a large binary file and does something with it but without ever making a blocking call. I have come up with the following:

var fs = require('fs');
var BUFFER_SIZE = 1024;
var path_of_file = "somefile"

fs.open(path_of_file, 'r', (error_opening_file, fd) =>
{
    if (error_opening_file)
    {
        console.log(error_opening_file.message);
        return;
    }

    var buffer = new Buffer(BUFFER_SIZE);
    fs.read(fd, buffer, 0, BUFFER_SIZE, 0, (error_reading_file, bytesRead, buffer) =>
    {
        if (error_reading_file)
        {
            console.log(error_reading_file.message);
            return;
        }

        // do something e.g. print or write to another file
    })
})

I know I need to put a while loop in order to read complete file but in the above code I am reading just the first 1024 bytes of the file and cannot formulate how to continue reading the file without using a blocking loop. How could we do it?

rgvassar · Accepted Answer · 2016-03-21 22:14:06Z

8

Use fs.createReadStream instead. This will call your callback over and over again until it has finished reading the file, so you don't have to block.

var fs = require('fs');

var readStream = fs.createReadStream('./test.exe');
readStream.on('data', function (chunk) {
  console.log(chunk.length);
})

edited Mar 21, 2016 at 22:14

answered Mar 21, 2016 at 21:35

rgvassar

5,8612 gold badges27 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Muhammad Ali Over a year ago

Hey rgvassar thanks for your response. How much data does it read in a single call? Doesn't this function fetch complete file in memory? Isn't it meant to read the text files only?

rgvassar Over a year ago

I was mistaken readFile reads the whole file in one go. I've edited my answer to use createReadStream, which will pass back chunks. Each chunk is 65536 bytes. Neither readFile nor createReadStream are limited to just text files. They can read any type of file.

Collectives™ on Stack Overflow

How to read large binary files in node js without a blocking loop?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related