NodeJS Async JSON parsing causing Buffer.toString() failure

Question

I'm attempting to parse a fairly large JSON file (~500Mb) in NodeJS. My implementation is based on the Async approach given in this answer:

var fileStream = require('fs');
var jsonObj;

fileStream.readFile('./data/exporttest2.json', fileCallback);

function fileCallback (err, data) {
    return err ? (console.log(err), !1):(jsonObj = JSON.parse(data));
    //Process JSON data here
}

That's all well and good, but I'm getting hit with the following error message:

buffer.js:495
    throw new Error('"toString()" failed');
    ^

Error: "toString()" failed
    at Buffer.toString (buffer.js:495:11)
    at Object.parse (native)
    at fileCallback (C:\Users\1700675\Research\Experiments\NodeJS\rf_EU.js:49:18)
    at FSReqWrap.readFileAfterClose [as oncomplete] (fs.js:445:3)

I understand from this answer that this is caused by the maximum buffer length in the V8 engine set at 256Mb.

My question then is this, is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?

can you share how you used the JSONstream module?

Max Koretskyi
– Max Koretskyi

2017-04-19 16:02:29 +00:00
Commented Apr 19, 2017 at 16:02 — Max Koretskyi
– Max Koretskyi, Commented Apr 19, 2017 at 16:02

rsp · Accepted Answer · 2017-04-19 10:28:01Z

3

is there a way I can asynchronously read my JSON file in chunks that do not exceed the buffer length of 256Mb, without manually disseminating my JSON data into several files?

This is acommon problem and there are several modules than can help you with that:

Example with JSONStream:

const JSONStream = require('JSONStream');
const fs = require('fs');

fs.createReadStrem('./data/exporttest2.json')
  .pipe(JSONStream.parse('...'))...

See the docs for details of all of the arguments.

answered Apr 19, 2017 at 10:28

rsp

112k31 gold badges210 silver badges185 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Scott Over a year ago

Literally just stumbled across a couple of these libraries. I'll use one until the NodeJS community address the issue. Thanks for your response nonetheless.

Max Koretskyi Over a year ago

do you know why the buffer is not exceeded with JSONStream? does it try to parse it as data goes in?

Max Koretskyi · Accepted Answer · 2017-04-19 10:22:56Z

-1

Try using streams:

let fs = require("fs");

let s = fs.createReadStream('./a.json');
let data = [];
s.on('data', function (chunk) {
    data.push(chunk);
}).on('end', function () {
    let json = Buffer.concat(data).toString();
    console.log(JSON.parse(json));
});

answered Apr 19, 2017 at 10:22

Max Koretskyi

106k69 gold badges356 silver badges524 bronze badges

1 Comment

Scott Over a year ago

This still causes the same issue, when Buffer.concat(data).toString() executes there is >256Mb in the buffer

Collectives™ on Stack Overflow

NodeJS Async JSON parsing causing Buffer.toString() failure

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related