NodeJS JSON.stringify of very long array of objects errors "invalid string length"

Question

I've got a very long array of objects that is about 100,000 items in size, and just before I got to write it to file, I pass the data into JSON.stringify

I get this error:

JSON.stringify(
     ^
RangeError: Invalid string length

How can I stringify a very large json object successfully?

Update:

since I initially had written this question, I've become more familiar with processing large amounts of data, and I prefer to use Apache Spark for this process. In Spark, it is customary to not use large JSON arrays, but rather a stream or file containing long strings of JSON, one object per line. A file would look like this:

{"name":"kristian", "favoriteSong":"enter sandman"}
{"name":"joshua", "favoriteSong":"penny lane"}
{"name":"susan", "favoriteSong":"all about that bass"}

This is good because:

you can arbitrarily deal with files that are terabytes large, and just as easily do some logicless file splitting and turn that into a thousand smaller files, were the logic is to simply break the file at some line break number N.
This is better you wont ever have the potentially horrible issue of having a corrupted data file on your hands, caused by an un-closed JSON array.
a large JSON array in a file tends to not have line breaks, and thus large files are a pain to open in VIM or other editors because it tries to load the whole thing into memory, as opposed to a file with linebreaks, where it can easily load the first 10 lines or so to the screen without issue.

Uh, make two separate objects of the org obj, stringify each then concat results back together. — user1971598
– user1971598, Commented Nov 12, 2015 at 16:42

fsaint · Accepted Answer · 2017-03-25 01:42:13Z

15

I find JSONStream to be a reliable alternative to the native JSON.stringify that works well with large objects. For example:

var fileSystem = require( "fs" );
var JSONStream = require( "JSONStream" );
var records = [
    { id: 1, name: "Terminator" },
    { id: 2, name: "Predator" },
    { id: 3, name: "True Lies" },
    { id: 4, name: "Running Man" },
    { id: 5, name: "Twins" }
    // .... hundreds of thousands of records ....
];

var transformStream = JSONStream.stringify();
var outputStream = fileSystem.createWriteStream( __dirname + "/data.json" );
transformStream.pipe( outputStream );    
records.forEach( transformStream.write );
transformStream.end();

outputStream.on(
    "finish",
    function handleFinish() {
        console.log("Done");
    }
);

Took the sample code from here.

answered Mar 25, 2017 at 1:42

fsaint

8,7593 gold badges39 silver badges48 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Srinivas Over a year ago

This didn't work for me. I have nested array of Objects that is about 70 Megabytes of data. Could you help me with that?

Jannis Froese Over a year ago

This works great, but be aware that node.js doesn't seem to write anything to disk while you are busy filling up the transformStream. So either start your process with a high memory limit or pause occasionally to give the nodejs event loop some time to empty the pipe.

T.J. Crowder · Accepted Answer · 2015-11-12 16:48:04Z

7

How can I stringify a very large json object successfully?

In parts? E.g., break the array up into smaller parts, use JSON.stringify on the smaller parts, and append each segment to the file (you'll have to do some [, ,, and ] handling).

edited Nov 12, 2015 at 16:48

answered Nov 12, 2015 at 16:42

T.J. Crowder

1.1m201 gold badges2k silver badges2k bronze badges

1 Comment

Ketan Patil Mar 13 at 13:52

This answer helped. Thanks. Writing to a file is the key here. One cant make a string in memory even with this approach.

Collectives™ on Stack Overflow

NodeJS JSON.stringify of very long array of objects errors "invalid string length"

Update:

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Update:

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related