JSON.stringify large object optimization

Question

I'd like to convert a large JSON object to string using the JSON.stringify, but due to the size of the object I got an error of

<--- Last few GCs --->

[20817:0x2cc2830]   295727 ms: Scavenge 1335.8 (1423.9) -> 1335.6 (1427.9) MB, 7.7 / 0.0 ms  (average mu = 0.255, current mu = 0.170) allocation failure 
[20817:0x2cc2830]   295966 ms: Mark-sweep 1339.5 (1427.9) -> 1339.3 (1422.9) MB, 227.1 / 0.0 ms  (average mu = 0.272, current mu = 0.291) allocation failure scavenge might not succeed
[20817:0x2cc2830]   295990 ms: Scavenge 1343.2 (1422.9) -> 1343.0 (1426.9) MB, 9.2 / 0.0 ms  (average mu = 0.272, current mu = 0.291) allocation failure 


<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x16f9b389e549 <JSObject>
    0: builtin exit frame: stringify(this=0x16f9b38877f9 <Object map = 0x39fb09082ba1>,0x1da31d6826f1 <undefined>,0x2762ba9be041 <JSFunction replacer (sfi = 0x1bc744b67dc9)>,0x1deddcf2e701 <Object map = 0x6bcf8da0341>,0x16f9b38877f9 <Object map = 0x39fb09082ba1>)

    1: stringify [0x16c685254241] [/home/blingga/Documents/integrations/integration-flow17employment-hrplus-to-datalake/dist/flow17em...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x8c02c0 node::Abort() [node]
 2: 0x8c030c  [node]
 3: 0xad15de v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [node]
 4: 0xad1814 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [node]
 5: 0xebe752  [node]
 6: 0xebe858 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [node]
 7: 0xeca982 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [node]
 8: 0xecb2b4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [node]
 9: 0xecdf21 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [node]
10: 0xe962a5  [node]
11: 0xe9dc0b v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [node]
12: 0x11cd6b0 v8::internal::IncrementalStringBuilder::Extend() [node]
13: 0xf92c20 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
14: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
15: 0xf977ee v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [node]
16: 0xf93e96 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
17: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
18: 0xf977ee v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [node]
19: 0xf93e96 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
20: 0xf9719d v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [node]
21: 0xf97ef9 v8::internal::JsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [node]
22: 0xba0051 v8::internal::Builtin_JsonStringify(int, v8::internal::Object**, v8::internal::Isolate*) [node]
23: 0x2639339dc17d 
Aborted (core dumped)

Is there any way I can optimize the JSON.stringify? Or is there anyway I can convert my JSON to string? Thanks.

can u provide a minimal example of payload?

Hitmands
– Hitmands

2019-04-02 07:07:58 +00:00
Commented Apr 2, 2019 at 7:07 — Hitmands
– Hitmands, Commented Apr 2, 2019 at 7:07
Hmm... the total size of my JSON is around 500mb

Abdullah
– Abdullah

2019-04-02 07:49:25 +00:00
Commented Apr 2, 2019 at 7:49 — Abdullah
– Abdullah, Commented Apr 2, 2019 at 7:49

Hitmands · Accepted Answer · 2019-04-02 07:09:49Z

4

did you try big-json?

const fs = require('fs');
const path = require('path');
const json = require('big-json');

const readStream = fs.createReadStream('big.json');
const parseStream = json.createParseStream();

parseStream.on('data', function(pojo) {
    // => receive reconstructed POJO
});

readStream.pipe(parseStream);

answered Apr 2, 2019 at 7:09

Hitmands

14.2k4 gold badges41 silver badges78 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Abdullah Over a year ago

Hmm, not yet. Looks promising. I'll try this one.

riv Over a year ago

Unfortunately it is extremely slow, writing at about 2MB per second. Since it's only application is writing JSON above 500MB (otherwise you can just use the native stringify), it is completely impractical.

ecdeveloper · Accepted Answer · 2020-03-24 23:57:26Z

3

You may also want to try JSONStream. It let's you stringify or parse a JSON virtually of any size.

Here's a parse example from their docs:

var request = require('request')
, JSONStream = require('JSONStream')
, es = require('event-stream')

request({url: 'http://isaacs.couchone.com/registry/_all_docs'})
.pipe(JSONStream.parse('rows.*'))
.pipe(es.mapSync(function (data) {
  console.error(data)
  return data
}))

edited Mar 24, 2020 at 23:57

answered Mar 19, 2020 at 22:51

ecdeveloper

2,8951 gold badge25 silver badges17 bronze badges

Collectives™ on Stack Overflow

JSON.stringify large object optimization

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related