Avoiding Node.js event loop block while running JSON.stringify [duplicate]

Question

I have a huge data structure that I need to dump to file:

fs.writeFile('dump.json', JSON.stringify(bigData));

The resulting file is close to 100MB and it takes several seconds to generate. While JSON.stringify runs, it blocks the event loop and my server does not handle any requests.

Is there a way to somehow split the JSON.stringify call? My bigData var is an array of objects, so I could probably write a function to serialize them separately and then stitch the JSON together, to make sure that requests can be handled in between - but are there any solutions that are already existing (external modules is fine)?

whoan · Accepted Answer · 2015-12-02 10:38:18Z

0

Try stream-json-stringify. It should do the trick.

edited Dec 2, 2015 at 10:38

whoan

8,6094 gold badges43 silver badges50 bronze badges

answered Dec 2, 2015 at 9:48

Gonen

211 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Ahmed Ashour Over a year ago

While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review

Rytis Over a year ago

@AhmedAshour - you cannot seriously expect the whole module source to be pasted in the answer. The link is to the module's homepage, and that is just fine.

Rytis Over a year ago

As for the module itself, it writes some data, but not all - I get only some 50MB instead of all 100MB of data. Quite hard to trace what's missing though!

Gonen Over a year ago

I'm new here. Don't know the rules yet. Just wanted to help. Anyway, if there is a bug or an issue in the package, why not send a message to the owner? (For you and for the sake of others that will use this package in the future).

Rytis Over a year ago

Thanks for pointing me in the right direction - I found code at stackoverflow.com/a/13504078/979 which seems to work.

Collectives™ on Stack Overflow

Avoiding Node.js event loop block while running JSON.stringify [duplicate]

1 Answer 1

5 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Linked

Related