How to read large json file in node js?

Question

I have a JSON file about 300MB in size and I'm trying to read it but it's not working.

How can I read a large JSON file? If anyone can guide me with a small piece of code then it would be great.

I have already tried fs.readFile but no luck, it's working fine for smaller files but not for large ones.

Below is the code I have tried so far:

app.get('/getData', function (req, res) {
  fs.readFile('./uploads/test.json', function (err, data) {
      if (err) throw err
      var jsonData = data;
      var jsonParsed = JSON.parse(jsonData);
      res.json(jsonParsed);
    });
});

What "does not work"? Do you get an error? Run out of memory? Take too long? What exactly is the problem? For starters, if (err) throw err should log. And, you can put a try/catch with logging in the catch around JSON.parse(). You should at least start by logging possible errors in this code. This takes some elemental debugging on your part before any of us could help you. You should do that before you come here and then tell us what you found. — jfriend00
– jfriend00, Commented Jun 22, 2019 at 0:43
It's possible that 300MB of JSON is too large for your environment. There are stream parsers for JSON that might be more memory efficient. At 300MB, I'd personally be looking at a database, not a JSON file for storage. — jfriend00
– jfriend00, Commented Jun 22, 2019 at 0:45
Possible duplicate of Upload a large file (1GB) with node and express — blurfus
– blurfus, Commented Jun 22, 2019 at 0:48
FYI, CSV is simpler to read serially line by line if that's appropriate for what you're storing. — jfriend00
– jfriend00, Commented Jun 22, 2019 at 0:52

Ashen Gunaratne · Accepted Answer · 2019-06-22 04:00:16Z

3

Speculating from the code snippet, I believe there is no requirement to do any modification (i.e. filtering) to the JSON data prior dispatching the response. If that is the case, parsing of the JSON would be unnecessary and Node.js's built-in readable streams could be used to provide more efficiency.

const fs = require('fs');
const path = require('path');
const express = require('express');
const app = express();

app.get('/data', (req, res) => {
  res.setHeader('Content-Type', 'application/json');
  fs.createReadStream(path.join(__dirname, 'public', 'citylots.json')).pipe(res);
});

app.listen(process.env.PORT || 8080);

answered Jun 22, 2019 at 4:00

Ashen Gunaratne

4553 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

slebetman Over a year ago

Is res a WritableStream? I would have just used res.sendFile(jsonFile)

Ashen Gunaratne Over a year ago

@slebetman Yes, the res object is an enhanced version of Node’s http.ServerResponse class which inherits Stream.

Ashen Gunaratne Over a year ago

@slebetman The sendFile method implemented via send utilises the exact same technique above but since the content type is know beforehand the above saves few function calls. Regardless, the sendFile is a commodious alternate too!

rajat meena Over a year ago

@AshenGunaratne with your solution i'm getting the data but it's taking too much time to load , is there any way to make it process faster ?

jfriend00 Over a year ago

@rajatmeena - What are you expecting? It takes awhile to stream 300MB to the client. Fix your design so you don't need to send 300MB to the client. Keep the data on the server and let the client make ajax requests to fetch what it needs. Or, send the data in small chunks and let the client request only what it needs when it needs it.

Collectives™ on Stack Overflow

How to read large json file in node js?

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related