Reading serialized array of objects from a file in node js

Question

I have a file in which I have serialized array of objects. I need to stream the file object by object.

My question regarding nodejs stream is that, do I have to take care of my data boundaries? That is, nodejs readable stream emits line by line. So after every line, should I be checking if it is a valid JSON object?

My current implementation is something like this

f = fs.createReadStream('file.txt');
let buff = '';
f.on('data', (data) => {
   buff += data;
   try {
    process(JSON.parse(buff));
    buff = '';
   } catch (e) {
     return;
   }
}

Is there a better alternative? Also do I need to define my own data boundaries? Note that the objects I need to serialize are continuous. That is, they will be received over time and not once.

Tamas Hegedus · Accepted Answer · 2017-02-25 08:39:06Z

In your current implementation, JSON.parse will only succeed once, when the entire json is in the buff variable, as this is the only prefix of the whole file that is a valid json.

Instead, you should use oboejs, a streaming json parser. Here is how to use it:

const parser = oboe().node("!.*", node => {
  console.log(node);
  return oboe.drop;
});

Rx.Observable.from(`[
{"name": "foo"},
{"name": "bar"},
]`).zip(Rx.Observable.interval(10), a=>a).subscribe(char => {
  // console.log(char);
  parser.emit("data", char);
});

<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/5.0.1/Rx.js"></script>
<script src="https://cdnjs.cloudflare.com/ajax/libs/oboe.js/2.1.3/oboe-browser.js"></script>

Or adapted to your example:

const oboe = require("oboe");
oboe(fs.createReadStream('file.txt')).node("!.*", node => {
    process(node);
    return oboe.drop;
});

Collectives™ on Stack Overflow

Reading serialized array of objects from a file in node js

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related