What's the fastest way to parse Node.JS buffer stream chunks' binary data into structures?

Question

Googling for "parse nodejs binary stream" I see lots of examples when the emitted data fits neatly into the expected return size, but zero examples of how to handle the case when the next chunk contains the remaining bytes from the first structure and the new header.

What's the "right" way to parse binary streams when I expect something like:

record-length: 4bytes data1: 8bytes data2: 8bytes 4-byte-records[(record-length - 16) * 4];

The data will come in as various sized chunks. But is there a way to call data.readUInt32(0) and have it wait for the chunk to fill? I'd hate to have to write a pipe stage that emits bytes and a receiving state machine, that seems so very wrong.

This has got to be solved, it is such a basic concept.

Help?

Thanks, PT

Michał Karpacki · Accepted Answer · 2018-09-14 13:53:10Z

2

Hmm... thats's something that can be solved using the async version of stream..read and a transform stream.

Now you can write (and it will probably be fun) your own version, but the framework I wrote, scramjet already has that async read and I gather you'd want to make this easy.

Here's the easiest I can think of, using AsyncGenerator:

const {BufferStream} = require('scramjet'); // or es6 import;
const input = BufferStream.from(getStreamFromSomewhere());

const output = DataStream.from(async function* () {
  while(true) {
    const recordLength = (await input.whenRead(4)).readUInt32(0);  // read next chunk length
    if (!recordLength) return;                                     // stream ends here;
    const data1 = await input.whenRead(8);
    const data2 = await input.whenRead(8);
    const restOfData = [];
    for (let i = 0; i < recordLength; i += 4)
      restOfData.push((await input.read(4)).readUInt32(0))

    yield {data1, data2, restOfData};
  }
})
  .catch(e => output.end()); // this is a handler for an option where any of the reads past
                             // recordLength was to return null - perhaps should be better.

This is super easy in node v10 or with babel, but if you like I can add the non AsyncGenerator version here.

answered Sep 14, 2018 at 13:53

Michał Karpacki

2,66624 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user3909192 Over a year ago

Thanks. I'm a little bummed that this requires yet another external module for something so seemingly basic but this did the trick.

Michał Karpacki Over a year ago

I wouldn't say it actually needs scramjet - but the implementation is kinda tricky. You can grab the sources and see how I implemented from for AsyncGenerator and redo that without the framework. And it has only 3 dependencies, so the bloat is minimal.

Collectives™ on Stack Overflow

What's the fastest way to parse Node.JS buffer stream chunks' binary data into structures?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related