Tokenising binary data in java

Question

I am writing some code to handle a stream of binary data. It is received in chunks represented by byte arrays. Combined, the byte arrays represent sequential stream of messages, each of which ends with the same constant terminator value (0xff in my case). However, the terminator value can appear at any point in any given chunk of data. A single chunk can contain part of a message, multiple messages and anything in between.

Here is a small sampling of what data handled by this might look like:

[0x00, 0x0a, 0xff, 0x01]
[0x01, 0x01]
[0xff, 0x01, 0xff]

This data should be converted into these messages:

[0x00, 0x0a, 0xff]
[0x01, 0x01, 0x01, 0xff]
[0x01, 0xff]

I have written a small class to handle this. It has a method to add some data in byte array format, which is then placed in a buffer array. When the terminator character is encountered, the byte array is cleared and the complete message is placed in a message queue, which can be accessed using hasNext() and next() methods (similar to an iterator).

This solution works fine, but as I finished it, I realized that there might already be some stable, performant and tested code in an established library that I could be using instead.

So my question is - do you know of a utility library that would have such a class, or maybe there is something in the standard Java 6 library that can do this already?

Byte array searching and manipulation is something which I sometimes miss in Java. One can use a buffered byte stream child class which can do a per byte test for 0xff. In some cases one can utilise a pipeline: see PipedInputStream etc. — Joop Eggen
– Joop Eggen, Commented Oct 11, 2012 at 13:54

Peter Lawrey · Accepted Answer · 2012-10-11 13:48:21Z

1

I don't think you need a framework as a custom parser is simple enough.

InputStream in = new ByteArrayInputStream(new byte[]{
        0x00, 0x0a, (byte) 0xff,
        0x01, 0x01, 0x01, (byte) 0xff,
        0x01, (byte) 0xff});

ByteArrayOutputStream baos = new ByteArrayOutputStream();
for (int b; (b = in.read()) >= 0; ) {
    baos.write(b);
    if (b == 0xff) {
        byte[] bytes = baos.toByteArray();
        System.out.println(Arrays.toString(bytes));
        baos = new ByteArrayOutputStream();
    }
}

prints as (byte) 0xFF == -1

[0, 10, -1]
[1, 1, 1, -1]
[1, -1]

answered Oct 11, 2012 at 13:48

Peter Lawrey

535k83 gold badges770 silver badges1.2k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Sevas Over a year ago

Thanks for the response. It looks promising and certainly more elegant than what I have, but the data I have comes in as separate byte arrays, not as one byte array like in the example you have given.

Peter Lawrey Over a year ago

Where does it come from as byte[]? Are you reading from a device or network?

Sevas Over a year ago

It is actually read from a serial port, so the data looks like a series of byte arrays

Peter Lawrey Over a year ago

So when you read from the serial device it returns a new byte[] each time? You cannot read it like an InputStream?

Sevas Over a year ago

The data does come from an InputStream. I wasn't looking at this the right way. The solution you have given actually works brilliantly for what I need to do. Thanks!

Collectives™ on Stack Overflow

Tokenising binary data in java

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related