Creating a (good) struct from a javascript typed array?

Question

I'm thinking about typed arrays in javascript; in particular, reading binary file headers. Reading the MDN Typed Arrays documentation, they blithely say we can represent the C struct:

struct someStruct {
  unsigned long id;
  char username[16];
  float amountDue;
};

with the javascript:

var buffer = new ArrayBuffer(24);

// ... read the data into the buffer ...

var idView = new Uint32Array(buffer, 0, 1);
var usernameView = new Uint8Array(buffer, 4, 16);
var amountDueView = new Float32Array(buffer, 20, 1);

and then you can say amountDueView[0] but seem oblivious to the fact that treating a scalar as a one-element array is terrible and being able to say something like sizeof(someStruct) is useful as well.

So... is there something better? MDN makes reference to js-ctypes but it's no standard javascript.

Edit: Yes, I know about javascript objects. Modeling the data isn't the problem; reading and writing binary data elegantly is.

You will want to have a look at DataView. Btw, from what I read js-ctypes are a library wrapper on standard javascript, so it should be save (and easy) to use. — Bergi
– Bergi, Commented Jul 3, 2014 at 2:27
you would want to composite the arrays into an object with it's own ArrayBuffer, and possibly add some getters onto the properties to avoid the extra "[0]" each time. — dandavis
– dandavis, Commented Jul 3, 2014 at 2:36

Nate · Accepted Answer · 2014-12-21 04:33:43Z

6

For what it's worth, there's jBinary, which lets you do stuff like:

var binary_data = '\x0f\x00\x00\x00Bruce\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xecQXA';

var myTypeset = {
    'jBinary.all': 'someStruct',
    'jBinary.littleEndian': true,
    someStruct: {
        id: 'uint32',
        username: ['string0', 16],
        amountDue: 'float'
    }
};

var binary = new jBinary(binary_data, myTypeset);
header = binary.readAll(); // Object {id: 15, username: "Bruce", amountDue: 13.520000457763672}

header.username = "Jim";
binary.writeAll(header);
binary.readAll(); // Object {id: 15, username: "Jim", amountDue: 13.520000457763672}

It's not particularly well-documented (an example like the above would be very welcome) but it's quite good.

The biggest caveat is that it relies on javascript objects having properties remain in the order they're defined, which is practically the case but not part of the javascript spec. I suspect this will never actually change because it would break a kajillion things, but still.

answered Dec 21, 2014 at 4:33

Nate

4,7672 gold badges38 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ciprian Tomoiagă Over a year ago

can't believe this is the only example on SO with jbinary. Thank you ! Do you know if it can read from an arrayBuffer (or typedarray, same thing) instead of a string ?

Nate Over a year ago

I'm quite sure it does; you'll need to try though.

Engineer · Accepted Answer · 2024-09-14 15:51:37Z

You do not need binary string conversion to get array-of-struct access. That is slow.

Do as C, Rust, Java, C# etc. have always done at compile time (albeit at runtime in JS): use offsets and sizes to working with your structs inside the array. Add some member names et voila!

This works for both interleaved (multi-member structs) and non-interleaved arrays. Obviously you are now using function calls instead of array[] (unlike C & Rust, TMK), but these may (?) be inlined under the hood. I suspect any slowdown due to read/write functions is far outweighed by performance gains associated with accessing contiguous data.

const systemWordSize = 8; //bytes
const strideLength = 1; //bytes
//...these functions can be optimised to take advantage of 4 or 8 byte reads / writes, depending on system word size.

//Utility functions

function padAndCalcMemberOffsets(type)
{
    let structSize = 0;
    for (let memberName in type)
    {
        let member = type[memberName];
        console.log(memberName, member);
        member.offset = structSize;
        structSize += member.size;
    }
    
    let remainder = structSize % systemWordSize;
    if (remainder != 0)
    {
        let padSize = systemWordSize - remainder;
        type._padding = {size: padSize, offset: structSize};
    
        structSize += padSize;
    }
    
    return structSize;
}

function read(array, index, type, memberName)
{
    let bytesLength = type[memberName].size;
    let ibase = index * type._size;
    let i = ibase + type[memberName].offset;
    let iStart = i;
    let value = 0;
    
    //read out sequentially into a single multibyte value.
    for (let b = 0; b < bytesLength; b += strideLength)
    {
        let bitsToShiftBy = b * 8; //bits per byte
        value |= array[i] << bitsToShiftBy;
        i += strideLength; //1 byte at a time
    }
    
    console.log("read ", type._name, "at ["+ibase+"],",  memberName, "at ["+iStart+"] (length="+bytesLength+"):", value);
    
    return value;
}

//write a multibyte field.
function write(array, index, type, memberName, value)
{
    let bytesLength = type[memberName].size;
    let ibase = index * type._size;
    let i = ibase + type[memberName].offset;
    let iStart = i;
    let valueNew = 0;
    
    //write out sequentially into a single multibyte value.
    for (let b = 0; b < bytesLength; b += strideLength)
    {
        let bitsToShiftBy = b * 8; //bits per byte
        let valueSegment = (value >> bitsToShiftBy) & 255;
        
        array[i] = valueSegment;
        i += strideLength; //1 byte at a time
    }
    
    console.log("write", type._name, "at ["+ibase+"],",  memberName, "at ["+iStart+"] (length="+bytesLength+"):", value);
}

function prepareStructTypes(structTypes)
{
    for (let structTypeName in structTypes)
    {
        let structType = structTypes[structTypeName];
        
        structType._size = padAndCalcMemberOffsets(structType);
        structType._name = structTypeName; //optional, only used for console.log
        console.log(structTypeName, structType);
    }   
}

Usage:

let structTypes = //incomplete for now
{
    Boxer:
    {
        id:   {size: 2, offset: 0},
        hits: {size: 3, offset: 0},
        time: {size: 4, offset: 0},
        //total size of this struct would be 1+4+2=7 bytes, which does not align with
        //typical 4 or 8 byte word boundaries, which make access (very) inefficient. So pad it.
    },
    
    Javelineer:
    {
        throws: {size: 3, offset: 0},
        misses: {size: 1, offset: 0},
    },

}

prepareStructTypes(structTypes); //complete them: pad, deduce member offsets and struct total size.

//Test

const Boxer = structTypes.Boxer;
const numBoxers = 8;
const arrayOfBoxer = new Uint8Array(numBoxers * Boxer._size);
console.log("Interleaved Boxer array length:", arrayOfBoxer.length, "size per element", Boxer._size);

//arbitrary values
let i = 3;
let id = 888;
let time = 23983291;

//write
write(arrayOfBoxer, i, Boxer, "id", id);
write(arrayOfBoxer, i, Boxer, "time", time);
 
console.log(arrayOfBoxer);

//read
read(arrayOfBoxer, i, Boxer, "id");
read(arrayOfBoxer, i, Boxer, "time");

Output:

write Boxer at [80], id at [82]: 888
write Boxer at [80], name at [89]: 23983291
Uint8Array(128) [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 120, 3, 0, 0, 0, 187, 244, 109, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, …]
read  Boxer at [80], id at [82]: 888
read  Boxer at [80], name at [89]: 23983291

TGH · Accepted Answer · 2014-07-03 02:28:51Z

-1

Can't you just define an object in JS to model this?

var someObject =  {
  Id:2,
  username: [],
  amountDue:5
};

answered Jul 3, 2014 at 2:28

TGH

39.4k13 gold badges106 silver badges140 bronze badges

2 Comments

dandavis Over a year ago

he can, but will that mount the header of a binary file in the same way?

Engineer Over a year ago

The OP can't do as you suggest. The reason they asked about using TypedArrays as the underlying mechanism for storing structs is contiguous allocation in AoS (Array of structs) for performance reasons. AoS implies a contiguously allocated memory block in languages such as C, C++, Rust, GLSL, etc., containing the structs themselves as values. This is not the case with JS objects, which are allocated elsewhere, and referenced from the array; requiring double-dereferencing which is costly.

maček · Accepted Answer · 2014-07-03 02:34:29Z

-4

You more than likely don't need typed arrays for your application; the rest of the world has been surviving just fine without them. IE < 10 doesn't even support them.

You can write the object like this

var data = {id: 123, username: "bob", amountDue: 123.45};

And read from it like this

data.id;
// 123

data.username;
// "bob"

Try to read an unset key

data.foo;
// undefined

And set a key after it's defined

data.foo = "bar";
// "bar"

And delete the key

delete data.foo;
// true

Try to read a deleted key

data.foo;
// undefined

answered Jul 3, 2014 at 2:34

maček

78k37 gold badges172 silver badges200 bronze badges

2 Comments

jfriend00 Over a year ago

I think you're missing the fact that the OP is reading some binary data (which has a C-like binary format) into a ArrayBuffer and then trying to interpret that data in javascript. This problem isn't just about statically declared javascript data.

Nate Over a year ago

Yes -- I'm not interested in doing this for performance reasons, but to read (and write) binary data structures elegantly.

Collectives™ on Stack Overflow

Creating a (good) struct from a javascript typed array?

4 Answers 4

2 Comments

Comments

2 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related