Node fails to read raw binary data into Float32Array

Question

I have a file with binary representations of single-precision floating point numbers (it's actually a FITS file in case that interests you).

Here's some sample code and output to orient you:

begin = 2880;
end = 2884;
console.log('HEX');
console.log(fileBuffer.slice(begin, end));
console.log('DEC');
console.log(fileBuffer.slice(begin, end).toJSON());
console.log(new Float32Array(fileBuffer.buffer.slice(begin, end)));

outputs

HEX
<Buffer 7f c0 00 00>
DEC
{ type: 'Buffer', data: [ 127, 192, 0, 0 ] }
Float32Array [ 6.905458702346266e-41 ]

So in the first line, I'm looking at a Node buffer and the next the decimal representation.

This is an IEEE NaN for single-precision, which in binary is: 0111 1111 1100 0000 0000 0000 0000 0000 so why am I not getting NaN for my value at that point?

But that's not all! Let's go to a non NaN spot in the data:

console.log(`start and stop: ${startByte} ${stopByte}`);
testArr = new Uint8Array(dataBuffer, startByte, stopByte);
console.log(testArr.slice(760, 774));
testArr = new Float32Array(dataBuffer, startByte, stopByte);
console.log(testArr.slice(190, 192));

with output

start and stop: 3072 4096
Uint8Array [ 127, 192, 0, 0, 199, 80, 6, 235, 199, 83, 165, 123, 199, 101 ]
Float32Array [ 6.905458702346266e-41, -1.6237752004906055e+26 ]

You can see that the first 4 bytes are again NaN (I had this in as a sanity check that I wasn't off by one byte or something) and the next four come out to 11000111010100000000011011101011 which is -53254.918, NOT the -1.62e+26 you see above.

What gives?

Note: I checked my work via: https://www.h-schmidt.net/FloatConverter/IEEE754.html and https://www.binaryhexconverter.com/decimal-to-binary-converter

OK, yup, just figured out that it's an endianness issue. The FITS standard is big-endian floats and I'm on a little-endian machine. I'll post a self-answer explaining how to byte swap once I come up with it unless someone beats me to it :) — KCE
– KCE, Commented Apr 12, 2018 at 7:44

KCE · Accepted Answer · 2018-04-13 12:47:31Z

1

If you take a look at the following you might notice something interesting:

a = new Float32Array([NaN]);
b = a.buffer;
u = new Uint8Array(b);

output is

Uint8Array(4) [0, 0, 192, 127]

In other words, NaN here is exactly reversed from the expectation stated in the question.

This reversal is the difference between big-endian and little-endian. The first order (with high numbers at the front) is called big-endian and the second ordering (with high numbers at the end) is called little-endian.

Since your FITS data is stored in big-endian byte order and the computer you're running Node on is little-endian, you'll need to use a DataView and the getFloat32 method in a loop (or in a byte stream if that's your setup). This method (and its relatives getFloat64, getInt16, etc.) enforces the endianness of your choice. Big-endian is the default so I ended up going with something like:

result = new Float32Array(256);
dataView = new DataView(fileBuffer.buffer.slice(2880));
byteNum = startByte;
for (let i = 0; i < result.length; i++) {
  result[i] = dataView.getFloat32(byteNum);
  byteNum = byteNum + Float32Array.BYTES_PER_ELEMENT;
}

edited Apr 13, 2018 at 12:47

answered Apr 12, 2018 at 11:38

KCE

1,2099 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Bergi Over a year ago

Perfect, thanks for the self-answer! I would even have explicitly passed false to getFloat32.

Collectives™ on Stack Overflow

Node fails to read raw binary data into Float32Array

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related