Different result writing to binary file with python and matlab

Question

I am attempting to convert a matlab script to python. The script involves writing data to a binary file. I have noticed differences between the files, specifically when writing matrices/numpy arrays. When writing other variable types (int, string etc) the files are identical as desired.

Matlab code:

fid = fopen("test2.txt", "wb");
a = [[1 2];[3 4]];
fwrite(fid, a, "float64");
fclose(fid);

Matlab result (as shown by notepad):

~~ð? @ @ @~~

Python code:

import numpy as np
with open("test2.txt", "wb") as fid:
    a = np.array([[1, 2], [3, 4]])
    fid.write(a.astype("float64"))
    # a.astype("float64").tofile(fid) # also doesn't give correct result

Python result (as shown by notepad):

~~ð? @ @ @~~

The results you posted look identical to me, can you point out the difference? — BatWannaBe
– BatWannaBe, Commented Aug 28, 2021 at 0:12
@BatWannaBe Looks like SO strips the characters (the squares) which show the difference. Screenshots have been added which show the difference. — L.Clarkson
– L.Clarkson, Commented Aug 28, 2021 at 13:56
Your NumPy and MATLAB matrices are not the same, they are transposed versions of each other. Consequently they are written to file in a different order. — Cris Luengo
– Cris Luengo, Commented Aug 28, 2021 at 15:08
Note also that in MATlAB, a = [[1 2];[3 4]]; has a lot of redundant brackets, it is better to write a = [1 2; 3 4];. — Cris Luengo
– Cris Luengo, Commented Aug 28, 2021 at 15:09
I don't think they are transposed versions of each other. To confirm this, a = [1 2 3; 4 5 6]; and a = np.array([[1, 2, 3], [4, 5, 6]]) give the same size/shape but show differences in the binary file. — L.Clarkson
– L.Clarkson, Commented Aug 28, 2021 at 15:19

BatWannaBe · Accepted Answer · 2021-08-28 18:49:34Z

2

The characters still look very similar due to how Notepad is attempting to read integers as text, but I think it gave enough of a hint. For easier typing, let's call Matlab's text d? [@ _@ [@ and Python's text d? _@ [@ [@.

Computer memory is linear, so all multidimensional arrays are actually stored as 1D arrays. What you're seeing is NumPy arrays being C order (by default) versus Matlab matrices being Fortran order (by default). This order is how multidimensional arrays are flattened to 1D arrays in memory.

matrix   notepad text
1  2     d?  _@               
3  4     [@  [@

Matlab Fortran order goes by columns
 1  3  2  4
d? [@ _@ [@

NumPy C order goes by rows
 1  2  3  4
d? _@ [@ [@

Since you're converting code between MATLAB and Python, you should be very aware of array orders being different. Iteration is faster when you don't jump around in memory, so nested for-loops may have to be reordered. It won't make much of a difference for vectorized code someScalar * myArray because it's handled for you. NumPy does provide functions and optional arguments to create Fortran order arrays numpy.asfortranarray(), ndarray.copy(order = 'F') and to check the order ndarray.flags.f_contiguous, ndarray.flags.c_contiguous, but coding with that is still tougher because C order is the default.

answered Aug 28, 2021 at 18:49

BatWannaBe

4,5301 gold badge17 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

L.Clarkson Over a year ago

I was able to achieve the correct output using a.T.astype("float64").tofile(fid), but fid.write(a.T.astype("float64") causes a ValueError: ndarray is not C-contiguous

BatWannaBe Over a year ago

I'm not sure if using Python's open to save binary files is the way to go here. I'm sure you can figure out how to make it work, but there's a reason NumPy provides numpy.save/.npy files and MATLAB provides save/.mat files. I don't use MATLAB so I can't tell you if MATLAB has something similar, but SciPy, which builds on NumPy, actually provides a way to work with .mat files: scipy.io.savemat, scipy.io.loadmat, scipy.io.whosmat

L.Clarkson Over a year ago

The MATLAB script I am converting is used to create Finite Element models files (with a prescriptive format/data layout), so working in .mat or .npy isn't possible. With your explanation, I have been able to solve my issue.

Collectives™ on Stack Overflow

Different result writing to binary file with python and matlab

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related