how to append numpy array of different dimension to already existing text file in python

Question

I have a code which requires 2xN dimensional matrix written to a text file for every iteration. But the problem is N keeps changing from one iteration to other.

I want to write this data to text file in python, and keep appending newly generated matrix to the old file. But it seems i have to convert numpy array to string before i can append to text file and its hugely time consuming as array to string function for large matrices is really slow.

Hence can somebody please tell me how do i append numpy arrays directly into the text file without converting them to string or list/

Thanks

I am not sure if pickle - docs.python.org/2/library/pickle.html#module-cPickle - supports numpy arrays, but if it does - it may be a solution — volcano
– volcano, Commented Feb 13, 2015 at 12:12

hpaulj · Accepted Answer · 2015-02-13 18:48:40Z

You can use savetxt multiple times with different size arrays:

In [34]: f = open('test.txt', 'wb')  # python3, write bytes not unicode
In [35]: np.savetxt(f,np.ones((2,3),dtype=int))
In [36]: np.savetxt(f,np.ones((2,2),dtype=int))
In [37]: np.savetxt(f,np.ones((2,1),dtype=int))
In [38]: f.close()
In [39]: cat test.txt
1.000000000000000000e+00 1.000000000000000000e+00 1.000000000000000000e+00
1.000000000000000000e+00 1.000000000000000000e+00 1.000000000000000000e+00
1.000000000000000000e+00 1.000000000000000000e+00
1.000000000000000000e+00 1.000000000000000000e+00
1.000000000000000000e+00
1.000000000000000000e+00

If you prefer to open and close the file each time, use the 'a', append mode.

I don't know how np.savetxt compares in speed with what you have tried. 'text' == 'string'. Someone has to convert the numeric data to string.

In savetxt this is done row by row:

for row in X:
   fh.write(asbytes(format % tuple(row) + newline))

There are fast ways of writing numpy arrays as binary data, but those will not be 'text' files.

`

Collectives™ on Stack Overflow

how to append numpy array of different dimension to already existing text file in python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related