How to append 3D numpy array to file?

Question

I am trying to save a 100x100x100 array of integers to a file with the date and time it was saved as a header. It doesn't need to be human-readable (except for the time-stamp header) so I was planning to use numpy.save(), take one slice at a time and save it to the file, but this does not append to the end of the file, it overwrites each time so the file only ends up containing the last slice.

Is there something like save() or savetxt() which appends to a file rather that overwrites?

Note: if it makes it easier, could I put the date/time into the filename when it saves instead of into the header?

My current attempt looks something like this:

with open("outfile.txt",'w') as mfile:
    mfile.write(strftime("%x %X\n"))
for i in range(len(x)):
    np.savetxt("outfile.txt",x[i])

Could you please show the code you use to save your data? Thanks! numpy.save takes a file as first argument, did you open with an 'a' flag ? — galath
– galath, Commented Jul 20, 2015 at 13:49
Ok, I have done that and it now saves all the slices, but I can't use np.save() if I want the date/time as a header in the file as it confuses the np.load() function when I want to read it back in. — FJC
– FJC, Commented Jul 20, 2015 at 13:58
Aha! Figured out how to do it if I read in the header first with readline(), then use np.load(), but now load() seems to only pick up one slice... can it deal with 3D data or do I have to read each slice in and then stack them? — FJC
– FJC, Commented Jul 20, 2015 at 14:04
You could also use reshape or flatten on your array in order to obtain low-dimensional versions. You need to remember your array shape while loading however (but that is probably the case anyway). — Christoph
– Christoph, Commented Jul 20, 2015 at 14:35

galath · Accepted Answer · 2015-07-20 14:38:10Z

1

Use the 'a'flag to append to a file. numpy.savetxt takes an array structure as input, so we need to reshape it.

p,q,r = x.shape
with open("outfile.txt",'ab') as mfile:
    header = strftime("%x %X\n")
    np.savetxt(mfile, x.reshape(p*q*r), header=header)

edited Jul 20, 2015 at 14:38

answered Jul 20, 2015 at 13:59

galath

6,00510 gold badges31 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Christoph Over a year ago

I think you would like to have that for block deeper indented.

Christoph Over a year ago

Uh, did not see that when I wrote my comment

Aaron · Accepted Answer · 2015-07-20 14:55:27Z

1

I'm a fan of Pickles :)

import cPickle
import time
import numpy as np

arr = np.array(xrange(100)).reshape(10,10)

#write pickle file
with open('out.p', 'wb') as f:
    t = time.asctime()
    cPickle.dump(t, f, cPickle.HIGHEST_PROTOCOL)
    cPickle.dump(arr, f, cPickle.HIGHEST_PROTOCOL)

#read pickle file
with open('out.p', 'rb') as f:
    t = cPickle.load(f)
    arr = cPickle.load(f)

edited Jul 20, 2015 at 14:55

answered Jul 20, 2015 at 14:38

Aaron

11.2k1 gold badge27 silver badges43 bronze badges

Comments

FJC · Accepted Answer · 2015-07-20 14:27:15Z

With the help of @galath's advice (although actually it doesn't seem to matter if I use 'a' or 'w' flag now...), I have figured out the following method using np.save/load and having the date as a header:

outfile="out.npy"

with open(outfile,'w') as mfile:
    mfile.write(strftime("%x %X\n"))      #write date and time as header
    for i in range(len(x)):
        np.save(mfile,x[i])               #save each slice

readx=np.zeros((len(x),len(y),len(z)) #empty array to be filled from file    

with open(outfile,'r') as mf:                #reopen file (for reading in)
    dt=mf.readline()                      #read date/time header
    for i in range(len(x)):
        readx[i,:,:]=np.load(mf)          #read in each slice and save it to array

If anyone has a more elegant solution feel free to share, but this way does what I need for now.

Collectives™ on Stack Overflow

How to append 3D numpy array to file?

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related