5

I have a numpy array with dimension 1000*30*150. I am trying to save it as txt file. So far I have tried this

np.savetxt("test.txt", mydata, fmt='%.5f', delimiter=",")
#and 
with open('test.txt', 'w') as f:
    for row in mydata:
        np.savetxt(f, row, delimiter=',', fmt='%.5f')

both method give me error

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py", line 1254, in savetxt
    fh.write(asbytes(format % tuple(row) + newline))
TypeError: only length-1 arrays can be converted to Python scalars

During handling of the above exception, another exception occurred:

Traceback (most recent call last):


        np.savetxt("test.txt", train, fmt='%.5f', delimiter=",")
      File "/usr/local/lib/python3.5/dist-packages/numpy/lib/npyio.py", line 1258, in savetxt
        % (str(X.dtype), format))
    TypeError: Mismatch between array dtype ('float64') and format specifier ('%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f,%.5f')

5 Answers 5

4

The problem is your array is 3 dimensional and can't be saved in a 2 dimensional format. Either reshape it, so that it is 2d:

mydata = mydata.reshape(mydata.shape[0],mydata.shape[1]*mydata.shape[2])
np.savetxt('text.txt',mydata,fmt='%.5f',delimiter=',')

or if you do not need to read it as a text file and want to just reload it later in python use:

np.save('text.npy',mydata)
Sign up to request clarification or add additional context in comments.

2 Comments

.@djk47463 - I get similar error Mismatch between array dtype ('float32') and format specifier ('%.18e,%.18e,%.18e'). Can you please explain reshape for this (I am new to numpy). The data I have can be saved in .npy, but I need raw text files to check the weights it's storing.
@ChetanArvindPatil, when using reshape, you are either reducing or increasing the dimensions. In this example, OP has a 3d array, with a length of 1000. Keeping the first dimision the same, we combine the 2nd and 3rd dimensions. So in a given row, in the csv 150 observations would be equivalent to one timestep from the second dimension
2

You did not mention what is the purpose of writing the 3-d array to a text file, would you be reading it back in the future, and what format are you looking for, but that is one possibility:

import json
print(json.dumps(mydata, default=lambda x: list(x), indent=4))

If you clarify the purpose, people will be able to suggest better suited solutions.

2 Comments

how can I save this in a text or json file?
with open('test.txt', 'w') as f: f.write(json.dumps(mydata, default=lambda x: list(x), indent=4))
1

Tell us about mydata. In particular its dtype and shape.

To save with %.5f format it needs to be a 2d array of numbers.

savetxt does, roughly:

 for row in arr:
   print(format % tuple(row))

where format is constructed from your fmt parameter, and the number columns in the array. It looks like your array has a large number of columns, so format is that '%.5f,%.5f,%.5f,%.5f,%.5f,%... string.

tuple is needed to turn that 1d array row into a tuple that works with format%().

If the array is higher dimensions, or an array of objects, it would have problems.


edit - so you say the array is 1000*30*150. So it tries to iterate on the 1000 rows, 30 looks like the size of that format. But it can't apply that to a (30,150) array.

With the open and row iteration, do you get the same error? In Py3 you might need to open with 'wb'. Iterating yourself on the first dimension means eachsavetxtcall works with a 30x150 array. It will iterate on the 30, and try to format rows of 150. The would create a largerformat`, but I think that would run.

In any case, savetxt is designed for 2d numeric arrays. 3d requires some sort of fudge. Keep in mind also that csv readers aren't designed for 3d arrays either. They expect rows with consistent columns separated by a simple delimiter.


In [260]: arr = np.arange(24).reshape(4,3,2)

It can work with 3d - if allowed to format each subrow with %s:

In [261]: np.savetxt('test',arr, fmt='%s')
In [262]: cat test
[0 1] [2 3] [4 5]
[6 7] [8 9] [10 11]
[12 13] [14 15] [16 17]
[18 19] [20 21] [22 23]

3d numeric format - error

In [263]: np.savetxt('test',arr, fmt='%d')
....
TypeError: Mismatch between array dtype ('int32') and format specifier ('%d %d %d')

Reshape 3d to 2d - save works:

In [264]: np.savetxt('test',arr.reshape(-1,2), fmt='%d')
In [265]: cat test
0 1
2 3
4 5
6 7
8 9
...
22 23

With an extra iteration; could add a blank line between blocks

In [267]: with open('test','wb') as f:
     ...:     for row in arr:
     ...:         np.savetxt(f, row, '%d',delimiter=', ')
     ...:         
In [268]: cat test
0, 1
2, 3
4, 5
6, 7
...
22, 23

Comments

0

An alternative to np.savetxt() could be using the csv module:

with open("filename.","w+") as my_csv:            # writing the file as my_csv
    csvWriter = csv.writer(my_csv,delimiter=',')  # using the csv module to write the file
    csvWriter.writerows(array_2d)                 # write every row in the matrix

I have encountered a similar TypeError problem with numpy, but the CSV method seems to work fine.

Comments

0

If you're looking to write the data out in formatted rows and columns along the axis at mydata[i,:,:] with the intention of producing something in a more readable table format, see this answer: How to write a multidimensional array to a text file? by @JoeKington. My code adds a loop through the rows and columns of each slice because I couldn't find any other resolution to a TypeError I was getting when implementing the original code:

    with open('test.txt', 'w') as outfile:
        # Add header giving shape of array
        # Any line starting with "#" will be ignored by numpy.loadtxt
        outfile.write('# Array shape: {0}\n'.format(x_train.shape))

        # Iterating through a ndimensional array produces slices along
        # the last axis. This is equivalent to data[i,:,:] in this case
        sliceCount = 0
        for data_slice in x_train:
            # Keep track of the slice numbers
            outfile.write('# New slice %d\n'%sliceCount)

            # Work through each row and column of the 2d numpy array inside the 
            # slice, writing each column number to file in format of your choosing
            for row in data_slice:
                for col in row:
                    itemStr = "%8.6f,"%col
                    outfile.write(itemStr)
                outfile.write("\n")

            sliceCount += 1

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.