ValueError: could not convert string to float, When loading a list of arrays with different shape

Question

So I have a list of 2D arrays and they have different shapes. For example,(60,40) and (42,40).

A sample code for starting can be

y = []
y.extend([np.ones((60,40)),np.ones((42,40)),np.ones((60,40))])

I saved them using np.savetxt(os.path.join("D:\A\code_testing\dataset",'mfccs.csv'), y,fmt='%s',delimiter=",") for later use.

However, a ValueError occurred when I tried to load them back.

ValueError                                Traceback (most recent call last)
d:\A\Pycodes\Dataset_prep(Watkins) 2.0.ipynb Cell 9 in <cell line: 1>()
----> 1 X_mfcc = np.loadtxt(os.path.join("D:\A\code_testing\dataset",'mfccs.csv'),  delimiter=",")

File e:\Anaconda\lib\site-packages\numpy\lib\npyio.py:1148, in loadtxt(fname, dtype, comments, delimiter, converters, skiprows, usecols, unpack, ndmin, encoding, max_rows, like)
   1143 # read data in chunks and fill it into an array via resize
   1144 # over-allocating and shrinking the array later may be faster but is
   1145 # probably not relevant compared to the cost of actually reading and
   1146 # converting the data
   1147 X = None
-> 1148 for x in read_data(_loadtxt_chunksize):
   1149     if X is None:
   1150         X = np.array(x, dtype)

File e:\Anaconda\lib\site-packages\numpy\lib\npyio.py:999, in loadtxt.<locals>.read_data(chunk_size)
    995     raise ValueError("Wrong number of columns at line %d"
    996                      % line_num)
    998 # Convert each value according to its column and store
--> 999 items = [conv(val) for (conv, val) in zip(converters, vals)]
   1001 # Then pack it according to the dtype's nesting
   1002 items = pack_items(items, packing)

File e:\Anaconda\lib\site-packages\numpy\lib\npyio.py:999, in <listcomp>(.0)
    995     raise ValueError("Wrong number of columns at line %d"
...
    734 if '0x' in x:
    735     return float.fromhex(x)
--> 736 return float(x)

ValueError: could not convert string to float: '[[-4.4259256e+02  4.0493172e+01  2.5947336e+01 ...  3.9495504e+00'

So i later tried using method(Python: How to save lists of 2D numpy arrays of different lengths)

np.savez("D:\A\code_testing\dataset\mfcc.npz",y)
data = np.load("D:\A\code_testing\dataset\mfcc.npz")
list2 = list2 = [v for k, v in data.items()]
print(len(list2))

I got error ValueError: Object arrays cannot be loaded when allow_pickle=False

So, what exactly is y? It can't be a list of 2D arrays, because savetxt doesn't work on that. Couldn't you save each array to a separate csv file? Saving arrays of different sizes to a single CSV file is never going to work reliably. How could it tell where one ends and another starts? — Tim Roberts
– Tim Roberts, Commented Jul 27, 2022 at 22:00
@hpaulj Hi hpauji, I have added a sample code for starting. Thanks for your attention. — Leo
– Leo, Commented Jul 27, 2022 at 22:36
I want to see a sample of the file, not the code that creates it. Did you look at it yourself? — hpaulj
– hpaulj, Commented Jul 27, 2022 at 22:42

Tim Roberts · Accepted Answer · 2022-07-27 22:03:44Z

1

I think you need to do these separately. Again, ASSUMING y really is a Python list of numpy arrays:

for i,a in enumerate(y):
    name = 'mfccs%02d.csv' % i
    np.savetxt( name, a, delimiter=',', fmt='%s')

...

y = [np.loadtxt(name, delimiter=',') for name in glob.glob('mfcc*.csv')]

answered Jul 27, 2022 at 22:03

Tim Roberts

55.3k4 gold badges28 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Leo Over a year ago

I am creating a dataset that contains time series features extracted from audio files. Saving one by one may be a little troublesome as there are thousands NumPy arrays. Is there a way for me to save them into a single file? By the way, i have edited the question, adding a sample code for starting. Thanks

Leo Over a year ago

Sorry, I forgot to mention that it doesn't have to be a CSV file, anything that can help me save and load the list of arrays will work.

Tim Roberts Over a year ago

You might checkout the h5py module. HDF files are a common way to store non-homogenous data sets in data science.

Leo Over a year ago

The problem is solved as a * is needed when I run np.savez

Leo · Accepted Answer · 2022-07-28 17:22:29Z

1

The solution is to put a * before the list when doing the saving

np.savez("D:\A\code_testing\dataset\mfcc.npz",*y)

edited Jul 28, 2022 at 17:22

answered Jul 28, 2022 at 17:22

Leo

1532 silver badges10 bronze badges

Collectives™ on Stack Overflow

ValueError: could not convert string to float, When loading a list of arrays with different shape

2 Answers 2

4 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related