convert list with arrays from csv file into floats

Question

I have a csv file with list of samples of various parameters of accelerometer and gyroscopic data. It contains floats(IQR,Range,MAD,...) and arrays(ZCD values). When I read it back, they are returned as strings and for my processing,I need numbers (int/float). I tried

    results = map(int, results)

and

results = [int(i) for i in results]

I got them from here, but I am having a problem with the arrays.

Traceback (most recent call last):
 File "/home/pi/Desktop/imu/mpu_v16.py", line 458, in <module>
  main()
 File "/home/pi/Desktop/imu/mpu_v16.py", line 439, in main
  compare()
 File "/home/pi/Desktop/imu/mpu_v16.py", line 305, in compare
  nrow = [float(i) for i in nrow]
ValueError: could not convert string to float: [ 53  73  79 139]

How can I solve this?

I am adding a sample of my data:

    C,3.3440452548950501e-17,0.99999999999999645,0.89244290726827058,1.7947499048650259,3.3651560716219242,[108 123 149 220 235],0.99999999999999822,110223.99999999971,331.99999999999915,-5.9524005537131887e-17,0.99999999999999989,0.81051947660768831,1.4756039753111405,4.4268119259334213,[103 122 160 205 212],0.99999999999999989,110224.00000000001,332.00000000000006,-6.4540073419474463e-17,0.999999999999999,0.74198651253618131,0.63512619216067612,4.256170326687128,[106 164 192 226],0.99999999999999933,110224.00000000001,332.00000000000006,2.083131190971185e-16,1.0000000000000009,0.66659374901400581,0.52759419283475883,4.5104130995285256,[  7  14  45  56 150 327],1.0000000000000002,110223.99999999994,331.99999999999983,-3.0890618042093025e-17,0.99999999999999667,0.58289607514346964,0.21669963911591134,4.7919240951669444,[ 82 149 208],0.99999999999999822,110223.99999999942,331.99999999999824,-3.2771643497971487e-16,1.0000000000000009,0.58746356061392535,0.29681486739557372,5.2741718744905794,[ 26  48  59  66 114 171 231 242],1.0000000000000002,110224.00000000036,332.00000000000108,-0.57536274345915739,0.147595080030029,0.13018399571123057

the problem I see in the error you posted is that you are trying to convert an array of strings into float, whereas you should apply int() one by one. As a quick and dirty solution, apply another for loop. — giosans
– giosans, Commented Jul 7, 2016 at 13:59
Your csv does not contain numbers, it contains space separated sequences of numbers. — MisterMiyagi
– MisterMiyagi, Commented Jul 7, 2016 at 14:06
Looks like you wrote numpy array to the file as strings.you will need to strip the [] and then split each line on whitespace and cast each element — Padraic Cunningham
– Padraic Cunningham, Commented Jul 7, 2016 at 14:08
post your input file data, and the code you wrote to read it — joel goldstick
– joel goldstick, Commented Jul 7, 2016 at 14:12
You should give a sample of your data if you are having conversion problems. — MisterMiyagi
– MisterMiyagi, Commented Jul 7, 2016 at 14:12

MisterMiyagi · Accepted Answer · 2016-07-07 14:11:56Z

Your csv is malformed, or you are reading it incorrectly. What you have extracted aren't individual numbers, it's multiple numbers boxed into brackets. See the error message:

ValueError: could not convert string to float: [ 53  73  79 139]
            ###              This is your data ^               ^

So you have a list like ["[ 53 73 79 139]", "[ 123 12 11]"] when you should actually have [["53", "73", "79", "139"], ["123", "12", "11"]] or ["53", "73", "79", "139", "123", "12", "11"].

If your csv is malformed, you can correct this on the fly by splitting the erroneous values. This will give you a flat list like [53, 73, 79, 139]:

nrow = [float(i) for elem in nrow for i in elem.strip('[] ').split()]
#                ^              ^ extract every sequence of numbers
#                                          ^              ^ remove [] and whitespace around sequences
#                                                            ^     ^ split the actual numbers

This will give you a list of lists like [[ 53, 73, 79, 139], ...]:

    nrow = [[float(i) for i in elem.strip('[] ').split()] for elem in nrow ]

Community · Accepted Answer · 2017-05-23 11:58:59Z

1

Because your input consists of strings of floats AND strings of arrays, you need a way of differentiating between the two. One way is to perhaps check the string for spaces? Then, for float strings, simply use float(i) like you have been doing. Then for array strings, try splitting on the spaces to separate the array elements and then running float on each individual string float (convert a string to an array).

Arbitrary float input: "54.6"
Check for spaces: False
Call: float("54.6")

Arbitrary array input: "[ 53 73 79 139]"
Check for spaces: True
Remove brackets: " 53 73 79 139"
Split on spaces and store in an array: ["53","73","79","139"]
Call float on each element: [53,73,79,139]

edited May 23, 2017 at 11:58

CommunityBot

11 silver badge

answered Jul 7, 2016 at 14:08

brotatotes

1217 bronze badges

1 Comment

Kranthi Kumar Over a year ago

,Your solution not exactly solved my problem, but helped me somewhat.I had changed my storing method to overcome this array problem.

Collectives™ on Stack Overflow

convert list with arrays from csv file into floats

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related