Reformat a text file so it can be used a numpy array, in python?

Question

I have a small chunk of code that i'm using to find the confidence interval from a data set.

from scipy import stats
import numpy as np

a = np.loadtxt("test1.txt")
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
    scale=sigma)

print(conf_int)

However, my text file (test1.txt) is a list of numbers that a) has a square bracket at the start and finish b)is not in equal columns.

"[-10.197663 -22.970129 -15.678419 -15.306197
-12.09961 -11.845362 -18.10553 -25.370747
-19.34831 -22.45586]

np.loadtxt really doesn't seem to like this, so is there any way i can use a function to either read and use the data as is or reformat it?

Any help would be greatly appreciated!

Update so i manged to remove my brackets with the code below

with open('test1.txt', 'r') as my_file:
text = my_file.read()
text = text.replace("[", "")
text = text.replace("]", "")


with open('clean.txt', 'w') as my_file:
my_file.write(text)


a = np.loadtxt("clean.txt")
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
   scale=sigma)

print(conf_int)

Just need to reformat clean.txt so its in one single column now so np.array will recognise it.

Final update

I managed to get it working, using @David Hoffman suggested code and my long work around from above; see below

from scipy import stats
import numpy as np

with open('test1.txt', 'r') as my_file:
    text = my_file.read()
    text = text.replace("[", "")
    text = text.replace("]", "")


with open('clean.txt', 'w') as my_file:
    my_file.write(text)


a = np.array(list(map(float, text.strip("[]").split())))
mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean,
   scale=sigma)

print(conf_int)

Thank you to everyone for taking the time to help, it was very much appreciated, especially to a new coder like me.

You can use ordinary Python text processing to load the strings, strip off brackets and quotes (if needed), split numbers etc. But to step back a bit. What was the source of this file? Something you generated earlier, or were given? — hpaulj
– hpaulj, Commented Aug 7, 2020 at 15:32
Hey, thanks for getting back to me. The file is one I've generated as part of a larger script. I'm pretty new to python and I've been working on it for the better part of a month. I've tried for a couple of days to change the output so the text file is formatted but I believe that it would be easier to do it the other way. Although I'm not sure. You said I could use pyhon text processing to strip off the brackets and reformat could you give me an example? — Amy D3
– Amy D3, Commented Aug 7, 2020 at 15:55

David Hoffman · Accepted Answer · 2020-08-08 18:42:00Z

1

This is what I would do:

import numpy as np
from scipy import stats
import requests

link = "https://pastebin.pl/view/raw/929f5228"

response = requests.get(link)
text = response.text

# with open("test1.txt", "r") as my_file:
#     text = my_file.read()

a = np.array(list(map(float, text.strip("[]").split())))

mean, sigma = np.mean(a), np.std(a)

conf_int = stats.norm.interval(0.95, loc=mean, scale=sigma)

print(conf_int)

The commented lines are for if you have a file.

There's a lot packed into the string handling line:

The text string is cleaned (removing brackets)
The clean text is split by white space (any length of consecutive whitespace characters are treated as delimiters)
Each split token is converted to a float (this is the map part)
The map generator is converted to a list and passed to the numpy array function

As @Dishin said, there's some weirdness with how your input file is formatted. If you have any control over how the file is written (say from a LabVIEW program or other Python script) it might be worth formatting the data in a more widely accepted format like .csv so that functions like np.loadtxt (or programs like Excel) can read it more easily.

If you're stuck with the files as is you can just make a little utility function like:

def loader(filename):
    with open(filename, "r") as my_file:
        text = my_file.read()

    return np.array(list(map(float, text.strip("[]").split())))

to reuse in your scripts.

answered Aug 8, 2020 at 18:42

David Hoffman

2,3931 gold badge18 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Amy D3 Over a year ago

Hi Thank you so much for getting back to me and giving me such a thorougher answer! However, every time i try the first code you suggested i get an error line 8, in <module> a = np.array(list(map(float, text.strip("[]").split()))) ValueError: could not convert string to float: ']' Is there anything you could suggest?

David Hoffman Over a year ago

@Amy it looks like the file you’re working with is different from the one you posted on pastebin. Could you post the file that’s giving the issue? Could you also update your post to show the code that generates the file?

Dishin Goyani · Accepted Answer · 2020-08-08 13:01:47Z

1

You can read it as string then replace space with , to make it like list and use eval to convert string list to list type and at last to numpy array.
For your given dummy input

li = """[-10.197663 -22.970129 -15.678419 -15.306197
-12.09961 -11.845362 -18.10553 -25.370747
-19.34831 -22.45586]"""

np.array(eval(li.replace(' ',',')))
array([-10.197663, -22.970129, -15.678419, -27.405807, -11.845362,
       -18.10553 , -44.719057, -22.45586 ])

For given input file - here solution would be

import re
li = open('test1.txt', 'r').read()

np.array(eval(re.sub(r'(\n +| +)',',',li)))
array([-10.197663  , -22.970129  , -15.678419  , -15.306197  ,
        -0.38851437, -12.09961   , -11.845362  , -18.10553   ,
       -25.370747  , -20.575884  , -19.34831   , -22.45586   ,
       -31.209     , -19.68507   , -31.07194   , -28.4792    ,
        ...])

edited Aug 8, 2020 at 13:01

answered Aug 7, 2020 at 16:05

Dishin Goyani

7,7533 gold badges33 silver badges42 bronze badges

4 Comments

Amy D3 Over a year ago

Hi, thanks so much for your help! BUT I keeping getting an error ,unexpected EOF while parsing - which i think is due to the way i'm indenting....i'm not sure

David Hoffman Over a year ago

Could you post the txt file?

Amy D3 Over a year ago

Sure! here you go pastebin.pl/view/929f5228

Dishin Goyani Over a year ago

@AmyD3 please check i have updated my answer for given input file. Also NOTE at end of the file you are missing ] en close square bracket which may cause unexpected EOF exception so try new solution after fixing that.

Collectives™ on Stack Overflow

Reformat a text file so it can be used a numpy array, in python?

2 Answers 2

2 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related