0

I need to read data from a text file, manipulate it, and store it all in array or list, or some other data structure so that I can tabulate it, and plot it using matplotlib.

I intend to have an input statement, to store a treatment date and time. This dates and times in the input file will be subtracted from this datetime, to give the time since treatment (either in minutes or hours).

Firstly, the input file I am working with is in the following format:

!05/04/2014
@1332
Contact Angle (deg)     106.87
Contact Angle Left (deg)    106.90
Contact Angle Right (deg)   106.85
Wetting Tension (mN/m)      -21.13
Wetting Tension Left (mN/m) -21.16
Wetting Tension Right (mN/m)    -21.11
Base Tilt Angle (deg)       0.64
Base (mm)           1.7001
Base Area (mm2)         2.2702
Height (mm)         1.1174
Sessile Volume (ul)     2.1499
Sessile Surface Area (mm2)  6.3842
Contrast (cts)          255
Sharpness (cts)         186
Black Peak (cts)        0
White Peak (cts)        255
Edge Threshold (cts)        105
Base Left X (mm)        2.435
Base Right X (mm)       4.135
Base Y (mm)         3.801
RMS Fit Error (mm)      2.201E-3

@1333
Contact Angle (deg)     105.42
Contact Angle Left (deg)    106.04
Contact Angle Right (deg)   104.80
Wetting Tension (mN/m)      -19.36
Wetting Tension Left (mN/m) -20.12
Wetting Tension Right (mN/m)    -18.59
Base Tilt Angle (deg)       0.33
Base (mm)           1.6619
Base Area (mm2)         2.1691
Height (mm)         0.9837
Sessile Volume (ul)     1.6893
Sessile Surface Area (mm2)  5.3962
Contrast (cts)          255
Sharpness (cts)         190
Black Peak (cts)        0
White Peak (cts)        255
Edge Threshold (cts)        105
Base Left X (mm)        2.397
Base Right X (mm)       4.040
Base Y (mm)         3.753
RMS Fit Error (mm)      3.546E-3

In the file, each new date starts with an '!' and is in the format shown (dd/mm/yyyy).

The tables should contain the datetime from the input file, the contact angle and finally minutes since treatment.

The code below extracts the relevant information needed from the text file and writes it to another file, but I don't know what the best way to store the information is.

with open(infile) as f, open(outfile, 'w') as f2:
    for line in f:
        if line.split():
            if line.split()[0][0] == '!':
                for i in range(1,11):
                    current_date += (line.split()[0][i])
                f2.write(current_date[:2] + ' ' + current_date[3:5] + ' ' + current_date[6:] + '\n')
            current_date = ""
            if line.split()[0][0] == '@':
                for i in range(0,5):
                    measure_time += (line.split()[0][i])
                f2.write(measure_time[1:3] + ":" + measure_time[3:] + '\n')
            if line.split()[0] == "Contact" and line.split()[2] == "(deg)":
                contact_angle = line.split()[-1].strip()
                f2.write("Contact Angle (deg): " + contact_angle + '\n\n')
            measure_time = ""
        else:
            continue

I've been playing with datetime too, and have some code that calculates the time since the treatment from a single input, but I would need this to apply for each date and time in the input file.

from datetime import datetime
import numpy as np

dt = input("Enter treatment date and time in format: dd mm yyyy hh:mm\n")
#dt = '27 03 2014 12:06'

dob = datetime.strptime(dt,'%d %m %Y %H:%M')



b = datetime(2014,3,27,16,22,0)
c = b-dob
print(c.seconds)
print(c.seconds/60)
print(c.seconds//3600)

Finally, I would like to use matplotlib to plot the contact angle versus time since treatment.

If anyone could help me with this I would greatly appreciate it.

Improve this question
5
  • You didn't state your question. What is your problem? Commented Apr 7, 2014 at 18:59
  • Sorry if it wasn't clear from the question. I can pull the relevant data into a file, but I want to be able to tabulate the points mentioned above (datetime, contact angle, time since treatment) and plot a graph. I have no idea how to go about this, and how to store the data in Python for manipulation. Commented Apr 7, 2014 at 19:12
  • There are few problems in your question, and general rule on SO is one problem at time. Give me a moment, I give you example how to parse data similar to yours (but don't expect complete solution, please). Commented Apr 7, 2014 at 19:17
  • One thing: are you sure, that data in your file is separated by spaces? If there are tabulators, parsing this file would be much easier for you to understand. Commented Apr 7, 2014 at 19:19
  • Thank you very much. The space between the label and the value are separated by tabs. Commented Apr 7, 2014 at 19:21

2 Answers 2

Reset to default
4

Here is how you can parse such file. All is stored in dictionary containing dictionaries (turtles all the way down :). Main keys ar IDs (@smth).

Alternative to that would be to store by date, each item being list of dictionaries by ID. But that would be easiest to do with collections.defauldict, which would probably confuse you a bit. Thus solution below might not be the best, but should be easier to understand for you.

data = {}

date = ID = values = None

for line in datafile:
    if line.lstrip().startswith('!'):
        date = line[1:].strip()
        print date, line
    elif line.lstrip().startswith('@'):
        ID = line[1:].strip()
        data[ID] = {}
        data[ID]['date'] = date
    elif line.strip(): # line not all whitespace
        if not ID: 
            continue # we skip until we get next ID
        try:
            words = line.split()
            value = float(words[-1]) # last word
            unit = words[-2].lstrip('(').rstrip(')')
            item = {'value': value, 'unit': unit}
            key = ' '.join(words[:-2])
            data[ID][key] = item
        except (ValueError) as err:
            print "Could not parse this line:"
            print line
            continue
    else: # if 'empty' line
        ID = None

I encourage you to analyse this line, by line, looking up methods in https://docs.python.org/2/. If you really get stuck ask in comments and someone can give you a link to more specific page. GL.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I've been through the code that you posted, and can understand what is happening with most of it. I was wondering now, if you could help me plotting the contact angle versus the time since treatment with matplotlib in an iPython environment?
1

You clearly have records, so your data would optimally be organized like that.

Each record in your example starts with @ (and then what I'm assuming is a measurement index). Each of these records has an extra field: the date listed at the top.

records = []
record = {}
for line in f:
    kv = line.strip().split('\t')
    if kv[0].startswith('@'):
        record['measurement_date'] = msr_date
        records.append(record)  # store the last record
        record = {}  # make a new empty record
        for n in range(21):
            kv = f.next().strip().split('\t')
            quantity = kv[0].split('(')[0].strip()
            value = float(kv[1])
            record[quantity] = value
    elif kv[0].startswith('!'):
        msr_date = datetime.strptime(kv[0][1:], "%d/%m/%Y")   # append it to the record later
    else: 
        pass  # empty line
records.pop()  # The first record is a dummy record
# the last record has nog been appended yet
record['measurement_date'] = msr_date
records.append(record)

At the end, you'll end up with a list records of dictionaries. You could then cycle over these to store them in a more efficient form, for example with numpy structured arrays.

arr = np.array([ (d['Contact Angle'], d['msr_date'], d['msr_date'] - treatment_date)
    for d in records ], dtype=[
    ('contact_angle', 'f4'),
    ('msr_date', 'datetime64'),
    ('lapse_time', 'timedelta64')])

Remark that you'll have to look up if datetime64 is your needed format (have a look at this SO question for that.

With this last arr you have everything neatly placed in "columns", but you can access them by name. You could for example plot

plt.plot(arr['lapse_time'], arr['contact_angle']) but you'll have to tell matplotlib to use timedelta arguments for its independant variable, as shown here for example.

Improve this answer

7 Comments

I haven't tried your solution yet, but am just writing this for clarity. The @1200 (for example) is just just notation for measured at 12:00. The four numbers after the '@' is the time in 24h format. Also, the date (!07/04/2014) line gets added on a new date. There can be several measurements that day, at different times.
Sorry, just an error that I'm getting. On line 24, value = float(kv[1]) - ValueError: could not convert string to float.
Is there by any chance also a tab between the unit and the quantity? In that case, you'll have to use kv[2] everywhere instead of kv[1].
I'd also like to point out that I think @m.wasowski 's solution for parsing the file is more elegant, as it gracefully tracks the HHMM specification (which he called ID) and takes special care of non-parseable content.
@Matthew if you get an IndexError, it's because you're trying to access an array element that does not exist. You could find out what's going wrong (and learning the way Python works) by adding a print(repr(kv[1])) before that line. You should see all your numbers being returned, with single quotes. Using the data you've provided and changing the spaces between the unit and the value into single tabs, I don't get this error, so either debug or edit the first post to reflect the actual file. However, I suggest you parse the data with the program given by @m-wasowski.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.