Data Processing in Python

Question

I'm dealing with a text data file that has 8 columns, each listing temperature, time, damping coefficients, etc. I need to take lines of data only in the temperature range of 0.320 to 0.322. Here is a sample line of my data (there are thousands of lines):

time    temp    acq. freq.  amplitude   damping     etc....
6.28444 0.32060 413.00000   117.39371   48.65073    286.00159

The only columns I care about are time, temp, and damping. I need those three values to append to my lists, but only when the temperature is in the specified range (there are some lines of my data where the temperature is all the way up at 4 kelvins, and this data is garbage).

I am using Python 3. Here are the things I have tried thus far

f = open('alldata','r')
c = f.readlines()
temperature = []
newtemp = []
damping = []
time = []

for line in c [0:]:
 line = line.split()
 temperature.append(line[1])
 damping.append(line[4])
 time.append(line[0])

for i in temperature:
 if float(i)>0.320 and float(i)<0.325:
   newtemp.append(float(i))

when I printed the list newtemp, I could see that this code did correctly fill the list with temperature values only in that range, however I also need my damping list and time list to now only be filled with values that correspond to that small temperature range. I'm not sure how to achieve that with this code.

I have also tried this, recommended by someone here:

output = []
lines = open('alldata', 'r')
for line in lines:
 temp = line.split()
 if float(temp[1]) > 0.320 and float(temp[1]) < 0.322:
    output.append(line)
print(output)

And I get an error that says:

IOPub data rate exceeded. The notebook server will temporarily stop sending output to the client in order to avoid crashing it. To change this limit, set the config variable --NotebookApp.iopub_data_rate_limit.

I will note that I am very new to coding, so I apologize if this turns out to be a silly question.

What is the format of the file? Is it, for example, CSV, TSV or Excel? — tuomastik
– tuomastik, Commented Jul 21, 2017 at 18:47
Do you want to extract the data from the file using a Python script? Have you tried to implement something already? What kind of file do you have? You need to provide more information so we can help you. Please provide a small example of the data. — KelvinS
– KelvinS, Commented Jul 21, 2017 at 18:59
What type of file do you have ? excel, csv, txt? I posted an answer covering the csv and excel case using pandas module in python. — seralouk
– seralouk, Commented Jul 21, 2017 at 19:07

mdasari · Accepted Answer · 2017-07-21 20:23:52Z

1

Data:

temperature, time, coeff...
0.32, 12:00:23, 2,..
0.43, 11:22:23, 3,..

Here, temperature is in the first column.

output = []
lines = open('data.file', 'r')
for line in lines:
    temp = line.split(',')
    if float(temp[0]) > 0.320 and float(temp[0]) < 0.322:
        output.append(line)
print output

edited Jul 21, 2017 at 20:23

answered Jul 21, 2017 at 18:51

mdasari

4334 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

KelvinS Over a year ago

This is a great example, but it depends on the file type and the data separator. Moreover, I think the temp variable should be converted to float.

seralouk · Accepted Answer · 2017-07-21 19:09:13Z

0

You can use pandas module:

import pandas as pd

# if the file with the data is an excel file use:
df = pd.read_excel('data.xlsx')

# if the file is csv
df = pd.read_csv('data.csv')

# if the column name of interest is named 'temperature'
selected = df['temperature'][(df['temperature'] > 0.320) & (df['temperature'] < 0.322)]

If you do not have pandas installed see here

edited Jul 21, 2017 at 19:09

answered Jul 21, 2017 at 19:03

seralouk

33.6k10 gold badges127 silver badges141 bronze badges

Collectives™ on Stack Overflow

Data Processing in Python

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related