Extracting data from a string where the data structure is embedded in the string itself

Question

In a project we are doing we encounter log files of which each line has the following structure:

2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309

The structure of the string is embedded in the string itself.

First we have some metadata:

date: 2012/01/02
time: 12:50:32
measurement number: 658
number of measurement groups: 2

This is then followed by the data of each group sequentially.

Measurement group 1: 1,2,0,0,0,0,1556,1555,62,60
Measurement group 2: 2,3,0,0,0,0,1559,1557,1557,63,64,65

Group data has the following structure (measurement group 1 used below as an example):

number of the measurement group:1
number of sensors in this group:2
control field 1 to 4 (0 most of the time):0,0,0,0
raw values of type 1 for each sensor (>1500 in the examples):1556,1555
raw values of type 2 for each sensor (~60 in the examples),62,60

The line continues with the calculated values for all sensors mentioned above consecutively (i.e. no more control values, or raw values)

In the example, the total number of sensors = 2 + 3 = 5 so the calculated line is:

0.305,0.265,0.304,0.308,0.309

My question is this: If we want to normalize the values for each sensor like this:

date, time, number of measurement group, number sensor in group, (raw value type 1, raw value type 2, calculated value)

What would be a flexible solution, given that a any date-time each variable is well... variable (meaning that the number of measurement group is variable, and the number of sensors in each group can also be variable?

For the example final output should be something like:

2012/01/02,12:50:32,1,1,(1556,62,0.305)
2012/01/02,12:50:32,1,2,(1555,60,0.265)
2012/01/02,12:50:32,2,1,(1559,63,0.304)
2012/01/02,12:50:32,2,2,(1557,64,0.308)
2012/01/02,12:50:32,2,3,(1557,65,0.309)

What I did up to now is to segment the measurement into cases over time and define "statically" which columns are to be inserted for a line belonging to a case, which group a sensor belongs to, what its sensornumber is,...

This is hardly a good solution as each change in the measurement setup results in more changes to the code.

line="""2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309"""
parts=line.split(",")
date=parts[0]
groupnames=[1,1,2,2,2]
sensornumbers=[1,2,1,2,3]
raw_type1_idx=[10,11,20,21,22]
raw_type2_idx=[12,13,23,24,25]
calc_idx=[26,27,28,29,30]
for i,j,k,l,m in zip(groupnames,sensornumbers,raw_type1_idx,raw_type2_idx,calc_idx):
    output_tpl= parts[k],parts[l],parts[m]
    print "%s,%s,%s,%s" % (date,i,j,output_tpl)

Is there a better Python way of doing stuff like this?

MattH · Accepted Answer · 2012-01-06 14:53:57Z

2

Not a particularly nice data structure. Assuming there are always 4 control values, the following should work with arbitrary numbers of groups and sensors within.

sample = "2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309"

def parse_line(line):
  line = line.split(',')
  sensors = []
  date = line[0]
  time = line[1]
  row = line[2]
  groups = int(line[3])
  c = 4
  for i in range(groups):
    group_num    = line[c]
    sensor_count = int(line[c+1])
    sensor_data_len = 4 + sensor_count * 2
    sensor_data  = line[c+2+4:c+2+sensor_data_len]
    c += 2 + sensor_data_len
    for j in range(sensor_count):
      sensors.append([group_num,str(j+1)] + sensor_data[j::sensor_count])
  for s,v in zip(sensors,line[c:]):
    s.append(v)
  # Now have a list of lists, one per sensor sensor containing all the data
  for s in sensors:
    print ",".join([date,time]+s)

parse_line(sample)

Yielding:

2012-01-02,12:50:32,1,1,1556,62,0.305
2012-01-02,12:50:32,1,2,1555,60,0.265
2012-01-02,12:50:32,2,1,1559,63,0.304
2012-01-02,12:50:32,2,2,1557,64,0.308
2012-01-02,12:50:32,2,3,1557,65,0.309

answered Jan 6, 2012 at 14:53

MattH

38.4k11 gold badges85 silver badges84 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Speediro Over a year ago

MattH, looks like a clean solution.

Don Question · Accepted Answer · 2012-01-06 17:42:45Z

It's a non-trivial task. Probably the most "pythonic" way would be to create a class.

I took the liberty and time to make an example:

from collections import namedtuple

class DataPack(object):
    def __init__(self, line, seperator =',', headerfields = None, groupfields = None):        
        self.seperator = seperator
        self.header_fields = headerfields or ('date', 'time', 'nr', 'groups')
        self.group_fields = groupfields or ('nr', 'sensors','controlfields',
                                            't1values', 't2values')
        Header = namedtuple('Header', self.header_fields)

        self.header_part = line.split(self.seperator)[:self.data_start]
        self.data_start = len(self.header_fields)
        self.data_part = line.split(self.seperator)[self.data_start:]
        self.header = Header(*self.header_part)
        self.groups = self._create_groups(self.data_part, self.header.groups)

    def _create_groups(self, datalst, groups):
        """nr, sensors controllfield * 4, t1value*sensors, t2value*sensors """        
        Group = namedtuple('DataGroup', self.group_fields)
        _groups = []
        for i in range(int(groups)):
            nr = datalst[0]
            sensors = datalst[1]
            controlfields = datalst [2:6]
            t1values = datalst[6:6+int(sensors)]
            t2values = datalst[6+int(sensors):6+int(sensors)*2]
            _groups.append(Group(nr, sensors, controlfields, t1values, t2values))
            datalst = datalst[6+int(sensors)*2:]
        return _groups

    def __str__(self):
        _return = []        
        for group in self.groups:
            for sensor in range(int(group.sensors)):
                _return.append('%s, ' % self.header.date.replace('-','/'))
                _return.append('%s, ' % self.header.time)
                _return.append('%s, ' % group.nr)
                _return.append('%s, ' % (int(sensor) + 1,))
                _return.append('(%s, ' % group.t1values[int(sensor)])
                _return.append('%s)\n' % group.t2values[int(sensor)])
        return u''.join(_return)

if __name__ == '__main__':
    line = """2012-01-02,12:50:32,658,2,1,2,0,0,0,0,1556,1555,62,60,2,3,0,0,0,0,1559,1557,1557,63,64,65,0.305,0.265,0.304,0.308,0.309"""
    data = DataPack(line)
    for i in data.header: print i,
    for i in data.groups: print '\n',i
    print '\n',data
    print 'cfield 0:2 ', data.groups[0].controlfields[2]
    print 't2value 1:2 ', data.groups[1].t2values[2]

On bigger changes to the input-data you would have to subclass and overwrite the _create_groups and __str__ methods.

Don, this approach certainly makes it more reusable. Not being trained in OO makes that I tend to "forget" this approach. Thx
@Don Question: It's a non-trivial task - Clearly you have a different boundary for triviality than most engineers I've encountered, especially considering that there are two answers on this page, written in a matter of minutes after the question was asked.

Collectives™ on Stack Overflow

Extracting data from a string where the data structure is embedded in the string itself

2 Answers 2

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related