3

I'm trying to get my head around the amazing list processing abilities of python (And eventually numpy). I'm converting some C code I wrote to python.

I have a text datafile where first row is a header, and then every odd row is my input data and every even row is my output data. All data space separated. I'm quite chuffed that I managed to read all the data into lists using nested list comprehensions. amazing stuff.

with open('data.txt', 'r') as f:
    # get all lines as a list of strings
    lines = list(f)

    # convert header row to list of ints and get info
    header = map(int, lines[0].split(' '))
    num_samples = header[0]
    input_dim = header[1]
    output_dim = header[2]
    del header    

    # bad ass list comprehensions 
    inputs = [[float(x) for x in l.split()] for l in lines[1::2]]
    outputs = [[float(x) for x in l.split()] for l in lines[2::2]]
    del x, l, lines

Then I want to produce a new list where each element is a function of a corresponding input-output pair. I couldn't figure out how to do this with any python specific optimizations. Here it is in C-style python:

# calculate position
pos_list = [];
pos_y = 0
for i in range(num_samples):
    pantilt = outputs[i];
    target = inputs[i];

    if(pantilt[0] > 90):
        pantilt[0] -=180
        pantilt[1] *= -1
    elif pantilt[0] < -90:
        pantilt[0] += 180
        pantilt[1] *= -1

    tan_pan = math.tan(math.radians(pantilt[0]))
    tan_tilt = math.tan(math.radians(pantilt[1]))

    pos = [0, pos_y, 0]
    pos[2] = tan_tilt * (target[1] - pos[1]) / math.sqrt(tan_pan * tan_pan + 1)
    pos[0] = pos[2] * tan_pan
    pos[0] += target[0]
    pos[2] += target[2]
    pos_list.append(pos)
del pantilt, target, tan_pan, tan_tilt, pos, pos_y

I tried to do it with a comprehension, or map but couldn't figure out how to:

  • draw from two different lists (both input and output) for each element of the pos_list array
  • put the body of the algorithm in the comprehension. would it have to be a separate function or is there a funky way of using lambdas for this?
  • would it even be possible to do this with no loops at all, just stick it in numpy and vectorize the whole thing?
Improve this question

3 Answers 3

Reset to default
2

One vectorized approach using boolean-indexing/mask -

import numpy as np

def mask_vectorized(inputs,outputs,pos_y):
    # Create a copy of outputs array for editing purposes
    pantilt_2d = outputs[:,:2].copy()

    # Get mask correspindig to IF conditional statements in original code
    mask_col0_lt = pantilt_2d[:,0]<-90
    mask_col0_gt = pantilt_2d[:,0]>90

    # Edit the first column as per the statements in original code
    pantilt_2d[:,0][mask_col0_gt] -= 180
    pantilt_2d[:,0][mask_col0_lt] += 180

    # Edit the second column as per the statements in original code
    pantilt_2d[ mask_col0_lt | mask_col0_gt,1] *= -1

    # Get vectorized tan_pan and tan_tilt 
    tan_pan_tilt = np.tan(np.radians(pantilt_2d))

    # Vectorized calculation for: "tan_tilt * (target[1] .." from original code 
    V = (tan_pan_tilt[:,1]*(inputs[:,1] - pos_y))/np.sqrt((tan_pan_tilt[:,0]**2)+1)

    # Setup output numpy array
    pos_array_vectorized = np.empty((num_samples,3))

    # Put in values into columns of output array
    pos_array_vectorized[:,0] = inputs[:,0] + tan_pan_tilt[:,0]*V
    pos_array_vectorized[:,1] = pos_y
    pos_array_vectorized[:,2] = inputs[:,2] + V

    # Convert to list, if so desired for the final output
    # (keeping as numpy array could boost up the performance further)
    return pos_array_vectorized.tolist()

Runtime tests

In [415]: # Parameters and setup input arrays
     ...: num_samples = 1000
     ...: outputs = np.random.randint(-180,180,(num_samples,5))
     ...: inputs = np.random.rand(num_samples,6)
     ...: pos_y = 3.4
     ...: 

In [416]: %timeit original(inputs,outputs,pos_y)
100 loops, best of 3: 2.44 ms per loop

In [417]: %timeit mask_vectorized(inputs,outputs,pos_y)
10000 loops, best of 3: 181 µs per loop
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

Suppose you read your file into a list, like so:

lines = open('data.txt', 'r').readlines()

The header is this:

lines[0]

The even lines are:

even = lines[1:][::2]

and the odd lines are:

odd = lines[2:][::2]

Now you can create a list using itertools.izip from these two lists:

itertools.izip(even, odd)

This is a sort of list-like thingy (you can loop over it, or just write list( ... ) around it to make it into a true list), whose each entry is a pair of your input-output data.

Improve this answer

7 Comments

wow that's pretty cool. So I could do a comprehension directly on the resulting iterator. e.g. pos_list = [myfunc(d) for d in itertools.izip(even, odd)] where d[0] is the even row and d[1] is the corresponding odd row?
You absolutely should! Note that d is a pair, so to access it, you should use d[0] and d[1]. Ain't Python cool?
this can be written much simpler: with open('data.txt') as f: header = next(f); pairs = itertools.izip(f, f)
well interestingly doing it with a list comprehension on itertools.izip is 0.08ms slower than doing it with a big for loop like I wrote above! perhaps because of the function call (in fact when I put a function call in the for loop, it's identical to the list comprehension and map). But I guess we wouldn't do this for performance. It's more about readibility (I was just curious about performance).
Not sure that's correct - when you read it more than once, the cache can change stuff. It's a bit tricky to measure it correctly. It's possible, though.
|
0

If anyone stumbles upon the same question, here are four variations based on Ami's suggestion (functions do1, do1b, do2, do3)

And for those curious, here are the benchmarks (I have ~1000 input-output pairs of data. Maybe with radically more data the benchmarks would vary more)

  • %timeit do3() - 100 loops, best of 3: 2.72 ms per loop
  • %timeit do2() - 100 loops, best of 3: 2.73 ms per loop
  • %timeit do1b() - 100 loops, best of 3: 2.74 ms per loop
  • %timeit do1() - 100 loops, best of 3: 2.67 ms per loop

....

def load_file(filename = 'Sharpy_7.txt'):
    global file_data, num_samples, input_dim, output_dim
    with open(filename, 'r') as f:
        # get all lines as a list of strings
        file_data = list(f)  

        # convert header row to list of ints and get info
        header = map(int, file_data[0].split(' '))
        num_samples = header[0]
        input_dim = header[1]
        output_dim = header[2]
        f.close()


def calc_pos2(d):
    target = d[0]
    pantilt = d[1]

    if(pantilt[0] > 90):
        pantilt[0] -=180
        pantilt[1] *= -1
    elif pantilt[0] < -90:
        pantilt[0] += 180
        pantilt[1] *= -1

    tan_pan = math.tan(math.radians(pantilt[0]))
    tan_tilt = math.tan(math.radians(pantilt[1]))

    pos = [0, 0, 0]
    pos[2] = tan_tilt * (target[1] - pos[1]) / math.sqrt(tan_pan * tan_pan + 1)
    pos[0] = pos[2] * tan_pan
    pos[0] += target[0]
    pos[2] += target[2]
    return pos


def calc_pos(target, pantilt):
    if(pantilt[0] > 90):
        pantilt[0] -=180
        pantilt[1] *= -1
    elif pantilt[0] < -90:
        pantilt[0] += 180
        pantilt[1] *= -1

    tan_pan = math.tan(math.radians(pantilt[0]))
    tan_tilt = math.tan(math.radians(pantilt[1]))

    pos = [0, 0, 0]
    pos[2] = tan_tilt * (target[1] - pos[1]) / math.sqrt(tan_pan * tan_pan + 1)
    pos[0] = pos[2] * tan_pan
    pos[0] += target[0]
    pos[2] += target[2]
    return pos


def calc_stats():
    global pos_array, pos_avg, pos_std       
    pos_array = np.asarray(pos_list)
    pos_avg = np.mean(pos_array, 0)
    pos_std = np.std(pos_array, 0)


# map on itertools.izip
def do3():
    global pos_list

    # bad ass list comprehensions 
    target_list = [[float(x) for x in l.split()] for l in file_data[1::2]]
    pantilt_list = [[float(x) for x in l.split()] for l in file_data[2::2]]

    # calculate position
    pos_list = map(calc_pos2, itertools.izip(target_list, pantilt_list))


# list comprehension on itertools.izip
def do2():
    global pos_list

    # bad ass list comprehensions 
    target_list = [[float(x) for x in l.split()] for l in file_data[1::2]]
    pantilt_list = [[float(x) for x in l.split()] for l in file_data[2::2]]

    # calculate position
    pos_list = [calc_pos(d[0], d[1]) for d in itertools.izip(target_list, pantilt_list)]


# for loop with function call
def do1b():
    global pos_list

    # bad ass list comprehensions 
    target_list = [[float(x) for x in l.split()] for l in file_data[1::2]]
    pantilt_list = [[float(x) for x in l.split()] for l in file_data[2::2]]

    # calculate position
    pos_list = [];
    for i in range(num_samples):
        pos_list.append(calc_pos(target_list[i], pantilt_list[i]))


# for loop with unrolled algorithm
def do1():
    global pos_list

    # bad ass list comprehensions 
    target_list = [[float(x) for x in l.split()] for l in file_data[1::2]]
    pantilt_list = [[float(x) for x in l.split()] for l in file_data[2::2]]

    # calculate position
    pos_list = [];
    for i in range(num_samples):
        pantilt = pantilt_list[i];
        target = target_list[i];

        if(pantilt[0] > 90):
            pantilt[0] -=180
            pantilt[1] *= -1
        elif pantilt[0] < -90:
            pantilt[0] += 180
            pantilt[1] *= -1

        tan_pan = math.tan(math.radians(pantilt[0]))
        tan_tilt = math.tan(math.radians(pantilt[1]))

        pos = [0, 0, 0]
        pos[2] = tan_tilt * (target[1] - pos[1]) / math.sqrt(tan_pan * tan_pan + 1)
        pos[0] = pos[2] * tan_pan
        pos[0] += target[0]
        pos[2] += target[2]
        pos_list.append(pos)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.