Faster way to access 2D lists/arrays and do calculations in Python?

Question

I am running 2 nested loops (first one 120 runs, second one 500 runs). In most of the 120x500 runs I need to access several lists and lists-in-lists (I will call them 2D arrays).

At the moment the 120x500 runs take about 4 seconds. Most of the time is taken by the three list appends and several 2D array accesses. The arrays are prefilled by me outside of the loops.

Here is my code:

    #Range from 0 to 119
    for cur_angle in range(0, __ar_angular_width-1):

        #Range from 0 to 499
        for cur_length in range(0, int(__ar_length * range_res_scale)-1):

            v_x = (auv_rot_mat_0_0*self.adjacent_dx[cur_angle][cur_length])+(auv_rot_mat_0_1*self.opposite_dy[cur_angle][cur_length])
            v_y = (auv_rot_mat_1_0*self.adjacent_dx[cur_angle][cur_length])+(auv_rot_mat_1_1*self.opposite_dy[cur_angle][cur_length])

            v_x_diff = (v_x+auv_trans_x) - ocp_grid_origin_x
            v_y_diff = (v_y+auv_trans_y) - ocp_grid_origin_y

            p_x = (m.floor(v_x_diff/ocp_grid_resolution))
            p_y = (m.floor(v_y_diff/ocp_grid_resolution))

            data_index = int(p_y * ocp_grid_width + p_x)

            if data_index >= 0 and data_index < (len(ocp_grid.data)-1):
                probability = ocp_grid.data[data_index]

                if probability == 100:
                    if not m.isnan(self.v_directions[cur_angle]):
                        magnitude = m.pow(probability, 2) * self.magnitude_ab[cur_length]

                        ov_1 = self.v_directions[cur_angle]
                        ov_2 = magnitude
                        ov_3 = self.distances[cur_length]

                        obstacle_vectors.append(ov_1)
                        obstacle_vectors.append(ov_2)
                        obstacle_vectors.append(ov_3)

I tried to figure out the processing times via time.time() and building differences, but it didn't work reliable. The calculated times were fluctuating quite a lot.

I am not really a Python pro, so any advices are welcome. Any ideas how to make the code faster?

EDIT: The initialization of the arrays was done with this code:

        self.adjacent_dx = [i[:] for i in [[0]*(length_iterations-1)]*(angular_iterations-1)]
        self.opposite_dy = [i[:] for i in [[0]*(length_iterations-1)]*(angular_iterations-1)]

Do not use time.time() directly to time the results. Wrap your code in a specific function and use the timeit module. In particular it's pretty simple to profile when using the IPython console and its %timeit magic. — Bakuriu
– Bakuriu, Commented Aug 26, 2013 at 9:34
Thanks for the quick tips. I will look into timeit. I am not sure about NumPy though, isn't it only quicker for big calculations? — Josch
– Josch, Commented Aug 26, 2013 at 9:53
For this kind of operations NumPy seems to need about 2-10 times more than python lists. I assume NumPy is only a good choice for larger and more complex operations. — Josch
– Josch, Commented Aug 29, 2013 at 12:09

HankMoody · Accepted Answer · 2013-08-26 10:18:45Z

2

Some general tips:

Try to optimize your algorithm the best You can.
If it's possible, then use PyPy instead of regular Python. This usually doesn't require any code modification if all Your external dependencies work with it.
Static typing and compilation to C can add additional boost, but requires some simple code modifications. For this purpose You can use Cython.

Note that step 1 is very ofter hard to do and most time consuming, giving you small amounts of boost if you have already good code, while the steps 2 and 3 give you dramatic boost without much additional effort.

edited Aug 26, 2013 at 10:18

answered Aug 26, 2013 at 10:08

HankMoody

3,1831 gold badge21 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Josch Over a year ago

Thank you, I will have a look into these possibilities. The algorithm is already very small, so as you said it's probably the best to look at step 2 and 3.

Collectives™ on Stack Overflow

Faster way to access 2D lists/arrays and do calculations in Python?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related