Optimizing nested Python for loops?

Question

I am having some trouble with optimizing the following for loops. I have looked into the map function, comprehension expressions, and a bit of itertools and generators, but am unsure at how to approach optimization for nested loops. Any help or suggestions is greatly appreciated. Thanks in advance!

Note that object architecture is:

self.variables.parameters.index1/index2/value
self.variables.rate
self.state.num

Loop 1:

mat1 = np.zeros(m, n)
for idx, variable in enumerate(self.variables):
    for parameter in variable.parameters:
        tracking_idx = parameter.index1 + parameter.index2
        mat1[tracking_idx, idx] = parameter.value

Loop 2:

mat2 = []
for variable in self.variables:
    rate = variable.rate
    for parameter in variable.parameters:
        if parameter.value < 0 and self.state.num[parameter.index1, parameter.index2] <= 0:
            rate = 0
    mat2.append(rate)

For the second loop, if you have a large number of parameters, adding a break after the rate = 0 statement might help, as it stops it checking the other parameters — Ignacy Debicki
– Ignacy Debicki, Commented Jan 12, 2017 at 23:46

hpaulj · Accepted Answer · 2017-01-13 17:20:52Z

With a numpy tag I assume 'optimize' means cast these loops as compiled numpy array expressions. I don't think there's a way to do without first collecting all the data as numpy arrays, which will require the same loops.

You have a list of n variables. Each variable has a list of ? parameters. Each parameter has 3 or 4 attributes.

So

rates = np.array([v.rate for v in variables])
values = np.array([[p.value for p in v.parameters] for v in variables]
index1s = <dito>   (?,n) array
index2s = <dita>   (?,n) array

self.state.num is already a 2d array, size compatible with the range of values in index1s and index2s.

Given those 1 and 2d arrays we should be able to derive mat1 and mat2 with whole array operations. If ? is small relative to m and the range of values in index1 and index2 it might be worth do this. I don't have a realistic feel for your data.

===========

You mention the map function, comprehension expressions, and a bit of itertools and generators. These can make the code look a little cleaner but don't make much difference in speed.

I demonstrated the use of list comprehensions. Those expression can be more elaborate, but often at the cost of readability. I like writing helper functions to hide details. A generator comprehension can replace a list comprehension that feeds another. Maps cover the same functionality.

Since variables and parameters have attributes I assume you have defined them in classes. You could write methods that extract those attributes as simple lists or arrays.

class Variable(....):
   ....
   def get_values(self):
       return [p.value for p in self.parameters]
   def get_rate(self, state):
       rate = self.rate
       for parameter in self.parameters:
            if parameter.value < 0 and                
                state.num[parameter.index1, parameter.index2] <= 0:
            rate = 0
       return rate

values = [v.get_values() for v in variables]
rates = [v.get_rate(self.state) for v in variables]

You could even write these helper function without the class structure.

That doesn't speed up anything; it just hides some details in the object.

Thanks! I used a list comprehension for mat1 and kept the for loop for mat2. It reduced run time by 20% or so!

Collectives™ on Stack Overflow

Optimizing nested Python for loops?

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related