Efficient way to loop through large numpy arrays

Question

I'm working on building a fuzzy inference system with skfuzzy and I need to find a way to speed up my code:

import skfuzzy as fuzz
from skfuzzy import control as ctrl
import numpy as np


def FIS(s, r):
    #Generate universe variables
    a = ctrl.Antecedent(np.arange(0, 70.1, 0.1), 'a')
    b = ctrl.Antecedent(np.arange(0, 6.01, 0.01), 'b')
    c = ctrl.Consequent(np.arange(0, 12.01, 0.01), 'c')

    #Generate fuzzy membership functions
    #a
    a['l'] = fuzz.trapmf(a.universe, [0.0, 0.0, 3.0, 6.0])
    a['m'] = fuzz.trapmf(a.universe,[3.0, 6.0, 16.0, 24.0])
    a['h'] = fuzz.trapmf(a.universe, [16.0, 24.0, 30.0, 45.0])
    a['e'] = fuzz.trapmf(a.universe, [30.0, 45.0, 70.0, 70.0])

    #b
    b['l'] = fuzz.trapmf(b.universe, [0, 0, 0.01, 0.02])
    b['m'] = fuzz.trapmf(b.universe,[0.01, 0.02, 0.03, 0.05])
    b['h'] = fuzz.trapmf(b.universe, [0.03, 0.05, 0.10, 0.12])
    b['e'] = fuzz.trapmf(b.universe, [0.10, 0.12, 6.00, 6.00])

    #c
    c['l'] = fuzz.trapmf(c.universe, [0, 0, 0.01, 0.02])
    c['m'] = fuzz.trapmf(c.universe,[0.01, 0.02, 0.04, 0.05])
    c['h'] = fuzz.trapmf(c.universe, [0.04, 0.05, 0.10, 0.20])
    c['e'] = fuzz.trapmf(c.universe, [0.10, 0.20, 12.00, 12.00])

    #FUZZY RULES
    rule1 = ctrl.Rule(a['l'] & b['l'], c['l'])
    rule2 = ctrl.Rule(a['l'] & b['m'], c['m'])
    rule3 = ctrl.Rule(a['l'] & b['h'], c['h'])
    rule4 = ctrl.Rule(a['m'] & b['l'], c['m'])
    rule5 = ctrl.Rule(a['m'] & b['m'], c['m'])
    rule6 = ctrl.Rule(a['m'] & b['h'], c['h'])
    rule7 = ctrl.Rule(a['h'] & b['l'], c['h'])
    rule8 = ctrl.Rule(a['h'] & b['m'], c['h'])
    rule9 = ctrl.Rule(a['h'] & b['h'], c['e'])
    rule10 = ctrl.Rule(a['e'], c['e'])
    rule11 = ctrl.Rule(b['e'], c['e'])

    #CONTROL SYSTEM
    c_ctrl = ctrl.ControlSystem([rule1, rule2, rule3, rule4, rule5, rule6,
    rule7, rule8, rule9, rule10, rule11])
    c_simulation = ctrl.ControlSystemSimulation(c_ctrl)

    c_simulation.input['a'] = s
    c_simulation.input['b'] = r

    c_simulation.compute()

    value = c_simulation.output['c']

    return value

#Fake data
s_data = np.random.RandomState(1234567890)
s_data = s_data.randint(0, 70, size=600000)

r_data = np.random.random_sample(600000)

vec1 = s_data.flatten().astype('float')
vec2 = r_data.flatten().astype('float')

#pre allocate output array
cert = np.zeros(np.shape(vec1))*np.nan

#Find index of all finite elements of the array
ind = np.where(np.isfinite(vec1))[0]

# classify 
for k in xrange(len(ind)):
   cert[ind[k]] = FIS(vec1[ind[k]], vec2[ind[k]])

My calculations are taking more than 10+ hours to complete. How do I perform these calculations without a for loop? Ideally, I would be looking for a numpy solution, but I am open to alternative solutions.

So you are calling FIS thousands of times? What takes up time in FIS? 2/3 of that function is setup which could be performed just once. I have no idea what simulation.compute does. Anyways, my guess is that you should worry about time spent in each FIS call rather than worrying about the looping mechanism. — hpaulj
– hpaulj, Commented Nov 10, 2016 at 20:30
It certainly looks like everything above c_simulation.input['a'] = s in FIS should not be within that loop... you're recomputing all of that len(ind) times. — TemporalWolf
– TemporalWolf, Commented Nov 10, 2016 at 20:37
Good point. I took out what you suggested outside of the for loop but still have the same issue. The issue is that there are 600,000 calculations to be made. — brainier7
– brainier7, Commented Nov 10, 2016 at 20:51
There's no skfuzzy tag, which implies that there is little knowledge about that package on this board. I glanced at its github, and the develop activity is low key. You'll need to seek help from one of the developers. — hpaulj
– hpaulj, Commented Nov 11, 2016 at 1:01
This isn't about looping over large array; try cert[ind[k]] = vec1[ind[k]]+vec2[ind[k]]) (or some simple operation on the 2 vec. That will be much faster. It's about performing FIS(...) many times. try a timeit for FIS(vec1[ind[0]], vec2[ind[0]]). — hpaulj
– hpaulj, Commented Nov 11, 2016 at 1:24

brainier7 · Accepted Answer · 2016-11-17 18:48:43Z

1

I'd like to give credit to JDWarner from the scikit-fuzzy google group for an excellent answer to the above question. I think other scikit-fuzzy users will find this posting useful in the future.

https://groups.google.com/forum/#!topic/scikit-fuzzy/AOO0iNuHEuo

answered Nov 17, 2016 at 18:48

brainier7

514 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mankoff Over a year ago

And a new comment showing that scikit-fuzzy now supports arrays: groups.google.com/g/scikit-fuzzy/c/d4MXJ3h8inc/m/JZQAqarwDAAJ

Collectives™ on Stack Overflow

Efficient way to loop through large numpy arrays

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related