Python dataframe vectorizing for loop

Question

I would like to vectorize this piece of python code with for loop conditioned on current state for speed and efficiency.

values for df_B are computed based on current-state (state) AND corresponding df_A value.

Any ideas would be appreciated.

import pandas as pd
df_A = pd.DataFrame({'a': [0, 1, -1, -1, 1, -1, 0, 0] ,})
df_B = pd.DataFrame( data=0, index=df_A.index, columns=['b'])
print(df_A)

state = 0
for index, iter in df_A.iterrows():
    if df_A.loc[index ,'a'] == -1:
        df_B.loc[index ,'b'] = -10 -state
    elif df_A.loc[index, 'a'] == 1:
        df_B.loc[index, 'b'] = 10 - state
    elif df_A.loc[index, 'a'] == 0:
        df_B.loc[index, 'b'] = 0 - state
    temp_state = state
    state += df_B.loc[index, 'b']
print(df_B)

Quang Hoang · Accepted Answer · 2020-11-11 19:31:56Z

4

This seems overkill. Your state variable basically is the previous value in df_A['a']*10. So we can just use shift:

s = df_A['a'].mul(10) 

df_B['b'] = s - s.shift(fill_value=0)

edited Nov 11, 2020 at 19:31

answered Nov 11, 2020 at 19:17

Quang Hoang

151k11 gold badges64 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

collinb9 · Accepted Answer · 2020-11-11 18:45:41Z

You can make a class where state is a class variable. This will allow you to write a function which can be fed to an apply statement. This isn't a vectorized solution, but it is faster than iterrows. For example:

class ComputeB:
    def __init__(self, state=0):
        self.state = state
    
    def compute_b(self, row):
        row["b"] = row["a"]*10 - self.state
        self.state +=  row["b"]
        return row
df = pd.concat([df_A, df_B], axis = 1)
cb = ComputeB()
df = df.apply(lambda row: cb.compute_b(row), axis = 1)

And now df["b"] contains the values you wanted to compute. This does assume that df_A["a"] can only contain 0, 1 and -1. On my machine with a column of 40000 values the approach in the question took 10.4 seconds and this approach took 2.95 seconds.

Collectives™ on Stack Overflow

Python dataframe vectorizing for loop

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related