Creating a conditional dataframe column with recursive action in pandas

Question

I have a df like below:

time               sb       bn          bs
2020-04-07         True   False
2020-04-08         True   False
2020-04-09         True   False
2020-04-13        False   False
2020-04-14        False   False
2020-04-15        False   False
2020-04-16        False   False
2020-04-17        False   False
2020-04-20        False   False
2020-04-21        False    True
2020-04-22        False   False
2020-04-23        False   False

And condition like below:

if -> bn and !sb then True
else if -> bs[1] and sb then False
else -> bs[1]

And my code:

conditions = [
    (df["bn"] == True) & (df["sb"] == False),
    (df["bs"].shift(1) == True) & (df["sb"] == True)]
choices = ["True", "False"]
df["bs"] = np.select(conditions, choices, default=df["bs"].shift(1))

I am getting an object not callable error. I think it is because of .shift(1). (I am using shift(1) for previous days data) The desired df with new bs column should be like:

time              sb       bn          bs
2020-04-07        True     False       False
2020-04-08        True     False       False
2020-04-09        True     False       False
2020-04-13        False    False       False	
2020-04-14        False    False       False
2020-04-15        False    False       False
2020-04-16        False    False       False
2020-04-17        False    False       False
2020-04-20        False    False       False
2020-04-21        False    True        True
2020-04-22        False    False       True
2020-04-23        False    False       True

What am I doing wrong? Thank you

Your logic is flawed to get that result. When you run the first mask you get a 'True' at column 'bs' and 2020-04-21. It's the only True in the column. When you shift 1 it moves to 2020-04-22, where sb and bn are false. So when you use the condition of True when bs shift 1 is Ture and sb is True, there are no rows that meet that condition. Not sure how you are trying to get the above df result, but it's not with that logic. — run-out
– run-out, Commented May 25, 2020 at 11:54
On a more technical note, df["bs"].shift(1) is a Series, but np.select() expects a scalar value as the default parameter. — Arne
– Arne, Commented May 25, 2020 at 12:03

Arne · Accepted Answer · 2020-05-25 12:49:21Z

2

Since you need the previous value in the condition that defines the assignment, I don't see a better way than just using a for loop. Note that we need to handle the first row separately, because there is no previous value in that case. And you can shorten the conditional expressions a bit by setting all bs values to False initially.

df['bs'] = np.repeat(False, len(df))

if df.bn[0] and not df.sb[0]:
    df.bs[0] = True

for i in range(1, len(df)):
    if df.bn[i] and not df.sb[i]:
        df.bs[i] = True
    else:
        if not (df.bs[i-1] and df.sb[i]):
            df.bs[i] = df.bs[i-1]          

df

            sb      bn      bs
2020-04-07  True    False   False
2020-04-08  True    False   False
2020-04-09  True    False   False
2020-04-13  False   False   False
2020-04-14  False   False   False
2020-04-15  False   False   False
2020-04-16  False   False   False
2020-04-17  False   False   False
2020-04-20  False   False   False
2020-04-21  False   True    True
2020-04-22  False   False   True
2020-04-23  False   False   True

answered May 25, 2020 at 12:49

Arne

10.6k2 gold badges22 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

saltuklev Over a year ago

Ame, "np.select() expects a scalar value as the default parameter" this is a neat info. Thank you for the path

Collectives™ on Stack Overflow

Creating a conditional dataframe column with recursive action in pandas

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related