Recursive definitions in Pandas

Question

I have a time-series A holding several values. I need to obtain a series B that is defined algebraically as follows:

B[t] = a * A[t] + b * B[t-1]

where we can assume B[0] = 0, and a and b are real numbers.

Is there any way to do this type of recursive computation in Pandas? Or do I have no choice but to loop in Python as suggested in this answer?

As an example of input:

> A = pd.Series(np.random.randn(10,))

0   -0.310354
1   -0.739515
2   -0.065390
3    0.214966
4   -0.605490
5    1.293448
6   -3.068725
7   -0.208818
8    0.930881
9    1.669210

here the open issue to cythonize it: github.com/pydata/pandas/issues/4567, but some links are their as well — Jeff
– Jeff, Commented Oct 8, 2014 at 23:26
You can use scipy.signal.lfilter. See stackoverflow.com/questions/21336794/… for an example. — Warren Weckesser
– Warren Weckesser, Commented Oct 8, 2014 at 23:49

Warren Weckesser · Accepted Answer · 2015-03-17 00:35:55Z

22

As I noted in a comment, you can use scipy.signal.lfilter. In this case (assuming A is a one-dimensional numpy array), all you need is:

B = lfilter([a], [1.0, -b], A)

Here's a complete script:

import numpy as np
from scipy.signal import lfilter


np.random.seed(123)

A = np.random.randn(10)
a = 2.0
b = 3.0

# Compute the recursion using lfilter.
# [a] and [1, -b] are the coefficients of the numerator and
# denominator, resp., of the filter's transfer function.
B = lfilter([a], [1, -b], A)

print B

# Compare to a simple loop.
B2 = np.empty(len(A))
for k in range(0, len(B2)):
    if k == 0:
        B2[k] = a*A[k]
    else:
        B2[k] = a*A[k] + b*B2[k-1]

print B2

print "max difference:", np.max(np.abs(B2 - B))

The output of the script is:

[ -2.17126121e+00  -4.51909273e+00  -1.29913212e+01  -4.19865530e+01
  -1.27116859e+02  -3.78047705e+02  -1.13899647e+03  -3.41784725e+03
  -1.02510099e+04  -3.07547631e+04]
[ -2.17126121e+00  -4.51909273e+00  -1.29913212e+01  -4.19865530e+01
  -1.27116859e+02  -3.78047705e+02  -1.13899647e+03  -3.41784725e+03
  -1.02510099e+04  -3.07547631e+04]
max difference: 0.0

Another example, in IPython, using a pandas DataFrame instead of a numpy array:

If you have

In [12]: df = pd.DataFrame([1, 7, 9, 5], columns=['A'])

In [13]: df
Out[13]: 
   A
0  1
1  7
2  9
3  5

and you want to create a new column, B, such that B[k] = A[k] + 2*B[k-1] (with B[k] == 0 for k < 0), you can write

In [14]: df['B'] = lfilter([1], [1, -2], df['A'].astype(float))

In [15]: df
Out[15]: 
   A   B
0  1   1
1  7   9
2  9  27
3  5  59

edited Mar 17, 2015 at 0:35

answered Oct 9, 2014 at 0:06

Warren Weckesser

116k20 gold badges207 silver badges224 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Josh Over a year ago

Fantastic answer. Thanks Warren. I took classes in signals & systems (Oppenheim's book), and this feels so right. I will look into this answer carefully, since it looks like it is the right way to solve the problem. I presume this approach can only handle linear recursions, correct?

Warren Weckesser Over a year ago

Yes, only linear. (The l in lfilter stands for linear.)

Collectives™ on Stack Overflow

Recursive definitions in Pandas

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related