Using a list of conditions to filter a DataFrame in Pandas

Question

I wish to have a function which takes a list of conditions, of any length, and places an ampersand between all the conditions. Example code below.

df = pd.DataFrame(columns=['Sample', 'DP','GQ', 'AB'],
         data=[
               ['HG_12_34', 200, 35, 0.4],
               ['HG_12_34_2', 50, 45, 0.9],
               ['KD_89_9', 76, 67, 0.7],
               ['KD_98_9_2', 4, 78, 0.02],
               ['LG_3_45', 90, 3, 0.8],
               ['LG_3_45_2', 15, 12, 0.9]
               ])


def some_func(df, cond_list):

    # wrap ampersand between multiple conditions
    all_conds = ?

    return df[all_conds]

cond1 = df['DP'] > 40
cond2 = df['GQ'] > 40
cond3 = df['AB'] < 0.4


some_func(df, [cond1, cond2]) # should return df[cond1 & cond2]
some_func(df, [cond1, cond3, cond2]) # should return df[cond1 & cond3 & cond2]

I would appreciate any help with this.

Community · Accepted Answer · 2017-05-23 10:30:52Z

9

You can use functools.reduce for that:

from functools import reduce

def some_func(df, cond_list):
    return df[reduce(lambda x,y: x&y, cond_list)]

Or, like @AryaMcCarthy says, you can use and_ from the operator package:

from functools import reduce
from operator import and_

def some_func(df, cond_list):
    return df[reduce(and_, cond_list)]

or with numpy - like @ayhan says - which has also a logical and reduction:

from numpy import logical_and

def some_func(df, cond_list):
    return df[logical_and.reduce(cond_list)]

All three versions produce - for your sample input - the following output:

>>> some_func(df, [cond1, cond2])
       Sample  DP  GQ   AB
1  HG_12_34_2  50  45  0.9
2     KD_89_9  76  67  0.7
>>> some_func(df, [cond1, cond2, cond3])
Empty DataFrame
Columns: [Sample, DP, GQ, AB]
Index: []

edited May 23, 2017 at 10:30

CommunityBot

11 silver badge

answered Apr 1, 2017 at 18:54

willeM_ Van Onsem

482k33 gold badges483 silver badges624 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Arya McCarthy Over a year ago

Might be even better to use operator.and_ instead of your custom lambda.

willeM_ Van Onsem Over a year ago

@AryaMcCarthy: yeah that is indeed more neat.

user2285236 Over a year ago

Or, from numpy: np.logical_and.reduce([cond1, cond2, cond3])

Collectives™ on Stack Overflow

Using a list of conditions to filter a DataFrame in Pandas

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related