2

Here is a minimal example:

import pandas as pd
df = pd.DataFrame({'x': [0, 0, np.NaN, 1], 'y': [1, 0, 0, np.NaN], 'z': [np.NaN, 1, 1, 0]}, dtype = int, index = ['a', 'a', 'b', 'b'])

       x      y      z
a      0      1    NaN
a      0      0      1
b    NaN      0      1
b      1    NaN      0

Values can only be 0, 1, or NaN. I want to add rows that have the same index, ignoring NaN values. The result would be here:

       x      y      z
a      0      1      1 
b      1      0      1

The way I am doing it:

df.max(level = 0)

Is there a faster way?

Improve this question

1 Answer 1

Reset to default
1

It is same, performace should be similar - mainly it depends of data:

df.groupby(level = 0).max()

Time comparison: 

In [15]: %timeit df.groupby(level = 0).max()
    ...: 
100 loops, best of 3: 8.08 ms per loop
In [12]: %timeit df.max(level = 0)
    ...: 
100 loops, best of 3: 8.04 ms per loop

Some bigger data:

N = 100000
idx = np.random.randint(10000, size=N).astype(str)
df = pd.DataFrame(np.random.choice([0,1,np.nan], size=(N,3)), index=idx)
df = df.sort_index()
print (df.head())

In [174]: %timeit df.max(level = 0)
100 loops, best of 3: 19.5 ms per loop

In [175]: %timeit df.groupby(level = 0).max()
10 loops, best of 3: 24 ms per loop
Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

@Arpit Solanki - Thank you for timings.
welcome. Deleted mine because does not make sense to give a slower solution
Thanks for the alternative and the timings @jezrael. Size of data is about (20000, 200). I can see the timings between the two approaches are similar, as you mentioned.
@unfolx - You are welcome. Yes, exactly. I think df.max(level = 0) is less typing mainly, maybe df.groupby(level = 0).max() is more readable? Hard to say what is the best ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.