Numpy inconsistent results with Pandas and missing values

Question

Why does numpy return different results with missing values when using a Pandas series compared to accessing the series' values as in the following:

import pandas as pd
import numpy as np

data = pd.DataFrame(dict(a=[1, 2, 3, np.nan, np.nan, 6]))
np.sum(data['a'])

#12.0

np.sum(data['a'].values)
#nan

Based on the answer by @coldspeed, this isn't quite a duplicate. I'm willing to remove it though if it doesn't add anything. — willk
– willk, Commented Mar 12, 2019 at 20:37

cs95 · Accepted Answer · 2019-03-12 20:28:35Z

5

Calling np.sum on a pandas Series delegates to Series.sum, which ignores NaNs when computing the sum (BY DEFAULT).

data['a'].sum()
# 12.0

np.sum(data['a'])
# 12.0

You can see this from the source code of np.sum:

np.sum??

def sum(a, axis=None, dtype=None, out=None, keepdims=np._NoValue, initial=np._NoValue):
    ...
    return _wrapreduction(a, np.add, 'sum', axis, dtype, out, keepdims=keepdims,

Taking a look at the source code for _wrapreduction, we see:

np.core.fromnumeric._wrapreduction??

def _wrapreduction(obj, ufunc, method, axis, dtype, out, **kwargs):
    ...
    if type(obj) is not mu.ndarray:
        try:
            reduction = getattr(obj, method)   # get reference to Series.add

reduction is then finally called at the end of the function:

            return reduction(axis=axis, out=out, **passkwargs)

edited Mar 12, 2019 at 20:28

answered Mar 12, 2019 at 20:25

cs95

406k106 gold badges744 silver badges797 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Matt Messersmith Over a year ago

Dang, beat me to it. Was just copying over the numpy source haha. +1

willk Over a year ago

Thanks! This applies to other numpy functions acting on Pandas series as well (such as mean, min, max, median, etc.)?

cs95 Over a year ago

@willk Yes, most ufuncs, actually. There might be exceptions that I am not aware of though.

run-out Over a year ago

There is a numpy function for summing up arrays with NaN: numpy.nansum(): Function is used when we want to compute the sum of array elements over a given axis treating Not a Numbers (NaNs) as zero.

willk Over a year ago

Thanks. I guess the safest option is to use data['a'].sum(skipna=True) to be explicit. Or use the np.nansum function.

|

Collectives™ on Stack Overflow

Numpy inconsistent results with Pandas and missing values

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related