In short, numpy.sum(a, axis=None) sums all cells of an array, but sums over rows of a data frame. I thought that pandas.DataFrame is built on top of numpy.array, and should not have this different behavior? What's the under-the-hood conversion?
a1 = numpy.random.random((3,2))
a2 = pandas.DataFrame(a1)
numpy.sum(a1) # Sums all cells
numpy.sum(a2) # Sums over rows
OK the following is a dump of my pdb debugging session which shows how this ends up in pandas land:
In [*]:
a1 = np.random.random((3,2))
import pdb
a2 = pd.DataFrame(a1)
print(np.sum(a1)) # Sums all cells
pdb.set_trace()
np.sum(a2) # Sums over rows
3.02993889742
--Return--
> <ipython-input-50-92405dd4ed52>(5)<module>()->None
-> pdb.set_trace()
(Pdb) b 6
Breakpoint 2 at <ipython-input-50-92405dd4ed52>:6
(Pdb) c
> <ipython-input-50-92405dd4ed52>(6)<module>()->None
-> np.sum(a2) # Sums over rows
(Pdb) s
--Call--
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1623)sum()
-> def sum(a, axis=None, dtype=None, out=None, keepdims=False):
(Pdb) print(axis)
None
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1700)sum()
-> if isinstance(a, _gentype):
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1706)sum()
-> elif type(a) is not mu.ndarray:
(Pdb) sssssss
*** NameError: name 'sssssss' is not defined
(Pdb) ss
*** NameError: name 'ss' is not defined
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1707)sum()
-> try:
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1708)sum()
-> sum = a.sum
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\numpy\core\fromnumeric.py(1713)sum()
-> return sum(axis=axis, dtype=dtype, out=out)
(Pdb) print(axis)
None
(Pdb) s
--Call--
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\pandas\core\generic.py(3973)stat_func()
-> @Substitution(outname=name, desc=desc)
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\pandas\core\generic.py(3977)stat_func()
-> if skipna is None:
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\pandas\core\generic.py(3978)stat_func()
-> skipna = True
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\pandas\core\generic.py(3979)stat_func()
-> if axis is None:
(Pdb) s
> c:\winpython-64bit-3.4.2.4\python-3.4.2.amd64\lib\site-packages\pandas\core\generic.py(3980)stat_func()
-> axis = self._stat_axis_number
(Pdb) print(self._stat_axis_number)
0
(Pdb)
So basically once it ends up in pandas land there are some integrity checks, one of which is that if axis is None then it's assigned the value from self._stat_axis_number which is 0, hence the difference in behaviour. I'm not a pandas dev so they may shed more light on this but this explains the difference in output
In order to achieve the same output you have to call sum twice:
In [6]:
a2.sum(axis=0).sum()
Out[6]:
3.9180334059883006
Or
In [7]:
np.sum(np.sum(a2))
Out[7]:
3.9180334059883006
pandas.DataFrame? Hitherto I've thought about it as numpy.array, but given this, it's not always appropriate.
axis=Noneis being overridden and set toaxis=0when it goes through the dfaxis=Noneit's being assigned toself._stat_axis_numberwhich is0, hence the difference in behaviournumpy.sum(a2.values)?