Python/numpy locating Runtime Warning

Question

I have a numpy script sitting on an app server, it gets called thousands of times and once in a blue moon I get a Runtime Warning:

/usr/local/lib/python2.7/dist-packages/scipy/stats/stats.py:2417: RuntimeWarning: invalid value encountered in double_scalars
r = (r_num / r_den)

Not sure where this occurs.
Why this occurs.
The consequence it has on the code if any. everything passes the eye test and unit test.

But again I'm not sure if I'm looking at the right place because the chance of this warrning happening is less than 1%

how can I get python to print out the location of the warning?

Community · Accepted Answer · 2017-05-23 10:28:08Z

If you put

np.seterr(all='raise')

near the beginning of your script, exceptions will be raised instead of warnings. That will halt your script with a nice traceback which will give you information about where the error is occurring.

You could then put a try...except around the line in your code that raises the exception, and use the except clause to log the value of relevant variables.

Also, the RuntimeWarning you posted says the warning is originating in stats.py, line 2417. This seems to be in the pearsonr function. Googling "invalid value encountered in double_scalars" yielded this SO question which suggests

from scipy.stats.stats import pearsonr

X = [4, 4, 4, 4, 4, 4]
Y = [4, 5, 5, 4, 4, 4]

pearsonr(X, Y)

raise the RuntimeWarning. This suggests that you are occasionally calling pearsonr with inputs that result in division by zero (as user3453425 stated)-- maybe due to one of the inputs being constant, and hence having a standard deviation of zero.

In this case pearsonr(X, Y) returns (nan, 1.0). So make sure you handle the case when the pearson correlation coefficient is undefined (nan).

Kiwi · Accepted Answer · 2014-03-26 20:04:56Z

3

Turn the warning into an exception :

import warnings
warnings.simplefilter('error')

This way, an error will be raised and you will see the traceback.

answered Mar 26, 2014 at 20:04

Kiwi

2,83619 silver badges16 bronze badges

Comments

Roland Smith · Accepted Answer · 2014-03-26 20:47:07Z

1

Where: looking at the code

r = (r_num / r_den)

This only occurs in two places in scipy;

linregress and pearsonr in stats.py.
pearsonr in mstats_basic.py.

Why: a double_scalar is a single double as opposed to a numpy array. I'm thinking that in some calls r_num and/or r_den is a single (invalid) floating point number. But it is not zero, because zero is not an invalid number, and that would raise a ZeroDivisionError exception. An invalid warning is generally issued when a calculation returns a NaN.

answered Mar 26, 2014 at 20:47

Roland Smith

43.8k3 gold badges69 silver badges98 bronze badges

Collectives™ on Stack Overflow

Python/numpy locating Runtime Warning

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related