1

I have a 3D array in numpy that includes nans. I need to return the value with the greatest index position along the 0 axis. The answer would reduce to a 2D array.

There are a lot of questions about finding the index position of a maximum value along an axis (How to get the index of a maximum element in a numpy array along one axis), but that is different than what I need.

Example 3D array: 

>>> import numpy as np
>>> foo = np.asarray([[[7,4,6],[4,2,11], [7,8,9], [4,8,2]],[[1,2,3],[np.nan,5,8], [np.nan,np.nan,10], [np.nan,np.nan,7]]])
>>> foo
array([[[  7.,   4.,   6.],
        [  4.,   2.,  11.],
        [  7.,   8.,   9.],
        [  4.,   8.,   2.]],

       [[  1.,   2.,   3.],
        [ nan,   5.,   8.],
        [ nan,  nan,  10.],
        [ nan,  nan,   7.]]])

I thought I was getting close using np.where but it returns all elements that are not nan. Not quite what I need because I want a (4,3) array.

>>> zoo = foo[np.where(~np.isnan(foo))]
>>> zoo
array([  7.,   4.,   6.,   4.,   2.,  11.,   7.,   8.,   9.,   4.,   8.,
     2.,   1.,   2.,   3.,   5.,   8.,  10.,   7.])

The answer I need is:

>>> ans = np.asarray([[1,2,3], [4,5,8], [7,8,10], [4,8,7]])
>>> ans
array([[ 1,  2,  3],
       [ 4,  5,  8],
       [ 7,  8, 10],
       [ 4,  8,  7]])

EDIT: I edited the foo example array to make the question more clear.

Improve this question
2
  • 1
    Use nanmax : np.nanmax(foo,axis=0)? Commented Dec 12, 2018 at 18:54
  • Thanks @Divakar but that isn't quite it. See my response to Edgar below. Commented Dec 12, 2018 at 19:48

2 Answers 2

Reset to default
1

You can use np.nanmax:

>>> np.nanmax(foo, axis=0)
array([[ 7.,  4.,  6.],
       [ 4.,  5., 11.],
       [ 7.,  8., 10.],
       [ 4.,  8.,  7.]])

The np.nanmax function returns the maximum of an array or maximum along an axis, ignoring any NaNs.

EDIT

As you rightly point out in your comment, you need the value at the maximum index and the code above doesn't return that.

Instead, you can use apply-along-axis:

>>> def highest_index(a):
...     return a[~np.isnan(a)][-1] # return non-nan value at highest index

>>> np.apply_along_axis(highest_index, 0, foo)
array([[ 1.  2.  3.]
       [ 4.  5.  8.]
       [ 7.  8. 10.]
       [ 4.  8.  7.]])
Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

Thanks Edgar, but that isn't what I need. Note the difference in your array and my ans. Your array returns the maximum value along the index (which at position 2,3 is 11) but I need the value at the maximum index (which at position 2,3 is 8).
You are right! I edited my answer to produce the right result.
Awesome! Thank you Edgar!
Glad I could help! Please consider upvoting :)
Just a note of warning, while apply_along_axis is convenient to use here, it isn't any faster than a more explicit iteration over the last 2 axes.
|
0

A vectored solution, only with indices:

def last_non_nan(foo):
    i = np.isnan(foo)[::-1].argmin(0)
    j,k = np.indices(foo[0].shape)
    return foo[-i-1,j,k]

i contains the index of the first not nan number in the reversed 'line'. so -i-1 is its index in the direct line. 

>>> last_non_nan(foo):
  [[  1.,   2.,   3.],
   [  4.,   5.,   8.],
   [  7.,   8.,  10.],
   [  4.,   8.,   7.]]

Faster than highest_index:

In [5]%timeit last_non_nan(foo)
133 µs ± 29.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

In [6]: %timeit np.apply_along_axis(highest_index,0,foo)
667 µs ± 90 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

up to 150x (40 ms vs 6 s) faster for a (10,400,400) array with 90 % nans.

It s essentially because last_non_nan just fetch the last non nan value in each line, when highest_index compute the index and fetch all non nan values.

Improve this answer

3 Comments

Thanks @B.M. I understand what j,k returns (the indices row wise and column wise). And it looks like returning i was the step I was really missing. Could you walk me through what the line that returns i is doing?
Also, I tried both your answer and @Edgar in my actual code problem (which is an array of shape (13, 414, 394). For some reason your answer works but Edgar's does not. I don't understand why...
Thanks so much for the extra info @BM. I upvoted your answer, it just isn't public because of my new reputation.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.