0

I am trying to apply a vectorised function to a 1D NumPy array (test).

If an element is greater than a certain threshold the function is applied, otherwise a 0 is returned.

Passing the function on the whole array gives an array of zeros (result #1)

Yet this works for some slices of the large array(result #2) and a small array (result #3)

Can you help me understand why this is? 

import numpy as np
test = np.array([-58.08281  , -47.07844  , -39.38589  , -38.244213 , -36.04118  ,
       -35.17719  , -47.651756 , -47.123497 , -38.47037  , -31.427711 ,
       -35.980206 , -39.04678  , -43.247276 , -29.217781 , -26.16616  ,
       -23.175611 , -19.073223 , -19.573145 , -19.291908 , -19.084608 ,
       -24.24286  , -26.768343 , -29.40547  , -42.254036 , -32.5126   ,
       -27.8232   , -26.521381 , -18.53816  , -16.300032 , -14.897881 ,
       -11.96727  , -11.895884 , -11.958228 , -11.689035 , -19.331993 ,
       -22.528988 , -14.850136 , -10.7898   , -10.738896 , -11.510415 ,
       -11.297523 , -14.9558525, -18.261246 , -20.11386  , -35.434853 ,
       -36.547577 , -29.713285 , -35.055378 , -19.717499 , -15.524372 ,
       -14.905738 , -11.690297 , -12.295127 , -14.571337 , -14.457521 ,
       -20.896961 , -35.145    , -39.106945 , -20.592056 , -19.292147 ,
       -21.957949 , -20.131369 , -31.953508 , -24.577961 , -23.88112  ,
       -16.549093 , -16.742077 , -22.181223 , -21.692726 , -34.572075 ,
       -20.111103 , -18.57012  , -12.833547 , -11.325545 , -12.807129 ,
       -11.844269 , -19.830124 , -21.79983  , -18.484238 , -12.855567 ,
       -11.830711 , -14.83697  , -14.618052 , -19.990686 , -30.934792 ,
       -27.72318  , -17.222315 , -14.099125 , -16.516563 , -15.129327 ,
       -19.21385  , -41.145554 , -37.12835  , -20.674335 , -17.670841 ,
       -26.641182 , -26.721628 , -29.708376 , -16.29707  , -15.220005 ,
       -11.475418 ,  35.859955 , -10.404102 ,  35.160667 , -11.339685 ,
       -17.627815 , -18.65314  , -25.346134 , -38.297813 , -22.460407 ,
       -21.334377 , -16.922516 , -10.733174 ,  35.263527 ,  35.078003 ,
        35.26928  ,  35.44266  ,  35.89205  , -10.965962 , -16.772722 ,
       -10.638295 ,  35.37294  ,  35.32364  ,  35.271263 ,  35.900078 ,
        35.145794 , -12.282233 , -14.206524 , -18.138363 , -37.339016 ,
       -26.27323  , -27.531588 , -25.00942  , -13.963585 , -12.315678 ,
       -10.978365 ,  35.439877 , -10.534686 , -11.77856  , -12.630129 ,
       -22.29188  , -32.74709  , -29.052572 , -16.526686 , -18.223225 ,
       -19.174236 , -18.920668 , -34.266537 , -23.23388  , -19.992903 ,
       -13.9729805, -16.85691  , -20.88271  , -21.805904 , -24.517344 ,
       -17.412155 , -15.050234 ,  35.047886 , -10.27907  , -10.765995 ,
       -11.394721 , -34.574    , -18.185272 , -15.156159 , -10.370025 ,
       -11.406872 , -13.781429 , -13.863158 , -24.35263  , -29.509377 ,
       -24.758411 , -14.150916 , -13.686075 , -15.366934 , -14.149103 ,
       -22.916718 , -35.810047 , -33.369896 , -17.931974 , -18.65556  ,
       -28.330248 , -27.015589 , -23.890095 , -15.020579 , -13.920487 ,
        35.49385  ,  35.613037 ,  35.326546 ,  35.1469   , -12.024554 ,
       -17.770742 , -18.414755 , -31.574192 , -35.00205  , -20.591629 ,
       -21.097118 , -14.166552 ,  35.61772  ,  35.196175 ,  35.884003 ,
        35.032402 ,  35.289963 ,  35.18595  , -36.364285 , -10.158181 ,
        35.040634 ,  35.349873 ,  35.31796  ,  35.87602  ,  35.88828  ,
        35.086105 , -12.404961 , -13.550255 , -20.19417  , -35.630135 ,
       -23.762396 , -27.673418 , -19.928736 , -12.206515 , -11.781338 ,
        35.307823 ,  35.67385  , -10.780588 , -11.199528 , -13.561855 ,
       -24.982666 , -30.838753 , -25.138466 , -16.61114  , -20.002995 ,
       -18.823566 , -21.581133 , -25.644733 , -22.914455 , -17.489904 ,
       -13.714966 , -18.483316 , -20.454823 , -25.238888 , -20.592503 ,
       -17.511456 , -13.5111885,  35.399975 , -10.711888 , -10.577221 ,
       -13.2071705, -27.878649 , -16.227467 , -13.394671 ,  35.33075  ,
       -10.933496 , -12.903596 , -13.261947 , -23.191689 , -36.082005 ,
       -26.252464 , -14.935854 , -14.955426 , -16.291502 , -15.563564 ,
       -27.91648  , -30.43707  , -27.09887  , -16.93166  , -19.03229  ,
       -26.68034  , -26.50705  , -22.435007 , -15.312309 , -13.67744  ,
        35.70387  ,  35.197517 ,  35.21866  ,  35.759956 , -12.934032 ,
       -18.348143 , -19.073929 , -36.864773 , -32.881073 , -20.560263 ,
       -20.530846 , -13.128365 ,  35.65545  ,  35.465275 ,  35.028538 ,
        35.842434 ,  35.676643 , -17.01441  , -17.217728 ,  35.667717 ,
        35.871662 ,  35.92965  ,  35.316013 ,  35.096027 ,  35.02661  ,
        35.988937 , -12.0597515, -13.201061 , -20.259245 , -28.855875 ,
       -21.791933 , -25.400242 , -17.618946 , -11.611944 , -11.329423 ,
        35.063614 ,  35.825493 , -10.553531 , -10.820301 , -13.883024 ,
       -22.231556 , -26.921532 , -31.872276 , -18.039211 , -19.713062 ,
       -20.517511 , -21.620483 , -26.919012 , -20.787134 , -17.330051 ,
       -13.198881 , -15.984946 , -19.181019 , -21.50328  , -25.311642 ,
       -18.11811  , -14.696231 , -10.136784 , -10.480961 , -11.110486 ,
       -13.739718 , -28.865023 , -15.966995 , -13.198223 ,  35.18759  ,
       -10.803377 , -12.718526 , -13.597855 , -23.472122 , -34.405643 ,
       -24.122065 , -14.643904 , -14.425363 , -15.651573 , -15.197855 ,
       -25.13602  , -33.207695 , -26.908777 , -17.217882 , -19.061764 ,
       -27.06517  , -28.88142  , -21.721449 , -14.84623  , -12.997027 ,
        35.853565 ,  35.51484  ,  35.660423 ,  35.982292 , -12.461762 ,
       -17.52755  , -19.008127 , -32.69878  , -30.82928  , -20.193447 ,
       -19.172876 , -12.901536 ,  35.05082  ,  35.915546 ,  35.254303 ,
        35.797028 , -14.470562 , -22.461277 , -15.07134  ,  35.970448 ,
        35.198704 ,  35.945583 ,  35.362762 ,  35.306732 ,  35.064957 ,
        35.10975  , -11.703257 , -13.411005 , -20.08778  , -28.905445 ,
       -22.59493  , -25.155657 , -17.814808 , -11.842859 , -11.154184 ,
        35.989094 ,  35.854362 , -10.2389765, -10.827884 , -14.010275 ,
       -25.168896 , -33.99552  , -22.858255 , -16.562387 , -19.22073  ,
       -18.317003 , -23.036928 , -25.22068  , -21.934307 , -16.469448 ,
       -13.88927  , -18.307293 , -20.485218 , -29.06332  , -19.628113 ,
       -16.496414 , -12.351503 ,  35.66623  , -10.330103 , -10.866837 ,
       -16.813847 , -21.454565 , -15.892494 , -12.269305 ,  35.174488 ,
       -11.898882 , -13.1494465, -15.517578 , -35.11971  , -29.069548 ,
       -19.153015 , -13.194953 , -14.334308 , -14.483275 , -15.592762 ,
       -30.123589 , -38.262245 , -24.752253 , -17.36696  , -22.627728 ,
       -29.787828 , -44.489254 , -17.438164 , -13.678364 , -11.26264  ,
        35.92086  ,  35.600876 ,  35.231567 ,  35.960655 , -13.438512 ,
       -16.794493 , -19.414097 , -33.008324 , -23.844492 , -18.63253  ,
       -17.060545 , -10.566847 ,  35.735447 ,  35.061024 ,  35.95225  ,
       -11.117262 , -18.978222 , -39.73106  , -11.048341 ,  35.58616  ,
        35.699783 ,  35.32885  ,  35.09172  ,  35.119743 ,  35.753242 ,
        35.73512  , -12.641587 , -14.861554 , -25.59355  , -29.808552 ,
       -24.463276 , -26.617489 , -15.665792 , -11.706967 , -11.054789 ,
        35.413254 ,  35.13033  , -10.968152 , -11.514641 , -17.074472 ,
       -31.623056 , -40.51703  , -18.116985 , -15.995826 , -18.33452  ,
       -17.266975 , -28.274193 , -24.104795 , -21.711021 , -15.209898 ,
       -15.003292 , -20.39471  , -21.562126 , -34.197975 , -16.957975 ,
       -14.923981 , -10.418877 ,  35.874657 , -10.214642 , -11.880876 ])

test2 = np.array([5,5,5])

Thresh = -10
Ratio = 5

#Defining function
def gr(x):
    if x >=Thresh:
        return Thresh + (x-Thresh)/Ratio
    else:
        return 0
#vectorising function
gr_v = np.vectorize(gr)

#RESULTS
##1
print(sum(gr_v(test))) #0
##2
print(sum(gr_v(test2)))
##3
print(sum(gr_v(test[200:250])))
Improve this question
2
  • This is interesting. It is the type inference. Quite a gotcha. See my answer below. Commented Mar 29, 2020 at 13:05
  • I know the vectorize docs are long and boring, but pays to read it. This error has come up several times recently, Commented Mar 29, 2020 at 14:46

3 Answers 3

Reset to default
3

Just as a remark, know that np.vectorize "function is provided primarily for convenience, not for performance. The implementation is essentially a for loop."

Therefore it is the same as:

l = [gr(x) for x in test]
sum(l)
-80.41708859999999

To use a vectorize implementation you can do:

np.sum(Thresh + (test[test>=Thresh]-Thresh)/Ratio)
-80.41708859999999

As time performance is concerned:

%timeit np.sum(Thresh + (test[test>=Thresh]-Thresh)/Ratio)
13.8 µs ± 864 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

%timeit sum(gr_v(test))
217 µs ± 5.32 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

The actual vectorized implemantation is 15x faster with this test size.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

This has to do with the inherent property of vectorisation. If the first output of your vectorised function is integer 0, then the entire output is off. While this behaviour of vectorise is a bit memorising, I can provide a rather simple fix. Instead of using 

def gr(x):
    if x >=Thresh:
        return Thresh + (x-Thresh)/Ratio
    else:
         return 0

which returns:

0
-21.0
-12.044083

rewrite the function as :

def gr(x):
    if x >=Thresh:
         return Thresh + (x-Thresh)/Ratio
    else:
        return 0.0

it returns:

-80.41708859999999
-21.0
-12.044083

The purpose of returning 0.0 instead of 0 is to have a float(0) instead of int(0) as your output. It also explains why your codes work for your test2 and test[200:250] arrays, because both have positive numbers as their first element, therefore what is returned by gr_v is a float, but not int(0). Hope that it helps!

Improve this answer

1 Comment

You can just do 0.0 or specify the output type when defining the vectorized function: gr_v = np.vectorize(gr, otypes=[float])
0

It is something to do with type inference:

Try this:

gr_v = np.vectorize(gr, otypes=[np.float])

And this fixes it (note the .0): 

Thresh = -10.0
Ratio = 5.0

#Defining function
def gr(x):
    if x >=Thresh:
        return Thresh + (x-Thresh)/Ratio
    else:
        return 0.0
#vectorising function
gr_v = np.vectorize(gr)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.