Implementation of NumPy exponential function

Question

I'm trying to perform an evaluation of total floating-point operations (FLOPs) of a neural network.

My problem is the following. I'm using a sigmoid function. My question is how to eval the FLOPs of the exponential function. I'm using Tensorflow which relies on NumPy for the exp function.

I tried to dig into the Numpy code but didn't find the implementation ... I saw some subjects here talking about fast implementation of exponential but it doesn't really help.

My guess is that it would use a Taylor implementation or Chebychev.

Do you have any clue about this? And if so an estimation of the amount of FLOPs. I tried to find some references as well on Google but nothing really standardized ...

Thank you a lot for your answers.

Most likely, the implementation used will be the C math library. — 9769953
– 9769953, Commented Jan 31, 2022 at 12:17
I suppose this depends on how tensorflow was compiled and built. Remember that x86 has F2XM1 and FYL2X instructions that compute 2^x and y*log2(x) respectively and can be considered as a single floating point operation each — Dima Chubarov
– Dima Chubarov, Commented Jan 31, 2022 at 12:36
The default implementation of exp in numpy is taken from C's math.h: github.com/numpy/numpy/blob/… Depending on architecture I think they have custom SIMD versions used in loops that will be faster (and you probably want to use those). — unddoch
– unddoch, Commented Jan 31, 2022 at 12:53

Eumel · Accepted Answer · 2022-01-31 13:02:55Z

1

I looked into it for a bit and what i found is that numpy indeed uses the C implementation as seen here.

Tensorflow though doesnt use nmpy implementation, instead it uses the scalar_logistics_opfunction from the C++ library called Eigen. The source for that can be found here.

answered Jan 31, 2022 at 13:02

Eumel

1,4431 gold badge10 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

boubasse Over a year ago

Many thanks for this. I realize that sigmoid function requires a lot of operations to be derived now (much more than ReLU) haha. Most NN papers do not often take this aspect in consideration ...

Collectives™ on Stack Overflow

Implementation of NumPy exponential function

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related