Why do I get an overflow error multiplying Numpy product outputs?

Question

I've encountered an overflow warning as a result of multiplying the output of Numpy products that I'm looking to understand. A simplified version of their actual use within my larger project is detailed below:

import numpy as np

class MyClass(object):
    def __init__(self,
                 array_1,
                 array_2):

        # Assigning arrays to be used in later methods
        self.array_1 = array_1
        self.array_2 = array_2

        # Assigning some scaling factors to be used in later methods.
        self.value_1 = np.prod(self.array_1.shape)
        self.value_2 = np.prod(self.array_2.shape)
        print("Numpy Product Assignment: {0}, {1}".format(self.value_1, self.value_2))

        # Alternative assignment of scaling factors
        self.alt_value_1 = self.array_1.shape[0] * self.array_1.shape[1]
        self.alt_value_2 = self.array_2.shape[0] * self.array_2.shape[1]
        print("Regular Product Assignment: {0}, {1}".format(self.alt_value_1, self.alt_value_2))

        pass

    def mymethod(self):
        print("Direct Multiplication: {0}".format(80160 * 262144))
        print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
        print("Regular Product Multiplcation {0}".format(self.alt_value_1 * self.alt_value_2))

if __name__ == '__main__':
    test_array_1 = np.zeros([512, 512], dtype=complex)
    test_array_2 = np.zeros([1002, 80], dtype=complex)

    test_class = MyClass(test_array_1, test_array_2)
    test_class.mymethod()

Including the use with the class structure for completeness, though heavily edited down to the bare minimum. If I run this code (on Python 3.6.0) I get the following output:

C:/somepath/scratch.py:247: RuntimeWarning: overflow encountered in long_scalars
  print("Numpy Product Multiplication: {0}".format(self.value_1 * self.value_2))
Numpy Product Assignment: 262144, 80160
Regular Product Assignment: 262144, 80160
Direct Multiplication: 21013463040
Numpy Product Multiplication: -461373440
Regular Product Multiplcation 21013463040

Process finished with exit code 0

Clearly I can get around the problem using the regular multiplication, but I would like to understand why there is a problem and if it can be fixed as is. I think there is some dtype=X subtlety that I have missed, so my question is what is causing these overflow errors?

I think the problem occurs because numpy uses c-type integers internally, while the regular multiplication uses python integers, which have a much larger maximum value. — Thomas Kühn
– Thomas Kühn, Commented Jan 9, 2018 at 11:57
@ThomasKühn "Much larger maximum value" Indeed. Integers have in Python3 theoretically no maximum value, only your system sets the limits. — Mr. T
– Mr. T, Commented Jan 9, 2018 at 12:35
@Piinthesky Yes, I formulated it badly, but the answer I linked includes that information. — Thomas Kühn
– Thomas Kühn, Commented Jan 9, 2018 at 12:39

Anand · Accepted Answer · 2020-07-11 18:33:00Z

5

This looks like an overflow caused by 32 integers. You can convert your values to 64 bit like this:

    self.value_1 = np.prod(self.array_1.shape, dtype=np.int64)
    self.value_2 = np.prod(self.array_2.shape, dtype=np.int64)

Numpy automatically selects a 32 bit integer type if the values used for array construction are small enough. During multiplication they are not automatically cast to 64 bit.

edited Jul 11, 2020 at 18:33

Anand

431 silver badge7 bronze badges

answered Jan 9, 2018 at 11:58

MB-F

23.8k5 gold badges71 silver badges127 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Folau Over a year ago

So it uses the size of the np.prod result to determine the output dtype, not any properties of the input array, unless you have defined the dtype argument within np.prod?

MB-F Over a year ago

Actually, prodnormally uses the properties of the input array - but you pass in array.shape, which is a tuple. The tuple needs to be converted to an array, and this conversion looks at the values. (Oh, and it looks like prod takes a dtype argument that you can use instead of calling .astype on the result.)

Folau Over a year ago

Ah I had missed that shape is a tuple, but that does make sense. Thanks.

Collectives™ on Stack Overflow

Why do I get an overflow error multiplying Numpy product outputs?

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related