Performance issues when iterating numpy array

Question

I have a 3D array of image such as

[
    [
        [225, 0, 0],
        [225, 225, 0],
        ...
    ],
    [
        [225, 0, 0],
        [225, 225, 0],
        ...
    ],
    ...
]

The size of this array is 500x500x3 which is 750.000 elements. These are simple nested loops to iterate over the array

for row in arr:
    for col in row:
        for elem in col:
            elem = (2 * elem / MAX_COLOR_VAL) - 1

But it takes a lot of time (> 5 min) to iterate.

I'm new in numpy so may be I'm iterating arrays wrong way? How can I optimize these loops?

You should not iterate over a numpy array. In fact it is very likely that iterating over a numpy array will be slower than iterating over a Python list. The idea of a numpy array is to do operations in bulk. Your program will not even set the items anyway. — willeM_ Van Onsem
– willeM_ Van Onsem, Commented Sep 26, 2019 at 16:57

willeM_ Van Onsem · Accepted Answer · 2019-09-26 16:59:46Z

1

Numpy arrays are not designed to do iteration over the elements. Likely it will even be slower than iterating over a Python list, since that will result in a lot of wrapping and unwrapping of elements.

Numpy arrays are designed to do processing in bulk. So for example calculate the elementwise-sum of two 1000×1000 matrices.

If you want to multiply all elements with 2, divide these by MAX_COLOR_VAL and subtract one from these, you can simply construct a new array with:

arr = (2 * arr.astype(float) / MAX_COLOR_VAL) - 1

This will apply this operation to all elements.

Note: note that if you iterate over a numpy array, you do not iterate over the indices, you iterate over the rows itself. So the row in for row in arr will return a 2d array, not the index of a 2d array.

answered Sep 26, 2019 at 16:59

willeM_ Van Onsem

482k33 gold badges483 silver badges624 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kmario23 Over a year ago

I think casting the array to float (arr.astype(float)) is bit more expensive than just doing something like 2.0 * arr / MAX_COLOR_VAL) - 1. Why is this explicit casting necessary?

willeM_ Van Onsem Over a year ago

@kmario23: when I run this 10'000 times with timeit, the .astype(float) gives 3.6568590000097174, whereas the 2.0 * ... gives 3.686493999994127. I think it does not matter much. As a functional programmer. I do not really like this float times int idea, since it reminds too much of weakly typed languages like C#. As Python says: "explicit over implicit" :)

Collectives™ on Stack Overflow

Performance issues when iterating numpy array

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related