Memory issue with numpy.copy array

Question

I have a big numpy array my_array that I copy into another temporary array temp_my_array in order to use it in calculations inside a loop, as following:

my_array = [10.1, 20.3, ..., 11.2] # a large numpy array
temp_my_array = np.copy(my_array)

for i in range(200000):
     for item in np.where(my_array> 5): 
           temp_my_array[item] = f(my_array[some other items]) 
     my_array = np.copy(temp_my_array)

I have memory error with np.copy when my_array is so big. Besides, profiling showed that np.copy is the slowest part of my code. Any ideas please?

I need a deep copy, so that when I change an item in temp_my_array it doesn't change the same item in my_array. — Behzad Jamali
– Behzad Jamali, Commented Nov 30, 2017 at 9:06
Cope the array ONCE (what you did on line 2), that will copy the entire array, i.e. all values, all of which you can change at will. — Lærne
– Lærne, Commented Nov 30, 2017 at 9:07
Sorry, I didn't realize you were using the values from my_array in your function call. — alkasm
– alkasm, Commented Nov 30, 2017 at 9:07
In the second for loop can't you just do my_array[item] = f(my_array[item]) and not bother with your temp array? The iterator you've created with np.where will not reference the changing my_array inside the loop. — alkasm
– alkasm, Commented Nov 30, 2017 at 9:08

alkasm · Accepted Answer · 2017-11-30 09:30:52Z

2

I suggest you only copy the values that actually changed. With your code this is only a slight change:

my_array = [10.1, 20.3, ..., 11.2] # a large numpy array
temp_my_array = np.copy(my_array)

for i in range(200000):
    inds = np.where(my_array > 5)
    for item in inds: 
        temp_my_array[item] = f(my_array[some other items]) 
    my_array[inds] = temp_my_array[inds]

Otherwise you can vectorize your function, but that might be annoying if [some other items] relies on your current index or impossible if it relies on the previous my_array result.

edited Nov 30, 2017 at 9:30

answered Nov 30, 2017 at 9:24

alkasm

23.1k6 gold badges83 silver badges99 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Coolness · Accepted Answer · 2017-11-30 09:24:06Z

1

Does this sound reasonable? Mass assign to original array without making copies

my_array = [10.1, 20.3, ..., 11.2] # a large numpy array

for i in range(200000):
     my_array[np.where(my_array>5)] = f(my_array[some other items]) # Mass assign instead of for-loop

You'll need to make sure f() returns an array now.

answered Nov 30, 2017 at 9:24

Coolness

1,99216 silver badges25 bronze badges

5 Comments

Behzad Jamali Over a year ago

This is an interesting solution! Is this kind of doing calculations in a vectorised version?

alkasm Over a year ago

If some other items depend on the prior result of my_array then you cannot vectorize in this fashion, so it really depends what the other items are.

Coolness Over a year ago

@BehzadJamali Yeah assignment with index-array is vectorized with numpy, so this should be very fast.

Coolness Over a year ago

@AlexanderReynolds Yeah that's true, I assumed that it doesn't because it's "some other items" as in from someplace else.

Behzad Jamali Over a year ago

I am dependent on the prior value of my_array .Is true to say that vectorization will speed up my code by removing the second loop but I still need do my_array = np.copy(temp_my_array)?

Collectives™ on Stack Overflow

Memory issue with numpy.copy array

2 Answers 2

Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related