Avoid creating new arrays as results for numpy/scipy operations?

Question

For doing repeated operations in numpy/scipy, there's a lot of overhead because most operation return a new object.

For example

for i in range(100):
   x = A*x

I would like to avoid this by passing a reference to the operation, like you would in C

for i in range(100):
   np.dot(A,x,x_new) #x_new would now store the result of the multiplication
   x,x_new = x_new,x

Is there any way to do this? I would like this not for just mutiplication but all operations that return a matrix or a vector.

Check the functions's docs. Some take an out parameter, especially the ufunc ones. And basic operators have versions like +=, *=. ufuncs also have reduce and accumulate methods to handle repeated actions. — hpaulj
– hpaulj, Commented Apr 26, 2015 at 2:42
The example code you gave using np.dot already does exactly this - the third argument to np.dot is the out= parameter, so x_new will be filled in place with the result of np.dot(A, x) — ali_m
– ali_m, Commented Apr 26, 2015 at 11:51
You might also want to consider np.dot(np.linalg.matrix_power(A, 100), x). — Dietrich
– Dietrich, Commented Apr 26, 2015 at 11:55
Note that your intuition that there is "a lot of overhead" is not correct in most cases. Creating new Python objects or allocating blocks of memory is cheap compared to computations required for the matrix multiplications itself. — pv.
– pv., Commented Apr 26, 2015 at 16:32

EelkeSpaak · Accepted Answer · 2015-06-10 08:50:23Z

See Learning to avoid unnecessary array copies in IPython Books. From there, note e.g. these guidelines:

a *= b

will not produce a copy, whereas:

a = a * b

will produce a copy. Also, flatten() will copy, while ravel() only copies if necessary and returns a view otherwise (and thus should in general be preferred). reshape() also does not produce a copy, but returns a view.

Furthermore, as @hpaulj and @ali_m noted in their comments, many numpy functions support an out parameter, so have a look at the docs. From numpy.dot() docs:

out : ndarray, optional Output argument.

This must have the exact kind that would be returned if it was not used. In particular, it must have the right type, must be C-contiguous, and its dtype must be the dtype that would be returned for dot(a,b). This is a performance feature. Therefore, if these conditions are not met, an exception is raised, instead of attempting to be flexible.

Collectives™ on Stack Overflow

Avoid creating new arrays as results for numpy/scipy operations?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related