Numpy array.resize() - zeros 'first'

Question

I can use array.resize(shape) to resize my array and have zeros added to those indices without any value. If my array is [1,2,3,4] and I use array.resize[5,0] I get [1,2,3,4,0]. How can I append / pad the zeros to the front, yielding [0,1,2,3,4]?

I am doing this dynamically - trying to use:

array.resize(arrayb.shape)

I would like to avoid (at all costs) making an in-memory copy of the array. That is to reverse the array, resize, and reverse again. Working with a view would be ideal.

@RussellBorogove The first reverse needs to make a new in-mem copy though for resize to work. This is not ideal if the arrays are large. Will update the question to account for this. — Jzl5325
– Jzl5325, Commented Sep 4, 2012 at 18:49
thats just not how array.resize works ... there is no pad at front option... I believe this is due to at the c level the are doing realloc which appends slots at the end of the original array size... — Joran Beasley
– Joran Beasley, Commented Sep 4, 2012 at 18:52
You cannot avoid a copy "at all costs", since the resize method might have to make a copy. — Fred Foo
– Fred Foo, Commented Sep 4, 2012 at 20:53

seberg · Accepted Answer · 2012-09-04 20:57:45Z

3

You could try working on an array with negative strides (though you can never be sure that resize may not have to make a copy):

_a = np.empty(0) # original array
a = _a[::-1] # the array you work with...

# now instead of a, resize the original _a:
del a # You need to delete it first. Or resize will want refcheck=False, but that
      # will be dangerous!
_a.resize(5)
# And update a to the new array:
a = _a[::-1]

But I would really suggest you make the array large enough if in any way possible, this does not seem very beautiful, but I think this is the only way short of copying around data. Your array will also have a negative stride, so it won't be contiguous, so if that means that some function you use on it must make copy, you are out of luck.

Also if you slice your a or _a you have to either make a copy, or make sure you delete them before resizing. While you can give refcheck=False this seems to invalidate the data.

edited Sep 4, 2012 at 20:57

answered Sep 4, 2012 at 20:50

seberg

9,0352 gold badges34 silver badges30 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Fred Foo Over a year ago

Nice trick, but it does rely on a fair amount of CPython implementation details. Besides, ndarray.resize might still do a copy.

seberg Over a year ago

@larsmans numpy is pretty much limited to CPython anyways, though I am unsure where CPython is involved, ndarray.resize is the point. reallocs might need to make a copy, that can never be avoided, unless the array is big enough to begin with.

Fred Foo Over a year ago

the del trick relies on CPython's reference-counting GC. Yes, NumPy is currently tied to CPython, but there are attempts to get it working on PyPy. I admit I don't know how PyPy does GC.

senderle · Accepted Answer · 2012-09-04 21:19:51Z

2

I believe you can use slice assignment to do this. I see no reason why numpy would need to make a copy for an operation like this, as long as it does the necessary checks for overlaps (though of course as others have noted, resize may itself have to allocate a new block of memory). I tested this method with a very large array, and I saw no jump in memory usage.

>>> a = numpy.arange(10)
>>> a.resize(15)
>>> a[5:] = a[:10]
>>> a[0:5] = 0
>>> a
array([0, 0, 0, 0, 0, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

The following showed no jump in memory usage for the assignment operation:

>>> a = numpy.arange(100000000)
>>> a.resize(150000000)
>>> a[50000000:] = a[:100000000]

I don't know of a better way, and this is just a conjecture. Let me know if it doesn't work.

edited Sep 4, 2012 at 21:19

answered Sep 4, 2012 at 19:05

senderle

152k36 gold badges218 silver badges244 bronze badges

1 Comment

Jzl5325 Over a year ago

I ended up going with this solution. Thanks for the suggestion and the memory testing. Long term, it looks like I'll move to a scalar as I can safely discard some of the data, but for now this is working quite well!

Collectives™ on Stack Overflow

Numpy array.resize() - zeros 'first'

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related