I need a good buffer for numpy ndarray data. I need to push there some arrays, the queue should somehow concatenate them into one, and then I want to get arbitrary length arrays out of it.
Something like this:
# push some numpy arrays into buffer
buffer.push(in_array_1)
buffer.push(in_array_2)
buffer.push(in_array_3)
# return a new numpy array with arbitrary length num_of_elements_to_get
out_array = buffer.get(num_of_elements_to_get)
Cannot you advice me something like that?
numpy and the ndarray can do what you describe, without any additional classes:
If that is good enough for you depends on what you expect of a "good buffer"...
It can work like this:
buffer = np.array([])
# I assume the arrays are one-dimensional.
buffer = np.concatenate([buffer, in_array_1])
buffer = np.concatenate([buffer, in_array_2])
buffer = np.concatenate([buffer, in_array_3])
out_array = buffer[:num_of_elements_to_get]
buffer = np.delete(buffer, slice(0, num_of_elements_to_get)) # remove elements (optional)
If you want to hide the details, you can create a class that stores buffer in an attribute and implement push and pop or get methods. Don't forget error checking in the getter (num_of_elements_to_get <= buffer.size). I would call the method get if it only returns elements, and pop if it removes elements.
As hpaulj pointed out, it is possible combine multiple concatenations in one function call:
buffer = np.concatenate([buffer, in_array_1, in_array_2, in_array_3])
This causes less re-allocations of the buffer. However, if allocations are an issue it may be better to invest a bit more effort and implement a ring buffer.
As an additional note, a setter method can be designed to take multiple arrays:
def push(self, *args):
self.buffer = np.concatenate([self.buffer] + args)
buffer = np.concatenate([in_array_1, in_array_2, in_array_3]) would be better numpy code. It just doesn't look like the buffer.push... model.