Python numpy array vs list

You first need to understand the difference between arrays and lists.

An array is a contiguous block of memory consisting of elements of some type (e.g. integers).

You cannot change the size of an array once it is created.
It therefore follows that each integer element in an array has a fixed size, e.g. 4 bytes.

On the other hand, a list is merely an "array" of addresses (which also have a fixed size).

But then each element holds the address of something else in memory, which is the actual integer that you want to work with. Of course, the size of this integer is irrelevant to the size of the array. Thus you can always create a new (bigger) integer and "replace" the old one without affecting the size of the array, which merely holds the address of an integer.

Of course, this convenience of a list comes at a cost: Performing arithmetic on the integers now requires a memory access to the array, plus a memory access to the integer itself, plus the time it takes to allocate more memory (if needed), plus the time required to delete the old integer (if needed). So yes, it can be slower, so you have to be careful what you're doing with each integer inside an array.

For most uses, lists are useful. Sometimes working with numpy arrays may be more convenient for example.

a=[1,2,3,4,5,6,7,8,9,10]

b=[5,8,9]

Consider a list 'a' and if you want access the elements in a list at discrete indices given in list 'b' writing

a[b]

will not work.

but when you use them as arrays, you can simply write

a[b]

to get the output as array([6,9,10]).

Do array.array or numpy.array offer significant performance boost over typical arrays?

I tried to test this a bit with the following code:

import timeit, math, array
from functools import partial
import numpy as np

# from the question
def calc1(x):
    for i in range(1,len(x)):
        x[i] = x[i-1] + 1

# a floating point operation
def calc2(x):
    for i in range(0,len(x)):
        x[i] = math.sin(i)

L = int(1e5)

# np
print('np 1: {:.5f} s'.format(timeit.timeit(partial(calc1, np.array([0] * L)), number=20)))
print('np 2: {:.5f} s'.format(timeit.timeit(partial(calc2, np.array([0] * L)), number=20)))

# np but with vectorized form
vfunc = np.vectorize(math.sin)
print('np 2 vectorized: {:.5f} s'.format(timeit.timeit(partial(vfunc, np.arange(0, L)), number=20)))

# with list
print('list 1: {:.5f} s'.format(timeit.timeit(partial(calc1, [0] * L), number=20)))
print('list 2: {:.5f} s'.format(timeit.timeit(partial(calc2, [0] * L), number=20)))

# with array
print('array 1: {:.5f} s'.format(timeit.timeit(partial(calc1, array.array("f", [0] * L)), number=20)))
print('array 2: {:.5f} s'.format(timeit.timeit(partial(calc2, array.array("f", [0] * L)), number=20)))

And the results were that list executes fastest here (Python 3.3, NumPy 1.8):

np 1: 2.14277 s
np 2: 0.77008 s
np 2 vectorized: 0.44117 s
list 1: 0.29795 s
list 2: 0.66529 s
array 1: 0.66134 s
array 2: 0.88299 s

Which seems to be counterintuitive. There doesn't seem to be any advantage in using numpy or array over list for these simple examples.

To OP: For your use case use lists.

My rules for when to use which, considering robustness and speed:

list: (most robust, fastest for mutable cases) Ex. When your list is constantly mutating as in a physics simulation. When you are "creating" data from scratch that may be unpredictable in nature.

np.arrary: (less robust, fastest for linear algebra & data post processing) Ex. When you are "post processing" a data set that you have already collected via sensors or a simulation; performing operations that can be vectorized.

use a=numpy.array(number_of_elements, dtype=numpy.int64) which should give you an array of 64-bit integers. These can store any integer number between -2^63 and (2^63)-1 (approximately between -10^19 and 10^19) which is usually more than enough.

The application of vectorized numpy seems to be affected by either timeit or partial. Try this instead:

`# not using time it
total_time=0
for i in range(20):
    t0=time.process_time()
    np.sin(np.arange(0,L))
    total_time+=time.process_time()-t0
    print('np 2 vectorized - no timeit or partial: {:.5f} s'.format(total_time/20))`

Results:

np 1: 0.67521 s
np 2: 0.39538 s
np 2 vectorized: 0.26964 s
list 1: 0.19755 s
list 2: 0.26914 s
array 1: 0.27625 s
array 2: 0.35779 s
np 2 vectorized - no timeit or partial: 0.00077 s