The code
import array, itertools
a = array.array('B', itertools.repeat(0, 3715948544))
takes almost 7 minutes to run on my machine (6m44s). The computer has 8 Gb of RAM and runs Linux with CPython 3.4.3. How can I obtain an array-like object with 1-byte unsigned int entries faster, preferably using the Python standard library? Numpy can allocate it instantly (in less than 1 millisecond).
a = array.array('B', [0]) * 3715948544
Sequence multiplication, analogous to how you'd create a giant list of zeros. Note that anything you want to do with this giant array is probably going to be as slow as your initial attempt to create it.
If you really can't use NumPy, you can try how far you can get with the built-in bytearray:
a = bytearray(3715948544)
This should finish in a couple of seconds at most.
bytearray vs array.array vs numpy. In my particular case, bytearray is the fastest, array.array is a close second and numpy loses by a factor of almost 2. I am running a particular calculation, where this array is basically a huge counter: I read a value, add +1, write it back and check if it has exceeded a threshold. With bytearray I get about 7.5 sec/iteration, array.array is about 7.75 seconds and numpy is 14.6 seconds.
collections.Counter? Or just a regular dict? The 7.5 seconds number sounds like you aren't using anywhere near all the array cells. (That, or maybe you mean that one read/increment/write/check cycle takes 7.5 seconds, which is crazy slow.)At first I thought numpy would be fastest, but as pointed out by Sven, bytearray is pretty quick for 10000. Try your luck with bytearray on 3billion.
In [1]: import numpy as np
In [2]: import array, itertools
In [3]: %timeit array.array('B', itertools.repeat(0, 10000))
1000 loops, best of 3: 456 µs per loop
In [4]: %timeit np.zeros(10000, dtype='uint8')
1000000 loops, best of 3: 924 ns per loop
In [5]: %timeit bytearray(10000)
1000000 loops, best of 3: 328 ns per loop
