Array is too big error in Python

Question

I have this code in python:

from numpy import *
import itertools

m, n  = 6, 10
set_m = [i + 1 for i in range(m + 1)]
comb  = zeros(((m + 1) ** n, n), dtype=int)

k = 0
for i in itertools.product(set_m, repeat=n):
    comb[k][:] = i
    k += 1

But when I run it, I got this error:

Traceback (most recent call last):
  File "main.py", line 33, in <module>
    comb = zeros(((m + 1) ** n, n), dtype=int)
ValueError: array is too big.

Just to make sure, you really want 2 billion elements, ((m + 1) ** n)*n? — Zero
– Zero, Commented May 25, 2015 at 15:00
Yes, sure. You are trying to create an array of integers with size (282475249, 10), which not surprisingly is too big. — Jblasco
– Jblasco, Commented May 25, 2015 at 15:00
Are you using the 64 bit version of Python? How much memory does your machine have? — cfh
– cfh, Commented May 25, 2015 at 15:21

Jblasco · Accepted Answer · 2015-05-25 15:04:31Z

4

If you are sure you MUST have a billion element array, and there is no way around it whatsoever (sometimes happens, but not every day), you can use memmap to create the array in the hard drive instead of the RAM memory. But I think it would be wise to search for ways to re-write your code to avoid doing such a slow thing.

answered May 25, 2015 at 15:04

Jblasco

3,98724 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Array is too big error in Python

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related