forrtl: severe (179): Cannot allocate array - overflow on array size calculation

Question

I am trying to run a simulation program on my laptop (Linux 3.8.0-25-generic x86_64, with Ubuntu 13.04).

It compiles okay, but when getting to allocation of some array sizes I get:

forrtl: severe (179): Cannot allocate array - overflow on array size calculation.\

Some googling about this message led me to conclude that this is generated because my program is out of memory.

To check this assumption I tried to allocate the same array a smaller dimension, but still got the same problem.

What I tried next was to increase my stack memory, but still got the same problem.

This is the code:

program memoria

implicit none

integer :: n,num
complex, dimension(:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:,:), allocatable :: ddptrj

n=4

num=21


allocate(ddptrj(n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,n,num))

deallocate(ddptrj)


endprogram memoria

How can I solve this problem?

My guess is you are doing something physically/mathematically wrong if you are trying to use over 1 petabyte of memory for your work. — Kyle Kanos
– Kyle Kanos, Commented Sep 15, 2013 at 17:29
I would like to point out that it's really a bad idea to use 20+ dimensions for a Fortran array, even if your compiler allows it. The code is going to be unreadable. You really need to use lower dimensional arrays and implement your own indexing scheme. — Sean Patrick Santos
– Sean Patrick Santos, Commented Sep 15, 2013 at 18:09
What makes you think you need 22 dimensions and 1344TB for your problem? It must be an extraordinary problem, or it is extraordinarily badly designed... You should first explain what you are trying to do. — user1220978
– user1220978, Commented Sep 16, 2013 at 5:56

Alexander Vogt · Accepted Answer · 2013-09-15 17:42:17Z

2

The Fortran 2008 Standard defines a maximum rank of 15. ifort allows 31. gfortran still does not support more than 7 dimensions as of May 2013 . So - if you really need all 22 dimensions, use ifort - but be aware that this feature is an extension of the standard, other compilers might not compile your code.

EDIT: To sum up the discussion in the comments:

There are two problems - one related to the maximum dimensions and one related to the array size.

For compilers that do not allow for 22 dimensions, one could use one large array of rank 1 that holds all elements and calculate the index inside the element oneself using strides.

Internally all arrays are stored contiguous in memory - think of it as a large array of rank one and size N*M*... Fortran actually stores the resulting array in column-major format. So a matrix Z(NxM) is stored as a vector A(N*M), and access to the elements is calculated by strides: Z(3,4) is at index A( M*(4-1)+3 ). Note that the memory is the same for both arrays!

The next problem is the array size:

For n=2 the code works because than it fits into memory: 2**21*21=44040192 at complex (16 bytes) this amounts to 672 MB. At n=4 this becomes 4**21*21 which is 1344TB.

edited Sep 15, 2013 at 17:42

answered Sep 15, 2013 at 16:06

Alexander Vogt

18.2k13 gold badges56 silver badges72 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

Mencia Over a year ago

what do you mean by "but be aware that this is not covered by the standard" ?

Alexander Vogt Over a year ago

You will probably not be able to compile it with compilers that do not have this specific extension.

Mencia Over a year ago

what would you do if you were in my situation?I need 22 dimensions

Alexander Vogt Over a year ago

You could use one large array of rank 1 that holds all elements (though it would amount to 1344TB in your example) and calculate the index inside the element yourself using strides. Apart from that - try to simplify your problem. Do you really need that many dimensions?

Mencia Over a year ago

I think I do need all of them. Are you a physicist?

|

JoeZuntz · Accepted Answer · 2013-09-17 07:52:20Z

If you're using a method that requires 22 dimensions then you almost certainly need to change your algorithm. Not only will you use enormous amounts of memory, as people have noted elsewhere, but the runtime itself will become absolutely enormous if you want to do something with all the elements in that array.

Because you're using Fortran, my guess is that you are looking at some kind of parameter space with 22 dimensions in, and recording a goodness of fit or likelihood for each combination of parameters. In that case you should consider other algorithms like MCMC for exploring the space.

But this is just a guess - if you tell us what underlying problem you're trying to solve people may have more suggestions.

Collectives™ on Stack Overflow

forrtl: severe (179): Cannot allocate array - overflow on array size calculation

2 Answers 2

12 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

12 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related