8

I am working to optimize a fluid flow and heat transfer analysis program written in Fortran. As I try to run larger and larger mesh simulations, I'm running into memory limitation problems. The mesh, though, is not all that big. Only 500,000 cells and small-peanuts for a typical CFD code to run. Even when I request 80 GB of memory for my problem, it's crashing due to insufficient virtual memory.

I have a few guesses at what arrays are hogging up all that memory. One in particular is being allocated to (28801,345600). Correct me if I'm wrong in my calculations, but a double precision array is 8 bits per value. So the size of this array would be 28801*345600*8=79.6 GB?

Now, I think that most of this array ends up being zeros throughout the calculation so we don't need to store them. I think I can change the solution algorithm to only store the non-zero values to work on in a much smaller array. However, I want to be sure that I'm looking at the right arrays to reduce in size. So first, did I correctly calculate the array size above? And second, is there a way I can have Fortran show array sizes in MB or GB during runtime? In addition to printing out the most memory intensive arrays, I'd be interested in seeing how the memory requirements of the code are changing during runtime.

Improve this question
5
  • 4
    How much actual memory do you have on the machine you're running this on? Also, you're wrong in your assumption double precision is 8 bytes, not 8 bits. That comes out to around 74.16 GB of data (powers of 1024, not 1000). Also, am I correct in assuming you're doing 4 days worth of data (345600 seconds = 60 * 60 * 24 * 4) Commented Jun 21, 2012 at 16:43
  • Mike, this is being run on a cluster with up to 96 GB of memory per node that I can request. Sorry about the bytes vs. bits confusion and thanks for clearing that up, but I'm in the right ballpark there so that array size is definitely a problem. And no, that 345600 is related to the number of cells in the model mesh and has nothing to do with time. Commented Jun 21, 2012 at 17:56
  • @user104629: One reason why could be that it can't allocate a contiguous array of 80 GB of memory. Commented Jun 21, 2012 at 18:00
  • It is seems to me there is something terribly wrong if you need such a big array in a CFD model with 0.5 Mcells. Commented Jun 23, 2012 at 11:02
  • Yes of course. As I suspected, the problem was that non-zero entries were being stored in the pressure coefficient matrix. TotalView revealed this array to be consuming 80 GB. I fixed this by using a compressed storage format. TotalView also revealed two other very large arrays that were taking up 10 GB of space each and not doing anything at all in the code. Eliminating these problems enabled me to reduce my memory usage from over 100 GB to 2.5 GB. Commented Jun 26, 2012 at 0:15

1 Answer 1

Reset to default
4

Memory usage is a quite vaguely defined concept on systems with virtual memory. You can have large amounts of memory allocated (large virtual memory size) but only a small part of it actually being actively used (small resident set size - RSS).

Unix systems provide the getrusage(2) system call that returns information about the amount of system resources in use by the calling thread/process/process children. In particular it provides the maxmimum value of the RSS ever reached since the process was started. You can write a simple Fortran callable helper C function that would call getrusage(2) and return the value of the ru_maxrss field of the rusage structure.

If you are running on Linux and don't care about portability, then you may just open and read from /proc/self/status. It is a simple text pseudofile that among other things contains several lines with statistics about the process virtual memory usage:

...
VmPeak:     9136 kB
VmSize:     7896 kB
VmLck:         0 kB
VmHWM:      7572 kB
VmRSS:      6316 kB
VmData:     5224 kB
VmStk:        88 kB
VmExe:       572 kB
VmLib:      1708 kB
VmPTE:        20 kB
...

Explanation of the various fields - here. You are mostly interested in VmData, VmRSS, VmHWM and VmSize. You can open /proc/self/status as a regular file with OPEN() and process it entirely in your Fortran code.

See also what memory limitations are set with ulimit -a and ulimit -aH. You may be exceeding the hard virtual memory size limit. If you are submitting jobs through a distributed resource manager (e.g. SGE/OGE, Torque/PBS, LSF, etc.) check that you request enough memory for the job.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

Cool, thanks for the advice. Portability isn't a concern for me, so I think I'll go with the /proc/self/status route for watching memory usage. ulimit -a showed virtual memory was unlimited. Someone also suggested to me that DDT and TotalView can be used to check where in a code memory is being eaten up.
TotalView has some advanced memory debugging stuff but I haven't used it extensively. At least any debugger could show you where the memory error occurs. In the end it might turn out to be something different from memory exhaustion.
It just crossed my mind - are your cluster nodes swap-less? If yes, you might be exhausting the total system memory and the Linux OOM killer might be kicking in...
I tried TotalView out and it has quite a few bells and whistles. I'm going to have to spend some time learning how to use them all. Using the 'heap status source report' shows the array that I figured was the culprit as consuming a ton of memory. Another odd behavior I notice when running with a debugger is that almost every time I pause, it is stopped on a particular array initialization e.g. "array = 0.0". Is this something I should investigate further? I don't know about the swap issue, but I imagine I want to design the code to avoid swap anyway, right?
Allocating an array just reserves a portion of the virtual memory. Zeroing it actually commits pages from the physical memory. Zeroing 70+ GB takes some time, especially if other memory is being swapped out or dirty I/O cache is getting flushed in order to free physical memory for your program.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.