System memory exception on for loop

Question

I have the following code below:

List<long> numbers = new List<long>();

for (long i = 1; i <= 300000000; i++)
{
    numbers.Add(i);
}

What I wanted to do is to populate the list from 1-300 million. But when it hit the 67108865, it throws an exception on line 4: Exception of type 'System.OutOfMemoryException' was thrown.

I tried using ulong but still no luck.

I believe the maximum range for long data type is 9,223,372,036,854,775,807 but why am I having an error here?

Thanks in advance!

EDIT Thanks for all the answers. It helped my realized that my design is not good. I ended up changing my code design.

If this is a 32-bit app, you probably hit the limit for your available address space — Roger Rowland
– Roger Rowland, Commented Dec 12, 2013 at 6:13
Thanks but I still got an error but now on the 134217729. I used the 64-bit settings. — Gerald
– Gerald, Commented Dec 12, 2013 at 6:16
Gerald what are you doing that you require that many object in memory at once? You may need to use a different design strategy, like partitioning the data in to smaller chunks or using a database instead of storing the entire list in memory. — Scott Chamberlain
– Scott Chamberlain, Commented Dec 12, 2013 at 6:49
@ScottChamberlain I am solving one of the problems in the Project Euler. I agree with you because I tried partitioning the list, using x64-bit app but my computer gave up because my RAM is not sufficient (4GB only). — Gerald
– Gerald, Commented Dec 12, 2013 at 7:06

Community · Accepted Answer · 2017-05-23 12:19:08Z

2

Well, it is not that your number are to large, but your list is...
Lets calculate it's size:

300,000,000 * 64 bits (size of long) = 19,200,000,000 bits
19,200,000,000 /8 (size of byte) = 2,400,000,000
2,400,000,000 / 2^10 = 2,343,750 KB
2,343,750 / 2^10 = 2,288~ MB
2,288/ 2^10 = 2.235~ GB

You wanted a list of about 2.35 GB. The current CLR limitation is 2 GB (see this or this SO thread)

If you need a list with size of 300,000,000, split it into 2 lists (either in place or using a wrapping object that will handle managing the lists).

edited May 23, 2017 at 12:19

CommunityBot

11 silver badge

answered Dec 12, 2013 at 6:28

Avi Turner

10.5k8 gold badges51 silver badges79 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gerald Over a year ago

I ended up splitting it into 3 lists but my machine ended up not responding because of too much memory being consumed up. Thanks for the answer btw!

Deepak Bhatia · Accepted Answer · 2013-12-12 14:27:16Z

1

First note System.OutOfMemoryException is not thrown when limit is reached for a variable,
Secondly, it is thrown because there is not enough memory available to continue the execution of a program.
Sadly you can not configure, .the NET runtime makes all the decisions about heap size and memory.
Either switch to 64-bit machine.

For info, on 32 bit machine you can increase the memory by using /3GB boot switch option in Boot.ini

EDIT While searching I found in MSDN Documentation under Remarks section

By default, the maximum size of an Array is 2 gigabytes (GB). In a 64-bit environment, you can avoid the size restriction by setting the enabled attribute of the gcAllowVeryLargeObjects configuration element to true in the run-time environment. However, the array will still be limited to a total of 4 billion elements, and to a maximum index of 0X7FEFFFFF in any given dimension (0X7FFFFFC7 for byte arrays and arrays of single-byte structures).

A List<long> is backed by an long[]. You will fail as soon as the backing array cannot be allocated, during the reallocation, there has to be enough total memory for both the old and the new arrays.

But, if you want a collection with more than 2³¹ elements, you have to write your own implementation by using multiple arrays or List, and managing them.

edited Dec 12, 2013 at 14:27

answered Dec 12, 2013 at 6:15

Deepak Bhatia

6,2962 gold badges28 silver badges60 bronze badges

2 Comments

Gerald Over a year ago

Is there any way to configure my machine to allocate enough memory on my application to run?

Avi Turner Over a year ago

@Gerald There is no way to do so because it is a CLR limitation, not a "application setting"

Paul Draper · Accepted Answer · 2013-12-12 06:13:59Z

0

Your numbers are not too large; your list is too long. It is using all the available memory.

Reduce 300000000 to 3000000, and it will (probably) work fine.

answered Dec 12, 2013 at 6:13

Paul Draper

84.2k53 gold badges216 silver badges303 bronze badges

1 Comment

Gerald Over a year ago

Yes, it will work fine with 3000000 but what I need is 300000000 as part of the requirements.

skjcyber · Accepted Answer · 2013-12-12 06:24:28Z

0

It should ideally hold number of elements that you are telling.

But, as per current CLR implementation each object can have max 2 GB size. As you are storing long values (size 8 bytes), then while populating the List, it is somewhere trying to exceed the 2GB limit. That's why you are getting System.OutOfMemoryException

answered Dec 12, 2013 at 6:24

skjcyber

6,00712 gold badges44 silver badges60 bronze badges

Collectives™ on Stack Overflow

System memory exception on for loop

4 Answers 4

1 Comment

2 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related