Compiler hang when initializing large std::arrays

Question

I need to initialize a very large multidimensional std::array of data:

class Thing;

class World
{
public:
    World() : space{nullptr} {};
    ~World() = default;
private:
    static unsigned int const size = 1000;
    std::array<std::array<std::array<std::unique_ptr<Thing>, size>, size>, size> space;
};

If you try to instantiate this, G++ 4.8.2 chokes: it consumes all the available memory and will not return. That is, the compiler hangs and I never get an executable. Why is this? Note that clang++ has no trouble.

Note: I fully realize that putting this much data on the stack can overflow it. What is the best way to initialize it on the heap? I think making space a reference (to allocated memory) would be the best way, but I can't figure the syntax out.

If this is instantiated as simply World world; you're consuming slightly more than 1000^3 * sizeof(std::unique_ptr<>) bytes. on a 64 bit system that would be a minimum 7.629 GB. So yeah, I'd say you crossed a line on the automatic variable space limit. I am dying to know the problem this is intended to solve. — WhozCraig
– WhozCraig, Commented Mar 25, 2014 at 3:36
I just tried compiling this (Changing Thing to int) and it compiled. Can you give a small example that demonstrates the compiler failing? — FDinoff
– FDinoff, Commented Mar 25, 2014 at 3:42
@WhozCraig not running locally hopefully no body at my school is using the machine for anything. Just got this g++: internal compiler error: Killed (program cc1plus) Please submit a full bug report, with preprocessed source if appropriate. See <bugzilla.redhat.com/bugzilla> for instructions. Used all 24GB of ram on the computer — FDinoff
– FDinoff, Commented Mar 25, 2014 at 3:53
@thirtythreeforty I can't explain why this pukes on g++ other than my suspicions about it building the object module with all those DDs embedded. I did post an alternative that might work for you in the interim. Hope it helps. — WhozCraig
– WhozCraig, Commented Mar 25, 2014 at 4:03
This looks like GCC Bug #59659: the compiler generates enormous initializer code for large std::array by initializing each element directly. — Casey
– Casey, Commented Mar 25, 2014 at 7:30

WhozCraig · Accepted Answer · 2014-03-25 04:02:14Z

2

Ok, i can't explain the nuance of why g++ is puking on this, but until you work that out, consider this for your member declaration:

std::vector<std::array<std::array<std::unique_ptr<Thing>,size>,size>> space;

and in the constructor initializer list:

World() : space{size}

That should at least get you compiling and move all this to the heap. Note: this better be a 64bit process. I'm going to have to hunt to find out why g++ is doing what I suspect it is doing.

answered Mar 25, 2014 at 4:02

WhozCraig

66.5k11 gold badges82 silver badges148 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

FDinoff Over a year ago

This compiled with size = 4000000000 in less than a second.

George Hilliard Over a year ago

I feel like there is a way to do this with space as a unique_ptr to the 3D array construct and initializing it with the proper call to new (and modifying all the usage appropriately to dereference) but I can't work that constructor syntax out either just yet. Regardless, this works.

WhozCraig Over a year ago

@thirtythreeforty so long as you have something to work with. You certainly could do this with a std::unique_ptr<> at the outer reaches and allocate yourself, but if you need that at runtime, you can do the same just by reserving and emplacing as needed through World members and hit up the vector. At least this gets you on the heap and out of automatic storage. I'll try and do some bugs hunting and see if there is something on this already reported when I get the spare cycles. Glad you have something interim to work with. Wish you could prop FDinoff for this as well. He put some work in it.

FDinoff Over a year ago

@thirtythreeforty or WhozCraig do either of you have an account at the gnu bug tracker and would be willing to submit the bug report there? Or should I just do it?

George Hilliard Over a year ago

I may have an account but if you would rather handle it that would be fine with me.

|

Community · Accepted Answer · 2017-05-23 11:49:22Z

2

As you are using unique_ptr it looks like you are looking for a sparse 3d-matrix like type. For implementing a sparse matrix you could have a look at What is the best way to create a sparse array in C++? and as an implementation detail you could use Boost Multi-index for implementing fast access to all dimensions.

edited May 23, 2017 at 11:49

CommunityBot

11 silver badge

answered Mar 25, 2014 at 8:27

Jan Herrmann

2,80519 silver badges22 bronze badges

Comments

ebasconp · Accepted Answer · 2014-03-25 03:50:57Z

0

vector<vector<vector<unique_ptr<Thing>>>> space;

and when initializing it:

for (int i = 0; i < 1000; i++)
{
   vector<vector<unique_ptr<Thing>>> sp1;
   for (int j = 0; j < 1000; j++)
   {
      vector<unique_ptr<Thing>> sp2;
      for (int k = 0; k < 1000; k++)
         sp2.push_back(make_unique<Thing>("params", "to", "construct", "thing"));

       sp1.push_back(move(sp2));
   }

   space.push_back(move(sp1));
}

It is similar to your approach, but it constructs the vectors in the heap instead of stack.

answered Mar 25, 2014 at 3:50

ebasconp

1,6582 gold badges17 silver badges27 bronze badges

Collectives™ on Stack Overflow

Compiler hang when initializing large std::arrays

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

7 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related