How to detect the one function causing a stack overflow quickly in Visual C++?

Question

I have a huge C++ codebase. On a certain set of data there's a stack overflow. If I run the program under Visual Studio debugger I get a call stack 30 unfamiliar functions deep - one (or more) of those functions created a too big object on stack and this lead to stack exhaustion. I looked at all functions and there's nothing obvious - nothing like

char buffer[512 * 1024];

I though I could add a variable at the beginning of each of those functions and dump that variable address and recompile and then look at difference between adjacent functions, but that's lots of manual labor.

How do I quickly identify the function that created a too large set of objects on stack and causes a buffer overflow?

Are you sure it is a single function overflowing the stack, or is it due to some unintended recursion? — Stack Overflow is garbage
– Stack Overflow is garbage, Commented Aug 12, 2011 at 9:09
I might be missing something, but isn't it always the function at the top of the stack that causes the overflow? — fredoverflow
– fredoverflow, Commented Aug 12, 2011 at 9:09
@FredOverflow: obviously that's the proximal cause, but if it's strlen then you'd probably want to look for something else rather than concluding, "oh dear, strlen uses too much stack" ;-) — Steve Jessop
– Steve Jessop, Commented Aug 12, 2011 at 9:16
@sharptooth: long time since I've used Visual Studio, and nothing like the latest version, but can you put esp in a debug window and then roam around the call stack, looking to see where it changes a lot? — Steve Jessop
– Steve Jessop, Commented Aug 12, 2011 at 9:17
@FredOverflow: Well, it triggers the overflow, but the reason can be some other function used too much stack before this function has been called. — sharptooth
– sharptooth, Commented Aug 12, 2011 at 9:19

Ajay · Accepted Answer · 2011-08-12 11:16:28Z

4

You can use Code Analysis in Visual C++ which is available in higher editions. A warning (C6262) is generated if function uses stack higher than some limit. You may use /analyze:stacksize switch, where stacksize is limit you want.

answered Aug 12, 2011 at 11:16

Ajay

18.6k14 gold badges65 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Matthieu M. Over a year ago

Nice ! There are definitely some good tools integrated in VC++!

sharptooth Over a year ago

There's one problem with this thing. It doesn't account for temporary objects. It just ignores them. And (guess what) it was a temporary object that caused a stack overflow in our code.

Matthieu M. · Accepted Answer · 2011-08-12 10:05:46Z

4

If you have a stack trace (and you should be able to get one), you might be able to access the addresses of the frames.

The one function causing the issue should lead to a huge leap in the frames pointers.

If there is none, check the stack size, it might simply be much too small.

EDIT: How to debug non-obvious issues with VC++ ? (hum... I code on Unix :/)

Elan Rusking made a great talk on investigation in his 2011 GDC presentation (PDF).

The stack pointer is (on x86) stored in the ESP register. If you have a look at the disassembly and check the changes of ESP, then you should be able to see which function incremented/decremented it with a large value.

Example on wikibooks:

mov eax, DWORD PTR SS:[esp]
add esp, 4

This add esp is what you want to track. Unless you are using VLAs, the values added/substracted are hardcoded, so easy to check on.

edited Aug 12, 2011 at 10:05

answered Aug 12, 2011 at 9:19

Matthieu M.

303k58 gold badges495 silver badges754 bronze badges

7 Comments

sharptooth Over a year ago

How do I check frame addresses?

RedX Over a year ago

In debug mode and with frame pointers enabled (default case IIRC) check EBP. See here for how it works codeguru.com/cpp/misc/misc/stack/article.php/c14799

Matthieu M. Over a year ago

@sharptooth: good question, probably with the Watch Window (there's lot of things there). Otherwise you can inspect the disassembly and track esp changes, I've added some references / code to my answer.

Martin Ba Over a year ago

+1 / Note: EBP can be checked with a watch expression (just add EBP to the watch window)

Matthieu M. Over a year ago

@Martin: Yes (and ESP too), however you can compile without frame pointers ( ~ -fomit-frame-pointer on gcc) and then EBP is just another register so not so interesting :)

|

Eric Z · Accepted Answer · 2011-08-12 09:14:58Z

2

A thread that exceeds its stack allocation will raise an exception. This exception can be trapped with the __try and __except keywords in Microsoft Visual C++. You can wrap your functions inside this try-except block to see if they cause the stack overflow.

Take a look here: How to trap stack overflow in a Visual C++ application

answered Aug 12, 2011 at 9:14

Eric Z

14.7k7 gold badges49 silver badges77 bronze badges

Collectives™ on Stack Overflow

How to detect the one function causing a stack overflow quickly in Visual C++?

3 Answers 3

2 Comments

7 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related