The exception is still being thrown, but before Python can show it you are calling f2() again.
So each time the exception is raised, you sneak in another call. That recursive call is allowed (because we are one step below the limit), we cross the limit, the exception is raised again, the finally handler sneaks in another call, almost ad infinitum.
CTRL-C doesn't end the program for the same reasons; an exception is raised (KeyboardInterrupt), but again the finally: handler sends you back into recursion.
You are now falling at such a velocity you entered orbit around the interpreter.
It all does end, but the finally handlers add an exponentially growing number of extra calls before the stack can fully unwind:
>>> import sys
>>> def f2(depth=0, final=0):
... try:
... print depth
... f2(depth + 1, final)
... finally:
... print 'finally:', final
... f2(depth, final + 1)
...
>>> sys.setrecursionlimit(5)
>>> f2()
0
1
2
3
finally: 0
finally: 0
2
finally: 1
finally: 0
1
2
finally: 1
finally: 1
1
finally: 2
finally: 0
0
1
2
finally: 1
finally: 1
1
finally: 2
finally: 1
0
1
finally: 2
finally: 2
0
finally: 3
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 7, in f2
File "<stdin>", line 7, in f2
File "<stdin>", line 7, in f2
File "<stdin>", line 7, in f2
RuntimeError: maximum recursion depth exceeded