The reason you're getting infinite recursion is because you're telling Python to store the result of the recursive call rec('a') in the dictionary. This means it recurses unconditionally, since you always build the dictionary before doing the lookup.
One way to solve this would be to store lambda functions in the dictionary, and then only call the one you get. In this version, the recursive call is only made when the appropriate x value is passed in:
def rec(x):
return {
'a': lambda: 'one',
'b': lambda: 'two',
'ac': lambda: rec('a'),
}.get(x, lambda: x)()
Unfortunately, that's a bit cumbersome. I'm not sure I'd bother if there were just have three cases and a default. I'd just use a series of if statements, even though that might not be as efficient as a dictionary lookup.