There is nothing mysterious about the order of execution here, and your counter doesn't get incremented. To understand what happens, let's walk through the code line by line. I'll take append([], 2) to make it quicker.
# you call append([], 2)
return array if n < 0
# n >= 0 so we continue
array << n*2
# array is now [4]
append(array, n - 1)
# you call append(array, 1) which will mutate array,
# lets call x what will be appended to it
# array is now [4, x]
array << n*2
# array is now [4, x, 4]
# you get [4, x, 4] as a returned value from the append method
# because after the first line there is no return statement,
# so the return value of the last line is returned
# let's now see what x, that is append(array, 1) is
return array if n < 0
# n >= 0 so we continue
array << n*2
# array is now [4, 2] because at that time, array is [4]
append(array, n - 1)
# you call append(array, 0) which will mutate array,
# lets call y what will be appended to it
# array is now [4, 2, y]
array << n*2
# array is now [4, 2, y, 2]
# this is what you return to the first method invocation
# so we can replace [4, x, 4] with [4, 2, y, 2, 4]
# let's now see what y, that is append(array, 0) is
return array if n < 0
# n >= 0 so we continue
array << n*2
# array is now [4, 2, 0] because at that time, array is [4, 2]
append(array, n - 1)
# you call append(array, -1) which will mutate array,
# lets call z what will be appended to it
# array is now [4, 2, 0, z]
array << n*2
# array is now [4, 2, 0, z, 0]
# this is what you return to the second method invocation
# so we can replace [4, 2, y, 2, 4] with [4, 2, 0, z, 0, 2, 4]
# now in the last invocation, z is nothing because -1 < 0,
# so nothing is appended to the array
# the first method invocation returns [4, 2, 0, 0, 2, 4]
The return statement only returns from its immediate method invocation. The fact that a method is recursive doesn't change that. It will not somehow find the top level invocation of itself and return from it.
If I can give you an advice when working with recursion, it would be to not mutate your arguments. Working with pure functions is much easier and intuitive, especially in this context. This is how your append method would look like without mutations :
def append n
return n < 0 ? [] : [n * 2, append(n - 1), n * 2].flatten
end
and you would call it like this :
array = append(3)
# [6, 4, 2, 0, 0, 2, 4, 6]
This way your array doesn't get mutated and you get a much clearer image of what the method returns.
In case you don't find it clearer, visualize it this way
# append(3)
[6,
# append(2)
[4,
# append(1)
[2,
# append(0)
[0,
# append(-1)
[]
, 0].flatten
, 2].flatten
, 4].flatten
, 6].flatten