This was an interesting algorithm, I tried this out to figure out what was going on and in the example I tried I had values that were greater then the length of the array which resulted in java.lang.ArrayIndexOutOfBoundsException. I then realized in your example all array values were less then the length of the array itself.
The reason this works is because the algorithm is using the array value itself as the index to mark in the array to determine if there is a duplicate. As it is iterating through each element it sets the value of the element at the index of that element value negative, so as it iterates if the same value is there it goes and checks that index if it was ever set to a negative value then we know that a duplicate was found.
This becomes more visibly clear if you print out the array itself after each iteration:
[1, -6, 1, 1, 2, 2, 5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
[1, -6, 1, 1, 2, 2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 1
[1, -6, 1, 1, 2, 2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 1
[1, -6, 1, 1, 2, 2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
[1, -6, -1, 1, 2, 2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 2
[1, -6, -1, 1, 2, 2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
[1, -6, -1, 1, 2, -2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 6
[1, -6, -1, 1, 2, -2, -5, 6, 8, 9, 6, 6, 6, 6, 10, 10]
[1, -6, -1, 1, 2, -2, -5, 6, -8, 9, 6, 6, 6, 6, 10, 10]
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 6
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 6
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 6
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, 6, 6, 6, 6, 10, 10]
Array has duplicates: 6
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, 6, 6, 6, 6, 10, 10]
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, -6, 6, 6, 6, 10, 10]
Array has duplicates: 10
[1, -6, -1, 1, 2, -2, -5, 6, -8, -9, -6, 6, 6, 6, 10, 10]
From this we can see that the first iteration (i=0) the value of the array element is 1. The value of the array at index 1 = 6. This value is > 0 so it goes to the else condition and sets the value at this index to -6. The second iteration it does the same thing at takes the value at index 6 (note Math.abs is used to prevent negative index) and sets the value of 5 to -5 yet again in the second iteration it went to the else condition.
Now at the third iteration the value of the array[2] = 1 which has a value of -6. Since this is negative we know that we had to have seen this value prior as it set the value to a negative number and therefore must be a duplicate.
Please note that in order for this algorithm to work the following preconditions must be met:
1) The values of the array elements must be [0,n) where n = the length of the array