Code explanation
Removing ?: makes the group capturing, which means that it will capture anything matched by [^\)]+, and the matched text is made available for replacement and backreference.
var output = 'text(-) 3'.replace(/text\(([^\)]+)\)/g, RegExp.$1);
When String.replace is called, RegExp.$1 is not yet initialized, so the result in output will be " 3". Then the regex is executed, and the capturing group captures - in text(-) and place it in capturing group 1.
The deprecated attribute RegExp.$1 is also updated with the content of the capturing group 1 of the last RegExp execution.
Then on the next line:
output = 'text(-) 3'.replace(/text\((?:[^\)]+)\)/g, RegExp.$1);
Since RegExp.$1 now contains "-", the replacement give the result "- 3".
RegExp.$1 is deprecated, since it is shared between all execution of RegExp and extremely error prone. So are a bunch of other properties RegExp.$2 to RegExp.$9 which contains the content of the capturing group 2 to 9 of the latest execution.
Solution
For your purpose, you should do this:
var output = 'text(-) 3'.replace(/text\(([^\)]+)\) */g, "$1");
It is not clear how many spaces after ), so I added *. To refer to the content of a capturing group in the replacement, use $1 in the replacement string.