There are three questions to consider here: What if the user expects some inputs
to be invalid? What if the invalid inputs are unintentional? and What patterns
are commonly found in the language?
When invalid inputs are expected
The user might expect certain inputs to be invalid. If the function returns NaN, these cases are easy to deal with.
let values = [[2, 4, 6], [3, 5, 1], [7, 5, 9]];
let clamped = values.map(args => clamp(...args)).filter(n => !isNaN(n));
console.log(clamped); // [4, 7]
If the function were to throw an exception, the user would have to use a
try...catch statement, which would be more verbose and less performant. Or she
could filter out the invalid inputs before applying clamp(), but then she
would have to know exactly which kinds of inputs are invalid, which could be
more complicated than it is in this example.
This is clearly a point in favor of the NaN option.
When invalid inputs are unintended
Genuine errors on part of the user will be more easily debugged if the function
throws an exception. For example, the user could easily make a mistake like
this:
clamp(-4, 0, -10);
If the above were to return NaN, debugging could be nightmarish, especially in
a large code base with lots of other things going on. If the function were to
throw an exception, the user would see this:
RangeError: min may not be greater than max
at clamp ...
Much simpler. This is clearly a win for throw.
Common language patterns
So we have a tie. Our best bet is to ask ourselves, WWJD (What would JavaScript
do)? What happens, for example, when we pass a negative number to Math.sqrt()?
console.log(Math.sqrt(-4)); // NaN
It returns NaN. This is the tie-breaker. JavaScript programmers will expect
functions like these to return NaN if the inputs are invalid. And they will
be used to dealing with errors that originate from this behavior, or if not,
then they had better get used to it.
So return NaN.