This is a bit tricky to find in the spec, because it's not with the rest of the Date stuff.
If you take a look at section 11.6.1, "The Addition operator," you'll find the following note:
NOTE 1 No hint is provided in the calls to ToPrimitive in steps 5 and 6. All native ECMAScript objects except Date objects handle the absence of a hint as if the hint Number were given; Date objects handle the absence of a hint as if the hint String were given. Host objects may handle the absence of a hint in some other manner.
In context, that means using the addition operator (+) with a Date object will use the string value rather than the numeric value. In this sense, Date objects are special and unlike any other kind of objects.
Note that there is no such exception for the subtraction operator, as it is unambiguous — it only works for numeric subtraction; it doesn't operate on strings.
Also note that this applies to the addition operator, a + with operands on both sides. The unary "plus" operator does not work like this, so +myDateObj with nothing on the left hand side will result in the numeric value.
+can be the addition or the string concatenation operator, depending on context. (If you have a string value and try to “add” anything to it, the second argument gets usually converted to string and concatenation happens instead of addition.)