I can't think of any particularly nasty way to mess up your server too much using just numbers and a handful of operators, however, there are some things you need to look out for:
Given that the [^...] is a character class, you do not need to separate every value with |. This is probably what you really want: [^^()\d*\/+-]. This will match everything you do not want to allow.
Additionally, it is important to remember that, in JavaScript, ^ does not represent powers but rather "exclusive or". This means, for example, that 2 ^ 3 == 1. So you probably do not want to whitelist ^: [^()\d*\/+-].
You might encounter invalid syntax like (1 * (2 + 3), so you should watch out for that as well. You can probably just have a try catch block and meaningfully deal with things like that (report the problem back to the user or something).
^at the beginning negates it, and you can't use repetition operators in a character class. Use[()^\d*/+-]instead.[,]and!to that set, we can start to do some damage!([][1]+[])[2]+(![]+[])[4]+([][1]+[])[1]+([][1]+[])[5]+(![]+[])[4]+([][1]+[])[2]==='denied'