I second BrettAM's comment. The Arduino IDE sets most of all the timers up for PWM, and I had to comment out a section once to use my own ISR with Timer 3 on a 32u4, because no matter where or how I set it, the CTC interrupt never went. The WGM50 and CS51 are set at the start of the Arduino files basically, so that messes with your settings.
You will need go into the Arduino core files. This is $install_path/arduino-1.x.x/hardware/arduino/avr/cores/arduino/, and under that directory you need to find file wiring.c. This file contains all the setup of the timers for PWM and the millis() or or micros() functions.
Go to line 342. Located there is this:
#if defined(TCCR5B) && defined(CS51) && defined(WGM50)
sbi(TCCR5B, CS51); //set ter 5 prescale factor to 64
sbi(TCCR5B, CS50);
sbi(TCCR5A, WGM50); // put timer 5 in 8-bit phase correct pwm mode
#endif
Here you can place another #if defined() macro around this, resulting in:
#if !defined(doNotSetTimer5)
#if defined(TCCR5B) && defined(CS51) && defined(WGM50)
sbi(TCCR5B, CS51); //set ter 5 prescale factor to 64
sbi(TCCR5B, CS50);
sbi(TCCR5A, WGM50); // put timer 5 in 8-bit phase correct pwm mode
#endif
#endif
Now when you want to use your custom settings, you will put the following at the top of your .ino file:
#define doNotSetTimer5
#include <wiring.c>
That does not block the Arduino environment from setting up Timer 5 when compiled with other code, but only when you want to use custom settings (changed because of @Gerbens@Gerben's suggestion). When you change these settings you will lose PWM on 3 pins: 44 45 46. But you will be able to make use of setting your own PWM up or a CTC ISR().
This doesn't mess with anything else and is the easiest way to do it.