You really shouldn't need to worry about this unless you are doing this millions and millions of times per frame. This graph shows some empirical data about the speed of casting:
Casting Speeds http://img707.imageshack.us/img707/5835/casta.png
http://www.codeproject.com/KB/cs/csharpcasts.aspx
Keep in mind that there are about 17 milliseconds per frame at 60 frames per second, and about 33 milliseconds at 30 frames per second.