toBuff = BitConverter.GetBytes(intNumBuffer);
The call to BitConverter.GetBytes() returns a byte array of length 4, because intNumBuffer is an int, which has size 4.
So, that means that the valid indices of toBuff are 0, 1, 2 and 3. Hence the error when you use index 4.
Now, I suppose that you imagined that when you wrote:
byte[] toBuff = new byte[20];
that toBuff would have length 20. Well, it does at this point. But when you subsequently overwrite toBuff, then you have a new and different array.
Probably what you need to do is as follows:
byte[] toBuff = new byte[20];
Array.Copy(BitConverter.GetBytes(intNumBuffer), toBuff, sizeof(int));
Or perhaps:
byte[] toBuff = new byte[20];
byte[] intBytes = BitConverter.GetBytes(intNumBuffer);
Array.Copy(intBytes, toBuff, intBytes.Length);
Either of these will copy the bits returned by the call to GetBytes() into toBuff.