Sending file via TCP with binary serialization/deserialization

I am currently writing a client/server application and need to send files (music, video, text, and any other kind) from the server to the client. Currently, I wrote the two following functions:

let sendFile (client:TcpClient) (srcFilePath:string) = 
    let formatter = new BinaryFormatter()

    // Deserialize message content to obj
    use mStream = new MemoryStream()
    (   
        use fStream = File.OpenRead srcFilePath
        fStream.CopyTo mStream 
    )
    let deserializedContent = formatter.Deserialize mStream

    // Create and serialize message into network stream
    let message = { Type = FileSync; Content = Some deserializedContent }
    formatter.Serialize (client.GetStream (), message)

Which sends the file to the client, and on the client's side to receive it:

let readStreamToFile (client:TcpClient) outputPath =
    let formatter = new BinaryFormatter()

    try
        let message = (formatter.Deserialize (client.GetStream ())) :?> Message
        match message.Type with
        | FileSync ->
            use fileStream = File.Open(outputPath, FileMode.Create)
            match message.Content with
            | Some content ->
                formatter.Serialize (fileStream, message.Content)
            | None ->
                failwith "There was no content in the FileSync message!!!"
        | _ ->
            ()
    with
    | :? InvalidCastException as ex ->
        failwith "Message format unknown!!!"

I am getting the following exception when calling formatter.Deserialize mStream:

System.Runtime.Serialization.SerializationException: End of Stream encountered before parsing was completed.

If I bypass copying the FileStream to the MemoryStream, and call formatter.Deserialize fStream instead, I am getting the following exception at the same location:

System.Runtime.Serialization.SerializationException: The input stream is not a valid binary format. The starting contents (in bytes) are: bytes

Why is my BinaryFormatter not able to deserialize my file content?

EDIT:

To ensure that my stream contains bytes, I have also tried the following, but I am getting the same The input stream is not a valid binary format exception:

// Deserialize message content to obj
let fileBytes = File.ReadAllBytes srcFilePath
use mStream = new MemoryStream()
mStream.Write (fileBytes, 0, fileBytes.Length)
mStream.Position <- (int64)0
let deserializedContent = formatter.Deserialize mStream