Problem with Socket and Serialization C#

You never use the message that you read from the stream. The memory stream you are reading from is thus empty.

On a side note, why do you use these intermediate MemoryStreams?

To answer your second question: for maximum maintainability, the class Command should be in a separate assembly that both Client and Server reference.

To answer your first question: you are attempting to deserialize from an empty stream on your server, just as the exception tells you. You need to copy the bytes you read from the clientStream into the memoryStream before you deserialize from the memoryStream. Alternatively, use the clientStream directly rather than using the memoryStream; this may require reconsidering your protocol.

Finally, I wholeheartedly agree with @Andrey: consider using WCF. It's way way way better than raw sockets.

If you change your server code to use a different MemoryStream constructor, the problem will go away.

MemoryStream memoryStream = new MemoryStream(message, 0, bytesRead);

However, I agree with Stephen and others. Either use WCF, or use Message framing.

Another question: I copied the class Command from Client and pasted at Server to deserialize. Is this correct?

No. The namespace for your Client and Server are likely different, so the Server will be trying to deserialize a stream that matches its namespace.

You should create your Command Class using its own namespace and reference that from both your Client and the Server.

After that...

Client:

static void StreamToServer(TcpClient client, Command obj) {
  using (NetworkStream ns = client.GetStream()) {
    using (MemoryStream ms = new MemoryStream()) {
      BinaryFormatter formatter = new BinaryFormatter();
      formatter.Serialize(ms, obj);
      byte[] buf = ms.ToArray();
      ns.Write(buf, 0, buf.Length);
    }
  }
}

Server:

static Command ReadStream(TcpListener listener) {
  Command obj = null;
  using (TcpClient client = listener.AcceptTcpClient()) { // waits for data
    using (NetworkStream ns = client.GetStream()) {
      byte[] buf = new byte[client.ReceiveBufferSize];
      int len = ns.Read(buf, 0, buf.Length);
      using (MemoryStream ms = new MemoryStream(buf, 0, len)) {
        BinaryFormatter formatter = new BinaryFormatter();
        obj = formatter.Deserialize(ms) as Command;
      }
    }
  }
  return obj;
}

WCF or Message framing may be easier, but I don't often have the opportunity at work to sit around and read a book.

Instead of passing bytesRead to the MemoryStream which is actually the length of the byte stream, you should pass 'message', as it is the actual stream of bytes. Like,

MemoryStream memoryStream = new MemoryStream(message);

As you are passing an integer variable, the compiler is throwing exception that the stream is empty.

As for WCF, it is remarkable framework, but for applications that require low latency and high performance, WCF is a horrible answer because of its overheads, it is built upon sockets. So if you use sockets, that will be the lowest level implementation and thus, the fastest. That depends on your application which paradigm you should choose...