9

I get an Out of Memory Exception when using Http.Put of a large file. I am using an asynchronous model as shown in the code. I am trying to send 8K blocks of data to a Windows 2008 R2 server. The exception consistently occurs when I attempt to write a block of data that exceeds 536,868,864 bytes. The exception occurs on the requestStream.Write method in the code snippet below.

Looking for reasons why?

Note: Smaller files are PUT OK. Logic also works if I write to a local FileStream. Running VS 2010, .Net 4.0 on Win 7 Ultimate client computer.

   HttpWebRequest request = (HttpWebRequest)WebRequest.Create("Http://website/FileServer/filename");
   request.Method = WebRequestMethods.Http.Put;
   request.SendChunked = true;
   request.AllowWriteStreamBuffering = true;
   ...

   request.BeginGetRequestStream( new AsyncCallback(EndGetStreamCallback), state);
   ...

   int chunk = 8192; // other values give same result
   ....

   private static void EndGetStreamCallback(IAsyncResult ar) {
        long limit = 0;
        long fileLength;
        HttpState state = (HttpState)ar.AsyncState;

        Stream requestStream = null;
        // End the asynchronous call to get the request stream.

        try {
            requestStream = state.Request.EndGetRequestStream(ar);
            // Copy the file contents to the request stream.

            FileStream stream = new FileStream(state.FileName, FileMode.Open, FileAccess.Read, FileShare.None, chunk, FileOptions.SequentialScan);

            BinaryReader binReader = new BinaryReader(stream);
            fileLength = stream.Length;

            // Set Position to the beginning of the stream.
            binReader.BaseStream.Position = 0;

            byte[] fileContents = new byte[chunk];

            // Read File from Buffer 
            while (limit < fileLength)
            {
                fileContents = binReader.ReadBytes(chunk);

                // the next 2 lines attempt to write to network and server
                requestStream.Write(fileContents, 0, chunk);   // causes Out of memory after 536,868,864 bytes
                requestStream.Flush();  // I get same result with or without Flush

                limit += chunk;
            }

            // IMPORTANT: Close the request stream before sending the request.
            stream.Close();

            requestStream.Close();
        }
    }
2
  • 1
    Does the same happen with POST? Does your code actually send any data? Commented Jun 6, 2011 at 2:17
  • You may want to look into .NET 4's new CopyTo() method for streams Commented Jun 6, 2011 at 2:55

1 Answer 1

18

You apparently have this documented problem. When AllowWriteStreamBuffering is true, it buffers all the data written to the request! So, the "solution" is to set that property to false:

To work around this issue, set the HttpWebRequest.AllowWriteStreamBuffering property to false.

Sign up to request clarification or add additional context in comments.

6 Comments

It's also mentioned in the documentation: “Setting AllowWriteStreamBuffering to true might cause performance problems when uploading large datasets because the data buffer could use all available memory.”
Yes, we tried this and continue to have challenges with either setting that to true or false. I expect that the real solution is to not use PUT or POST to do updates, but instead use a more low level technology like WCF or POS (Plain Ole Sockets lol).
@pearcewg: Add an update to your question (or your own answer!) if you figure out a sure-fire way of getting it to work -- I'm curious too, and it'd be good to have a reference to refer to in the future ;-)
we ended up dumping this, and replaced it with uploading via WCF, which works just great! I would recommend looking at the same solution.
@pearcewg: Cool, good to know. Thanks! What class did you use?
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.