Download Large File Using Web API Returning OutOfMemoryException

Question

I've tried using numerous questions/answers on SO - but can't seem to overcome the OutOfMemoryException I receive when trying to download a 200MB zip file via the web api.

I've drastically simplified my code in order to test:

    [HttpPost]
    public async Task<HttpResponseMessage> ExportReports(OrderExportFilter filterJson)
    {

        var filename = "C:\\pdftemp\\1128d0ff-a4b7-440d-9e3b-dd152445eb62.zip";

        var fileStream = File.OpenRead(filename);

        var content = new StreamContent(fileStream, 4096);

        resp.Content = content;

        resp.Content.Headers.ContentType = new System.Net.Http.Headers.MediaTypeHeaderValue("application/pdf");
        resp.Content.Headers.ContentDisposition = new System.Net.Http.Headers.ContentDispositionHeaderValue("attachment")
        {
            FileName = "OrdersExport.pdf"
        };

        return resp;
     }

This is just one of many way's I've attempted to download the zip file, I've also referenced this link and duplicated the code to no avail.

I'm at a loss as to what I'm doing incorrectly to enable large zip files to be downloaded.

Update: Per request in comments i've tried using "application/octet-stream" instead - still no luck - same error.

Also - one more thing to note - when i download smaller zip files using this code it works fine, it just seems once the file is too large is starts to bomb.

Exception Update:

I was able to pull the exception details:

    {
        "Message": "An error has occurred.",
        "ExceptionMessage": "Exception of type 'System.OutOfMemoryException' was thrown.",
        "ExceptionType": "System.OutOfMemoryException",
        "StackTrace": "   at System.IO.MemoryStream.set_Capacity(Int32 value)\r\n   at System.IO.MemoryStream.EnsureCapacity(Int32 value)\r\n   at System.IO.MemoryStream.Write(Byte[] buffer, Int32 offset, Int32 count)\r\n   at System.IO.Compression.DeflateStream.WriteDeflaterOutput(Boolean isAsync)\r\n   at System.IO.Compression.DeflateStream.PurgeBuffers(Boolean disposing)\r\n   at System.IO.Compression.DeflateStream.Dispose(Boolean disposing)\r\n   at System.IO.Stream.Close()\r\n   at System.IO.Compression.GZipStream.Dispose(Boolean disposing)\r\n   at System.IO.Stream.Close()\r\n   at System.IO.Stream.Dispose()\r\n   at System.Net.Http.Extensions.Compression.Core.Compressors.BaseCompressor.<Compress>d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Net.Http.Extensions.Compression.Core.Models.CompressedContent.<SerializeToStreamAsync>d__4.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<BufferResponseContentAsync>d__13.MoveNext()"
    }

because we are posting over filters that define the type of data returned — 99823
– 99823, Commented Dec 5, 2017 at 20:26
This code should work fine. Is it exact code which produces exception? What is stack trace? How do you call this code? — Evk
– Evk, Commented Dec 5, 2017 at 20:39
Try changing the media type header to application/octet-stream. If the file is very huge, I think a better way to redirect the user to a normal file download service. — Yuan Z
– Yuan Z, Commented Dec 5, 2017 at 20:42

Martin Liversage · Accepted Answer · 2017-12-05 22:53:50Z

Your stack trace indicates that the exception is thrown when a DeflateStream is closed. This stream apparently writes to a MemoryStream and closing the DeflateStream outputs the entire (compressed) response to the MemoryStream. That is, the byte array buffer in the MemoryStream is allocated and it fails with an OutOfMemoryException if the compressed output is too big.

It is somewhat surprising that the compressed response is buffered in memory because a web server at least conceptually is able to stream data from a file through a compressor and to the network without having to read and compress the entire file into memory first.

From the comments I guess that you are using the NuGet package Microsoft.AspNet.WebApi.Extensions.Compression.Server in your Web API application. While I can't claim that I fully understand how it works I did try to decompile it and from what I can see sending a response through this middleware results in the output being buffered not once but actually twice. (The BaseCompressor.Compress() method writes to a MemoryStream created by StreamManager.GetStream() and this MemoryStream is then read into an array of bytes effectively doubling the memory requirement of the compressor.)

Based on this analysis you should turn off compression for big responses and this seems to have fixed your problem by applying the attribute [Compression(Enabled = false)] to your controller action.

It is a bit ironic that you are unable to compress big responses which might benefit the most from being compressed. However, if you run your Web API on top of a web server like IIS that natively supports response compression you can completely remove the middleware and instead turn compression on in the web server (it is called dynamic compression in IIS). Native compression probably also performs better than compression done in managed code.

Collectives™ on Stack Overflow

Download Large File Using Web API Returning OutOfMemoryException

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related