c# Huge memory build-up with HttpWebRequest

Question

Hello I am having problems with httpwebrequest, huge memory buildups when running a rather large request based program. The only real place the build up can be happening is reading the stream back after sending the request, how would I combat this or better yet set a limit to this sort of thing?

I also have various (10-15) if statements looking for various things in the body of the response but about to change them to a switch statement, although I doubt they're causing such high memory buildups I thought it'd be best to mention them. My request looks like so:

HttpWebResponse response = null;

            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);


                request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
                request.Headers.Add("Upgrade-Insecure-Requests", @"1");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";


                response = (HttpWebResponse)request.GetResponse();

                string _responseData = new StreamReader(response.GetResponseStream()).ReadToEnd();

            catch (WebException e)
            {

            }
            catch (UriFormatException p)
            {


            }

As you can see my stream is read back via _responseData which I then use .contains to see if what I'm looking for is there.

Although it seems like the culprit, you should never make assumptions about memory usage. Profile it. Collecting information about .NET memory allocation helps you identify memory leaks and provides information about what type of objects need how much memory. Then you can take action. — Dennis VW
– Dennis VW, Commented Sep 28, 2019 at 23:56

scopolamin · Accepted Answer · 2019-09-29 01:39:44Z

3

Set HttpWebRequest.AllowWriteStreamBuffering to false.
Dispose HttpWebResponse, Stream and StreamReader.

            try
            {
                HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
                request.AllowWriteStreamBuffering = false;

                request.Headers.Set(HttpRequestHeader.CacheControl, "max-age=0");
                request.Headers.Add("Upgrade-Insecure-Requests", @"1");
                request.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.132 Safari/537.36 OPR/63.0.3368.88";

                string _responseData;

                using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                {
                    using (Stream stream = response.GetResponseStream())
                    {
                        using (StreamReader sr = new StreamReader(stream, Encoding.UTF8))
                        {
                            _responseData = sr.ReadToEnd();
                        }
                    }
                }
            }
            catch (WebException e)
            {

            }
            catch (UriFormatException p)
            {

            }

edited Sep 29, 2019 at 1:39

answered Sep 29, 2019 at 1:12

scopolamin

6046 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

James D Over a year ago

Hello, thank you for your answer. Could you please explain Dispose to me a little, I wasn't aware that you needed to clean up used resources in C# like C++. If that's what it even dose.

Collectives™ on Stack Overflow

c# Huge memory build-up with HttpWebRequest

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related