C# Not getting proper response from HttpWebResponse. Encoding?

Question

I'm trying to fetch some webpages using the code below:

    public static string FetchPage(string url)
    {

         HttpWebRequest req = (HttpWebRequest)WebRequest.Create(url);

        req.Method = "GET";

        req.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 6.0; sv-SE; rv:1.9.2.12) Gecko/20101026 Firefox/3.6.12 (.NET CLR 3.5.30729";
        req.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
        req.Headers.Add("Accept-Language", "sv-se,sv;q=0.8,en-us;q=0.5,en;q=0.3");
        req.Headers.Add("Accept-Encoding", "gzip,deflate");
        req.Headers.Add("Accept-Charset", "ISO-8859-1,utf-8;q=0.7,*;q=0.7");
        req.Headers.Add("Keep-Alive", "115");
        req.Headers.Add("Cache-Control: max-age=0");
        req.AllowAutoRedirect = true;

        req.IfModifiedSince = DateTime.Now;

        using (HttpWebResponse resp = (HttpWebResponse)req.GetResponse())
        {
           using (Stream resStream = resp.GetResponseStream())
           {
              StreamReader reader = new StreamReader(resStream);
              return reader.ReadToEnd();
            }
        }
    }

Some pages work (W3C, example.com) while most others I've tried do not (BBC.co.uk, CNN.com, etc). Wireshark shows that I'm getting a proper reponse.

I've tried setting the encoding of the reader to the expected encoding of the response (CNN - utf8) as well as every possible combination but I have had no luck.

What am I missing out on here?

The first bytes of my response are always "1f ef bf bd" if you're able to tell something based on that.

tvanfosson · Accepted Answer · 2010-11-20 15:24:48Z

1

I suspect the most likely explanation is that you are getting compressed data and not uncompressing it. Try using a stream filter to deflate/unzip it. See Rick Strahl's blog article for more info.

answered Nov 20, 2010 at 15:24

tvanfosson

534k102 gold badges703 silver badges801 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

pastapockets Over a year ago

This was indeed the case. Thanks guys.

steinar · Accepted Answer · 2010-11-20 15:29:25Z

1

Loading http://bbc.co.uk worked for me when leaving out the "Accept-Encoding" header:

req.Headers.Add("Accept-Encoding", "gzip,deflate");

answered Nov 20, 2010 at 15:29

steinar

9,7011 gold badge27 silver badges39 bronze badges

Collectives™ on Stack Overflow

C# Not getting proper response from HttpWebResponse. Encoding?

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related