Problem pulling data from website in .NET and C#

Question

I have written a web scraping program to go to a list of pages and write all the html to a file. The problem is that when I pull a block of text some of the characters get written as '�'. How do I pull those characters into my text file? Here is my code:

string baseUri = String.Format("http://www.rogersmushrooms.com/gallery/loadimage.asp?did={0}&blockName={1}", id.ToString(), name.Trim());

// our third request is for the actual webpage after the login.
HttpWebRequest request =
(HttpWebRequest)WebRequest.Create(baseUri);
request.Method = "GET";
request.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1)";
//get the response object, so that we may get the session cookie.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
StreamReader reader = new StreamReader(response.GetResponseStream());

// and read the response
string page = reader.ReadToEnd();

StreamWriter SW;
string filename = string.Format("{0}.txt", id.ToString());
SW = File.AppendText("C:\\Share\\" + filename);

SW.Write(page);

reader.Close();
response.Close();

Encoding issues. Check out SO - stackoverflow.com/questions/2700638/… — Mikael Svenson
– Mikael Svenson, Commented Jun 14, 2010 at 20:14

Joel Coehoorn · Accepted Answer · 2010-06-14 20:17:20Z

2

You're saving a page named loadimage to a text file. Are you sure that's really all text?

Either way, you can save yourself a lot of code by using System.Net.WebClient.DownloadFile().

answered Jun 14, 2010 at 20:17

Joel Coehoorn

418k114 gold badges582 silver badges820 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Gregoire · Accepted Answer · 2010-06-14 20:19:20Z

1

You need to specify your encoding in this line:

StreamReader reader = new StreamReader(response.GetResponseStream());

and

File.AppendText("C:\\Share\\" + filename); uses UTF-8

answered Jun 14, 2010 at 20:19

Gregoire

24.9k7 gold badges50 silver badges74 bronze badges

Comments

Antony · Accepted Answer · 2010-06-14 20:17:44Z

0

Specify Unicode encoding, like so:

New StreamReader(response.GetResponseStream(), Text.Encoding.UTF8)

..same for the StreamWriter

answered Jun 14, 2010 at 20:17

Antony

1,4611 gold badge12 silver badges25 bronze badges

Collectives™ on Stack Overflow

Problem pulling data from website in .NET and C#

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related