How to get HTML code from webpage?

Question

I'm trying to get HTML code from a specific webpage, but when I do it using

        HttpWebRequest request;
        HttpWebResponse response;
        StreamReader streamReader;
        request = (HttpWebRequest)WebRequest.Create(pageURL);
        response = (HttpWebResponse)request.GetResponse();
        streamReader = new StreamReader(response.GetResponseStream(), Encoding.GetEncoding("windows-1251"));
        htmlCode = streamReader.ReadToEnd();
        streamReader.Close();

or using WebClient, I get redirected to a login page and I get its code. Is there any other way to get HTML code?

I read some information here: How to get HTML from a current request, in a postback , but didn't understand what should I do, or how and where to specify URL.

P.S.: I'm logged-in in a browser. Notepad++ perfectly gets what I need via "right click - view source code".

Thanks.

Sounds like the page expect you to be in a login session to access the page. You'll have to mimic the login first to get the session (Cookies most likely - Use a CookieContainer for this) and then access the page. — Quintium
– Quintium, Commented Oct 23, 2012 at 13:36

Quentin · Accepted Answer · 2012-10-23 13:36:03Z

2

If you get redirected to a login page, then presumably you must be logged in before you can get the content.

So you need to make a request, with suitable credentials, to the login page. Get whatever tokens are sent (usually in the form of cookies) to maintain the login. Then request the page you want (sending the cookies with the request).

Alternatively (and this is the preferred approach), most major sites that expect automated systems to interact with them provide an API (often using OAuth for authentication). Consult their documentation to see how their API works.

answered Oct 23, 2012 at 13:36

Quentin

949k136 gold badges1.3k silver badges1.4k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Swomble · Accepted Answer · 2012-10-23 13:38:43Z

1

If the page you want to get to is behind a login screen - you're going to need to do the login mechanism through code. And add an associated CookieCollection to hold the login cookie that the website will try to drop on your Request.

Alternatively, if you have a user who can help the program along, you could try listing the cookies for the site once they've logged in through their browser. Copy that cookie across and add it to the CookieCollection.

Cheers Simon

answered Oct 23, 2012 at 13:38

Swomble

9098 silver badges17 bronze badges

Comments

Felice Pollano · Accepted Answer · 2012-10-23 13:43:36Z

0

If you want to scrap an html page that requires autentication, I suggest you to use Watin to fill the proper fields and navigate to the pages you want to download. Maybe iot seems a little overkilling at a first glance, but it will save a lot of troubles later.

answered Oct 23, 2012 at 13:43

Felice Pollano

33.4k10 gold badges77 silver badges125 bronze badges

Collectives™ on Stack Overflow

How to get HTML code from webpage?

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related