Web data scraper without httpwebrequest in .net

Question

In .net there are some .net classes which uses IE component.By them we can read htmldom and do login and data scraping. Kindly can any body give me the name of those classes /assemblies where it is present.I myself used it to login and data scrapping quite a way back

Richard · Accepted Answer · 2012-11-07 11:20:31Z

1

If you want to scrape data from existing web pages, consider the HtmlAgilityPack

answered Nov 7, 2012 at 11:20

Richard

30.7k12 gold badges79 silver badges121 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Kamran Shahid Over a year ago

I were looking for mshtml and SHDocvw solution. your post helped in reminding about it.What I need a mshtml/sHdocvw solution where simply i can put data in certain login field then invoke some click event of a button.Then in the fresh page i have to read the htmldom.

Kamran Shahid · Accepted Answer · 2012-11-07 14:47:14Z

I were able to it richard's.Just minor thing about HTMLTableRowCollection is left from HTMLTable. My code is like following

        object o = null;
        InternetExplorer ie = new InternetExplorerClass();
        IWebBrowserApp wb = ie;
        wb.Visible = chkShowBrowser.Checked;
        wb.Navigate("http://LoginPage.aspx", ref o, ref o, ref o, ref o);

        do
        {
            Thread.Sleep(10000);
        } while (wb.Busy);

        if (ie.Document != null)
        {
            var myDoc = ie.Document as HTMLDocument;

            if (myDoc != null)
            {
                var oUserName = (HTMLInputTextElement)myDoc.getElementById("ctl00_MainBodyPlaceholder_PublicPortalLogin_UserName");
                oUserName.value = ConfigurationManager.AppSettings.Get("userName");

                var oPassword =
                    (HTMLInputTextElement)
                    myDoc.getElementById("ctl00_MainBodyPlaceholder_PublicPortalLogin_Password");
                oPassword.value = ConfigurationManager.AppSettings.Get("password");

                var btnSubmitLogin =
                    (HTMLInputElement)myDoc.getElementById("ctl00_MainBodyPlaceholder_PublicPortalLogin_Login");
                btnSubmitLogin.click();

                do
                {
                    Thread.Sleep(10000);
                } while (wb.Busy);


                if (ie.Document != null)
                {
                    wb.Navigate("http://SearchPage.aspx", ref o, ref o, ref o, ref o);


                    do
                    {
                        Thread.Sleep(10000);
                    } while (wb.Busy);



                    if (ie.Document != null)
                    {

                        var oIncidentNumber =
                            (HTMLInputTextElement)
                            myDoc.getElementById("ctl00_MainBodyPlaceholder_txtIncidentNumber");
                        oIncidentNumber.value = ConfigurationManager.AppSettings.Get("incidentNumber");

                        var btnTicketNumberSearch =
                            (HTMLInputElement)myDoc.getElementById("ctl00_MainBodyPlaceholder_btnSearch");
                        btnTicketNumberSearch.click();

                        do
                        {
                            Thread.Sleep(10000);
                        } while (wb.Busy);

                        HTMLTable searchResultTable = myDoc.getElementById("ctl00_MainBodyPlaceholder_gdView_DXMainTable") as HTMLTable;



                        if (searchResultTable != null)
                        {
                            //foreach (var VARIABLE in searchResultTable.T)
                            //{

                            //}
                        }

                        if (chkRenderBody.Checked)
                        {
                            txtFinalTextBox.Text = myDoc.body.outerHTML;
                        }
                    }
                }
            }
        }

How Can i close the question with accepting one comment as answer in stackoverflow?

Collectives™ on Stack Overflow

Web data scraper without httpwebrequest in .net

2 Answers 2

1 Comment

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related