How to extract source html from webpage?

Question

I am trying to extract the html source of this page, http://www.fxstreet.com/rates-charts/currency-rates/

I want what I see when I save the page from chrome as a .html file.

I tried to do this in java, using bufferedreader, and then using jsoup. I also tried to do it in python, however I keep getting the following message:

"This site requires JavaScript and Cookies to be enabled. Please change your browser settings or upgrade your browser."

The end goal is to extract the values in the main table.

Xeon was right. It works fine with HtmlUnit, with JS enabled. — Spart
– Spart, Commented Jun 8, 2012 at 18:15

Community · Accepted Answer · 2017-05-23 12:09:14Z

4

Try using HtmlUnit and setting setJavascriptEnabled(true)

Look also at: this and this

JSoup isn't headless browser to execute Javascript so you must choose other library to get the page and then you can use JSoup to parse it.

edited May 23, 2017 at 12:09

CommunityBot

11 silver badge

answered Jun 1, 2012 at 21:51

Xeon

6,0195 gold badges34 silver badges52 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

JohnnyQ · Accepted Answer · 2012-12-09 15:56:57Z

1

Just to extract the main table can be easily done using Jsoup

here's a method that will take all the content from the main table on the page

public void parse(){
        try{

        Document doc = Jsoup.connect("http://www.fxstreet.com/rates-charts/currency-rates/").get();
        Element content = doc.getElementById("ddlPairsChoose");
        Elements table = doc.getElementsByClass("applet-content");      

        System.out.print(table);

        }

        catch(Exception e){

            System.out.print("error --> " + e);
        }       
    }

It prints out the table on the page

answered Dec 9, 2012 at 15:56

JohnnyQ

55310 silver badges28 bronze badges

1 Comment

Stefan Over a year ago

Like this too, as Jsoup can simulate a browser quite well if you use the connect mechanism. If this wont work either, run a local http proxy and browse the site manually recording the communication. Perhaps the Browser sends additional AJAX background requests to filter out robots or the site requires different user agents (then check this: stackoverflow.com/questions/6581655/…)

Collectives™ on Stack Overflow

How to extract source html from webpage?

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related