Javascript based dynamic content using htmlUnit

Question

I have been stuck in getting JavaScript based dynamic content using HtmlUnit. I am expecting to get (Signin, Registration html content) from the page. With the following code, I only get the static content.

I am new to HtmlUnit. Any help will be highly appreciated.

String strURL = "https://www.checkmytrip.com" ;
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);

final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31);
webClient.getOptions().setJavaScriptEnabled(true);
webClient.getCookieManager().setCookiesEnabled(true);
webClient.waitForBackgroundJavaScript(60 * 1000);
webClient.setAjaxController(new NicelyResynchronizingAjaxController());

HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));

String theContent = myPage.getWebResponse().getContentAsString();
System.out.println(theContent);

Ahmed Ashour · Accepted Answer · 2015-05-20 07:37:08Z

5

Two points:

You need to waitForBackgroundJavaScript() after you get the page, as hinted here

You should use myPage.asText() or .asXml() instead, because getWebResponse() returns the original content without JavaScript execution.

String strURL = "https://www.checkmytrip.com" ;
java.util.logging.Logger.getLogger("com.gargoylesoftware.htmlunit").setLevel(java.util.logging.Level.OFF);
java.util.logging.Logger.getLogger("org.apache.http").setLevel(java.util.logging.Level.OFF);

try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX_31)) {
    webClient.setAjaxController(new NicelyResynchronizingAjaxController());

    HtmlPage myPage = ((HtmlPage) webClient.getPage(strURL));
    webClient.waitForBackgroundJavaScript(10 * 1000);

    String theContent = myPage.asXml();
    System.out.println(theContent);
}

answered May 20, 2015 at 7:37

Ahmed Ashour

5,66310 gold badges41 silver badges63 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Javascript based dynamic content using htmlUnit

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related