HtmlUnit doesn't work on javascript processing

Question

I have tried almost all the methods mentioned in Stackoverflow, but none of them worked...

I'm trying to scraping following page using HtmlUnit: http://www.nseindia.com/corporates/offerdocument/past_issue_document.htm

Just an empty page returned. It should be caused by javascript issue. I tried following tricks in HtmlUnit: waitForBackgroundJavaScript, refresh, redirect, sleep, enable javascript, click(true, true, true), etc. None of them worked...

Any suggestion:

my code:

String url = "http://www.nseindia.com/corporates/offerdocument/past_issue_document.htm";
WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_8);
webClient.setJavaScriptEnabled(true);
HtmlPage page = (HtmlPage) webClient.getPage(url);
this.getWebClient().waitForBackgroundJavaScriptStartingBefore(5000);
System.out.println(page.asXml());

Thanks a lot!

If the page uses ajax you may need webClient.setAjaxController(new NicelyResynchronizingAjaxController());. This will cause the AJAX to block the call until it is completed. — Opal
– Opal, Commented Dec 20, 2012 at 2:54

R.Moeller · Accepted Answer · 2012-12-18 18:09:40Z

1

I had once similar issues. I workarounded it by using a firefox dev plugin, which logs all the requests the javascript page does. Then I emulated those requests directly from HtmlUnit (just grep the requests from the request log, paste them and inject sessionid misc parameters which are usually easy to identify. Especially useful when dealing with sites using a lot of ajax stuff.

answered Dec 18, 2012 at 18:09

R.Moeller

3,4461 gold badge19 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

WDong Over a year ago

Finally, I gave up to access it directly. Instead, I hijacked the response json file from their server and work on it. This India SE website really sucks! Always down!

Collectives™ on Stack Overflow

HtmlUnit doesn't work on javascript processing

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related