Ok, so Jason Smiley posted a solution but without an explanation as to what the problem was and how the solution works.
The first problem is that the script passed to executeScript does not return anything, period. What executeScript does is wrap the script you pass to it in function () { ... }. So something of the form $.ajax(...) would be executed as this in the browser:
function () {
$.ajax(...);
}
This obviously won't return anything in the same way that if the script were 2 + 2 it would not return 4 because it would execute as:
function () {
2 + 2;
}
The script passed to executeScript would have to be return 2 + 2.
However, here, just adding this return won't fix the problem. There is an additional problem that $.ajax is an asynchronous function. When $.ajax executes, it returns right away. The functions that are passed to it are stored for future execution. If $.ajax returns right away, then the value it returns cannot possibly be the return value of any of the functions that are given to it because these have not executed yet. Conversely, then the functions passed to $.ajax do execute, whatever they give to return cannot be retrieved by your code. The only thing you can do is use the values you care about inside of the function itself or pass it to another function.
The solution is to use executeAsyncScript which is designed for asynchronous execution. This function is just like executeScript but adds an argument which is a callback which should be called when the script has finished executing and it should be called with the value you want executeAsyncScript to return.
Note that it is absolutely not necessary to abandon $.ajax. The following should work:
public static String getHTMLResponse(String url){
String body = (String) js.executeAsyncScript(
"var url = arguments[0];" +
"var callback = arguments[1];" +
"$.ajax({url: url, success: callback});",
url);
return body;
}
Note that I've set this code so that url is passed as an argument instead of being concatenated into the script. I find this cleaner than concatenation. In case this needs be explained, arguments is an object automatically made available by the JavaScript virtual machine. It holds the list of arguments given to a function.
Whether $.ajax is used or XMLHttpRequest is used directly the principles are the same.