Just to expand on the reason that you are having problems here, you may want to read a little about javascript Scope (Very Helpful Blog). Essentially, consider the following code:
<script>
var thisOne=true;
function thatOne() {
alert("whizbang");
}
var theOther={foo:"bar"};
//More code here...
</script>
Once you reach the comment, you know you can access those variables and the function directly, like if (thisOne) {...}, element.onclick=thatOne; or console.log(theOther.foo). However, you can also access them as children of the root object which, in a web browser, is called window. So you can do:
console.log(window["thisOne"]);
window.thatOne.call(obj, params);
console.log(window.foo.bar);
so by defining open() as a function which is not inside another element (which is to say, is inside the root element), you overwrite the window.open() function. When you attempt to call the function later on, you get problems because the open function calls window.open, which calls window.open, which calls window.open...
There's a few ways to get around this -
Define the onclick handler inline
To do this, get rid of the whole <script>..</script> element then, using whichever element you choose (that supports it) add the onclick attribute:
onclick="window.open('hello.jsp','window','status=1,height=700, width=800');"
This is a nice and quick method, and it keeps all the logic right there with it's triggering element, but it is not easily extensible and you may find yourself sneered at by some. ("Oh, you use inline javascript? how quaint")
change the method name
This will take the least effort from you in terms of getting your page working now from what you have (it's also essentially what everyone else has suggested). Just change the name of the open method to something like openANewWindow() or gotoJSP() or anything that doesn't already exist in the root object, making sure to get both where you define it (in the script element) and where you use it (in the onclick attributes).
Use a closure
This is almost definitely not what you want in this case, its more complexity than you need for a single function. Just including this as an example of how to get out of the root object, seeing as being in it seems to be the heart of your problem.
You have probably already seen in javascript how to define an object, but you may not know that by defining an object, all you are really doing is adding an object property to the root object. You can use this behavior to your advantage, to give a hierarchical structure to your functions.
For example:
<script>
var MyFunctions = (function() {
function open(){
var x=window.open("hello.jsp","window","status=1,height=700, width=800");
x.focus();
}
return {open:open};
})();
</script>
This creates an anonymous function that is immediately run. Inside the scope of this function, another function, open() is defined, however it is defined within the scope of that anonymous function, not the root object (window). After open() is defined, a reference to it is returned as the value of the object property: open.
The result of all this is that the open property of the MyFunctions object is the function you need. You can then call it with MyFunctions.open() or even window.MyFunctions.open().