You have several major flaws...
1) Your code that calls .validate()...
$(document).on("click", '#loginbtn', function(e){
e.preventDefault();
$("#loginForm").validate();
alert('test');
//I want to do Ajax stuff
});
Do not call .validate() within a click handler because it only needs to be called one time. The .validate() method is not the method for calling validation on the form. It is only the method for initializing the plugin on the form. So just like in your working example, you must call .validate() as soon as the form is constructed, which is normally within the DOM ready event. The entire on('click') handler function above can be removed.
2) Your code that opens the dialog...
open:function(){
$(this).html($("#formDiv").html());
},
Within your open callback, you duplicate the HTML from a hidden div into your dialog div for the form. The main problem with this technique is that you now have more than one element on the same page with with same id. Not only is this invalid HTML, but the jQuery Validate plugin will only apply to the first instance, the original hidden instance, of this id. Remove the open callback and attach the hidden div to .dialog() like this: $("#formDiv").dialog( ...
3) Your comment indicates you want to submit this form with ajax(). If that's the case, use the submitHandler callback function of the jQuery Validate plugin. As per documentation, this is the "right place to submit a form via Ajax after it is validated."
$("#loginForm").validate({ // initialize the plugin
submitHandler: function (form) {
// your ajax here; // submit the data
$("#formDiv").dialog("close"); // close the dialog
alert('submitted with ajax'); // for demo
return false; // prevent regular form submit action
}
});
Working DEMO: http://jsfiddle.net/TRHxZ/
$(document).ready(function () {
$("#loginForm").validate({ // initialize the plugin
submitHandler: function (form) {
// your ajax here; // submit the data
$("#formDiv").dialog("close"); // close the dialog
return false; // prevent regular form submit action
}
});
$("a#loginLink").on("click", function (e) {
e.preventDefault();
$("#formDiv").dialog({
closeOnEscape: false,
title: "Login Form",
modal: true,
resizable: false,
width: '350px',
position: 'fixed',
close: function () {
$(".ui-dialog-content").dialog("close");
}
});
});
});