As most answers mention you make an asynchronous call, but thats not really the reason. So JavaScript is single threaded, only on think can be done per time.
So first you call your function and this function will put on the execution context stack. This function will be executed before any other function that will be added to stack can be executed. In this function you make your ajax call and on success the success function will be put on the execution context stack. So this function could never ever called before naviSet. As alert1 is made in the naviSet function it will be the called first.
And to your second question:
From your function I think you believe, when $.ajax() returns true, your ajax call was succesful and pageCount was set to data. But it isn't. $.ajax doesn't return true but the truethy value $. Its a function that return reference to the main jquery object, so you can chain function calls.
function naviSet()
{
//you create a new var which is undefined
var pageCount;
// return $ which is a truethy in JavaScript, but it does not mean the ajax call was successful
if($.ajax({
type: "POST",
url: "http://localhost/mywebsite/wp-content/themes/twentyeleven/more-projects.php",
success:function(data)
{
// now you in the context of your success function
// and set the value of your variable to data
pageCount = data;
alert(pageCount); //alert 1
return true;
},
error:function()
{
$("#direction").html("Unable to load projects").show();
return false;
}
}))
//here you are still in the context of your naviSet function where pageCount is undefined
alert(pageCount); //alert 2
}