mysql_num_rows() would tell you whether any users matched the provided username. You should also use mysql_real_escape_string to ensure that your username value is safely escaped for use in the query. Also -- be sure your strings are quotes (using single-quotes) inside the mysql query.
Something like this should get you pointed in the right direction:
$username = mysql_real_escape_string($_GET['username'], $con);
$usercheck = mysql_query("SELECT * FROM wp_users WHERE user_login='".$username."'",$con);
if( mysql_num_rows($usercheck) <= 0 ) {
// error: no such user was found
} else {
// found one or more matching users
}