try with
string checkuser = "select count(*) from [User] where UserName=@UserName";
your problem is ExecuteScalar return the first row , first column value of the result and it can't convert to a integer
if you have number column , for example age, do as below
string checkuser = "select age from [User] where UserName=@UserName";
your SQL statement widely open for sql injection attacks, you better use parameters
string sql= "select count(*) from [User] where UserName = @UserName";
using(SqlConnection con = new SqlConnection(conString))
using(SqlCommand cmd= new SqlCommand(sql, con))
{
con.Open();
cmd.Parameters.AddWithValue("@UserName", txtusername.Text);
int temp = Convert.ToInt32(cmd.ExecuteScalar().ToString());
if(temp == 1)
{}
}