http://en.wikipedia.org/wiki/Two%27s_complement - good read regarding signed numbers in binary format. It really isn't "complex"; the highest bit being "1" denotes a negative number. As there is no "-0" the negative numbers also have an offset of one - i.e. -1 is stored as "2" with the high bit set.
If you mix unsigned and signed - you simply get unexpected results.
You need to be very precise, i.e. check if the signed number is negative and handle it with care to make it "compatible" with the unsigned number before you use them together. What "care" ends up be in reality depends on what you expect to get out of it, as result.