... aa.getClass == a.Inner and ba.getClass == b.Inner, but in fact they are both Outer.Inner and equal
This is not true. Inner is a class member and is unique to its parent instance of Outer. This means that both a and b have their own unique version of Inner, which are incompatible types. So a.Inner is not the same type as b.Inner, and therefore an a.Inner can never be equal to a b.Inner. I cannot assign one for the other:
scala> val z: a.Inner = aa // aa is a.Inner, so this is ok
z: a.Inner = Outer$Inner@575d06dd
scala> val z: b.Inner = aa // aa is not b.Inner, so it fails to compile
<console>:14: error: type mismatch;
found : a.Inner
required: b.Inner
val z: b.Inner = aa
^
getClass just isn't very useful here.
We can prove this with reflection:
import scala.reflect.runtime.universe._
def tpeOf[A](a: A)(implicit tt: TypeTag[A]) = tt.tpe
scala> tpeOf(aa) =:= tpeOf(ba) // different Outer parents
res24: Boolean = false
scala> tpeOf(aa) =:= tpeOf(aa) // Same instance
res25: Boolean = true
scala> tpeOf(aa) =:= tpeOf(ab) // Same Outer parent
res26: Boolean = true
On the other hand, you can use Outer#Inner to specify that you don't care which Outer your Inner type belongs to.
val x: Outer#Inner = aa
val x: Outer#Inner = ab
val x: Outer#Inner = ba
So as stated by @BenReich, you could use aa.isInstanceOf[Outer#Inner] to check if you have any of those types, and they would all return true.
ab.type means something completely different. ab.type is a singleton type that contains nothing but ab. So naturally then, aa.isInstanceOf[ab.type] must be false, because aa is not ab, regardless of whether or not they are both a.Inner.