There are a lot of ways you could simplify this. For example, if you didn't allow layers (for example, a bridge you can both walk over and under, you could do all the collision detecting in X/Y dimensions, where you filter the collision set to anything of its height+depth or higher.