Let's introduce some more parentheses in your first example:
max3 a b c = (max a) ((max b) c)
Now compare that to the ones in your last one:
max3 a b c = (max a) . ((max b) c)
Or, if we write (.) in prefix notation:
max3 a b c = (.) (max a) ((max b) c)
And now we see why you get that error. In order to type-check, (max b) c needs to be a function:
(.) :: (b -> c ) -> (a -> b ) -> a -> c
max a :: Float -> Float
max b c :: Float
^^^^^^^^^^^
We get a better error message if we use a constrained version of max instead:
maxFloat :: Float -> Float -> Float
maxFloat = max
max3 a b c = max a . max b c
Now the error message is a lot better:
Couldn't match expected type ‘a0 -> Float’ with actual type ‘Float’
Possible cause: ‘maxFloat’ is applied to too many arguments
In the second argument of ‘(.)’, namely ‘maxFloat b c’
In the expression: maxFloat a . maxFloat b c
That being said, let's actually tackle the problem:
max3 a b c = max a ((max b) c)
= (max a . max b) c
Note that you can also write
max3 :: Ord a => a -> a -> a -> a
max3 a b = max a . max b
(.)isn't some built-in magic operator; it's a normal function defined as any other function would be.a0 -> Float.