Why is it strange? It is working as intended, perhaps due to the flag you passed as 2nd argument.
Let's take a look at your pattern: -
".*(?<!S)\\.a\\s*\\(\\s*\\)\\s*$"
Your pattern will match the string which does not have an S before .a. This is clear from the below part of your pattern.
(?<!S)\\.a // Match if `.a` is not preceded by `S`.
Now, when you use Pattern.CASE_INSENSITIVE, this condition will check for both s and S, before .a and if any of them is present, it will not match it. So your pattern literally becomes: -
(?<![sS])\\.a // Match if `.a` is not preceded by `S` or `s`.
Now, in your string: -
"attributes.a()"
You have a small s before .a. So, the pattern in your code will return true for this string, with case insensitivity flag enabled.
Just FYI, you can also use (?i) flag instead of passing a second parameter for Pattern.CASE_INSENSITIVE to see the same effect. So the below pattern is same as yours: -
Pattern p = Pattern.compile("(?i).*(?<!S)\\.a\\s*\\(\\s*\\)\\s*$");
The advantage of using (?i) is that, you can use it to make only a part of your pattern CASE_INSENSITIVE. For e.g., if you want that you only check for (?<!S), but the following string can be .a or .A, then you can use (?i) just before .a: -
Pattern p = Pattern.compile(".*(?<!S)(?i)\\.a\\s*\\(\\s*\\)\\s*$");
^^^
Now, the entire pattern after the (?i) flag will be matched in CASE_INSENSITIVE manner.
Also, note that, you don't really need a look-behind or case-insensitive matching here as pointed out in a comment. You can do it simply by using [^S] and [aA] if your letter is only supposed to be a. Because look-behinds and case-insensitivity brings with them some performance difference. So, you can simply change your pattern to: -
Pattern p = Pattern.compile(".*[^S][.][aA][ ]*[(][ ]*[)][ ]*$");
I also replaced backslashes with character class. I prefer using it, instead of double-escaping the meta-characters. But it's just a matter of taste. You can use whatever you like.