First of all, if you're trying to match some character in the range A-Z, you need to put it in square brackets. This
.replaceAll("A-Z|ss$", "SS")
will look for the three characters A-Z in the source, which isn't what you want. Second, I think you're confused about what | means. If you say this:
.replaceAll("[A-Z]|ss$", "SS")
it will replace any upper-case letter at the end of the word with SS, because | means look for this or that.
A third problem with your approach is that the second and third replaceAll's will look for any ss that was in the original string, even if it didn't come from a ß. This may or may not be what you want.
Here's what I'd do:
String replaceUml = str
.replaceAll("(?<=[A-Z])ß", "SS")
.replaceAll("ß", "ss");
This will first replace all ß by SS if the character before the ß is an upper-case letter; then if there are any ß's left over, they get replaced by ss. Actually, this won't work if the character before ß is an umlaut like Ä, so you probably should change this to
String replaceUml = str
.replaceAll("(?<=[A-ZÄÖÜ])ß", "SS")
.replaceAll("ß", "ss");
(There may be a better way to specify an "upper-case Unicode letter"; I'll look for it.)
EDIT:
String replaceUml = str
.replaceAll("(?<=\\p{Lu})ß", "SS")
.replaceAll("ß", "ss");
A problem is that it won't work if ß is the second character in the text, and the first letter of the word is upper-cased but the rest of the word isn't. In that case you probably want lower-case "ss".
String replaceUml = str
.replaceAll("(?<=\\b\\p{Lu})ß(?=\\P{Lu})", "ss")
.replaceAll("(?<=\\p{Lu})ß", "SS")
.replaceAll("ß", "ss");
Now the first one will replace ß by ss if it's preceded by an upper-case letter that is the first letter of the word but followed by a character that isn't an upper-case letter. \P{Lu} with an upper-case P will match any character other than an upper-case letter (it's the negative of \p{Lu} with a lower-case p). I also included \b to test for the first character of a word.
ss, the second one takesssat the end of a word (or the stringA-Z) and replaces it withSS(which is howFUSShappens to be right), but I cannot figure out what you think the third one is supposed to do... Can you clarify?ssinbetween. If all mentioned expressions are true then replace thesstoSSssif the string contains only lowercase characters and to upperSSif it is all uppercase, the third replace I still can't figure out what you are trying to achieve and you are probably confusing character class ranges with literal character matching.