I have already tried using Normalizer
String s = "口水雞 hello Ä";
String s1 = Normalizer.normalize(s, Normalizer.Form.NFKD);
String regex = Pattern.quote("[\\p{InCombiningDiacriticalMarks}\\p{IsLm}\\p{IsSk}]+");
String s2 = new String(s1.replaceAll(regex, "").getBytes("ascii"), "ascii");
System.out.println(s2);
System.out.println(s.length() == s2.length());
i want it to work in Unix/Linux ,
There is an ASCII character class for matching code points in the ASCII set:
String s = "口水雞 hello Ä";
String s1 = Normalizer.normalize(s, Normalizer.Form.NFKD);
String nonAscii = "[^\\p{ASCII}]+";
String s2 = s1.replaceAll(nonAscii, "");
System.out.println(s2);
System.out.println(s.length() == s2.length());
As Joop Eggan notes, Java string and char types are always UTF-16. You can only have ASCII-encoded data in byte form:
byte[] ascii = s2.getBytes(StandardCharsets.US_ASCII);
“ ”) and such by the ASCII quotes.Explanation
First in java text (String/Reader/Writer) is already Unicode. For the java source code (String literals) the editor and the javac compiler should use the same encoding. Ideally UTF-8.
The normalizer splits into base letter and combining diacritical mark(s) and regular expression removes those marks. Converting text with accents like ä é fi fl ĉ œ to a e fi fl c oe to ASCII.
Hence you would get - I think - "??? hello A".
Charset ascii = StandardCharsets.US_ASCII;
String s2 = new String(s1.replaceAll(regex, "").getBytes(ascii), ascii);
To prevent receiving the question marks (and distinguishing between a ? in the original string), you can use a Charset.newDecoder().
For ASCII you would still need some transliteration to latin script.
Answer
As most Linux operating systems of newer origin already use UTF-8 as operating system default, you probably can simply do:
System.out.println("We are using encoding: " + System.getProperty("file.encoding"));
System.out.println(s);
Here s is converted to the operating system encoding.
ff is \uFB00, and this ligature is normalized to ff. But still, normalizer doesn't do anything to ß. Normalizer("ß", Normalizer.Form.WHICHEVER) returns "ß", not "ss", and the regex does nothing afterwards.