Update for nested square bracket support

Since you need to also support nested square brackets, and the comma should be ignored inside the square brackets, you need a simple parser to collect the chunks of text you need.

public static List<String> splitWithCommaOutsideBrackets(String input) {
    int BracketCount = 0;
    int start = 0;
    List<String> result = new ArrayList<>();
    for(int i=0; i<input.length(); i++) {
        switch(input.charAt(i)) {
        case ',':
            if(BracketCount == 0) {
                result.add(input.substring(start, i).trim());// Trims the item!
                start = i+1;
            }
            break;
        case '[':
            BracketCount++;
            break;
        case ']':
            BracketCount--;
            if(BracketCount < 0) 
                return result; // The BracketCount shows the [ and ] number is unbalanced
            break;
        }
    }
    if (BracketCount > 0)
        return result; // Missing closing ]
    result.add(input.substring(start).trim()); // Trims the item!
    return result;
}

And use it as

String s = "ex1 , [ex2 , ex3 ] , [ hh3 , rt5 , w3 [ bn7 ] ] , ex 4 , ex 4, [ex , ex ]";
List<String> res = splitWithCommaOutsideBrackets(s);
for (String t: res) {
    System.out.println(t);
} 

Output of the sample Java code:

ex1
[ex2 , ex3 ]
[ hh3 , rt5 , w3 [ bn7 ] ]
ex 4
ex 4
[ex , ex ]

Note that trimming items is not necessary.

Also, where I return result, you may want to add code throwing an exception rather than returning the result as it is at that moment.

Original answer

In Java character classes, ] and [ must be escaped, unlike in JavaScript where you only have to escape ] symbol (inside the character class).

String pat = ",(?![^\\[]*])";
                    ^^

Here is an IDEONE demo:

String s = "ex1 , [ex2 , ex3 ] , ex 4 , ex 4, [ex , ex ]";
String pat = ",(?![^\\[]*])";
String[] result = s.split(pat);
System.out.println(Arrays.toString(result));

Note that neither in Java, nor in JS, the ], outside the character class, does not have to be escaped.