First of all, thanks for your opinions. During last weekend I have run stress test with real data using a revised program and so happy that my problem is solved (Many thanks to A.J. ^_^ ). I would like to share my findings.
After studying the example mentioned by A.J., I have run some test program to read and process data using StringTokenizer and "indexOf" (Regex is even worst compared to StringTokenizer in my situation). My test program would count how many mini second is needed to process 24 messages (~12000 tokens each).
StringTokenizer need ~2700ms to complete, and "indexOf" only take ~210ms!
I've then revised my program like this (with minimum changes) and tested with real volume during last weekend:
Original program:
public class MsgProcessor {
//Some other definition and methods ...
public void processMessage (String msg)
{
//...
StringTokenizer token = new StringTokenizer(msg, FieldSeparator);
while (token.hasMoreTokens()) {
my_data = token.nextToken();
// peformance different action base on token read
}
}
}
And here is updated program using "indexOf":
public class MsgProcessor {
//Some other definition and methods ...
private int tokenStart=0;
private int tokenEnd=0;
public void processMessage (String msg)
{
//...
tokenStart=0;
tokenEnd=0;
while (isReadingData) {
my_data = getToken(msg);
if (my_data == null)
break;
// peformance different action base on token read ...
}
}
private String getToken (String msg)
{
String result = null;
if ((tokenEnd = msg.indexOf(FieldSeparator, tokenStart)) >= 0) {
result = msg.substring(tokenStart, tokenEnd);
tokenStart = tokenEnd + 1;
}
return result;
}
}
- Please noticed that there is no "null" data in original tokens. If no FieldSeparator found, "getToken(msg)" will return null (as a signal for "no more token").
StringTokenizerthat would blow up for long inputs. It has to be something in the surrounding code.