How to break a file into tokens based on regex using Java

Question

I have a file in the following format, records are separated by newline but some records have line feed in them, like below. I need to get each record and process them separately. The file could be a few Mb in size.

 <?aaaaa>
 <?bbbb
     bb>
 <?cccccc>

I have the code:

 FileInputStream fs = new FileInputStream(FILE_PATH_NAME);
 Scanner scanner = new Scanner(fs);
 scanner.useDelimiter(Pattern.compile("<\\?"));
 if (scanner.hasNext()) {
     String line = scanner.next();
     System.out.println(line);
 } 
 scanner.close();

But the result I got have the begining <\? removed:

aaaaa>
bbbb
   bb>
cccccc>

I know the Scanner consumes any input that matches the delimiter pattern. All I can think of is to add the delimiter pattern back to each record mannully.

Is there a way to NOT have the delimeter pattern removed?

Bohemian · Accepted Answer · 2017-01-06 20:04:36Z

5

Break on a newline only when preceded by a ">" char:

scanner.useDelimiter("(?<=>)\\R"); // Note you can pass a string directly

\R is a system independent newline
(?<=>) is a look behind that asserts (without consuming) that the previous char is a >

Plus it's cool because <=> looks like Darth Vader's TIE fighter.

edited Jan 6, 2017 at 20:04

answered Jan 6, 2017 at 18:40

Bohemian♦

427k103 gold badges603 silver badges750 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

jlp Over a year ago

I tested with more records and this approach made some records on the same line. Can you please help ?

Bohemian Over a year ago

@jlp do you mean a "missing" newline like in "<?aaa>\n<?bbb><?ccc>\n<?ddd>" between bbb and ccc?

Bohemian Over a year ago

btw, if that is the case it's easy to handle - just add ? to the end of the regex

jlp Over a year ago

@Bohemian Hi, sorry, I have another question. If the data is like "<aaaa><bbbb><cccc>" with no space or newline between them, how do I make my regex so that it can break them into 3 lines like <aaaa> then <bbbb> and <cccc>. Please let me know if I should create another post for this questions. Thanks.

Bohemian Over a year ago

@jlp as per previous comment; add ? to the regex: scanner.useDelimiter("(?<=>)\\R?");. This makes the newline optional, but will consume it if it's there.

Community · Accepted Answer · 2017-05-23 11:54:39Z

1

I'm assuming you want to ignore the newline character '\n' everywhere.

I would read the whole file into a String and then remove all of the '\n's in the String. The part of the code this question is about looks like this:

String fileString = new String(Files.readAllBytes(Paths.get(path)), StandardCharsets.UTF_8);
fileString = fileString.replace("\n", "");
Scanner scanner = new Scanner(fileString);
...  //your code

Feel free to ask any further questions you might have!

edited May 23, 2017 at 11:54

CommunityBot

11 silver badge

answered Jan 6, 2017 at 18:41

KarelPeeters

1,46413 silver badges29 bronze badges

2 Comments

jlp Over a year ago

The file may be a few Mb large, not sure if it's going to cause any issues if storing the entire file into a string.

KarelPeeters Over a year ago

@jlp I wouldn't worry about files being a couple of megabytes in size, but you're right that this approach wouldn't scale very well.

ninja.coder · Accepted Answer · 2017-01-06 18:44:17Z

0

Here is one way of doing it by using a StringBuilder:

public static void main(String[] args) throws FileNotFoundException {
    Scanner in = new Scanner(new File("C:\\test.txt"));
    StringBuilder builder = new StringBuilder();

    String input = null;
    while (in.hasNextLine() && null != (input = in.nextLine())) {
        for (int x = 0; x < input.length(); x++) {
            builder.append(input.charAt(x));
            if (input.charAt(x) == '>') {
                System.out.println(builder.toString());
                builder = new StringBuilder();
            }
        }
    }

    in.close();
}

Input:

 <?aaaaa>
 <?bbbb
     bb>
 <?cccccc>

Output:

 <?aaaaa>
 <?bbbb     bb>
 <?cccccc>

answered Jan 6, 2017 at 18:44

ninja.coder

9,6685 gold badges40 silver badges55 bronze badges

Collectives™ on Stack Overflow

How to break a file into tokens based on regex using Java

3 Answers 3

5 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related