Spring batch filtering data inside item reader

Question

I'm writing a batch that reads log files which should take many types (format of log log file ) then I want to read every file based on some characters inside log files for example

15:31:44,437 INFO <NioProcessor-32> Send to <SLE- 15:31:44,437 INFO <NioProcessor-32> [{2704=5, 604=1, {0=023pdu88mW00007z}] 15:31:44,437 DEBUG <NioProcessor-32> SCRecord 2944

In such a log file I want to read only log lines which contain ' [{}] ' and ignore all others. I have tried to read it in item reader and split it to object but I can't figure how. I think that I should create a custom item reader or something like that; my Logline class looks too simple:

public class logLine {

String idOrder;

String time;

String Tags;

}

and my item reader look like:

public FlatFileItemReader<logLine> customerItemReader() {
        FlatFileItemReader<logLine> reader = new FlatFileItemReader<>();

        reader.setResource(new ClassPathResource("/data/customer.log"));

        DefaultLineMapper<LogLine> customerLineMapper = new DefaultLineMapper<>();

        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames(new String[] {"idOrder", "date", "tags"});

        customerLineMapper.setLineTokenizer(tokenizer);
        customerLineMapper.setFieldSetMapper(new CustomerFieldSetMapper());    
        reader.setLineMapper(customerLineMapper);

        return reader;
    }

How can I add a filter in this item reader to read only lines which contain [{

without doing the job in the item processor

Don't you like to do the filtering in CustomerFieldSetMapper class? You can do it there — The Guest
– The Guest, Commented Apr 2, 2018 at 18:59
im doing it inside the item processor now i can reach a single line inside item reader its all about f abstraction — patoCapongo93
– patoCapongo93, Commented Apr 2, 2018 at 22:07

Niraj Sonawane · Accepted Answer · 2018-04-04 14:27:56Z

3

filtering should be responsibility of processor and not the reader. You can use composite item processor and add First processor as Filtering.

Filtering processor should return null for log lines which does not contain ' [{}] ' .

These rows will be automatically ignore in next processor and in writer.

answered Apr 4, 2018 at 14:27

Niraj Sonawane

11.3k13 gold badges82 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Niladri · Accepted Answer · 2022-12-03 20:05:42Z

0

can implement a customfilereader extending FlatFileItemReader with partition num or filter criteria passed in constructor from config, and override the read() method -> https://github.com/spring-projects/spring-batch/blob/main/spring-batch-infrastructure/src/main/java/org/springframework/batch/item/support/AbstractItemCountingItemStreamItemReader.java#L90

Every slave step would instantiate based on a different constructor param.

answered Dec 3, 2022 at 20:05

Niladri

1

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

Spring batch filtering data inside item reader

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related