Logstash parse multiple instances of the same word into Elasticsearch Array

Question

I am trying to extract the following from a log line into 2 different array fields

03/07/2014 13:29:43.103 INFO NUM*0000001*DOS*0000001-000000001* TDKIIDKSSS NUM*0000002*DOS*0000002-000000001* TDKIIDKSSS NUM*0000003*DOS*0000003-000000001* TTDKIIDKSSS NUM*0000004*DOS*0000004-000000001* TDKIIDKSSS NUM*0000005*DOS*0000005-000000001* TDKIIDKSSS

My issue is that from my experience, grok works only on fixed patterns. I am trying to extract the above numbers from different log lines that would have NUM and DOS repeated ranging for 2 times or 100 times.

Is there a way to get multiple repetitions and put them in an array using logstash for elasticsearch to store?

emonik · Accepted Answer · 2014-03-08 10:28:36Z

1

I think your best shot with these log lines is the following approach :

filter{
  mutate{
    gsub=>["message","NUM\*(?:[0-9]+)\*DOS\*"," ","message","[*|-]"," "]
  }
  extractnumbers{}
}

The above will extract NUM & DOS numbers from the message field, it will put NUM in an odd values (int1,int3,...), & DOS in an even ones (int2,int4...). Be aware that any leading zero will be removed from the values ( 00001 => 1 ).
The output will be the following for your log :

# leaing zeros will be removed
int1=>1, #NUM
int2=>1, #DOS
int3=>2, #NUM
int4=>1, #DOS
int5=>3,
int6=>1,
int7=>4,
int8=>1,
int9=>5,
int10=>1

I hope this help :) .

answered Mar 8, 2014 at 10:28

emonik

1,5901 gold badge18 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Logstash parse multiple instances of the same word into Elasticsearch Array

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related