logstash log parsing with regex and grok

Question

Hello I have below logs

12-Apr-2021 16:11:41.078 WARNING [https-jsse-nio2-8443-exec-3] org.apache.catalina.realm.LockOutRealm.filterLockedAccounts An attempt was made to authenticate the locked user [uv19nb]
12-Apr-2021 16:01:01.505 FINE [https-jsse-nio2-8443-exec-8] org.apache.catalina.realm.CombinedRealm.authenticate Failed to authenticate user [uv19nb] with realm [org.apache.catalina.realm.JNDIRealm]
12-Apr-2021 17:12:45.289 FINE [https-jsse-nio2-8443-exec-5] org.apache.catalina.authenticator.FormAuthenticator.doAuthenticate Authentication of 'uv19nb' was successful

I am trying to build a pattern for these for logstash.

I have following

%{MY_DATE_PATTERN:timestamp}\s%{WORD:severity}\s\[%{DATA:thread}\]\s%{NOTSPACE:type_log}

which parses below

{
  "timestamp": [
    "12-Apr-2021 16:01:01.505"
  ],
  "severity": [
    "FINE"
  ],
  "thread": [
    "https-jsse-nio2-8443-exec-8"
  ],
  "type_log": [
    "org.apache.catalina.realm.CombinedRealm.authenticate"
  ]
}

and i would like to parse log as 2 parts as the bold ones and the user name what would you advise please?

An attempt was made to authenticate the locked user [uv19nb]

Failed to authenticate user [uv19nb] with realm [org.apache.catalina.realm.JNDIRealm]

Authentication of 'uv19nb' was successful

I have tried using (?<action>[^\[]*) and (?<action>[^']*) but they only capture if the next character is either [ or '.

I need some regex/grok pattern to catch all the sentence until any special character I believe and for user name I need to extract numbers and letters from [] and ''.

Wiktor Stribiżew · Accepted Answer · 2021-04-14 14:48:43Z

1

Provided the MY_DATE_PATTERN works well for you, you can use

%{MY_DATE_PATTERN:timestamp}\s+%{WORD:severity}\s+\[%{DATA:thread}\]\s+%{NOTSPACE:type_log}\s+(?<action>\w(?:[\w\s]*\w)?)

I added \s+(?<action>\w(?:[\w\s]*\w)?):

\s+ - one or more whitespaces
(?<action>\w(?:[\w\s]*\w)?) - Group "action":
- \w - a word char followed with
- (?:[\w\s]*\w)? - an optional occurrence of zero or more word and whitespace chars and then an obligatory word char.

answered Apr 14, 2021 at 14:48

Wiktor Stribiżew

631k41 gold badges502 silver badges632 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Hzl Aysen Over a year ago

perfect this seems to be working!! thank you ! would you have any idea how to parse the userid which is either inside of the '' or []

Wiktor Stribiżew Over a year ago

@HzlAysen Append (?:\s+['\[](?<user>[^\]']+))? at the end of the pattern.

Collectives™ on Stack Overflow

logstash log parsing with regex and grok

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related