Logstash / GROK: Creating a custom Variable when parsing a log file

Question

I am new to Grok, although I have managed to create custom regular expressions and write GROK filters in the logstash config file. My problem is as follows:

SOURCE FIELD - I am parsing a log file, where, every event includes a 'source' field, which is the name of the log file, e.g.:

test.YYYYMMDD_HHMMSS.log

What I want to do is: For each event, where 'source' contains this filename, extract the date and time in the following format within a new field within the Grok Filter:

DD/MM/YYYY HH:MM:SS

I know how to write custom Regular Expressions (REs) in GROK, but I cannot write an RE which will match the data and format it before storing it into a variable. So that is my problem.

Can anyone please help?

Thanks a lot!

I'm not quite sure why you need match the data and format it before storing it. How about matching it and then adding a new variable or replacing an old one with the mutate filter plugin — Phonolog
– Phonolog, Commented Sep 21, 2017 at 6:44
The trouble is that the application logs I am parsing, contain the Start TimeStamp as the first line in the file, e.g. "start: YYYY/MM/DD HH:MM:SS". And I don't know a how to use the mutate syntax to create a global variable. — sgrover
– sgrover, Commented Sep 21, 2017 at 10:51
What I asked above was the second option, which I thought might be easy, that is, to extract/manipulate the Start Timestamp from the Log file's name. — sgrover
– sgrover, Commented Sep 21, 2017 at 10:52

Phonolog · Accepted Answer · 2017-09-21 13:24:10Z

1

Extracting the date from the filename should work. You should be able to match the date parts in the source field with a grok filter and add a new field like so:

filter { 
    grok {
        match => [
        "source", "test.%{YEAR:year}%{MONTHNUM2:month}%{DATA:day}_%{HOUR:hour}%{MINUTE:minute}%{SECOND:second}.log"
        ]
    }
    mutate { add_field => { "your_new_date_field" => "%{day}/%{month}/%{year} %{hour}:%{minute}:%{second}" } }
}

I don't have the possibility to test this right now but I hope you get the idea.

This solution will create a lot of additional fields like year, month, day and so on. If you want to get rid of the additional fields you can use metadata fields.

answered Sep 21, 2017 at 13:24

Phonolog

6,5513 gold badges39 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Logstash / GROK: Creating a custom Variable when parsing a log file

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related