Not able to parse string to date in logstash/elasticSearch

Question

I had created a logstash script to read a logfile which is having various timestamp of format "2018-05-08T12:18:53.506+0530". I am trying to parse it to date using the date filter in log stash

date{
     match => ["edrTimestamp","yyyy-MM-dd'T'HH:mm:ss.SSS'Z'","ISO8601"]
     target => "edrTimestamp"
}

The running the above logstash script it creates a elastic search index. But still the string is not parsed to date. It is also showing date parsed exception in the index.

It creates output like this.

{
          "tags": [
            "_dateparsefailure"
          ],
          "statusCode": "805",
          "campaignRedemptionLimitTotal": 1000,
          "edrTimestamp": "2018-05-22T16:41:25.162+0530 ",
          "msisdn": "+919066231327",
          "timestamp": "2018-05-22T16:41:25.122+0530",
          "redempKeyword": "print1",
          "campaignId": "C910101-1527004962-1582",
          "category": "RedeemRequestReceived"
        }

Please tell me whats wrong in the above code> I had tried many others alternative but still it is not working.

Alcanzar · Accepted Answer · 2018-05-23 14:29:47Z

2

Your issue is that your timestamp has a space at the end of it "edrTimestamp": "2018-05-22T16:41:25.162+0530 ", which is causing the date parsing to fail. You need to add a:

mutate {
  strip => "edrTimestamp"
}

before your date filter.

answered May 23, 2018 at 14:29

Alcanzar

17.2k7 gold badges45 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Sufiyan Ghori · Accepted Answer · 2018-05-23 23:27:59Z

1

I don't think you should be escaping the Z. So you probably want something like:

yyyy-MM-dd'T'HH:mm:ss,SSS

Also you should not be using "Z" since your time is not Zulu (0 offset). You will want to contain the offset as part of the pattern. The Heroku grok debug app is useful for this.

If I pass your string

2018-05-08T12:18:53.506+0530

and use the filter %{TIMESTAMP_ISO8601} then it matches, this pattern is made up of the following sub-patterns:

 TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}?

edited May 23, 2018 at 23:27

Sufiyan Ghori

18.9k17 gold badges84 silver badges111 bronze badges

answered May 23, 2018 at 14:25

TheFiddlerWins

9327 silver badges24 bronze badges

Collectives™ on Stack Overflow

Not able to parse string to date in logstash/elasticSearch

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related