Malformed Date Error Indexing Data Into Elasticsearch

Question

I'm attempting to index data in elasticsearch that includes dates with time zones.

My date mapping is:

"date": {
    "type": "date",
    "format": "yyyy-MM-dd'T'HH:mm:ss.SSSZ"
}

I use the following code to index the data:

client.prepareIndex(INDEX, TYPE, id))
        .setSource(gson.toJson(object), XContentType.JSON)
        .setRefreshPolicy(RefreshPolicy.IMMEDIATE)
        .get();

I've created my own ZonedDateTime adapter as follows:

public class ZonedDateTimeAdapter implements JsonSerializer<ZonedDateTime> {

    public JsonElement serialize(ZonedDateTime date, Type typeOfSrc, JsonSerializationContext context) {
        DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
        return new JsonPrimitive(date.format(formatter));
    }

}

The resulting date is like 2005-01-01T13:35:50.596-0500. Which to me seems to match my format of yyyy-MM-dd'T'HH:mm:ss.SSSZ. However I receive an error stating:

Exception in thread "main" MapperParsingException[failed to parse [startDate]]; nested: IllegalArgumentException[Invalid format: "2005-01-01T13:35:50.596-0500" is malformed at ".596-0500"]

What's interesting is that if I change format in ZonedDateTimeAdapter to read yyyy-MM-dd'T'HH:mm:ss.000Z (forcing the second fraction to always be 000) I get a resulting date like 2005-01-01T13:31:06.000-0500. An object with this date is successfully indexed into elasticsearch.

Do you know why 2005-01-01T13:31:06.000-0500 index successfully but 2005-01-01T13:35:50.596-0500 does not? Aren't these the same format?

kgf3JfUtW · Accepted Answer · 2017-12-14 20:58:02Z

2

Update: I did a quick test on my ES 5.2 and it worked fine.

$curl -XPUT localhost:9200/myindex1 -d '
{"mappings": {"type1": {"properties": {"field1": {
    "type": "date", 
    "format": "yyyy-MM-dd'\''T'\''HH:mm:ss.SSSZ"
}}}}}
'

$curl -XPUT localhost:9200/myindex1/type1/id1 -d '
    { "field1":"2005-01-01T13:35:50.000-0500" }
'

$curl -XPUT localhost:9200/myindex1/type1/id2 -d '
    { "field1":"2005-01-01T13:35:50.596-0500" }
'

To confirm we have the same mapping:

$curl localhost:9200/myindex1/type1/_mapping
{"myindex1":{"mappings":{"type1":{"properties":
    {"field1":{"type":"date","format":"yyyy-MM-dd'T'HH:mm:ss.SSSZ"}}}}}}

From this thread on https://discuss.elastic.co

Currently, Elasticsearch is limited to millisecond precision, see this GH issue.

The post is from 2015 but the issue remains open. It looks like nanosecond precision is not yet supported.

edited Dec 14, 2017 at 20:58

answered Dec 14, 2017 at 19:51

kgf3JfUtW

15.2k12 gold badges63 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Gabriel Over a year ago

Thanks! but isn't .001 a millisecond? Whereas .000000001 is a nanosecond?

kgf3JfUtW Over a year ago

You're absolutely correct. Apologies for my oversight - I naturally assumed you are doing micro/nanoseconds since I ran into this problem before. That said, I tried creating a mock index (using default mapping), inserting two documents, each having just one field. One has value 2005-01-01T13:31:06.000-0500 and one 2005-01-01T13:31:06.596-0500. Both were inserted successfully. Let us see whether your mapping is causing the issue.

kgf3JfUtW Over a year ago

@Gabriel I tried on my local ES 5.2: Created same mapping, created index, inserted two documents with those values, everything worked fine. I updated my answer to reflect the logs. Is it possible for you to try a simple curl command? So that you could isolate the issue to either ES itself or the Java API.

Gabriel · Accepted Answer · 2017-12-19 17:36:56Z

0

I'm not sure if my "fix" will be appropriate for anyone seeing this problem. But it was getting worse. I couldn't parse out any dates that ended in 0. So yyyy-MM-dd'T'HH:mm:00 and yyyy-MM-dd'T'HH:00 would fail, but yyyy-MM-dd'T'HH:mm:01 or yyyy-MM-dd'T'HH:01 would be successful. I destroyed my entire elasticsearch database and upgraded to 6.1 and reindexed all of my data. I don't see the issue anymore.

edited Dec 19, 2017 at 17:36

answered Dec 19, 2017 at 16:25

Gabriel

6241 gold badge8 silver badges20 bronze badges

Collectives™ on Stack Overflow

Malformed Date Error Indexing Data Into Elasticsearch

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related