Storing Document with generic JsonNode in ElasticSearch using Spring Data

Question

I want to store unspecified json payload data in an Elasticsearch index using Spring Data with the following entity

    @Document(indexName = "message")
    public class Message {
    
        @Id
        private String id;
        private JsonNode payload;

        //getters and setters here
    }

The payload varies and needs to be stored in a generic way that can also be easily loaded again that's why I'd like to use the JsonNode here.

A document with "id" gets written but the field "payload" is empty.

When I look up the document written to the index in Kibana it looks like this:

_class:
    com.tryout.Message
payload:
id:
    30243006-0844-4438-a7f0-db93518b340f
_id:
    30243006-0844-4438-a7f0-db93518b340f
_type:
    _doc
_index:
    message
_score:
    0

In the index mapping "payload" also wasn't created and it looks like this:

{
  "mappings": {
    "_doc": {
      "properties": {
        "_class": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "id": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }
}

Any ideas how I can get my generic payload stored?

(I'm using Elastic v7.9.2 & Spring Booot + spring-boot-data-jpa v2.3.5 and spring-data-elasticsearch v4.1.1)

yes getters/setters are there. I just reduced it to the essentials here — Vincent
– Vincent, Commented Dec 7, 2020 at 8:30
what is the mapping for the payload field? Do you want to search on the fields within the generic string or just store? — ibexit
– ibexit, Commented Dec 7, 2020 at 14:11
I would like to search on the fields. I alsoalready tried to add @Field(type = Object) annotation which forces an Object mapping but still the json won't be stored — Vincent
– Vincent, Commented Dec 7, 2020 at 15:02
I just had a look at JsonNode (from Jackson?). This class has no properties, so there's nothing in a JsonNode that Spring Data Elasticsearch can write. You'll need a custom Converter that is able to convert from a JsonNode to a Map<String, Object> and back . — Abacus
– Abacus, Commented Dec 7, 2020 at 21:57

Alieh Mohtashami · Accepted Answer · 2021-05-23 13:30:54Z

I had this problem after upgrading spring-data-elastic-search version to 4, and unfortunately I didn't find a clear answer for it. Reading spring data elasticsearch documentation,I found out this:

As of version 4.0 only the Meta Object Mapping is used, the Jackson based mapper is not available anymore and the MappingElasticsearchConverter is used.

After searching several days, finally I solve my problem this way, I hope it could help others.

p.s : JsonNode can include jsonArray or JsonObject, so these two data-types should be handled while reading/writing.

class JsonElasticSearchConverter extends MappingElasticsearchConverter {

    private CustomConversions conversions = new ElasticsearchCustomConversions(Collections.emptyList());

    CustomElasticSearchConverter(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext) {
        super(mappingContext);
        setConversions(conversions);
    }

    CustomElasticSearchConverter(MappingContext<? extends ElasticsearchPersistentEntity<?>, ElasticsearchPersistentProperty> mappingContext, GenericConversionService conversionService) {
        super(mappingContext, conversionService);
        setConversions(conversions);
    }

    @Override
    protected <R> R readValue(@Nullable Object source, ElasticsearchPersistentProperty property,
                              TypeInformation<R> targetType) {

        if (source == null) {
            return null;
        }

        if (targetType.getType() == JsonNode.class) {
            return (R) mapStoredValueToJsonNode(source);
        }

        return super.readValue(source, property, targetType);
    }

    @Override
    protected Object getWriteComplexValue(ElasticsearchPersistentProperty property, TypeInformation<?> typeHint, Object value) {
        if (typeHint.getType() == JsonNode.class) {
            return getWriteJsonNodeValue(value);
        }
        return super.getWriteComplexValue(property, typeHint, value);
    }

    private JsonNode mapStoredValueToJsonNode(Object source) {
        JsonNode jsonNode = null;
        ObjectMapper mapper = new ObjectMapper();
        if (source instanceof ArrayList) {
            ArrayNode array = mapper.valueToTree(source);
            jsonNode = mapper.valueToTree(array);
        }
        if (source instanceof HashMap) {
            jsonNode = mapper.convertValue(source, JsonNode.class);
        }
        return jsonNode;
    }

    private Object getWriteJsonNodeValue(Object value) {
        JsonNode jsonNode = null;
        ObjectMapper mapper = new ObjectMapper();
        if (value instanceof ObjectNode)
            try {
                jsonNode = mapper.readTree(value.toString());
            } catch (IOException shouldNotHappened) {
                //log.warn("This error should not happened" + shouldNotHappened.getMessage());
            }
        else if (value instanceof ArrayNode) {
            ArrayNode array = mapper.valueToTree(value);
            jsonNode = mapper.valueToTree(array);
        }
        return jsonNode;
    }
}

Collectives™ on Stack Overflow

Storing Document with generic JsonNode in ElasticSearch using Spring Data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related