0

I'm in really weird situations, I need to create indexes in elasticsearch that contain typeless fields. I have a rails application that sends any data per second to my elasticsearch. about my architecture, I have to say I use elastic-stack on docker in ubuntu server and use socket to send data's to elk and all of them are the latest version.

In my rails application user could choose datatype for each field but the issues happen when the user want to change the datatype of one field right after it's created, logstash return this error

error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [field] of type [long] in document with id '5e760cac-cafc-4fd0-9e45-1c650967ccd4'. Preview of field's value: '2022-01-18T08:06:30'", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"For input string: \"2022-01-18T08:06:30\

I found deadly queue letter plugins to save wrong input in my server after that I think if I could index documents without any type the problem is solved so I start to googling and found Removal of mapping types in elasticsearch documents and I follow instructions which describe in tutorials I get the following error:

unknown setting [index.include_type_name] please check that any required plugins are installed, or check the breaking changes documentation for removed settings

even I put "include_type_name" in the request to send to the elastic noting change I have the latest version of elastic.

I think maybe it's helpful to edit the default elasticsearch template but noting the change. could you please help me with what should I do?

Improve this question
7
  • How are you creating the index and where are you putting the index.include_type_name? Are you running 7.16.3? Also, changing the data type of a field after it is created is not possible, the include_type_name is related to the mapping type that was used in previous versions, not to the data type of the fields. Can you give an example of what you are doing? Commented Jan 26, 2022 at 13:21
  • @leandrojmp, Yeah I use latest version 7.16.3, As I said I use AWS to send data to my Elasticsearch by socket protocol and As tutorials describe I put "index.include_type_name" in setting in default mapping. In fact I want to save data in elasticsearch something like this:{"_id":"61f3f40ac64c09b21e99a88d","index":0,"age":33,"eyeColor":"brown","name":"Foley Palmer","phone":"+1 (953) 489-2171","favoriteFruit":"apple"} but elastic save json with datatype, because in my scenario user send specifice fields strings after that want to change to integer I have to save both but I could not Commented Jan 28, 2022 at 14:02
  • 1
    You are mixing things, the include_type_name is a deprecated setting for when you want to use a type for your mapping, this was possible in older versions and it is not possible anymore. The data type of the field, if it is a string or an integer, has no relation with that setting, also you can't change a data type, if a field is created as a string you can't change it to integer, you would need to reindex the entire index, you cannot have a field that is both string and integer for example. Commented Jan 28, 2022 at 14:39
  • So how could I save any changes in the most efficient way? I could not reindex to change the fields name, And in some scenarios, user changes his/her datatype so late and I have a lot of index with wrong datatype fields, Do I have any chance to write a specific exception handler for this result? Commented Jan 29, 2022 at 5:25
  • 1
    Well, this will depend entirely on your code, you will need to get the elasticsearch error and deal with that, but since you can't change the data type of a field you shouldn't allow the users to do that. Commented Jan 29, 2022 at 12:31

1 Answer 1

Reset to default
1

As already mentioned in the comments, Elasticsearch does not support changing the data type of a field without a reindex or creating a new index.

For example, if a field is mapped as a numeric field like integer and the user wants to index a string value in this field, elasticsearch will return an mapping error.

You would need to change the mapping of the index and reindex it or create a entirely new index using the new mapping.

None of this is done automatically by elastic, you would need to deal with this in your application, you could catch the error and implement some logic to create a new index with the new mapping, but this also could lead to other problems as having too many indices in the cluster and query errors when the range of the query include index with the same field with different mappings.

One feature that Elasticsearch has that could help you in some way is the runtime fields, with runtime fields you can query a field that has a specific mapping using a different mapping.

For example, if you have a field that has date values, but was wrongly mapped as a keyword or text field, you could use a runtime field to query it as it was a date field.

But again, this will need that you implement a logic to build those runtime fields and also can lead to other problems, not all the data types are available to runtime fields and runtime fields can impact in the performance.

Another feature that could help you is to use of multi-fields, this, I think, is the closest you got of having a field with multiple data types.

Using multi-fields you could have a field named date with the date type and also a field named date.keyword with the keyword type, you also could have a field name code with the keyword type and a field name code.int with the integer type, you would also need to use the ignore_malformed setting in the mapping so elastic does not reject the entire document in case of mapping errors, just the field with the wrong mapping.

Just keep in mind that when use multi-fields, you will have a different field for each mapping, for example date is a field, date.keyword is another field, this will increase the storage usage.

But again, none of this is done automatically, it needs logic in your application, elasticsearch does not allows you to change the mapping of a field, if your application needs this, you will need to implement something in the application that can work with that limitations of elasticsearch.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.