How to import CSV data using Logstash for field type completion in Elasticsearch suggester

Question

ElasticSearch Index Creation

curl -XPOST 'http://localhost:9200/music/' -d '{}'

Field Mapping

curl -XPUT 'http://localhost:9200/music/_mapping/song' -d '
{
  "properties": {
    "name" : {
      "type" : "string"
    },
    "suggest": {
      "type" : "completion"
    }
  }
}'

LogStash config file, musicStash.config

input {
    file {
        path => "pathToCsv"
        start_position => beginning
    }
}

filter {  
    csv {
        columns => ["id", "name", "suggest"]
        separator => ","
    }
}

output {
    elasticsearch {
        hosts => "localhost"
        index => "music"
        document_id => "%{id}"
    }
}

Now while executing logstash config file, received following exception in elasticsearch console

failed to put mappings on indices [[music]], type [logs]
java.lang.IllegalArgumentException: Mapper for [suggest] conflicts with existing mapping in other types:
[mapper [suggest] cannot be changed from type [completion] to [string]]
at org.elasticsearch.index.mapper.FieldTypeLookup.checkCompatibility(FieldTypeLookup.java:117)

And error received in logstash console,

response=>{"index"=>{"_index"=>"music", "_type"=>"logs", "_id"=>"5", "status"=>400, 
"error"=>{"type"=>"illegal_argument_exception", 
"reason"=>"Mapper for [suggest] conflicts with existing mapping in other types:\n[mapper [suggest] cannot be changed from type [completion] to [string]]"}}}, :level=>:warn}

So how to achieve elasticsearch auto-complete feature by importing csv file through Logstash.

Sheel, How would you achieve importing multiple suggester's for a single document as appose to a single suggester per document when using logstash? — Birdy
– Birdy, Commented Jul 26, 2017 at 13:37

Val · Accepted Answer · 2016-02-16 12:55:51Z

3

You're missing the following setting in your elasticsearch output:

document_type => "song"

What happens is that logstash is creating a new type called logs (by default) and since as of ES 2.0 it is forbidden to have two fields with the same name but different types (string vs completion) in the same index, it's erroring out.

Just modify your output like this and it will work:

output {
    elasticsearch {
        hosts => "localhost"
        index => "music"
        document_type => "song"
        document_id => "%{id}"
    }
}

answered Feb 16, 2016 at 12:55

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MosheZada · Accepted Answer · 2016-10-07 14:13:56Z

1

I'm the author of elasticsearch_loader
If you just want to load CSV data into elasticsearch you can make use of elasticsearch_loader
After installation you will be able to load csv/json/parquet files into elasticsearch by issuing:

elasticsearch_loader \
  --index-settings-file mappings.json \
  --index completion \
  --type song \
  --id-field id \
  csv \
  input1.csv input2.csv

answered Oct 7, 2016 at 14:13

MosheZada

2,4291 gold badge17 silver badges17 bronze badges

1 Comment

Birdy Over a year ago

How would your solution work if you need to add multiple suggesters for a single document, For example an address document could have Line1, Line2, Line3 and Postcode all as suggester for the single address field.

Collectives™ on Stack Overflow

How to import CSV data using Logstash for field type completion in Elasticsearch suggester

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related