logstash output with document oriented elasticsearch

Question

First question about this topic is with the code below, we can pass logstash output to elasticsearch. As you know, elasticsearch is document oriented. But the code below, we can only define an index not a document id. I want to ask that is it possible to define different document id for every logstash pipeline output ? If your answer is yes how, if not why ?

Second question is, if i run the code below. I have got yellow state in elasticsearch node. I know how to change it with manuely or sending dsl query via python etc. But is it possible to solve this inside of below code ?

elasticsearch {
        hosts => ["localhost:9200"]
        index => "name_of_index" 
        http_compression => true
    }

Thanks for answering..

By "document name" do you mean "document id", i.e. the unique identifier of a given document? — Val
– Val, Commented Dec 9, 2020 at 14:44
Imagine that, I have a software that have large amount of logs and I want to pass that to elasticsearch with decomposing via date. So, my aim is having same index but so many documents to decomposed via date and if i run a "get query" to that document, i want to get only that day logs. Is it possible ? Thanks for your help. — Orkun
– Orkun, Commented Dec 9, 2020 at 14:50

Val · Accepted Answer · 2020-12-15 08:09:13Z

1

You can definitely specify the document ID using the document_id setting:

elasticsearch {
    hosts => ["localhost:9200"]
    index => "name_of_index-%{+YYYY.MM.dd}"
    manage_template => true
    template_name => "my-template"
    template => "/path/to/my-template.json"
    document_id => "%{my_id_field}" 
    http_compression => true
}

You can also make sure that every day a new index is created by specifying the date pattern in the index name (see above).

Also make sure to have the following in a file called my-template.json that is referenced in the elasticsearch output. Its role is to provide the specific settings to use when creating your index. Here, since you have a single node, we're instructing the index to not create any replica shards, to make sure that the cluster will be green.

my-template.json

{
  "index_patterns": ["name_of_index*"],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0
    }
  }
}

edited Dec 15, 2020 at 8:09

answered Dec 9, 2020 at 14:54

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Orkun Over a year ago

Thanks a lot this solved my question. But, one thing last, when i run the logstash config it sends the datas to elasticsearch its ok but how to solve yellow state ? I know manually solving this or sending dsl query via python etc. But is it possible to solve this inside of logstash elasticsearch output ?

Orkun Over a year ago

Yes, but i can make more. But how should i define how many nodes i should?

Collectives™ on Stack Overflow

logstash output with document oriented elasticsearch

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related