How to update an index/indice in Elasticsearch?

Question

I've already got my index (response_summary) created using logstash, which puts data into the index from a MySQL database.

My concern here is, how will I be able to update the index manually whenever a new set of records are being added to the database without deleting and recreating the index yet again.

Or is there a way that it can be done automatically, whenever a db change is done?

Any help could be appreciated.

You need your own way of finding out when your sql database changes (regular queries against it or similar) OR when you change something in the DB you do the same in ES. — Andrei Stefan
– Andrei Stefan, Commented Oct 25, 2016 at 20:14
@AndreiStefan I could do it through logstash by inserting a scheduler cron job, but wanted to know if there's any other way to do it using ES. Thanks — Kulasangar
– Kulasangar, Commented Oct 26, 2016 at 4:40
No way with ES. There were the rivers in ES, but they were removed in ES 2.0. The alternative is the Logstash JDBC input plugin: elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html — Andrei Stefan
– Andrei Stefan, Commented Oct 26, 2016 at 4:46
For db handling that's fine, what if i'm uploading log files as input, could I use the same scheduler within the input? — Kulasangar
– Kulasangar, Commented Oct 26, 2016 at 4:55
No, you use the file input from Logstash which picks up file changes automatically. — Andrei Stefan
– Andrei Stefan, Commented Oct 26, 2016 at 4:58

Andrei Stefan · Accepted Answer · 2016-10-26 05:19:45Z

1

No way with ES. There were the rivers in ES, but they were removed in ES 2.0. The alternative is the Logstash JDBC input plugin to automatically pickup changes based on a defined schedule.

For doing the same with files, you have the LS file input plugin which is tailing the files to pick up the new changes and, also, to keep track of where it left off in case LS is restarted.

answered Oct 26, 2016 at 5:19

Andrei Stefan

52.5k6 gold badges102 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Kulasangar Over a year ago

Thanks, spot on :)

Kulasangar Over a year ago

I tried with the jdbc schedule in logstash conf, but when i checked it with Kibana it only takes the last record which I added, not the whole! I added something like this: schedule => "* * * * *"

Andrei Stefan Over a year ago

And the query used covers all the data?

Kulasangar Over a year ago

Sorry @Andrei, it was my mistake. The document_id was wrong. It works perfectly now!

AlisonGrey Over a year ago

@AndreiStefan, i am facing the same issue, except when i try to do an update, any new row is getting updated in the index, the old row updation is also added as a new row. I want the old index row to get updated. Ive also posted a detailed question here . Could you please help me?

Collectives™ on Stack Overflow

How to update an index/indice in Elasticsearch?

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related