I am working on the project that involves searching different pattern in Remedy log files. Is it advantageous to have logstash break the message field into multiple fields JSON document or manually create a JSON document of only one field and fed it elasticsearch.As far as I know elasticsearch does preety well in the manual generated JSON document with only one field.So do I have to really use logstash in these application
1 Answer
By placing everything into a single field you will limit yourself to only search functionality of elasticsearch. While elasticsearch is quite powerful search engine, it doesn't make much sense to me to restrict yourself to just search. By splitting your data into fields you will be able to unlock the full analytics power of elasticsearch and use, for example, Kibana to visualize your data. Just to give you an example, by simply extracting a timestamp field, you will be able to not only search for certain events but also draw a histogram of how this events were distributed in time. By extracting severity of the event and source, you will be able to see which sources are causing most of the issues and so on. I think it is well worth the effort in a longer term.
