How to get ElasticSearch output?

Question

I want to add my log document to ElasticSearch and, then I want to check the document in the ElasticSearch. Following is the conntent of the log file :

Jan  1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<[email protected]>
Feb  2 06:25:43 mailserver15 postfix/cleanup[21403]: BEF25A72999: message-id=<[email protected]>
Mar  3 06:25:43 mailserver16 postfix/cleanup[21403]: BEF25A72998: message-id=<[email protected]>

I am able to run my logstash instance with following logstast configuration file :

input {
  file {
    path => "/Myserver/mnt/appln/somefolder/somefolder2/testData/fileValidator-access.LOG"
    start_position => "beginning"
    sincedb_path => "/dev/null"
  }
}

filter {
  grok {
    patterns_dir => ["/Myserver/mnt/appln/somefolder/somefolder2/logstash/pattern"]
    match => { "message" => "%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}" }
  }
}

output{
    elasticsearch{
       hosts => "localhost:9200"
       document_id => "test"
       index => "testindex"
       action => "update"
    }
stdout { codec => rubydebug }
}

I have define my own grok pattern as :

POSTFIX_QUEUEID [0-9A-F]{10,11}

When I am running the logstash instance, I am successfully sending the data to elasticsearch, which gives following output :

Now, I have got the index stored in elastic search under testindex, but when I am using the curl -X GET "localhost:9200/testindex" I am getting following output :

{
  "depositorypayin" : {
    "aliases" : { },
    "mappings" : { },
    "settings" : {
      "index" : {
        "creation_date" : "1547795277865",
        "number_of_shards" : "5",
        "number_of_replicas" : "1",
        "uuid" : "5TKW2BfDS66cuoHPe8k5lg",
        "version" : {
          "created" : "6050499"
        },
        "provided_name" : "depositorypayin"
      }
    }
  }
}

This is not what is stored inside the index.I want to query the document inside the index.Please help. (PS: please forgive me for the typos)

Joe Waller · Accepted Answer · 2019-01-18 08:18:14Z

1

The API you used above only returns information about the index itself (docs here). You need to use the Query DSL to search the documents. The following Match All Query will return all the documents in the index testindex:

curl -X GET "localhost:9200/testindex/_search" -H 'Content-Type: application/json' -d'
{
    "query": {
        "match_all": {}
    }
}
'

answered Jan 18, 2019 at 8:18

Joe Waller

2,0093 gold badges21 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

KidWithAComputer Over a year ago

I got this : {"took":3,"timed_out":false,"_shards":{"total":5,"successful":5,"skipped":0,"failed":0},"hits":{"total":0,"max_score":null,"hits":[]}}

KidWithAComputer Over a year ago

[2019-01-18T13:49:04,858][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>404, :action=>["update", {:_id=>"test", :_index=>"testindex", :_type=>"doc", :routing=>nil, :_retry_on_conflict=>1}, #<LogStash::Event:0x1b1ee3ed>], :response=>{"update"=>{"_index"=>"testindex", "_type"=>"doc", "_id"=>"test", "status"=>404, "error"=>{"type"=>"document_missing_exception", "reason"=>"[doc][test]: document missing", "index_uuid"=>"SYDH5d8eSUOvsdHbibdxHg", "shard"=>"0", "index"=>"testindex"}}}}

KidWithAComputer Over a year ago

I am getting the above warning when I am running the logstash, can u help me with this waning, beacuse it says **"reason"=>"[doc][test]: document missing", **

Joe Waller Over a year ago

I'm not very familiar with logstash, but I think you need to add doc_as_upsert => true to your output configuration. Otherwise logstash will only attempt to update an existing document, and will not add new documents.

KidWithAComputer · Accepted Answer · 2019-01-18 11:01:34Z

0

Actually I have edited my config file whic look like this now :

input {
. . .
}

filter {
 . . .
}

output{
    elasticsearch{
       hosts => "localhost:9200"
       index => "testindex"
    }

}

And now I am able to get fetch the data from elasticSearch using

curl 'localhost:9200/testindex/_search'

I don't know how it works, but it is now. can anyone explain why ?

answered Jan 18, 2019 at 11:01

KidWithAComputer

3311 gold badge2 silver badges12 bronze badges

1 Comment

Joe Waller Over a year ago

It works because by removing the action => "update" and document_id => "test" options logstash will use the default action => "insert". This will cause logstash to add the document to the index instead of trying to update an existing document in the index (which in your case the document did not exist, so it failed).

Collectives™ on Stack Overflow

How to get ElasticSearch output?

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related