Stream data to amazon elasticsearch using logstash?

Question

So I spinned up a 2 instance Amazon Elasticsearch cluster.

I have installed the logstash-output-amazon_es plugin. This is my logstash configuration file :

input {
    file {
        path => "/Users/user/Desktop/user/logs/*"
    }
}

filter {
  grok {
    match => {
      "message" => '%{COMMONAPACHELOG} %{QS}%{QS}'
    }
  }

  date {
    match => [ "timestamp", "dd/MMM/YYYY:HH:mm:ss Z" ]
    locale => en
  }

  useragent {
    source => "agent"
    target => "useragent"
  }
}

output {
    amazon_es {
        hosts => ["foo.us-east-1.es.amazonaws.com"]
        region => "us-east-1"
        index => "apache_elk_example"
        template => "./apache_template.json"
        template_name => "apache_elk_example"
        template_overwrite => true
    }
}

Now I am running this from my terminal:

/usr/local/opt/logstash/bin/logstash -f apache_logstash.conf

I get the error:

Failed to install template: undefined method `credentials' for nil:NilClass {:level=>:error}

I think I have got something completely wrong. Basically I just want to feed some dummy log inputs to my amazon elasticsearch cluster through logstash. How should I proceed?

Edit Storage type is Instance and access policy is set to accessible to all.

Edit

output {
    elasticsearch {
        hosts => ["foo.us-east-1.es.amazonaws.com"]
        ssl => true
    index => "apache_elk_example"
         template => "./apache_template.json"
          template_name => "apache_elk_example"
          template_overwrite => true

    }
}

anand · Accepted Answer · 2018-03-09 19:12:41Z

3

I also faced same problem, and I solved it by mentioning port after the hostname. This occurs because hostname hosts => ["foo.us-east-1.es.amazonaws.com"] points to foo.us-east-1.es.amazonaws.com:9200 which is not the default port in the case of aws elasticsearch. So by changing hostname to foo.us-east-1.es.amazonaws.com:80 solves the problem.

edited Mar 9, 2018 at 19:12

answered Feb 15, 2017 at 19:12

anand

72711 silver badges20 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Val · Accepted Answer · 2016-06-13 09:15:59Z

2

You need to provide the following two parameters:

aws_access_key_id and
aws_secret_access_key

Even though they are described as optional parameters, there is one comment in the code that makes it clear.

aws_access_key_id and aws_secret_access_key are currently needed for this >plugin to work right. Subsequent versions will have the credential resolution logic as follows:

answered Jun 13, 2016 at 9:15

Val

218k14 gold badges377 silver badges384 bronze badges

3 Comments

Val Over a year ago

That's probably not a logstash issue but more likely a wrong security setting on the AWS IAM side.

Karup Over a year ago

Yeah I am trying to figure that out. Meanwhile I thought of using just the elasticsearch output plugin ...with that I am getting: connect timed out {:class=>"Manticore::ConnectTimeout", :level=>:error} I have added code for that as an edit in case you could suggest something.

dejavu013 Over a year ago

I am going to update the comment. This has been fixed and we have lot of customers resolving credentials using instance profiles

cpanato · Accepted Answer · 2016-06-16 06:49:31Z

2

I was able to run logstash together with AWS Elasticsearch without the AccessKeys, I configured the policie in the ES service.

It worked without the Keys if you start the logstash manually, if you start logstash as a service the plugin doenst work.

https://github.com/awslabs/logstash-output-amazon_es/issues/34

answered Jun 16, 2016 at 6:49

cpanato

4295 silver badges8 bronze badges

2 Comments

tread Over a year ago

Not a great idea as that means that your elasticsearch instance is publicly available

cpanato Over a year ago

agree that is not the best solution :)

Collectives™ on Stack Overflow

Stream data to amazon elasticsearch using logstash?

3 Answers 3

Comments

3 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

3 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related