Logstash output to Elasticsearch on AWS EC2

Question

I'm having trouble configuring logstash to output to an Elasticsearch cluster on AWS EC2.

I'm using Logstash version 1.1.5 and Elasticsearch 1.19.8.

This is my output configuration in logstash:

output {
  stdout { debug => true debug_format => "json"}
  elasticsearch {
    cluster => "logstash-searcher"
    node_name => "logstash-indexer"
  }
}

and this is the corresponding configuration in elasticsearch.yml

cluster.name: logstash-searcher
path.data: /usr/local/elasticsearch/data
path.work: /usr/local/elasticsearch/tmp
path.logs: /usr/local/elasticsearch/logs
path.plugins: /usr/local/elasticsearch/plugins
bootstrap.mlockall: true
cloud.aws.region: eu-west-1
cloud.aws.access_key: --
cloud.aws.secret_key: --
discovery.type: ec2
discovery.ec2.host_type: public_ip
discovery.ec2.groups: elasticsearch
gateway.type: s3
gateway.s3.bucket: es-logstash

transport.tcp.port: 9300-9400

I start logstash using:

java -jar logstash-1.1.5-monolithic.jar agent -f shipper.conf

And after a while of startup I get these failures:

Failed to index an event, will retry {:exception=>org.elasticsearch.discovery.MasterNotDiscoveredException: waited for [1m],

My suspicion is that logstash needs to use something like the cloud-aws for its elasticsearch client to be able to find the cluster. Does anyone have an example configuration that works on aws?

JonasL · Accepted Answer · 2013-01-04 13:55:43Z

10

The problem is that the embedded elasticsearch instance of logstash was using its default discovery mode. Since the elasticsearch cluster is configured with cloud-aws the embedded elasticsearch of logstash needs to as well.

To do that you have to add an elasticsearch.yml configuration file to the working directory of logstash. You also need to supply the cloud-aws plugin by adding it to the class path.

java -cp logstash-1.1.7-monolithic.jar:cloud-aws/* logstash.runner agent -f shipper.conf

Using this configuration I managed to get logstash to output to my elasticsearch cluster.

answered Jan 4, 2013 at 13:55

JonasL

1811 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

zimbatm Over a year ago

For those wondering where to download the cloud-aws plugin, the url is like: download.elasticsearch.org/elasticsearch/…

Collectives™ on Stack Overflow

Logstash output to Elasticsearch on AWS EC2

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related