Unable to connect to AWS Elasticsearch instance from EC2 instance

Question

I have an EC2 instance and an Elasticsearch instance in the same vpc. I am trying to connect via requests_aws4auth package and python elasticsearch connector.

host = cfg.AWS_HOST # For example, my-test-domain.us-east-1.es.amazonaws.com
region = cfg.AWS_REGION # e.g. us-west-1

service = 'es'

awsauth = AWS4Auth(cfg.AWS_ACCESS_KEY, cfg.AWS_SECRET_KEY, region, service)

es = Elasticsearch(
hosts = [{'host': host, 'port': 443}],
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)

I am getting this ERROR when trying to connect

Traceback (most recent call last):
  File "writeData.py", line 27, in <module>
    print(es.info())
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/lib/python2.7/site-packages/elasticsearch/client/__init__.py", line 241, in info
    return self.transport.perform_request('GET', '/', params=params)
  File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 318, in perform_request
    status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_requests.py", line 90, in perform_request
    self._raise_error(response.status_code, raw_data)
  File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.AuthorizationException: AuthorizationException(403, u'{"message":"The security token included in the request is invalid."}')

Have you verified that your EC2 Instance Profile has permissions to call es? And that your AES permission policy allows your instance profile to call it? — Cheruvian
– Cheruvian, Commented Aug 29, 2018 at 19:07

SPrabhakaran · Accepted Answer · 2018-10-05 08:20:21Z

2

You may need to pass the session_token information too as a parameter.

Like this..

auth = AWS4Auth(credentials.access_key, credentials.secret_key, 
              region, service, session_token=credentials.token)

as shown here

answered Oct 5, 2018 at 8:20

SPrabhakaran

313 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ekmcd · Accepted Answer · 2019-04-17 21:55:39Z

0

If you are connecting within the same VPC, you do not want to use the external host address. Instead, use the internal ip of the elastic machine or an internal load balancer depending on your ES setup.

answered Apr 17, 2019 at 21:55

ekmcd

1828 bronze badges

Collectives™ on Stack Overflow

Unable to connect to AWS Elasticsearch instance from EC2 instance

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related