AWS Lambda interact with elastic search

Question

I'm new in AWS world and try to dev a proof of concept that allow lambda function to interact with elasticSearch (AWS service) and S3 bucket.

I'm not sure to understand really how it works. We have to set an IAM role linked to the Lambda function but no user so first question :

Who is the user running the Lambda function ?

I set my Elastic Search connexion like this (using https://github.com/DavidMuller/aws-requests-auth) :

def connectES(esEndPoint):
print ('Connecting to the ES Endpoint {0}'.format(esEndPoint))
try:
    # let's talk to our AWS Elasticsearch cluster
    auth = BotoAWSRequestsAuth(
                aws_host='vpc-poc-lambda-3opu7gwdw7bwvtmmazjmdgo7gi.eu-west-3.es.amazonaws.com',
                aws_region='eu-west-3',
                aws_service='es')
    esClient = Elasticsearch(
        hosts=[{'host': esEndPoint, 'port': 443}],
        use_ssl=True,
        verify_certs=True,
        connection_class=RequestsHttpConnection,
        http_auth=auth)
    return esClient
except Exception as E:
    print("Unable to connect to {0}".format(esEndPoint))
    print(E)
    exit(3)

It seams to work but I don't understand what credential it use. Then I create an index successfully with :

esClient.indices.create('test')

But when I try try to index a doc like this I get error :

esClient.index(index='test', doc_type='post', id=123456, body={
            'author': 'John Doe',
            'blog': 'Learning Elasticsearch',
            'title': 'Using Python with Elasticsearch',
            'tags': ['python', 'elasticsearch', 'tips'],
        })


[WARNING]   2018-02-15T10:39:28.460Z    7f1cc69d-123c-11e8-be8b-cbbfad79068b    PUT https://vpc-****-****.eu-west-3.es.amazonaws.com:443/test/post/123456 [status:406 request:0.039s]
Document not indexed
Error:  string indices must be integers

I really don't understant why it won't work and hope there is an easy way to interact with other AWS service from Lambda.

Can you help me ? Thanks

Amazon ElasticSearch supports IAM credentials. I would assume that you have configured ES to allow access from the IAM role under which the Lambda function runs. On the 'string index' error, the ES docs suggest that the parameter to the index function is a dictionary like client.index({ index: 'myindex', type: 'mytype', id: '1', body: { ... }, tags: [...] }) — jarmod
– jarmod, Commented Feb 15, 2018 at 12:06

Desnoxav · Accepted Answer · 2018-02-15 14:24:31Z

3

With the help of one of my coworker I finally found the issue. The error append when you try to use elastisearch python library in an old version with an elasticsearch server in a newer version. I upgraded the lib (https://elasticsearch-py.readthedocs.io/en/master/Changelog.html) to the last release and now it works great.

I'm sorry if I disturbed you and hope this will help others like me.

answered Feb 15, 2018 at 14:24

Desnoxav

3542 gold badges7 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

AWS Lambda interact with elastic search

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related