Making a signed HTTP request to AWS Elasticsearch in Python

Edit: Apparently this workaround has been broken by Elastic.

I struggled for a while to do a similar thing. Currently the boto3 library doesn't support making signed es requests, though since I raised an issue with them it's become a feature request.

Here's what I've done in the meantime using DavidMuller's library mentioned above and boto3 to get my STS session credentials:

import boto3
from aws_requests_auth.aws_auth import AWSRequestsAuth
from elasticsearch import Elasticsearch, RequestsHttpConnection

session = boto3.session.Session()
credentials = session.get_credentials().get_frozen_credentials()

es_host = 'search-my-es-domain.eu-west-1.es.amazonaws.com'
awsauth = AWSRequestsAuth(
    aws_access_key=credentials.access_key,
    aws_secret_access_key=credentials.secret_key,
    aws_token=credentials.token,
    aws_host=es_host,
    aws_region=session.region_name,
    aws_service='es'
)

# use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
    hosts=[{'host': es_host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

print(es.info())

Hope this saves somebody some time.

While other answers are perfectly fine, I wanted to eliminate the use of external packages. Obviously, botocore itself has all the required functionality to sign requests it was just a matter of looking at the source code. This is what I ended up with for sending AWS API requests directly (things are hardcoded for the demonstration purposes):

      import boto3
      import botocore.credentials
      from botocore.awsrequest import AWSRequest
      from botocore.endpoint import URLLib3Session
      from botocore.auth import SigV4Auth

      params = '{"name": "hello"}'
      headers = {
        'Host': 'ram.ap-southeast-2.amazonaws.com',
      }
      request = AWSRequest(method="POST", url="https://ram.ap-southeast-2.amazonaws.com/createresourceshare", data=params, headers=headers)
      SigV4Auth(boto3.Session().get_credentials(), "ram", "ap-southeast-2").add_auth(request)    


      session = URLLib3Session()
      r = session.send(request.prepare())

I recently published requests-aws-sign, which provides AWS V4 request signing for the Python requests library.

If you look at this code you will see how you can use Botocore to generate the V4 request signing.

why not just use requests?

import requests
headers = {'Content-Type': 'application/json',}
data = '{"director": "Burton, Tim", "genre": ["Comedy","Sci-Fi","R-rated"],"profit" : 98 , "year": 1996, "actor": ["Jack Nicholson","PierceBrosnan","Sarah Jessica Parker"], "title": "Mars Attacks!"}'
response = requests.post('https://search-your-awsendpoint.us-west-2.es.amazonaws.com/yourindex/_yourdoc/', headers=headers, data=data)

this worked for me