API Key Authentication in Elasticsearch with python

Question

I was hoping to find an answer to my problem with the elasticsearch python framework. Maybe I'm completely blind or doing something absolutely wrong, but I'm very confused right now and can't find an adequate answer.

I'm currently trying to establish a connection to my elastic search API using the elasticsearch python framework, my code looks like this:

from elasticsearch import Elasticsearch

def create_es_connection(host: str, port: int, api_key_id: str, api_key: str, user: str, pw: str) -> Elasticsearch:
    return Elasticsearch([f"https://{user}:{pw}@{host}:{port}"])

This is working fine. I then created an API key for my testuser, which I'm giving to the function above as well. I am now trying to do something similar like this: Elasticsearch([f"https://{api_key_id}:{api_key}@{host}:{port}"]) so, I want to leave out the user and password completely, because in regards to the greater project behind this snippet, I'm not feeling very well in saving the user/password credentials in my project (and maybe even pushing it to our git server). Sooner or later, these credentials have to be entered somewhere and I was thinking that only saving the API key and to authenticate with that could be more safe. Unfortunately, this isn't working out like I planned and I couldn't find anything about how to authenticate with the API key only.

What am I doing wrong? In fact, I found so little about this, that I'm questioning my fundamental understanding here. Thank you for your replies!

yamamoto92 · Accepted Answer · 2019-12-30 12:12:08Z

9

working configuration for me was :

es = Elasticsearch(['localhost:9200'], api_key=('DuSkVm8BZ5TMcIF99zOC','2rs8yN26QSC_uPr31R1KJg'))

answered Dec 30, 2019 at 12:12

yamamoto92

911 bronze badge

Sign up to request clarification or add additional context in comments.

3 Comments

CptSnuggles Over a year ago

This worked for me. api_key is apparently using api_key=('api_key_id', 'api_key'). This really should be documented somewhere.

DU_ds Over a year ago

@CptSnuggles I found this [link] (elasticsearch-py.readthedocs.io/en/master/…) in the documentation. Not sure when it was added, but it seems to be there now!

Anup Ash Over a year ago

this does not work rather i got whole range of http.client.RemoteDisconnected junk

Andreas · Accepted Answer · 2019-12-19 11:46:11Z

5

Elasticsearch's documentation shows how to generate and use (at the bottom of the page) an API key: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html

curl -H "Authorization: ApiKey ..." http://localhost:9200/_cluster/health

-H means "header", so to do the same in Python you will need to set this header. Rummaging through the elasticsearch module source code tells me that you might just be able to do the following:

Elasticsearch([f'http://{host}:{port}'], api_key=api_key)

The reason for this is that **kwargs of the Elasticsearch.__init__ method are passed to Transport.__init__, the **kwargs of that are passed to Connection.__init__, which takes an api_key arg which is then used in _get_api_key_header_val to construct the appropriate header.

The following code shows that the api key header gets added to the HTTP headers of the request:

import elasticsearch, logging, http.client
http.client.HTTPConnection.debuglevel = 5
logging.basicConfig(level=logging.DEBUG)
c = elasticsearch.Elasticsearch(['localhost'], api_key='TestApiKey')
print(c.cluster.health(wait_for_status='green'))

This is definitely something that should be added to Elasticsearch's docs.

edited Dec 19, 2019 at 11:46

answered Dec 18, 2019 at 13:51

Andreas

8,0692 gold badges32 silver badges38 bronze badges

4 Comments

CptSnuggles Over a year ago

Hey Andreas, thank you for your response! Unfortunately, just adding this like you suggested didn't work. Error: https://pastebin.com/raw/0x96LSJV Further information, I'm currently using all components in the newest version available (Elastic in 7.5.0 and the python framework updated as well).

Andreas Over a year ago

I updated my answer to show that passing the api_key parameter does set the correct HTTP header. There may be another reason your ES rejects the request - xpack security may be disabled, or you may have to enable HTTPS.

CptSnuggles Over a year ago

You were absolutely right, the key is passed down to the header correctly. I could reconstruct it now as well inside the http_urllib3. X-Pack and https are actually enabled, but maybe I've misconfigured something down the line somewhere there, gonna double check that.

CptSnuggles Over a year ago

So far, I've found some WARN messages in the elasticsearch logs:

elasticsearch_1  | {"type": "server", "timestamp": "2019-12-19T14:41:57,747Z", "level": "WARN", "component": "o.e.x.s.a.AuthenticationService", "cluster.name": "elasticsearch", "node.name": "elasticsearch", "message": "Authentication using apikey failed - invalid ApiKey value", "cluster.uuid": "0DNPuMBRQ7yN9SB34yYz9w", "node.id": "vj82JxvhTi6HbfZ5if54Ow" ,

Collectives™ on Stack Overflow

API Key Authentication in Elasticsearch with python

2 Answers 2

3 Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related