How to connect to Elasticsearch with python using SSL?

Question

I am trying to connect to an Elasticsearch node from Python with SSL.

I'm using the basic code for that:

from elasticsearch import Elasticsearch
from ssl import create_default_context

context = create_default_context(cafile="path/to/cafile.pem")
es = Elasticsearch("https://elasticsearch.url:port", ssl_context=context, http_auth=('elastic','yourpassword'))

From: https://github.com/elastic/elasticsearch-py

I need to supply cafile.pem, and http_auth parameters. On the server where my Python is running, SSL connection is already set up, so I can do basic queries to Elasticsearch. It was set up using keys in the ~/.ssh directory: id_rsa, id_rsa.pub.

So, now I am wondering whether I should supply id_rsa.pub key in place of path/to/cafile.pem, and if yes, then I would need to change permissions of ~/.ssh folder which seems like not a good idea from security perspective.

Then, I am not sure that .pub is the same as .pem, do I need to convert it first? Then, should http_auth just be omitted since I do not use any password when I do simple queries from the terminal?

How should I go about this issue of setting up access in Python to ES with SSL according to best practices?

I tried both .pub and generated from it pem: https://serverfault.com/questions/706336/how-to-get-a-pem-file-from-ssh-key-pair

But both failed to create_default_context with an unknown error in context.load_verify_locations(cafile, capath, cadata).

You are missing the things, the id_rsa, id_rsa.pub are your ssh keys which your user uses to connect to your server without the need to provide a password, to make a secure connection with elasticsearch using SSL you need to configure it in your cluster and use the certificate generated during the configuration process. — leandrojmp
– leandrojmp, Commented May 22, 2020 at 19:30
You can configure SSL in your cluster using search-guard.com — Assael Azran
– Assael Azran, Commented May 22, 2020 at 20:11

Nikita Vlasenko · Accepted Answer · 2020-05-22 20:42:52Z

7

The answer for my particular case turned out to be very simple. I found it here:

https://elasticsearch-py.readthedocs.io/en/master/

es = Elasticsearch(['https://user:secret@localhost:443'])

Just specified https url inside and it worked out right away.

answered May 22, 2020 at 20:42

Nikita Vlasenko

4,3828 gold badges55 silver badges96 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

rozacek Over a year ago

Thanks, it's simple but helped when switching from HTTP to HTTPS in Elasticsearch.

Johan Over a year ago

I had to also verify_certs=False , see stackoverflow.com/a/73277995/3755989

dobhareach · Accepted Answer · 2022-06-08 10:31:29Z

3

Elasticsearch Docker image & Python2.7. Have Copied ssl certificate file to root of the project. Made sure it's readable, ownership and group ownership will allow read access. Put pass and login to constants.

es = Elasticsearch(
    hosts=[
            "https://localhost:9200"
    ],
    http_auth=(USR_LOGIN, USR_PASS),
    use_ssl=True,
    verify_certs=True,
    ca_certs="./http_ca.crt",
)

answered Jun 8, 2022 at 10:31

dobhareach

3425 silver badges7 bronze badges

1 Comment

slhck Over a year ago

use_ssl is not a valid parameter.

John Nguyen · Accepted Answer · 2024-02-08 17:22:46Z

2

We are using Elasticsearch 8.12.0 deployed by docker-compose with self-signed certs

from elasticsearch import Elasticsearch

es = Elasticsearch(
    "https://hostname:9200/",
    ca_certs="path/to/ca.crt",
    basic_auth=("username", "password")
)
#print(es.info())
indices = es.indices.get(index="_all")
for index in indices:
    print(index)

edited Feb 8, 2024 at 17:22

answered Feb 8, 2024 at 17:18

John Nguyen

212 bronze badges

Comments

Gino Mempin · Accepted Answer · 2024-01-15 02:50:20Z

For self-signed certificates, using:

The official Python client for Elasticsearch (https://pypi.org/project/elasticsearch/) 8.7.0
The shared elastic-transport library (https://pypi.org/project/elastic-transport/) 8.4.0

from elastic_transport import NodeConfig
from elasticsearch import AsyncElasticsearch

client = AsyncElasticsearch(
    hosts=[
        NodeConfig(
            scheme= "https",
            host="<host URL>",
            port=443,
            verify_certs=False,
            ca_certs=None,
            ssl_show_warn=False,
        )
    ],
    http_auth=("username", "password"),
    verify_certs=False,
    ca_certs="/path/to/cafile.pem",  # PEM format
    client_cert="/path/to/tls.cert"  # PEM format
    client_key="/path/to/tls.key"    # PEM format
)
client.info()

Explanation:

The CA file and the TLS cert/key pair were generated based on the guide for Set up basic security for the Elastic Stack plus secured HTTPS traffic: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup-https.html#security-basic-setup-https. As mentioned in the comment you would need to properly generate your TLS certs/keys, then pass that to the Elasticsearch client library
Since this answer is for self-signed certificates, you would need to also pass your CA file (in PEM format).
The verify_certs=False disables the underlying Python SSL modules from verifying the self-signed certs, but properly sends it upstream to the server. For non-self-signed certificates, you should try enabling verify_certs=True.
The code above is using AsyncElasticsearch, but if you need the sync Elasticsearch version, it should be directly compatible as all the parameters are the same. See: https://elasticsearch-py.readthedocs.io/en/v8.8.1/async.html#getting-started-with-async)

So, now I am wondering whether I should supply id_rsa.pub key in place of path/to/cafile.pem, and if yes, then I would need to change permissions of ~/.ssh folder which seems like not a good idea from security perspective.

These SSH keys are likely not related to Elasticsearch, but for allowing you to connect (via SSH) to your nodes running your Elasticsearch instance.

Collectives™ on Stack Overflow

How to connect to Elasticsearch with python using SSL?

4 Answers 4

2 Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

2 Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related