Get all field names in an index elasticsearch python

Question

I am trying to search a particular string against all indexes and fields in elastic search. I was able to get all the indices and search a particular field against the string. How can I get the list of field names in a particular index?

    def check_name_query(self):
    es = Elasticsearch("http://localhost:9200")
    indexlist = es.indices.get_alias()
    print "indices:", indexlist
    text_val = self.textin.text
    for ind in es.indices.get('*'):
        print ind
        res = es.search(index=ind, body={'query': {'wildcard': {'name': "*" + text_val + "*"}}})
        print res['hits']['hits']

This code gets all indices and checks if there is a field name with the value in textinput. Ideally i want to get check against all fields. Thanks in advance!

Lupanoide · Accepted Answer · 2020-03-03 08:17:30Z

10

Try:

from elasticsearch import Elasticsearch

es = Elasticsearch()
indices_names = []
for elem in es.cat.indices(format="json"):
    indices_names.append( elem['index'] )

dict_index_fields = {}

for index in indices_names: 
    mapping = es.indices.get_mapping(index)
    dict_index_fields[index] = []
    for field in mapping[index]['mappings']['properties']:
        dict_index_fields[index].append(field)

edited Mar 3, 2020 at 8:17

answered Mar 2, 2020 at 9:06

Lupanoide

3,24225 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

MortenB · Accepted Answer · 2025-01-08 14:48:00Z

The above answer will miss subfields aka:data.person.status and include objects that are holders for subfields but not fields.

The below routine matches exactly the available fields if you pick the index up in kibana:

(just cut out the relevant parts)

import elasticsearch
client = elasticsearch.Elasticsearch('localhost:9200')

def get_fields(m:dict, fields:list=[], path:str=None) -> list:
    """ traverse mapping recursively and return fields """
    for k,v in m.items():
        add = f"{path}.{k}" if path else k
        if 'properties' in v:
            fields = get_fields(v['properties'], fields, path=add)
        elif 'type' in v and v['type'] != 'object':
            fields.append(add)
    return fields

if args.getfields:
    mapping = client.indices.get_mapping(index=args.index)
    for k,v in mapping.items():
        #print(f"fields: {v['mappings']['properties']}")
        fields = get_fields(v['mappings']['properties'], fields=[], path="")
        out = []
        for f in fields:
            if '/' in f: # must be quoted in ESQL
                out.append(f"`{f}`")
            else:
                out.append(f)
        print(f"fields: {",".join(out)}")
    sys.exit(0)

python .\bin\esql.py --index=fido2.trk.events-test1.0-2025.01 --getfields
fields: @timestamp,@version,data.changes.active.from,data.changes.external_id.from,data.changes.phone_number.from,data.changes.username.from,
data.credential.aaguid,data.credential.id,data.dispensation,data.document_check.document_number,data.document_check.issuer_country,data.document_check.last_heartbeat,
data.document_check.receipt,data.document_check.request_timestamp,data.document_check.status,data.error.type,data.freg.sequence_counter,data.group.name,
data.http.correlation_id,data.identity_check.contact_date,data.identity_check.contact_method,data.identity_check.contact_reason,
data.identity_check.identity_check_method,data.identity_check.identity_confirmed,data.identity_document.date_of_expiry,data.identity_document.issuer.country,
data.identity_document.number,data.identity_document.type,data.openid.auth_method,data.openid.auth_type,data.openid.bankid_originator,data.openid.bp_amr,
data.openid.bp_id,data.openid.cert_issuer,data.openid.cert_serial,data.openid.client_auth_method,data.openid.client_id,data.openid.code_id,data.openid.consent,data.openid.error,
data.openid.grant_type,data.openid.identity_provider,data.openid.identity_provider_identity,data.openid.incident,data.openid.ip_address,data.openid.issuer_dn,
data.openid.originator,data.openid.redirect_uri,data.openid.remember_me,data.openid.response_mode,data.openid.response_type,data.openid.restart_after_timeout,
data.openid.scope,data.openid.serial_number,data.openid.session_id,data.openid.session_state,data.openid.sid,data.openid.subject_dn,data.openid.token_id,
data.openid.trace_id,data.openid.verification_id,data.operator.id,data.operator.username,data.otp.expiry_time,data.otp.trace_id,data.passport_reader.auth_files_url,
data.passport_reader.date_of_expiry,data.passport_reader.document_number,data.passport_reader.document_type,data.passport_reader.facematch_level,
data.passport_reader.issuing_country,data.passport_reader.jti,data.passport_reader.nationality,data.passport_reader_auth_files.auth_files_url,
data.passport_reader_auth_files.azure_url,data.passport_reader_verification.id,data.passport_reader_verification.matching_method,
data.person.id,data.person.status,data.sms.phone_number,data.sms.provider,data.tenant.id,data.terms.accepted_at,data.user.active,
data.user.external_id,data.user.id,data.user.organization,data.user.phone_number,data.user.updated_at,data.user.username,
data.user_summary.users_with_credential,data.verification.method,data.verification.time,id,source,specversion,time,type

Collectives™ on Stack Overflow

Get all field names in an index elasticsearch python

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related