6

I'm using Python to add entries in a local ElasticSearch (localhost:9200)

Currently, I use this method:

def insertintoes(data):
"""
Insert data into ElasicSearch
:param data: dict
:return:
"""
timestamp = data.get('@timestamp')
logstashIndex = 'logstash-' + timestamp.strftime("%Y.%m.%d")
es = Elasticsearch()
if not es.indices.exists(logstashIndex):
    # Setting mappings for index
    mapping = '''
        {
            "mappings": {
                  "_default_": {
                    "_all": {
                      "enabled": true,
                      "norms": false
                    },
                    "dynamic_templates": [
                      {
                        "message_field": {
                          "path_match": "message",
                          "match_mapping_type": "string",
                          "mapping": {
                            "norms": false,
                            "type": "text"
                          }
                        }
                      },
                      {
                        "string_fields": {
                          "match": "*",
                          "match_mapping_type": "string",
                          "mapping": {
                            "fields": {
                              "keyword": {
                                "type": "keyword"
                              }
                            },
                            "norms": false,
                            "type": "text"
                          }
                        }
                      }
                    ],
                    "properties": {
                      "@timestamp": {
                        "type": "date",
                        "include_in_all": true
                      },
                      "@version": {
                        "type": "keyword",
                        "include_in_all": true
                      }
                    }
                  }
            }
        }
    '''
    es.indices.create(logstashIndex, ignore=400, body=mapping)

es.index(index=logstashIndex, doc_type='system', timestamp=timestamp, body=data)

data is a dict structure with a valid @timestamp defined like this data['@timestamp'] = datetime.datetime.now()

The problem is, even if there is a timestamp value in my data, Kibana doesn't show the entry in «discovery» field. :(

Here is an example of a full entry in ElasicSearch:

{ "_index": "logstash-2017.06.25", "_type": "system", "_id": "AVzf3QX3iazKBndbIkg4", "_score": 1, "_source": { "priority": 6, "uid": 0, "gid": 0, "systemd_slice": "system.slice", "cap_effective": "1fffffffff", "exe": "/usr/bin/bash", "hostname": "ns3003395", "syslog_facility": 9, "comm": "crond", "systemd_cgroup": "/system.slice/cronie.service", "systemd_unit": "cronie.service", "syslog_identifier": "CROND", "message": "(root) CMD (/usr/local/rtm/bin/rtm 14 > /dev/null 2> /dev/null)", "systemd_invocation_id": "9228b6c72e6a4624a1806e4c59af8d04", "syslog_pid": 26652, "pid": 26652, "@timestamp": "2017-06-25T17:27:01.734453" } }

As you can see, there IS a @timestamp field but it doesn't seems to be what Kibana expects.

And don't know what to do to make my entries visible in Kibana.

Any idea ?

1 Answer 1

11

Elasticsearch is not recognizing @timestamp as a date, but as a string. If your data['@timestamp'] is a datetime object, you can try to convert it to a ISO string, which is automatically recognized, try:

timestamp = data.get('@timestamp').isoformat()

timestamp should now be a string, but in ISO format

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.