Elasticsearch not showing running log in python

Question

I am new in Elasticsearch. I have a running Elasticsearch instance in cloud and accessing it via python, i want to see running logs which has a field - "type"- "filebeat". I have following lines of code:

import elasticsearch
from elasticsearch import Elasticsearch
import elasticsearch.helpers

# Creating the client instance
es = Elasticsearch(
    cloud_id=CLOUD_ID,
    basic_auth=("elastic", ELASTIC_PASSWORD)
)

# Successful response!
print(es.info())
ES_INDEX = <my index>
ES_TYPE="filebeat"
results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"match_all": {}}},
    index=ES_INDEX)

results = list(results_gen)
print(results)

The output shows the instance details and 4407 logs in result (obviously all logs). My question is how to obtain running logs and how to modify the query to show only logs with "type"-"filebeat"?

hkulekci · Accepted Answer · 2022-12-14 21:04:05Z

1

You need to do some filtering in your query.

results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"match_all": {}}},
    index=ES_INDEX)

In here, you are using match_all. This will return all data on your index.

Here is a query sample above. term query will filter the data according to type: filebeat.

results_gen = elasticsearch.helpers.scan(
    es,
    query={"query": {"term": {"type": "filebeat"}}},
    index=ES_INDEX)

Also, you can check the documentation for more.

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html

answered Dec 14, 2022 at 21:04

hkulekci

1,96215 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

mrin9san Over a year ago

Thank you. Got the idea very well. Built the json, passed into query field and now its giving me exact entries. Any idea how to get running logs?

hkulekci Over a year ago

What does it mean exactly with "running logs"?

mrin9san Over a year ago

I want a loop or something so that I can read new logs that the instance generates. Right now it generates a log entry in every 5 seconds. But my python code reads the recent 4407 logs and stops.

hkulekci Over a year ago

You need to build a loop that should work with a timeout. For example, you can say every 5 seconds; you need to the user search_after.

Collectives™ on Stack Overflow

Elasticsearch not showing running log in python

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related