How to send Serilog data to Elasticsearch with fields

Question

I'm new to ES and Serilog, but my searches haven't produced this answer yet. I am trying to figure out how to use Serilog to send data to Elasticsearch in such a way that, if the data contains fields (for instance, if it's an object that has public properties), the data shows up in ES with those properties as fields. So far, I've gotten as far as using a RenderedCompactJsonFormatter and anonymous types to be able to achieve this mostly (see below), but that still produces named fields where the data in the fields is everything but the "new" part of the anonymous type declaration:

            var log = new LoggerConfiguration()
                .MinimumLevel.Information()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200/test_srpostimes"))
                {
                    InlineFields = true,
                    IndexDecider = (@event,offset) => "test_elapsedtimes",
                    CustomFormatter = new RenderedCompactJsonFormatter()
                })
                .WriteTo.Console()
                .CreateLogger();
            var elapsedTime = new {Time = 64};
            var timeStamp = new {Timestamp = DateTime.Now};
            var transID = new {TransID = "551674"};

            log.Information("{timeStamp} {transID} {elapsedTime}", timeStamp, transID, elapsedTime);

This produces:

@t:
    2016-07-11T18:45:35.0349343Z
@m:
    "{ Timestamp = 7/11/2016 2:45:35 PM }" "{ TransID = 551674 }" "{ Time = 64 }"
@i:
    b3ee2c05
timeStamp:
    { Timestamp = 7/11/2016 2:45:35 PM }
transID:
    { TransID = 551674 }
elapsedTime:
    { Time = 64 }
_id:
    AVXbR11WjgSgCs5HSlYY
_type:
    logevent
_index:
    test_srpostimes
_score:
    1

Is there a better way to do this so that our data can be searched/visualized using fields in ES (and Kibana)?

Hi Ant, have you seen: github.com/serilog/serilog-sinks-elasticsearch ? May help with this task. — Nicholas Blumhardt
– Nicholas Blumhardt, Commented Jul 11, 2016 at 22:06
Yes, I've been using your wonderful tools :) but I guess I either missed where it explained how to do this, or maybe (more likely?) I'm too green with ES and Serilog and would otherwise know how to accomplish this with more experience. I will update the question momentarily to show how I constructed the logger and sink to clarify things better. — Ant
– Ant, Commented Jul 12, 2016 at 11:20

Ant · Accepted Answer · 2016-07-12 13:35:57Z

I figured it out. I changed the construction to use the ElasticsearchJsonFormatter. Since the logger seemed to be able to parse the field name from the message, I switched to an object and passed in the properties instead:

        var log = new LoggerConfiguration()
            .MinimumLevel.Information()
            .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://localhost:9200/test_srpostimes"))
            {
                IndexDecider = (@event,offset) => "test_elapsedtimes",
                CustomFormatter = new ElasticsearchJsonFormatter()
            })
            .WriteTo.Console()
            .CreateLogger();

            var elapsedTimeMessage = new ElapsedTimeMessage(DateTime.Now.Millisecond);

            log.Information("{EventTime} {EventId} {ElapsedTime}", elapsedTimeMessage.EventTime, elapsedTimeMessage.EventId, elapsedTimeMessage.ElapsedTime);

That produced a much more readable output in ES:

  "_source": {
    "@timestamp": "2016-07-12T09:03:21.5804873-04:00",
    "level": "Information",
    "messageTemplate": "{EventTime} {EventId} {ElapsedTime}",
    "fields": {
      "EventTime": "2016-07-12T09:03:21.5754873-04:00",
      "EventId": "575",
      "ElapsedTime": 575
    }
  }

Collectives™ on Stack Overflow

How to send Serilog data to Elasticsearch with fields

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related