Serilog + ElasticSearch Sink Connection Issues (.net core)

Question

I've gotten Serilog writing to a rolling file okay and started implementing a connection to a Docker-based ElasticSearch instance. However, I am having connection issues.

To start, my application is running locally and there's two big take-aways:

I can reach the Elastic service from my machine
Kibana is showing that my service created an index with the name I specified

However, when I look at the output of Serilog.Debugging.SelfLog.Enable(msg => Debug.WriteLine(msg));, I see there's some sort of connection issue with Elastic:

Caught exception while preforming bulk operation to Elasticsearch: Elasticsearch.Net.ElasticsearchClientException: Maximum timeout reached while retrying request

My appsettings.json (yes, I am configuring through json and not code):

"Serilog": {
    "Using": ["Serilog.Sinks.Console", "Serilog.Sinks.File", "Serilog.Sinks.ElasticSearch"],
    "MinimumLevel": {
      "Default": "Debug",
      "Override": {
        "Microsoft": "Warning",
        "System": "Warning"
      }
    },
    "Enrich": ["FromLogContext", "WithExceptionDetails"],
    "WriteTo": [
      { "Name": "Console" },
      { "Name": "Debug" },
      {
        "Name": "File",
        "Args": {
          "path": "%LogDir%\\someserverpath.xyz.com\\log-.txt",
          "rollingInterval": "Day",
          "shared": true
        }
      },
      {
        "Name":  "Elasticsearch",
        "Args": {
          "nodeUris": "http://sxdockertst1:9200",
          "indexFormat": "imaging4cast-index-{0:yyyy.MM}",
          "emitEventFailure": "WriteToSelfLog",
          "autoRegisterTemplate": true
        } 
      }
    ],
    "Properties": {
      "Application":  "xyz.yyy.Imaging4CastApi" 
    } 
  },

I feel like it's getting rudimentary connection because, otherwise, how else would the index have been created? There's no auth on the elastic server, either. But actually pushing a log message doesn't appear to be working...

I'm at a loss here...

I updated to use a fully-qualified URI in nodeUris but still gets a timeout, FYI. — Killnine
– Killnine, Commented Feb 22, 2019 at 16:58

Killnine · Accepted Answer · 2019-02-22 19:43:45Z

2

Okay, so I figured this out.

The reason it was not posting messages was because the hard drive (technically, var on the docker host) was full. I had to clean up about 15GB of logs and messages in var.

Then, I needed to run this command to get Elastic out of Read-only mode (I quickly encountered an issue where I couldn't created indexes in Kibana for dashboarding and it was because Elastic went into RO mode):

curl -XPUT -H "Content-Type: application/json" http://sxdockertst1:9200/_all/_settings -d '{"index.blocks.read_only_allow_delete": null}'

answered Feb 22, 2019 at 19:43

Killnine

5,9709 gold badges44 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Navin Over a year ago

what happens intermediately if the connection to ELK is lost? will you lose your logs as well? or will it store locally & retry to send when ELK connection is up?

Killnine Over a year ago

I think it depends on your logging implementation. I believe Serilog does have a buffer to store logs (which can also be why you might not see records in Kibana immediately after you know a log was triggered, because the buffer hasn’t flushed to Elastic yet). However, I suspect logs would be lost if elastic was in RO mode for any significant period of time. That’s why I always have a backup File sink setup.

Navin Over a year ago

Ok. so u have a file sink + ELK sink. That was mu initial idea too but now I thinking of using either logstash | fluentd and not the the code.

Killnine Over a year ago

Yep, and I think that's actually a more popular model: pulling via logstash rather than pushing from the code.

Collectives™ on Stack Overflow

Serilog + ElasticSearch Sink Connection Issues (.net core)

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related