Serilog to Elasticsearch exception on creating the builder

Question

I face a situation where after upgrading all my Serilog* nuget packages to

<PackageReference Include="Serilog" Version="2.12.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.4.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.1.0" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.2.1" />

when I create the logger using

Log.Logger = LoggerConfiguration()
    .ReadFrom.Configuration(config)
    .Enrich.WithElasticApmCorrelationInfo()
    .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri("http://xxxxxxxxxxxxxxxxxxx"))
    {
        AutoRegisterTemplate = true,
        IndexFormat = "mslogs-{0:yyyy.MM.dd}",
        DetectElasticsearchVersion = true,
        RegisterTemplateFailure = RegisterTemplateRecovery.IndexAnyway,
        AutoRegisterTemplateVersion = AutoRegisterTemplateVersion.ESv7,
        FailureCallback = e => Console.WriteLine($"Unable to submit event {e?.RenderMessage()} to ElasticSearch. Exception : " + e?.Exception?.ToString()),
        EmitEventFailure = EmitEventFailureHandling.WriteToSelfLog |
                                EmitEventFailureHandling.WriteToFailureSink |
                                EmitEventFailureHandling.RaiseCallback,
        BufferCleanPayload = (failingEvent, statuscode, exception) =>
        {
            dynamic e = JObject.Parse(failingEvent);
            return JsonConvert.SerializeObject(new Dictionary<string, object>()
                {
                    { "action", "DeniedByElasticSearch"},
                    { "@timestamp",e["@timestamp"]},
                    { "level","Error"},
                    { "message","Error: "+e.message},
                    { "messageTemplate",e.messageTemplate},
                    { "failingStatusCode", statuscode},
                    { "failingException", exception}
                });
        },
        CustomFormatter = new EcsTextFormatter()
    })
    .CreateLogger();

with the appsettings.json configuration

  "Serilog": {
    "Using": [],
    "MinimumLevel": {
      "Default": "Information",
      "Override": {
        "Microsoft": "Warning",
        "System": "Information",
        "Elastic": "Warning",
        "Apm": "Warning"
      }
    },
    "WriteTo": [
      {
        "Name": "Console"
      }
    ],
    "Enrich": [
      "FromLogContext",
      "WithMachineName",
      "WithProcessId",
      "WithThreadId"
    ],
    "Properties": {
      "ApplicationName": "myapplication"
    }
  }

It throws the following exception, apparently when calling the .ReadFrom.Configuration(config) extension method :

Unhandled exception. System.Reflection.CustomAttributeFormatException: Binary format of the specified custom attribute was 
invalid. 
at System.Reflection.CustomAttributeEncodedArgument.ParseAttributeArguments(IntPtr pCa, Int32 cCa, 
CustomAttributeCtorParameter[]& CustomAttributeCtorParameters, CustomAttributeNamedParameter[]& 
CustomAttributeTypedArgument, RuntimeAssembly assembly) 
at System.Reflection.RuntimeCustomAttributeData..ctor(RuntimeModule scope, MetadataToken caCtorToken, ConstArray& blob)  
at System.Reflection.RuntimeCustomAttributeData.GetCustomAttributes(RuntimeModule module, Int32 tkTarget) 
at System.Reflection.RuntimeCustomAttributeData.GetCustomAttributesInternal(RuntimeMethodInfo target) 
at System.Reflection.RuntimeMethodInfo.GetCustomAttributesData()   at System.Reflection.MemberInfo.get_CustomAttributes()  
at Serilog.Settings.Configuration.ConfigurationReader. FindConfigurationExtensionMethods>g__HasExtensionAttribute|32_0(MethodInfo m) 
at Serilog.Settings.Configuration.ConfigurationReader.<>c.<FindConfigurationExtensionMethods>b__32_3(MethodInfo m) 
at System.Linq.Utilities.<>c__DisplayClass1_0`1.<CombinePredicates>b__0(TSource x)
at System.Linq.Enumerable.WhereEnumerableIterator`1.ToList() 
at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source) 
at Serilog.Settings.Configuration.ConfigurationReader.FindConfigurationExtensionMethods(IReadOnlyCollection`1 
configurationAssemblies, Type configType)
at Serilog.Settings.Configuration.ConfigurationReader.FindEventEnricherConfigurationMethods(IReadOnlyCollection`1 
configurationAssemblies) 
at Serilog.Settings.Configuration.ConfigurationReader.ApplyEnrichment(LoggerConfiguration 
loggerConfiguration)
at Serilog.Settings.Configuration.ConfigurationReader.Configure(LoggerConfiguration 
loggerConfiguration)
at Serilog.Configuration.LoggerSettingsConfiguration.Settings(ILoggerSettings settings)
at Serilog.ConfigurationLoggerConfigurationExtensions.Configuration(LoggerSettingsConfiguration settingConfiguration, 
IConfiguration configuration, String sectionName, DependencyContext dependencyContext)
at Serilog.ConfigurationLoggerConfigurationExtensions.Configuration(LoggerSettingsConfiguration settingConfiguration, 
IConfiguration configuration, DependencyContext dependencyContext)
at ITF.SharedLibraries.Logging.LoggerFactory.GetELKLogger(IConfiguration config, String varEnv)
at MyApplication.Startup.ConfigureServices(IServiceCollection services) in /src/MyApplication/src/Startup.cs:line 46 
at System.RuntimeMethodHandle.InvokeMethod(Object target, Span`1& arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
at Microsoft.AspNetCore.Hosting.MethodInfoExtensions.InvokeWithoutWrappingExceptions(MethodInfo methodInfo, Object obj, Object[] parameters)
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0. <Invoke>g__Startup|0(IServiceCollection serviceCollection) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services) 
at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, 
IServiceCollection services, Object instance) 
at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass13_0.
<UseStartup>b__0(HostBuilderContext context, IServiceCollection services) 
at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
at Microsoft.Extensions.Hosting.HostBuilder.Build() 
at MyApplication.Program.Main(String[] args) in /src/MyApplication/src/Program.cs:line 12

Important notes

This is the exact same behavior for two independant microservices where we pushed the package upgrade to production.
It doesn't throw locally (docker environment), it doesn't throw in our K8S cluster in development environment, or in our uat environment neither. But it throws systematically in our production environment.
We systematically (commit triggered) build our microservices from a vanilla .net runtime docker image (mcr.microsoft.com/dotnet/sdk:6.0) and basically dotnet restore on the fly in our CI/CD pipeline on Azure Devops using Azure agents.
When I downgrade to the previous version of the Serilog* packages I had (before upgrading them), it works fine for both applications in all our environments
The microservices are developed in .net 6, they are hosted in a K8S cluster (AKS), and the Elasticsearch cluster is in version v 7.17.1

Here are the downgraded versions where the problem dispears

<PackageReference Include="Serilog" Version="2.11.0" />
<PackageReference Include="Serilog.Extensions.Hosting" Version="5.0.1" />
<PackageReference Include="Serilog.Extensions.Logging" Version="3.1.0" />
<PackageReference Include="Serilog.Settings.Configuration" Version="3.3.0" />
<PackageReference Include="Serilog.Sinks.Console" Version="4.0.1" />
<PackageReference Include="Serilog.Sinks.ElasticSearch" Version="8.4.1" />
<PackageReference Include="Serilog.Sinks.Seq" Version="5.1.1" />
<PackageReference Include="StackExchange.Redis" Version="2.6.48" />

As the problem only occurs in our production environment it makes me feel a problem of data/settings in Elk that the new Serilog* highlight but I don't know where to investigate.

Cladoo · Accepted Answer · 2022-10-13 11:05:48Z

1

Finally the Serilog team has provided a workaround https://github.com/serilog/serilog-settings-configuration/pull/333/files

By the way the origin of the error is not determined and hardly reproductible.

answered Oct 13, 2022 at 11:05

Cladoo

1,18411 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

J.M. · Accepted Answer · 2022-11-01 18:34:00Z

I ran into this same issue, with a dotnet 6 microservice. After reading this post, reviewing the recent change to SeriLog in gitlab, and looking at my stack trace - it was obvious that something is broken in the ReadFromConfig option.

I was only seeing the issue in cert, so I focused on the differences between my working envs (local, sbx, dev) vs. the broken one (cert). After review of the differences, I was able to resolve the issue with the following two tweaks:

Moved a shared config section in appsettings.json to each respective environment file (appsettings.sbx.json, appsettings.cert.json, etc.)
Removed Enrich.FromLogContext() from config file and coded it in Program.cs because it didn't need to be configurable by environment.

Collectives™ on Stack Overflow

Serilog to Elasticsearch exception on creating the builder

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related