Application logging .net core & docker

Question

I'm attempting to put together a set of services written in .net-core and running in docker. I'm POCing each aspect that is deemed critical fora potential production system. Currently i'm looking at logging, and came across this article: Automating Docker Logging: ElasticSearch, Logstash, Kibana, and Logspout.

This works fine and monitors system and container activity from my web api that is running inside a container on my windows machine, but I can't find any guidelines regarding how to log application info so that it will be picked up the same way as the container logs are (e.g. logging internal exceptions).

In windows world I used Log4Net with a rolling file appender and a log location defined in the app.config. I'm hoping its as easy as this inside a docker container too (bearing in mind that in my current setup logsprout is sending container logs to logstash), but if it is, I dont know which location to define.

So I have 2 issues to solve:

1. Logging output to the default log directory of a container

2. Discovering the best format to use for logstash integration

Does anybody have any insights into these?

Answer 1

If you want your .NET Core application to write logs to Elasticsearch, and be queryable via Kibana, then you can add a logger provider to the standard Microsoft.Extensions.Logging.

log4net has a sink that will write to Elasticsearch if you are most familiar with that, and there is now also a stand alone logger provider that will write .NET Core logging direct to Elasticsearch, following the Elasticsearch Common Schema (ECS) field specifications, https://github.com/sgryphon/essential-logging/tree/master/src/Essential.LoggerProvider.Elasticsearch

Disclaimer: I am the author of Essential.LoggerProvider.Elasticsearch

To use this from your .NET Core application, add a reference to the Essential.LoggerProvider.Elasticsearch package:

dotnet add package Essential.LoggerProvider.Elasticsearch

Then, add the provider to the loggingBuilder during host construction, using the provided extension method.

using Essential.LoggerProvider;
// ...
    .ConfigureLogging((hostContext, loggingBuilder) =>
    {
        loggingBuilder.AddElasticsearch();
    })

The default configuration will write to a local Elasticsearch running at http://localhost:9200/. You can change this configuration to where your Elasticsearch is running, in appsettings.json:

{
  "Logging": {
    "Elasticsearch": {
      "NodeUris": [ "https://elastic-staging.example.com:9200" ],
      "Tags": [ "Staging" ]
    }
  }
}

Once you have sent some log events, open Kibana (e.g. http://localhost:5601/) and define an index pattern for "dotnet-*" with the time filter "@timestamp".

Log events are written following the Elasticsearch Common Schema (ECS) conventions, so should be compatible with events logged from other sources, e.g. Beats.