Uploaded image for project: 'logback'
  1. logback
  2. LOGBACK-1734

Add support for NDJSON / structured logging - popular log collector/shipper format

    XMLWordPrintable

Details

    Description

      Modern centralized logging assumes you work with structured data and even beyond logger/message/exception you enhance data with MDC (like httpUrl / httpMethod or customerEmail, etc) and with keyValue of SLF4J fluent API.

      Some tracing solution (OpenTelemetry/OpenTracing) might add additional dimension to data, via MDC, for example: https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/docs/logger-mdc-instrumentation.md

      So:

      • logging data gets lots of independent dimensions (via MDC or keyValue)
      • plain text file ambiguity is no longer acceptable - who wants to deal with parsing multiline stack traces anymore?

      One of the popular structured format is NDJSON.

      http://ndjson.org/ tells that NDJSON is a file with valid JSON on each line.

      Popular file log collectors supports NDJSON parsers:

      Today to log in NDJSON format you need to relay on 3rd party layout providers, like https://github.com/logfellow/logstash-logback-encoder

      I wish Logback has support for NDJSON out of the box.

      https://github.com/qos-ch/logback-contrib/blob/master/json/classic/src/main/java/ch/qos/logback/contrib/json/classic/JsonLayout.java is not a solution:

      • it lacks customization: possibility to rename attributes
      • it lacks customization: possibility to flatten MDC
      • it lacks support for Markers
      • it lacks support for keyValue of fluent API
      • it lacks support for nanoseconds
      • it lacks auto-incremented sequence number - centralized logging solutions store events in random order, if a query returns several events within a millisecond (or nanosecond) - they will be presented in random order, not the original order!

      Attachments

        Activity

          People

            ceki Ceki Gülcü
            gavenkoa Oleksandr Gavenko
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: