Uploaded image for project: 'logback'
  1. logback
  2. LOGBACK-1402

SocketAppender not detecting dirty TCP disconnect. Log events go missing.

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.3
    • Fix Version/s: None
    • Component/s: logback-core
    • Labels:
    • Environment:

      Description

      It would appear that a remote container shutdown results in a dirty TCP disconnect. The remote TCP server disappears abruptly, leaving the client side with the socket in CLOSE_WAIT state. 

      While this is the case the client side is happily appending events to the SocketAppender which in turn seems to be writing this to the socket (the Send-Q on the socket keeps growing) without detecting any errors.

      This keeps going for several minutes (between 7 and 12 in my tests) during which log events go missing. Only at this very late point is an error raised and the reconnect logic triggered.

       

      I found that the AbstractLogstashTcpSocketAppender from the logstash project encountered and fixed this same problem.

      https://github.com/logstash/logstash-logback-encoder/issues/76

      https://github.com/logstash/logstash-logback-encoder/issues/85

       

      They effectively fixed the problem by attempting a read from the socket (with short read timeout) which results in a much more timely detection of the disconnect . This could take place on a separate watchdog thread to detect the problem.

      https://github.com/logstash/logstash-logback-encoder/blob/master/src/main/java/net/logstash/logback/appender/AbstractLogstashTcpSocketAppender.java#L385

      The call() method tries to constantly read from the socket. The expected outcome is that -1 bytes were read within the read timeout period, and this is where an exception will be much more reliably triggered when the socket is no longer available.

       

       

       

       

       

       

        Attachments

          Activity

            People

            • Assignee:
              logback-dev Logback dev list
              Reporter:
              CharlesL Charles Loots
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated: