Uploaded image for project: 'logback'
  1. logback
  2. LOGBACK-1402

SocketAppender not detecting dirty TCP disconnect. Log events go missing.

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.2.3
    • Fix Version/s: None
    • Component/s: logback-core
    • Labels:
    • Environment:

      Description

      It would appear that a remote container shutdown results in a dirty TCP disconnect. The remote TCP server disappears abruptly, leaving the client side with the socket in CLOSE_WAIT state. 

      While this is the case the client side is happily appending events to the SocketAppender which in turn seems to be writing this to the socket (the Send-Q on the socket keeps growing) without detecting any errors.

      This keeps going for several minutes (between 7 and 12 in my tests) during which log events go missing. Only at this very late point is an error raised and the reconnect logic triggered.

       

      I found that the AbstractLogstashTcpSocketAppender from the logstash project encountered and fixed this same problem.

      https://github.com/logstash/logstash-logback-encoder/issues/76

      https://github.com/logstash/logstash-logback-encoder/issues/85

       

      They effectively fixed the problem by attempting a read from the socket (with short read timeout) which results in a much more timely detection of the disconnect . This could take place on a separate watchdog thread to detect the problem.

      https://github.com/logstash/logstash-logback-encoder/blob/master/src/main/java/net/logstash/logback/appender/AbstractLogstashTcpSocketAppender.java#L385

      The call() method tries to constantly read from the socket. The expected outcome is that -1 bytes were read within the read timeout period, and this is where an exception will be much more reliably triggered when the socket is no longer available.

       

       

       

       

       

       

        Attachments

          Activity

            People

            Assignee:
            logback-dev Logback dev list
            Reporter:
            CharlesL Charles Loots
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated: