Uploaded image for project: 'logback'
  1. logback
  2. LOGBACK-1517

Reduce disk io when rolling between different filesystems

    XMLWordPrintable

Details

    • Icon: Improvement Improvement
    • Resolution: Unresolved
    • Icon: Major Major
    • None
    • 1.2.3
    • logback-core
    • None

    Description

      In "TimeBasedRollingPolicy#renameRawAndAsyncCompress", the temporary file ("*.tmp") is created based on the compressed file path.

       Future<?> renameRawAndAsyncCompress(String nameOfCompressedFile, String innerEntryName) throws RolloverFailure {
       String parentsRawFile = getParentsRawFileProperty();
       String tmpTarget = nameOfCompressedFile + System.nanoTime() + ".tmp";
       renameUtil.rename(parentsRawFile, tmpTarget);
       return compressor.asyncCompress(tmpTarget, nameOfCompressedFile, innerEntryName);
       }
      

      If the active log path and the compressed log path are located in different filesystems, this could add extra load to the infrastructure especially in cases where the active log path is under a fast disk (cf: locally-mounted SSDs, tmpfs) and the compressed log path is under a slower disk (cf: NFS, FUSE).

      /fast-disk/logs/example.log 
      /slow-disk/logs/example.log.gz.nano.tmp
      /slow-disk/logs/example.log.gz
      

      The rename writes a temp file into the slow-disk, then reads from the slow disk for in-memory compression. Both of these require accessing the slow-disk.

      This can be optimized by creating the temp file under the same directory as the active file (assuming that rename doesn't require as much disk io as copying).

      /fast-disk/logs/example.log 
      /fast-disk/logs/example.log.nano.tmp
      /slow-disk/logs/example.log.gz
      

      Attachments

        Activity

          People

            logback-dev Logback dev list
            ats Takanori Takase
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: