This was discussed briefly at http://marc.info/?l=logback-dev&m=124905434331308&w=2 and I forgot to file a ticket about this.

Currently, RandomFileAppender => FileAppender => WriterAppender is using the following method in WriterAppender to actually write the data:
protected void writerWrite(String s, boolean flush) throws IOException

Please add an additional method like
protected void writerWrite(byte[] bytes, boolean flush) throws IOException
to write to the underlying stream directly.

writerWrite(String, boolean) could call that method after performing the transformation internally, making this change transparent for the rest of the implementation.

Using a binary format for logfiles could have tremendous performance impact as can be seen here: http://sourceforge.net/apps/trac/lilith/wiki/SerializationPerformance