[#LBCLASSIC-140] Starvation on AppenderBase.doAppend

logback-classic

LBCLASSIC-135

Starvation on AppenderBase.doAppend

Created: 29/May/09 12:47 PM Updated: 23/Mar/10 09:55 AM

Component/s:

Appender

Affects Version/s:

0.9.15

Fix Version/s:

unspecified

File Attachments:

AppenderBaseLock3.patch (5 kb)
2.

NoLockThroughput.patch (2 kb)
3.

SynchronizedVsFairLock.java (4 kb)
4.

TimeWasted.java (7 kb)

Description

« Hide

The problem we are facing here is that several threads are trying to obtain the object monitor of the exact same resource, namely the Appender.

The assumption that multiple threads waiting for ownership of a monitor would receive the monitor in the order that they tried to acquire it is simply incorrect. This is documented behavior.

See the last paragraph of [1]:
"Likewise, no assumptions should be made about the order in which threads are granted ownership of a monitor or the order in which threads wake in response to the notify or notifyAll method. An excellent reference for these topics is Chapter 9, "Threads," in Joshua Bloch's book Effective Java Programming Language Guide. "

The documentation of Object.notifyAll() [2] states the following:
"[..] The awakened threads will compete in the usual manner with any other threads that might be actively competing to synchronize on this object; for example, the awakened threads enjoy no reliable privilege or disadvantage in being the next thread to lock this object."

The documentation in [5] also lists the following as a weak-spot of Built-in Synchronization in J2SE 1.4.x:
"No way to alter the semantics of a lock, for example, with respect to reentrancy, read versus write protection, or fairness."

In ~~LBCORE-96~~ Ceki stated the following:
"Logback is using the standard synchronization mechanism found in the JDK. You are saying that that mechanism is unsuitable which, truth be told, sounds quite suspicious."

Yes, it's unsuitable in this situation because obtaining the object monitor is not guaranteed to be fair. It's not working in a "first come, first serve" manner. I assumed that, too, but it isn't the case. I had the exact same problem in Lilith some time ago because I made exactly this very same wrong assumption.

Fairness of object monitor lock acquisition seems to be logical and a "good thing" but it's not specified that way, and for good reasons.

Without fairness in place, the VM can optimize the execution of an application much better. A context switch is a costly operation for a CPU so performance is increased significantly if such a switch can be omitted.

Concerning a test-case, this would be pretty hardcore to implement since it's 100% implementation dependent. One implementation *may* handle the locking of object monitors fairly while others don't.

Therefore I'll try the following first:
I assume I could convince you that object monitor acquisition (OMA) is not fair.

If we take that for granted the following scenario should show my point:
There are 4 Threads (e.g. 4 Threads handling concurrent webapp-requests) that have one single chokepoint, the appender.
Since OMA isn't fair, it's possible that only some of those threads can work at all (see [3]). Exactly that is happening right now in our webapp running on a 4-core Solaris on JDK6.

My next assumption is that we both agree that this isn't acceptable behavior.
Logging should be handled "first come, first served" so that if Thread A is waiting to append an event and Thread B is waiting to append an event subsequently, the actual appending order should be A, then B.
This is currently not the case. It *may* be the case but there is no guarantee of it.
One could even argue that the logging system is working incorrectly (aside from the starvation problem) because events are appended in a different order than the actual execution of the logger call.

The only way to prevent this is the introduction of fairness into the locking process. The way to do this is to use ReentrantLock [4].
From the ReentrantLock javadoc:
"The constructor for this class accepts an optional fairness parameter. When set true, under contention, locks favor granting access to the longest-waiting thread. Otherwise this lock does not guarantee any particular access order. Programs using fair locks accessed by many threads may display lower overall throughput (i.e., are slower; often much slower) than those using the default setting, but have smaller variances in times to obtain locks and guarantee lack of starvation."

This is exactly what is absolutely necessary for a logging framework.
Logging must *NOT* be able to introduce a starvation problem into a multi-threaded application! And it does exactly that right now.

I doubt that the performance will decrease in a significant manner due to the use of a fair ReentrantLock but even if this would be the case it would be necessary anyway.
Otherwise, appending simply isn't working correctly.

[1] http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html
[2] http://java.sun.com/javase/6/docs/api/java/lang/Object.html#notifyAll()
[3] http://en.wikipedia.org/wiki/Starvation_(computing)
[4] http://java.sun.com/javase/6/docs/api/java/util/concurrent/locks/ReentrantLock.html
[5] http://java.sun.com/developer/technicalArticles/J2SE/concurrency/

Description

The problem we are facing here is that several threads are trying to obtain the object monitor of the exact same resource, namely the Appender. The assumption that multiple threads waiting for ownership of a monitor would receive the monitor in the order that they tried to acquire it is simply incorrect. This is documented behavior. See the last paragraph of [1]: "Likewise, no assumptions should be made about the order in which threads are granted ownership of a monitor or the order in which threads wake in response to the notify or notifyAll method. An excellent reference for these topics is Chapter 9, "Threads," in Joshua Bloch's book Effective Java Programming Language Guide. " The documentation of Object.notifyAll() [2] states the following: "[..] The awakened threads will compete in the usual manner with any other threads that might be actively competing to synchronize on this object; for example, the awakened threads enjoy no reliable privilege or disadvantage in being the next thread to lock this object." The documentation in [5] also lists the following as a weak-spot of Built-in Synchronization in J2SE 1.4.x: "No way to alter the semantics of a lock, for example, with respect to reentrancy, read versus write protection, or fairness." In ~~LBCORE-96~~ Ceki stated the following: "Logback is using the standard synchronization mechanism found in the JDK. You are saying that that mechanism is unsuitable which, truth be told, sounds quite suspicious." Yes, it's unsuitable in this situation because obtaining the object monitor is not guaranteed to be fair. It's not working in a "first come, first serve" manner. I assumed that, too, but it isn't the case. I had the exact same problem in Lilith some time ago because I made exactly this very same wrong assumption. Fairness of object monitor lock acquisition seems to be logical and a "good thing" but it's not specified that way, and for good reasons. Without fairness in place, the VM can optimize the execution of an application much better. A context switch is a costly operation for a CPU so performance is increased significantly if such a switch can be omitted. Concerning a test-case, this would be pretty hardcore to implement since it's 100% implementation dependent. One implementation *may* handle the locking of object monitors fairly while others don't. Therefore I'll try the following first: I assume I could convince you that object monitor acquisition (OMA) is not fair. If we take that for granted the following scenario should show my point: There are 4 Threads (e.g. 4 Threads handling concurrent webapp-requests) that have one single chokepoint, the appender. Since OMA isn't fair, it's possible that only some of those threads can work at all (see [3]). Exactly that is happening right now in our webapp running on a 4-core Solaris on JDK6. My next assumption is that we both agree that this isn't acceptable behavior. Logging should be handled "first come, first served" so that if Thread A is waiting to append an event and Thread B is waiting to append an event subsequently, the actual appending order should be A, then B. This is currently not the case. It *may* be the case but there is no guarantee of it. One could even argue that the logging system is working incorrectly (aside from the starvation problem) because events are appended in a different order than the actual execution of the logger call. The only way to prevent this is the introduction of fairness into the locking process. The way to do this is to use ReentrantLock [4]. From the ReentrantLock javadoc: "The constructor for this class accepts an optional fairness parameter. When set true, under contention, locks favor granting access to the longest-waiting thread. Otherwise this lock does not guarantee any particular access order. Programs using fair locks accessed by many threads may display lower overall throughput (i.e., are slower; often much slower) than those using the default setting, but have smaller variances in times to obtain locks and guarantee lack of starvation." This is exactly what is absolutely necessary for a logging framework. Logging must *NOT* be able to introduce a starvation problem into a multi-threaded application! And it does exactly that right now. I doubt that the performance will decrease in a significant manner due to the use of a fair ReentrantLock but even if this would be the case it would be necessary anyway. Otherwise, appending simply isn't working correctly. [1] http://java.sun.com/j2se/1.5.0/docs/guide/vm/thread-priorities.html [2] http://java.sun.com/javase/6/docs/api/java/lang/Object.html#notifyAll() [3] http://en.wikipedia.org/wiki/Starvation_(computing) [4] http://java.sun.com/javase/6/docs/api/java/util/concurrent/locks/ReentrantLock.html [5] http://java.sun.com/developer/technicalArticles/J2SE/concurrency/

Show »

All

Comments

Change History

Subversion Commits

Sort Order:

Ascending order - Click to sort in descending order

[ Permlink | « Hide ]

Joern Huxhorn - 29/May/09 12:48 PM

Patch fixing the starvation problem using a fair ReentrantLock.

[ Permlink | « Hide ]

Ceki Gulcu - 29/May/09 04:38 PM - edited

> Concerning a test-case, this would be pretty hardcore to implement
> since it's 100% implementation dependent. One implementation *may*
> handle the locking of object monitors fairly while others don't.

Could you please supply a test case which demonstrates your point, even if it is for a specific JDK? I would at least like to see an example. FYI, I have several multi-core PCs available running Linux and/or Windows.

[ Permlink | « Hide ]

Joern Huxhorn - 29/May/09 05:43 PM

I could try to provide a sample program that shows the general problem of starvation.
Would this have to be logback related or is it sufficient if I show that what the documenation says is correct?

Or are you talking about a real JUnit testcase that's using logback? Would this be worth anything if it fails on some systems and works on others? As always, assumptions about absolute program execution are pretty hard to make when it comes to multi-threaded applications.

The example attached to ~~LBCORE-96~~ shows that the threads are "synchronized" with the patch in place, i.e. every pool gets the chance to log a message

2009-05-19 22:03:14,520 DEBUG [pool-1-thread-1] [EX1] 5936
2009-05-19 22:03:14,520 DEBUG [pool-3-thread-1] [MAIN] - [EX3] 5925
2009-05-19 22:03:14,520 DEBUG [pool-2-thread-1] [EX2] 5926
2009-05-19 22:03:14,520 DEBUG [pool-5-thread-1] [MAIN] - [EX5] 5900
2009-05-19 22:03:14,520 DEBUG [pool-4-thread-1] [MAIN] - [EX4] 5925

2009-05-19 22:03:14,521 DEBUG [pool-1-thread-1] [MAIN] - [EX1] 5937
2009-05-19 22:03:14,521 DEBUG [pool-3-thread-1] [EX3] 5926
2009-05-19 22:03:14,521 DEBUG [pool-2-thread-1] [MAIN] - [EX2] 5927
2009-05-19 22:03:14,521 DEBUG [pool-5-thread-1] [EX5] 5901
2009-05-19 22:03:14,521 DEBUG [pool-4-thread-1] [EX4] 5926

I inserted newlines to better show the point. It's always executing 1-3-2-5-4.

Without the patch, the threads are executed unevenly. This is solely caused by the non-fairness of synchronized as the rest of the code is untouched.

The only thing that I could do is to write some program that shows this more drastically, e.g. letting 10 threads run for some time in an endless loop, each containing a
synchronized(sameObject) block, counting the times they loop.
After a specific time, e.g. 10 sec, I could interrupt all threads and they would certainly have quite different counter values (the amount of difference would vary from system to system and VM to VM).

Running the same code with a fair ReentrantLock would generate results with (nearly, because of the interruption of the threads) identical numbers. That's the whole point of fair locks.

I had the exact same wrong assumption about fairness in a wait()/notifyAll() situation in Lilith. A thread with the purpose of removing received events from an intermittent cyclic buffer just wouldn't get the chance to remove the events. This was caused by updating the used JRE to Java 1.6. It took me a while to understand the problem and I was quite baffled.

[ Permlink | « Hide ]

Ceki Gulcu - 29/May/09 06:09 PM

No, the "test case " does not need to be logback specific. I am just looking for sample code with an accompanying analysis. If it can be reproduced on a Windows or Linux machine (with whichever JDK) that would be a step forward. Thank you in advance.

[ Permlink | « Hide ]

Joern Huxhorn - 29/May/09 11:37 PM

Execute the attached application on both 1.5 and 1.6 to see the difference. Check on both single- and multi-core systems.

[ Permlink | « Hide ]

Joern Huxhorn - 29/May/09 11:38 PM

Here are my personal result:

iMac
2.4GHz Intel Core 2 Duo
i.e. multi-core

Environment:
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_07-b06-153
java.vendor = Apple Inc.
java.version = 1.6.0_07
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Mac OS X
os.version = 10.5.7
os.arch = x86_64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=394, running=false]
runnables[1]: SynchronizedRunnable[counter=1, running=false]
runnables[2]: SynchronizedRunnable[counter=1, running=false]
runnables[3]: SynchronizedRunnable[counter=1, running=false]
runnables[4]: SynchronizedRunnable[counter=1, running=false]
runnables[5]: SynchronizedRunnable[counter=1, running=false]
runnables[6]: SynchronizedRunnable[counter=1, running=false]
runnables[7]: SynchronizedRunnable[counter=1, running=false]
runnables[8]: SynchronizedRunnable[counter=1, running=false]
runnables[9]: SynchronizedRunnable[counter=605, running=false]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=997, running=false]
runnables[1]: LockRunnable[counter=1, running=false]
runnables[2]: LockRunnable[counter=1, running=false]
runnables[3]: LockRunnable[counter=1, running=false]
runnables[4]: LockRunnable[counter=1, running=false]
runnables[5]: LockRunnable[counter=1, running=false]
runnables[6]: LockRunnable[counter=1, running=false]
runnables[7]: LockRunnable[counter=1, running=false]
runnables[8]: LockRunnable[counter=1, running=false]
runnables[9]: LockRunnable[counter=1, running=false]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=101, running=false]
runnables[1]: LockRunnable[counter=101, running=false]
runnables[2]: LockRunnable[counter=101, running=false]
runnables[3]: LockRunnable[counter=101, running=false]
runnables[4]: LockRunnable[counter=100, running=false]
runnables[5]: LockRunnable[counter=100, running=false]
runnables[6]: LockRunnable[counter=100, running=false]
runnables[7]: LockRunnable[counter=100, running=false]
runnables[8]: LockRunnable[counter=100, running=false]
runnables[9]: LockRunnable[counter=100, running=false]
##########################################

Environment:
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.5.0_16-b06-284
java.vendor = Apple Inc.
java.version = 1.5.0_16
java.vm.name = Java HotSpot(TM) Client VM
java.vm.info = mixed mode, sharing
os.name = Mac OS X
os.version = 10.5.7
os.arch = i386
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=101, running=false]
runnables[1]: SynchronizedRunnable[counter=101, running=false]
runnables[2]: SynchronizedRunnable[counter=101, running=false]
runnables[3]: SynchronizedRunnable[counter=101, running=false]
runnables[4]: SynchronizedRunnable[counter=101, running=false]
runnables[5]: SynchronizedRunnable[counter=101, running=false]
runnables[6]: SynchronizedRunnable[counter=100, running=false]
runnables[7]: SynchronizedRunnable[counter=100, running=false]
runnables[8]: SynchronizedRunnable[counter=100, running=false]
runnables[9]: SynchronizedRunnable[counter=100, running=false]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=485, running=false]
runnables[1]: LockRunnable[counter=495, running=false]
runnables[2]: LockRunnable[counter=19, running=false]
runnables[3]: LockRunnable[counter=1, running=false]
runnables[4]: LockRunnable[counter=1, running=false]
runnables[5]: LockRunnable[counter=1, running=false]
runnables[6]: LockRunnable[counter=1, running=false]
runnables[7]: LockRunnable[counter=1, running=false]
runnables[8]: LockRunnable[counter=1, running=false]
runnables[9]: LockRunnable[counter=1, running=false]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=101, running=false]
runnables[1]: LockRunnable[counter=101, running=false]
runnables[2]: LockRunnable[counter=101, running=false]
runnables[3]: LockRunnable[counter=101, running=false]
runnables[4]: LockRunnable[counter=101, running=false]
runnables[5]: LockRunnable[counter=100, running=false]
runnables[6]: LockRunnable[counter=100, running=false]
runnables[7]: LockRunnable[counter=100, running=false]
runnables[8]: LockRunnable[counter=100, running=false]
runnables[9]: LockRunnable[counter=100, running=false]
##########################################

####################################################################################

IBM ThinkPad
Intel Pentium M processor 1.86GHz
i.e. single-core

Environment:
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.5.0_13-b05
java.vendor = Sun Microsystems Inc.
java.version = 1.5.0_13
java.vm.name = Java HotSpot(TM) Client VM
java.vm.info = mixed mode, sharing
os.name = Windows XP
os.version = 5.1
os.arch = x86
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=65, running=false]
runnables[1]: SynchronizedRunnable[counter=65, running=false]
runnables[2]: SynchronizedRunnable[counter=65, running=false]
runnables[3]: SynchronizedRunnable[counter=65, running=false]
runnables[4]: SynchronizedRunnable[counter=65, running=false]
runnables[5]: SynchronizedRunnable[counter=65, running=false]
runnables[6]: SynchronizedRunnable[counter=65, running=false]
runnables[7]: SynchronizedRunnable[counter=64, running=false]
runnables[8]: SynchronizedRunnable[counter=64, running=false]
runnables[9]: SynchronizedRunnable[counter=64, running=false]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=50, running=false]
runnables[1]: LockRunnable[counter=86, running=false]
runnables[2]: LockRunnable[counter=63, running=false]
runnables[3]: LockRunnable[counter=54, running=false]
runnables[4]: LockRunnable[counter=72, running=false]
runnables[5]: LockRunnable[counter=72, running=false]
runnables[6]: LockRunnable[counter=54, running=false]
runnables[7]: LockRunnable[counter=72, running=false]
runnables[8]: LockRunnable[counter=70, running=false]
runnables[9]: LockRunnable[counter=57, running=false]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=65, running=false]
runnables[1]: LockRunnable[counter=65, running=false]
runnables[2]: LockRunnable[counter=65, running=false]
runnables[3]: LockRunnable[counter=65, running=false]
runnables[4]: LockRunnable[counter=64, running=false]
runnables[5]: LockRunnable[counter=64, running=false]
runnables[6]: LockRunnable[counter=64, running=false]
runnables[7]: LockRunnable[counter=64, running=false]
runnables[8]: LockRunnable[counter=64, running=false]
runnables[9]: LockRunnable[counter=64, running=false]
##########################################

Environment:
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_13-b03
java.vendor = Sun Microsystems Inc.
java.version = 1.6.0_13
java.vm.name = Java HotSpot(TM) Client VM
java.vm.info = mixed mode, sharing
os.name = Windows XP
os.version = 5.1
os.arch = x86
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=6, running=false]
runnables[1]: SynchronizedRunnable[counter=11, running=false]
runnables[2]: SynchronizedRunnable[counter=85, running=false]
runnables[3]: SynchronizedRunnable[counter=70, running=false]
runnables[4]: SynchronizedRunnable[counter=75, running=false]
runnables[5]: SynchronizedRunnable[counter=73, running=false]
runnables[6]: SynchronizedRunnable[counter=74, running=false]
runnables[7]: SynchronizedRunnable[counter=88, running=false]
runnables[8]: SynchronizedRunnable[counter=72, running=false]
runnables[9]: SynchronizedRunnable[counter=90, running=false]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=64, running=false]
runnables[1]: LockRunnable[counter=87, running=false]
runnables[2]: LockRunnable[counter=60, running=false]
runnables[3]: LockRunnable[counter=60, running=false]
runnables[4]: LockRunnable[counter=76, running=false]
runnables[5]: LockRunnable[counter=54, running=false]
runnables[6]: LockRunnable[counter=54, running=false]
runnables[7]: LockRunnable[counter=71, running=false]
runnables[8]: LockRunnable[counter=56, running=false]
runnables[9]: LockRunnable[counter=67, running=false]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=65, running=false]
runnables[1]: LockRunnable[counter=65, running=false]
runnables[2]: LockRunnable[counter=65, running=false]
runnables[3]: LockRunnable[counter=65, running=false]
runnables[4]: LockRunnable[counter=65, running=false]
runnables[5]: LockRunnable[counter=65, running=false]
runnables[6]: LockRunnable[counter=65, running=false]
runnables[7]: LockRunnable[counter=64, running=false]
runnables[8]: LockRunnable[counter=64, running=false]
runnables[9]: LockRunnable[counter=64, running=false]
##########################################

[ Permlink | « Hide ]

Joern Huxhorn - 29/May/09 11:41 PM

Conclusion:

synchronized was fair on 1.5, is a bit unfair on 1.6 single-core and very unfair on 1.6 multi-core.

Please apply my patch, thanks ;)

[ Permlink | « Hide ]

Maarten Bosteels - 30/May/09 11:17 AM

Joern, I have not yet looked at your appender, but I wonder if it wouldn't be better to extend UnsynchronizedAppenderBase ?

I have implemented an AsyncSocketAppender myself (not yet in use in production) and it takes care of syncronization itself : it adds all events to a BlockingQueue and another thread takes care of writing the events to the socket.
This way you can use whatever locking mechanism you prefer.

[ Permlink | « Hide ]

Joern Huxhorn - 30/May/09 01:49 PM

This problem is not at all related to my appender but bites *every* appender that extends AppenderBase, i.e. console, file and socket appender.

On a side-note, the Lilith appender is semi-asynchronous and has a buffer of 1000 already serialized events so it's possible to handle bursts of events gracefully. It blocks only after that buffer has been filled. The buffer content is attached to async sub-appenders that are either sending the bytes or discarding them in case of no connection. Since the events are already serialized all sub-appenders receive the exact same byte arrays to send. Take a look at http://apps.sourceforge.net/trac/lilith/wiki/MultiplexAppenderBackground if you are interested.

On another side-note, I still consider UnsynchronizedAppenderBase downright dangerous because I firmly (read: religiously) believe that async handling of LoggingEvent instances will introduce problems like http://www.nabble.com/Best-practices-to-avoid-deadlocks-while-logging-td23639031.html reloaded, i.e. on a whole new level.
Serialization (or, more generally, evaluation) *must* be handled synchronously because it's impossible to predict how objects in the MDC and message arguments will behave if converted to a string asynchronously. Even if no deadlock will occur, they *might* have changed in the meantime, essentially producing "lying" log statements in the end.

But all this is really 100% unrelated to this bug report.

This is a general problem that was introduced in Java 1.6. It's only showing up badly in case of real multi-core systems which made it a bit hard for me to analyze on my terribly outdated 3-year old laptop at work...
I guess that's the price a company has to pay if their developers are forced to work with flint stone devices.

Are you reading this, IT department?? :p

[ Permlink | « Hide ]

Ralph Goers - 30/May/09 08:50 PM

Your notion that UnsynchronizedAppenderBase is "downright dangerous" is complete nonsense. That is like saying multithreaded programming is dangerous so don't do it.

The Appender that extends UnsynchronizedAppenderBase is called on the same thread that called the logger method. All UnsynchronizedAppenderBase does is leave whatever synchronization is necessary up to the actual Appender. If the appender is going to pass work of to some other thread then the work queue elements need to be fully populated with data on the thread creating the work elememnts, i.e. the same thread that called the logging method. Trying to have the worker thread call methods on objects in a logger event is a very poor design.

In many cases the only synchronization needed in an appender will be in the JDK itself when it performs some kind of I/O.

There have been many demonstrated examples where the synchronization that is occuring within logback has caused deadlocks because the application using logback was unaware that the locking would occur. UnsynchronizedAppenderBase actually helps with avoiding deadlocks since the locking moves down to where it is actually needed, not at some higher level "just in case".

[ Permlink | « Hide ]

Joern Huxhorn - 31/May/09 03:27 AM

I replied on the dev list because this discussion is unrelated to the bug.

[ Permlink | « Hide ]

Joern Huxhorn - 31/May/09 03:00 PM - edited

Patch fixing the starvation problem using a fair ReentrantLock.

Same patch as before with the exception that lock is now protected instead of private so extending classes can (and should!) use that same lock for any synchronization.

[ Permlink | « Hide ]

Joern Huxhorn - 31/May/09 06:08 PM

I've written an article about this problem in the Lilith wiki: http://apps.sourceforge.net/trac/lilith/wiki/SynchronizedVsFairLock

[ Permlink | « Hide ]

Ceki Gulcu - 04/Jun/09 06:01 PM

Joern, thank you for supplying SynchronizedVsFairLock. Here are my results running SynchronizedVsFairLock. Only results for SynchronizedRunnable are shown.

Environment:
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_05-b13
java.vendor = Sun Microsystems Inc.
java.version = 1.6.0_05
os.name = Windows XP
os.version = 5.1
os.arch = x86
##########################################
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=97, running=false]
runnables[1]: SynchronizedRunnable[counter=43, running=false]
runnables[2]: SynchronizedRunnable[counter=33, running=false]
runnables[3]: SynchronizedRunnable[counter=96, running=false]
runnables[4]: SynchronizedRunnable[counter=99, running=false]
runnables[5]: SynchronizedRunnable[counter=134, running=false]
runnables[6]: SynchronizedRunnable[counter=184, running=false]
runnables[7]: SynchronizedRunnable[counter=78, running=false]
runnables[8]: SynchronizedRunnable[counter=131, running=false]
runnables[9]: SynchronizedRunnable[counter=44, running=false]

Environment:
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_11-b03
java.vendor = Sun Microsystems Inc.
java.version = 1.6.0_11
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Linux
os.version = 2.6.25-gentoo-r6
os.arch = amd64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=995, running=false]
runnables[1]: SynchronizedRunnable[counter=1, running=false]
runnables[2]: SynchronizedRunnable[counter=1, running=false]
runnables[3]: SynchronizedRunnable[counter=1, running=false]
runnables[4]: SynchronizedRunnable[counter=1, running=false]
runnables[5]: SynchronizedRunnable[counter=1, running=false]
runnables[6]: SynchronizedRunnable[counter=1, running=false]
runnables[7]: SynchronizedRunnable[counter=1, running=false]
runnables[8]: SynchronizedRunnable[counter=1, running=false]
runnables[9]: SynchronizedRunnable[counter=1, running=false]

Environment:
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = jvmxa6460-20081105_25433
java.vendor = IBM Corporation
java.version = 1.6.0
java.vm.name = IBM J9 VM
java.vm.info = J2RE 1.6.0 IBM J9 2.4 Linux amd64-64 jvmxa6460-20081105_25433 (JIT enabled, AOT enabled)
J9VM - 20081105_025433_LHdSMr
JIT - r9_20081031_1330
GC - 20081027_AB
os.name = Linux
os.version = 2.6.25-gentoo-r6
os.arch = amd64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=36, running=false]
runnables[1]: SynchronizedRunnable[counter=148, running=false]
runnables[2]: SynchronizedRunnable[counter=192, running=false]
runnables[3]: SynchronizedRunnable[counter=26, running=false]
runnables[4]: SynchronizedRunnable[counter=124, running=false]
runnables[5]: SynchronizedRunnable[counter=62, running=false]
runnables[6]: SynchronizedRunnable[counter=122, running=false]
runnables[7]: SynchronizedRunnable[counter=32, running=false]
runnables[8]: SynchronizedRunnable[counter=27, running=false]
runnables[9]: SynchronizedRunnable[counter=137, running=false]

The results show that on Windows and Sun's JDK 1.6, the various threads compete fairly for the monitor, whereas on a 64bit Dual Core AMD Opteron running Linux and Sun's JDK 1.6, the first thread almost always wins the monitor while the other threads go starving. On the same Dual Core AMD machine changing to IBM's JDK 1.6.0, the results while not completely fair are not as lopsided as with Sun's JDK.

I am tempted to say that it's a bug in Sun JDK. While we are not allowed to assume any particular ordering between competing threads, the behavior displayed by Sun's JDK is too extreme to be considered reasonable.

Regardless of what I or anyone else thinks about Sun's 1.6. scheduling behavior, it should be considered as an established fact that we have to deal with. The question is whether we ignore it until Sun comes to
their senses, or if we apply Joern's patch.

[ Permlink | « Hide ]

Joern Huxhorn - 04/Jun/09 07:11 PM

Have you done any benchmarking concerning the performance impact of the fair ReentrantLock?

I guess you have logback benchmarks at your fingertips, I'd be quite interested in the numbers.

Concerning the "bug in the Sun JDK" part: It's the same way in case of the Mac version of Java 6.
I highly doubt that Sun will come to it's senses ;) and I'm not even sure that it can be validly called a bug since it doesn't breach any documented behavior. This might really be a valid performance enhancement because switching threads means degrading performance.

I'd be really grateful if this patch could be applied before the next release. We are suffering quite bad because of it.
In our case it's - as mentioned - a Solaris and we can't just switch the VM or the OS as we like because of customer rules.

[ Permlink | « Hide ]

Philippe Marschall - 04/Jun/09 10:30 PM

Standard java synchronization was never guaranteed to be fair. Java 6 (or the OS scheduler) is simply doing more optimizations that it didn't do previously but was always allowed to do.

However I don't see why this is a problem unless logging is your bottleneck because there is no real ordering in multi thread applications anyway. If logging is your bottleneck I see this as a serious issue because synchronous logging will in this case turn your application into a single threaded one and you're effectively producing logging events faster than you can log them.

If this is the case I'd investigate why you're spending so much time in logging and if that's unfixable change to asynchronous logging. Sure it doesn't work out of the box with MDCs but there are ways to solve that. Either copy the MDC or format the message in the calling thread (without aquiring a lock) and adding the formatted instead of the unformatted string to the queue.

[ Permlink | « Hide ]

Joern Huxhorn - 05/Jun/09 12:24 AM

> Standard java synchronization was never guaranteed to be fair. Java 6 (or the
> OS scheduler) is simply doing more optimizations that it didn't do previously
> but was always allowed to do.

+1

> However I don't see why this is a problem unless logging is your bottleneck

Logging is, indeed, the bottleneck of our application at the moment, but this is expected, "by design" and will change at a later time.

- We are rolling out a new version of our app so logging is still unusually high
- We are doing a proprietary CMS with XSLT as the template language. There are several XSLT developers receiving compilation specific INFO, WARN and ERROR events via my MulticastSocketAppender.
- The slowest receiver essentially dictates the overall speed of the application

> If this is the case I'd investigate why you're spending so much time in
> logging and if that's unfixable change to asynchronous logging.

The MultiplexAppender is already semi-asynchronous, meaning that it has a buffer of fixed size that is filled before the appender blocks.

See http://apps.sourceforge.net/trac/lilith/wiki/AppenderPitfalls

We've done this "abusive" logging for over a year with *very* acceptable performance.

Then we moved from a single-core Java 1.5 server to a multi-core Java 1.6 server and, suddenly, performance was extremely crappy. I narrowed the problem down to the issue discussed here and never really expected any discussion whether this patch should be applied or not because - honestly - I think I did a pretty good job citing all relevant documentation, showing up that we all worked with false presumptions - and this explicitly includes myself!

I *knew* that fairness wasn't guaranteed but I *assumed* that something like nearly-fair or somewhat-unfair could be expected. This assumption was wrong.

> because synchronous logging will in this case turn your application into a
> single threaded one

There's some truth in this statement.;)
However, large amounts of code won't log anything while other parts log in relatively big bursts so, theoretically, all CPUs should be able to work quite a bit, anyway, even if parts of the application are running semi-single-threaded.

[ Permlink | « Hide ]

Joern Huxhorn - 05/Jun/09 08:17 AM

FYI, I changed MultiplexSocketAppenderBase to extend UnsynchronizedAppenderBase instead of AppenderBase to circumvent this problem.
Locking is now solely handled by a fair (!!) BlockingQueue.

This issue is still valid, though, since it's also apparent - to a lesser degree - if only file or console appender is used.

In addition to applying my patch I'd also strongly suggest that a comment is added to UnsynchronizedAppenderBase stating that any locking performed by an extending class *must* be fair to prevent starvation.

This is really the case, please believe me.

If this comment is added to UnsynchronizedAppenderBase, an extending class is effectively breaching contract if locking differently.

A logging system is, in fact, a prime example where locking just *has* to be fair to prevent starvation since multiple threads *are* trying to access the same resource (Logger/Appender) all the time.

http://java.sun.com/docs/books/tutorial/essential/concurrency/starvelive.html

[ Permlink | « Hide ]

Ralph Goers - 05/Jun/09 08:45 AM

I think that your recommendation makes sense if for no other reason than to make anyone extending the class aware of the issue. I tend to agree that logging, when enabled, is a high overhead activity and is far more likely to experience this behavior than most other scenarios.

It is too bad that this behavior can't be controlled through an annotation.

[ Permlink | « Hide ]

Ceki Gulcu - 08/Jun/09 09:37 PM - edited

Have you noticed that the starvation issue arises

1) only if the sleep duration is 3 milliseconds and over (on Linux)
2) with JDK 1.5 as well ?

I had a long exchange with Dave Dice on the topic of biased locking [1] in my blog. Apparently, the issue is not JDK 1.6 specific and was present in older JVMs as well.

Given that logback usually does not hold the lock (on itself) for more than 10 microseconds, the starvation condition should not occur in practice. (This assertion is based on experimentation done on Linux and Windows).

I'd appreciate if the 3 millisecond threshold could be confirmed for OS X and Solaris. To set a lower sleep duration, within the run() method of SynchronizedRunnable modify

  Thread.sleep(10);

to

  Thread.sleep(1);

or even a smaller value:

  Thread.sleep(0, 10*1000); // 10 microseconds

[1] http://ceki.blogspot.com/2009/06/biased-locking-in-java-se-60.html

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 08:39 AM - edited

I was able to get the following results after several runs of my modified SynchronizedVsFairLock ( http://apps.sourceforge.net/trac/lilith/browser/trunk/misc/synchronized-lock/SynchronizedVsFairLock.java ).

In other cases, locking wasn't as unfair as the numbers below.

The starvation behavior isn't as reproducible as before but it's still there. The chance of starvation increases with the amount of delay.

Given that there are Appenders that are candidates for a delay > 3ms, e.g. SocketAppender, DBAppender, SMTPAppender and even FileAppender (especially if the log file is located on a NAS), I'd still strongly suggest to change AppenderBase according to my suggestion.

We should really aim to switch over to a "locking as needed" strategy, as Ralph suggested. Changing more and more appenders to extend UnsynchronizedAppenderBase and handling locking - in a fair way - as needed by the respective implementation.

Depending on unfair locking is IMO a risk that is way to high and is, essentially, wrong.

Concerning "real life experience", leaving my proof of problem aside, I can tell you that the starvation effect is still happening with a RollingFileAppender in place, it's just not as bad as with a SocketAppender.

Taking into account LBCORE-62 and ~~LBCORE-96~~, I think it's save to assume that this is a general problem.

My results for Mac OS are below. I can't test on Solaris right now because I'm on vacation.

Environment:
delay = 3
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_07-b06-153
java.vendor = Apple Inc.
java.version = 1.6.0_07
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Mac OS X
os.version = 10.5.7
os.arch = x86_64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=967, running=false, delay=3]
runnables[1]: SynchronizedRunnable[counter=155, running=false, delay=3]
runnables[2]: SynchronizedRunnable[counter=753, running=false, delay=3]
runnables[3]: SynchronizedRunnable[counter=583, running=false, delay=3]
runnables[4]: SynchronizedRunnable[counter=471, running=false, delay=3]
runnables[5]: SynchronizedRunnable[counter=367, running=false, delay=3]
runnables[6]: SynchronizedRunnable[counter=1, running=false, delay=3]
runnables[7]: SynchronizedRunnable[counter=1, running=false, delay=3]
runnables[8]: SynchronizedRunnable[counter=1, running=false, delay=3]
runnables[9]: SynchronizedRunnable[counter=2, running=false, delay=3]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=375, running=false, delay=3]
runnables[1]: LockRunnable[counter=51, running=false, delay=3]
runnables[2]: LockRunnable[counter=1154, running=false, delay=3]
runnables[3]: LockRunnable[counter=38, running=false, delay=3]
runnables[4]: LockRunnable[counter=1583, running=false, delay=3]
runnables[5]: LockRunnable[counter=91, running=false, delay=3]
runnables[6]: LockRunnable[counter=1, running=false, delay=3]
runnables[7]: LockRunnable[counter=1, running=false, delay=3]
runnables[8]: LockRunnable[counter=1, running=false, delay=3]
runnables[9]: LockRunnable[counter=1, running=false, delay=3]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=328, running=false, delay=3]
runnables[1]: LockRunnable[counter=328, running=false, delay=3]
runnables[2]: LockRunnable[counter=328, running=false, delay=3]
runnables[3]: LockRunnable[counter=328, running=false, delay=3]
runnables[4]: LockRunnable[counter=328, running=false, delay=3]
runnables[5]: LockRunnable[counter=328, running=false, delay=3]
runnables[6]: LockRunnable[counter=328, running=false, delay=3]
runnables[7]: LockRunnable[counter=328, running=false, delay=3]
runnables[8]: LockRunnable[counter=328, running=false, delay=3]
runnables[9]: LockRunnable[counter=328, running=false, delay=3]
##########################################

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 08:54 AM

Below are value for a delay of 1:

Environment:
delay = 1
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_07-b06-153
java.vendor = Apple Inc.
java.version = 1.6.0_07
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Mac OS X
os.version = 10.5.7
os.arch = x86_64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=1005, running=false, delay=1]
runnables[1]: SynchronizedRunnable[counter=541, running=false, delay=1]
runnables[2]: SynchronizedRunnable[counter=796, running=false, delay=1]
runnables[3]: SynchronizedRunnable[counter=1669, running=false, delay=1]
runnables[4]: SynchronizedRunnable[counter=1543, running=false, delay=1]
runnables[5]: SynchronizedRunnable[counter=638, running=false, delay=1]
runnables[6]: SynchronizedRunnable[counter=197, running=false, delay=1]
runnables[7]: SynchronizedRunnable[counter=212, running=false, delay=1]
runnables[8]: SynchronizedRunnable[counter=2587, running=false, delay=1]
runnables[9]: SynchronizedRunnable[counter=486, running=false, delay=1]

While this isn't as critical and as obvious as before, the difference between the lowest value 197 and the highest value 2587 is still very high. The values should be relatively equal, after all, especially since the only thing preventing this is, in reality, the logging system.

That's a big no-no in my books.

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 09:08 AM

Quick try at improving documentation of UnsynchronizedAppenderBase.

[ Permlink | « Hide ]

Ceki Gulcu - 09/Jun/09 10:41 AM

Thank you for these new figures and I hope you enjoy your holidays.

> Given that there are Appenders that are candidates for a delay > 3ms,
> e.g. SocketAppender, DBAppender, SMTPAppender and even FileAppender
> (especially if the log file is located on a NAS), I'd still strongly
> suggest to change AppenderBase according to my suggestion.

Only DBAppender has a latency in the order of a millisecond. SMTPAppender causes delays in the order of a millisecond only when it sends email, which is rare and thus can be ignored. SocketAppender has a throughput of over 20'000 messages per second, translating to a latency of less than 50 microseconds. FileAppender, when writing to a local file, has a throughput of over 100'000 messages per second, corresponding to a latency of less than 10 microseconds.

FileAppender writing on a networked file system is an interesting case because delays can be significantly larger. However, unless the network awfully slow, we would still be situated well below the millisecond range.

By the way, 3 ms (milliseconds) is the threshold where things go awry, at least on Linux. At 2ms and lower, the time slicer distributes resources rather fairly.

> While this isn't as critical and as obvious as before, the difference
> between the lowest value 197 and the highest value 2587 is still very
> high. The values should be relatively equal, after all, especially
> since the only thing preventing this is, in reality, the logging
> system.

Yes, the ration between 2587 and 195, that is 13.13, is rather uneven. We could reasonably begin to talk about starvation. Nevertheless, one could also argue that, if you remove the two extreme values, 195 and 2597, and if you draw a frequency distribution where the x-axis represents frequency groupings of width 300, starting at 0 ending at 1500, you get a graph looking like a nice bell curve. (I've actually done this.)

Note that if you reduce the delay Thread.sleep() to the range of 10 microseconds, which in my opinion is more representative, you will see that the numbers are distributed more smoothly. Also note that, and this is quite important, a normal application would not spend all its time logging, it would have other activities which would cause the thread last holding the "appender" lock to lose it in favor of a competing thread.

You seem to be fairly convinced that the starvation problem observed in your application is due to logging. What makes you think this? Have you observed improved behavior when logging is disabled? You could replace logback-classic.jar with slf4j-nop.jar just to test.

As for a pluggable locking mechanism, it is an interesting idea. However, I would first like to establish that there is actually a problem in logback, especially as Java itself is not a real-time language.

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 11:34 AM

It's definitely related to logging.

We haven't replaced logback-classic with slf4j-nop,yet, but we have removed all appenders and the behavior of the app went back to normal.

I'm really convinced that the problem described in this issue is the root cause since the problems started when we switched to a multi-core system.

Concerning SocketAppender, I'm wondering about the use case where it's showing less than 50 microseconds latency. Are you running it against the loopback device?

In our case it's running over a VPN so a certain slowdown is expected.

The problem seems to be that:
a) some delay is happening, either due to a slow client (SocketAppender will block if the client isn't reading fast enough), network latency, large bursts of log-messages or a combination of all that.
b) this delay sends the logged application into "starvation mode" because the critical time, be it 1ms, 3ms or 10ms was reached.

It's not acceptable that certain threads are starving because of any of the above. The app, as a whole, may slow down significantly but starvation must not happen.

This is a prime-example of starvation where multiple threads are fighting over scarce resources, in this case just one, the appender. To prevent starvation, fairness must be established.

Have you had the chance to do benchmarks about the impact of the fair ReentrantLock, yet?
If the impact is negligible than this should be changed ASAP.

Are we really in disagreement that logging mustn't impact the execution of an application in a way that can result in starvation?

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 11:49 AM

Additional info:
As I said before, this does happen with just a simple RollingFileAppender in place, too. This may be caused by some huge debug messages but this shouldn't be able to render the whole application unusable.

[ Permlink | « Hide ]

Ceki Gulcu - 09/Jun/09 12:05 PM - edited

> I'm really convinced that the problem described in this issue is the
> root cause since the problems started when we switched to a multi-core
> system.

The multi-core system may exhibit a different time-slicing behavior, which does not necessarily implicate logback. There have been many instances where users have blamed log4j/logback for race-conditions or deadlocks in their application. In most, albeit not all cases, the culprit was the application itself, logging merely exposed a pre-existing problem. I am not saying that there is no problem, but that we should not jump to conclusions.

> Concerning SocketAppender, I'm wondering about the use case where it's
> showing less than 50 microseconds latency. Are you running it against
> the loopback device?

No, just sending events to a nearby host via a 100Mb Ethernet connection. The records show an average latency of 42 microseconds. However, these records are somewhat old and the current figures may be higher.

> This is a prime-example of starvation where multiple threads are
> fighting over scarce resources, in this case just one, the
> appender. To prevent starvation, fairness must be established.

You are assuming that starvation is caused by lack of fairness. It may also be caused because the shared resource is intrinsically slow. Could it be that the VPN is the culprit?

> Have you had the chance to do benchmarks about the impact of the fair
> ReentrantLock, yet? If the impact is negligible than this should be
> changed ASAP.

No I have not done any benchmarks about the impact of the fair ReentrantLock. However, I am still not convinced that the problem is related to the fairness of the appender lock. Have you tried applying your patch in production? And if you have, what have you observed? As far as I understood, thus far, disabling SocketAppender as well as other appenders, has improved your application's performance but this does not necessarily mean that the problem is related to fair time slicing. In this regards, Dave's last comment in my blog regarding the difference between spinning and blocking is quite informative.

Let me remind you that lock fairness issue is not related to JDK 1.6. It was present in older JVMs as well. Anyway, I understand that you are under pressure to resolve this issue.

I would encourage you to apply the patch in production and kindly report back to us the results.

[ Permlink | « Hide ]

Ceki Gulcu - 09/Jun/09 12:26 PM

> As I said before, this does happen with just a simple
> RollingFileAppender in place, too. This may be caused by some huge
> debug messages but this shouldn't be able to render the whole
> application unusable.

Writing a very large log message will obviously take time. Since RollingAppender writes only one message at a time, it is natural for other threads wishing to access RollingAppender to queue, regardless of the fairness policy.

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 01:08 PM

The problem isn't that the application is slowing down but that some threads (the starving) are literally grinding to a halt while others (the greedy) keep executing as expected.

I can't apply my patch to our production system, partly because I'm not at work and partly because I'm not sure I'd get the OK to do it.

But I did test it with the example application attached to http://jira.qos.ch/browse/LBCORE-63 and it solved the issue.

I just realized that I added the result of this test to ~~LBCORE-96~~ by mistake :(
I'll just copy the comment over:

I just did a test using the above example and it looks good with the patch applied.

Without the patch, threads are executed unevenly.

2009-05-19 22:21:49,481 DEBUG [pool-2-thread-1] [MAIN] - [EX2] 54803
2009-05-19 22:21:49,481 DEBUG [pool-4-thread-1] [MAIN] - [EX4] 54803
2009-05-19 22:21:49,481 DEBUG [pool-2-thread-1] [EX2] 54803
2009-05-19 22:21:49,481 DEBUG [pool-2-thread-1] [MAIN] - [EX2] 54803
2009-05-19 22:21:49,481 DEBUG [pool-4-thread-1] [EX4] 54803
2009-05-19 22:21:49,481 DEBUG [pool-3-thread-1] [EX3] 54803
2009-05-19 22:21:49,481 DEBUG [pool-1-thread-1] [MAIN] - [EX1] 54805
2009-05-19 22:21:49,481 DEBUG [pool-5-thread-1] [EX5] 54803
2009-05-19 22:21:49,482 DEBUG [pool-5-thread-1] [MAIN] - [EX5] 54804
2009-05-19 22:21:49,481 DEBUG [pool-4-thread-1] [MAIN] - [EX4] 54803
2009-05-19 22:21:49,482 DEBUG [pool-1-thread-1] [EX1] 54806
2009-05-19 22:21:49,481 DEBUG [pool-2-thread-1] [EX2] 54803
2009-05-19 22:21:49,481 DEBUG [pool-3-thread-1] [MAIN] - [EX3] 54803
2009-05-19 22:21:49,482 DEBUG [pool-5-thread-1] [EX5] 54804
2009-05-19 22:21:49,482 DEBUG [pool-4-thread-1] [EX4] 54804
2009-05-19 22:21:49,482 DEBUG [pool-1-thread-1] [MAIN] - [EX1] 54806
2009-05-19 22:21:49,482 DEBUG [pool-2-thread-1] [MAIN] - [EX2] 54804
2009-05-19 22:21:49,482 DEBUG [pool-3-thread-1] [EX3] 54804
2009-05-19 22:21:49,482 DEBUG [pool-5-thread-1] [MAIN] - [EX5] 54804
2009-05-19 22:21:49,482 DEBUG [pool-3-thread-1] [MAIN] - [EX3] 54804
2009-05-19 22:21:49,482 DEBUG [pool-1-thread-1] [EX1] 54806
2009-05-19 22:21:49,482 DEBUG [pool-1-thread-1] [MAIN] - [EX1] 54806
2009-05-19 22:21:49,482 DEBUG [pool-4-thread-1] [MAIN] - [EX4] 54804

With the patch they are running evenly as I'd expect it.

2009-05-19 22:03:14,520 DEBUG [pool-5-thread-1] [EX5] 5900
2009-05-19 22:03:14,520 DEBUG [pool-4-thread-1] [EX4] 5925
2009-05-19 22:03:14,520 DEBUG [pool-1-thread-1] [EX1] 5936
2009-05-19 22:03:14,520 DEBUG [pool-3-thread-1] [MAIN] - [EX3] 5925
2009-05-19 22:03:14,520 DEBUG [pool-2-thread-1] [EX2] 5926
2009-05-19 22:03:14,520 DEBUG [pool-5-thread-1] [MAIN] - [EX5] 5900
2009-05-19 22:03:14,520 DEBUG [pool-4-thread-1] [MAIN] - [EX4] 5925
2009-05-19 22:03:14,521 DEBUG [pool-1-thread-1] [MAIN] - [EX1] 5937
2009-05-19 22:03:14,521 DEBUG [pool-3-thread-1] [EX3] 5926
2009-05-19 22:03:14,521 DEBUG [pool-2-thread-1] [MAIN] - [EX2] 5927
2009-05-19 22:03:14,521 DEBUG [pool-5-thread-1] [EX5] 5901
2009-05-19 22:03:14,521 DEBUG [pool-4-thread-1] [EX4] 5926
2009-05-19 22:03:14,521 DEBUG [pool-1-thread-1] [EX1] 5937

What do you think?

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 01:11 PM

The important point here is that the threads are always running in the same order 3-2-5-4-1 so no thread delays another one. But I'm repeating myself :p

[ Permlink | « Hide ]

Joern Huxhorn - 09/Jun/09 01:27 PM

It's absolutely clear that a logging-system immune to starving effects (congestion is something different) has a serious advantage over one that suffers from one.

Congestion happens if one thread writes a really big log-message. The whole system is slowed down to a certain degree in that case. That's absolutely OK.

In case of starvation caused by unfair locking, that same thread will get the chance to immediately write *another* big log-message and so on. As we've seen, taking longer does even increase the chance to immediately get the lock again.

With fair locking, all other threads that waited in line would first get the chance to execute their logging code, being able to work on, before the "chatty" thread would get another chance of taking a long time.

This could very well be the case with our app because some parts are currently very chatty, writing many and large messages, but other parts can't go on, at all, because they are starving.

[ Permlink | « Hide ]

Joern Huxhorn - 13/Jun/09 03:43 AM

- Replacing synchronized with fair ReentrantLock to prevent starvation.
- Changing guard into ThreadLocal<Boolean> and checking guard before locking. (LBCORE-99)
- Added documentation to UnsynchronizedAppenderBase to encourage the use of fair mechanisms in extending appenders.

[ Permlink | « Hide ]

Joern Huxhorn - 14/Jun/09 05:08 AM

The example we discussed on Skype.
My results will be posted in an additional comment.

[ Permlink | « Hide ]

Joern Huxhorn - 14/Jun/09 05:09 AM

Environment:
delay = 1
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.5.0_16-b06-284
java.vendor = Apple Inc.
java.version = 1.5.0_16
java.vm.name = Java HotSpot(TM) Client VM
java.vm.info = mixed mode, sharing
os.name = Mac OS X
os.version = 10.5.7
os.arch = i386
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=9050, running=false, delay=1]
runnables[1]: WastedTimeSynchronizedRunnable[counter=5533, running=false, delay=1, minWasted=0, maxWasted=2035000, totalWasted=4307631000]
runnables[2]: WastedTimeSynchronizedRunnable[counter=5564, running=false, delay=1, minWasted=0, maxWasted=3340000, totalWasted=4274103000]
runnables[3]: WastedTimeSynchronizedRunnable[counter=5499, running=false, delay=1, minWasted=0, maxWasted=2613000, totalWasted=4341686000]
runnables[4]: WastedTimeSynchronizedRunnable[counter=5541, running=false, delay=1, minWasted=0, maxWasted=2614000, totalWasted=4298552000]
runnables[5]: WastedTimeSynchronizedRunnable[counter=5580, running=false, delay=1, minWasted=0, maxWasted=2115000, totalWasted=4257398000]
runnables[6]: WastedTimeSynchronizedRunnable[counter=5543, running=false, delay=1, minWasted=0, maxWasted=3344000, totalWasted=4297607000]
runnables[7]: WastedTimeSynchronizedRunnable[counter=5581, running=false, delay=1, minWasted=0, maxWasted=3332000, totalWasted=4254483000]
runnables[8]: WastedTimeSynchronizedRunnable[counter=5570, running=false, delay=1, minWasted=0, maxWasted=2118000, totalWasted=4267368000]
runnables[9]: WastedTimeSynchronizedRunnable[counter=5538, running=false, delay=1, minWasted=0, maxWasted=2518000, totalWasted=4305538000]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=9655, running=false, delay=1]
runnables[1]: WastedTimeLockRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3177612000, totalWasted=9992771000]
runnables[2]: WastedTimeLockRunnable[counter=11, running=false, delay=1, minWasted=0, maxWasted=3175541000, totalWasted=9989755000]
runnables[3]: WastedTimeLockRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3113132000, totalWasted=9992838000]
runnables[4]: WastedTimeLockRunnable[counter=10, running=false, delay=1, minWasted=0, maxWasted=2945695000, totalWasted=9990695000]
runnables[5]: WastedTimeLockRunnable[counter=11, running=false, delay=1, minWasted=0, maxWasted=2952869000, totalWasted=9989668000]
runnables[6]: WastedTimeLockRunnable[counter=13, running=false, delay=1, minWasted=0, maxWasted=3092394000, totalWasted=9987500000]
runnables[7]: WastedTimeLockRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3750551000, totalWasted=9992601000]
runnables[8]: WastedTimeLockRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3750598000, totalWasted=9992641000]
runnables[9]: WastedTimeLockRunnable[counter=10, running=false, delay=1, minWasted=0, maxWasted=3623426000, totalWasted=9990499000]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=8756, running=false, delay=1]
runnables[1]: WastedTimeLockRunnable[counter=8691, running=false, delay=1, minWasted=0, maxWasted=1384000, totalWasted=1141561000]
runnables[2]: WastedTimeLockRunnable[counter=8691, running=false, delay=1, minWasted=0, maxWasted=1240000, totalWasted=1139965000]
runnables[3]: WastedTimeLockRunnable[counter=8688, running=false, delay=1, minWasted=0, maxWasted=1373000, totalWasted=1142592000]
runnables[4]: WastedTimeLockRunnable[counter=8695, running=false, delay=1, minWasted=0, maxWasted=1406000, totalWasted=1134775000]
runnables[5]: WastedTimeLockRunnable[counter=8690, running=false, delay=1, minWasted=0, maxWasted=1447000, totalWasted=1138803000]
runnables[6]: WastedTimeLockRunnable[counter=8687, running=false, delay=1, minWasted=0, maxWasted=2188000, totalWasted=1146588000]
runnables[7]: WastedTimeLockRunnable[counter=8698, running=false, delay=1, minWasted=0, maxWasted=1395000, totalWasted=1134743000]
runnables[8]: WastedTimeLockRunnable[counter=8688, running=false, delay=1, minWasted=0, maxWasted=1455000, totalWasted=1140360000]
runnables[9]: WastedTimeLockRunnable[counter=8687, running=false, delay=1, minWasted=0, maxWasted=1226000, totalWasted=1145801000]
##########################################

Environment:
delay = 10
java.runtime.name = Java(TM) 2 Runtime Environment, Standard Edition
java.runtime.version = 1.5.0_16-b06-284
java.vendor = Apple Inc.
java.version = 1.5.0_16
java.vm.name = Java HotSpot(TM) Client VM
java.vm.info = mixed mode, sharing
os.name = Mac OS X
os.version = 10.5.7
os.arch = i386
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=989, running=false, delay=10]
runnables[1]: WastedTimeSynchronizedRunnable[counter=600, running=false, delay=10, minWasted=0, maxWasted=10844000, totalWasted=3976076000]
runnables[2]: WastedTimeSynchronizedRunnable[counter=610, running=false, delay=10, minWasted=0, maxWasted=10850000, totalWasted=3875304000]
runnables[3]: WastedTimeSynchronizedRunnable[counter=602, running=false, delay=10, minWasted=0, maxWasted=10851000, totalWasted=3954896000]
runnables[4]: WastedTimeSynchronizedRunnable[counter=604, running=false, delay=10, minWasted=0, maxWasted=10840000, totalWasted=3935276000]
runnables[5]: WastedTimeSynchronizedRunnable[counter=602, running=false, delay=10, minWasted=0, maxWasted=10838000, totalWasted=3955131000]
runnables[6]: WastedTimeSynchronizedRunnable[counter=606, running=false, delay=10, minWasted=0, maxWasted=10863000, totalWasted=3915602000]
runnables[7]: WastedTimeSynchronizedRunnable[counter=606, running=false, delay=10, minWasted=0, maxWasted=10841000, totalWasted=3915068000]
runnables[8]: WastedTimeSynchronizedRunnable[counter=603, running=false, delay=10, minWasted=0, maxWasted=10814000, totalWasted=3944606000]
runnables[9]: WastedTimeSynchronizedRunnable[counter=603, running=false, delay=10, minWasted=0, maxWasted=10840000, totalWasted=3944719000]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=996, running=false, delay=10]
runnables[1]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=5718199000, totalWasted=9981366000]
runnables[2]: WastedTimeLockRunnable[counter=3, running=false, delay=10, minWasted=0, maxWasted=8281023000, totalWasted=9971366000]
runnables[3]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990838000, totalWasted=9990838000]
runnables[4]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990895000, totalWasted=9990895000]
runnables[5]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990802000, totalWasted=9990802000]
runnables[6]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990792000, totalWasted=9990792000]
runnables[7]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990720000, totalWasted=9990720000]
runnables[8]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990713000, totalWasted=9990713000]
runnables[9]: WastedTimeLockRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990647000, totalWasted=9990647000]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=985, running=false, delay=10]
runnables[1]: WastedTimeLockRunnable[counter=972, running=false, delay=10, minWasted=0, maxWasted=10396000, totalWasted=250077000]
runnables[2]: WastedTimeLockRunnable[counter=972, running=false, delay=10, minWasted=0, maxWasted=10184000, totalWasted=248853000]
runnables[3]: WastedTimeLockRunnable[counter=970, running=false, delay=10, minWasted=0, maxWasted=11014000, totalWasted=268824000]
runnables[4]: WastedTimeLockRunnable[counter=971, running=false, delay=10, minWasted=0, maxWasted=10564000, totalWasted=260988000]
runnables[5]: WastedTimeLockRunnable[counter=970, running=false, delay=10, minWasted=0, maxWasted=10428000, totalWasted=268175000]
runnables[6]: WastedTimeLockRunnable[counter=973, running=false, delay=10, minWasted=0, maxWasted=10295000, totalWasted=241207000]
runnables[7]: WastedTimeLockRunnable[counter=974, running=false, delay=10, minWasted=0, maxWasted=10188000, totalWasted=231224000]
runnables[8]: WastedTimeLockRunnable[counter=972, running=false, delay=10, minWasted=0, maxWasted=10314000, totalWasted=249652000]
runnables[9]: WastedTimeLockRunnable[counter=970, running=false, delay=10, minWasted=0, maxWasted=10989000, totalWasted=268618000]
##########################################

Environment:
delay = 1
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_07-b06-153
java.vendor = Apple Inc.
java.version = 1.6.0_07
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Mac OS X
os.version = 10.5.7
os.arch = x86_64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=9673, running=false, delay=1]
runnables[1]: WastedTimeSynchronizedRunnable[counter=15, running=false, delay=1, minWasted=0, maxWasted=2590599000, totalWasted=9987373000]
runnables[2]: WastedTimeSynchronizedRunnable[counter=22, running=false, delay=1, minWasted=0, maxWasted=2675914000, totalWasted=9980257000]
runnables[3]: WastedTimeSynchronizedRunnable[counter=23, running=false, delay=1, minWasted=0, maxWasted=1871833000, totalWasted=9978626000]
runnables[4]: WastedTimeSynchronizedRunnable[counter=9, running=false, delay=1, minWasted=0, maxWasted=3243744000, totalWasted=9992860000]
runnables[5]: WastedTimeSynchronizedRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3243700000, totalWasted=9993815000]
runnables[6]: WastedTimeSynchronizedRunnable[counter=8, running=false, delay=1, minWasted=0, maxWasted=3243654000, totalWasted=9993713000]
runnables[7]: WastedTimeSynchronizedRunnable[counter=12, running=false, delay=1, minWasted=0, maxWasted=2670722000, totalWasted=9989535000]
runnables[8]: WastedTimeSynchronizedRunnable[counter=9, running=false, delay=1, minWasted=0, maxWasted=2959142000, totalWasted=9992392000]
runnables[9]: WastedTimeSynchronizedRunnable[counter=13, running=false, delay=1, minWasted=0, maxWasted=2959018000, totalWasted=9988193000]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=9607, running=false, delay=1]
runnables[1]: WastedTimeLockRunnable[counter=20, running=false, delay=1, minWasted=0, maxWasted=2059909000, totalWasted=9986792000]
runnables[2]: WastedTimeLockRunnable[counter=15, running=false, delay=1, minWasted=0, maxWasted=2059904000, totalWasted=9986182000]
runnables[3]: WastedTimeLockRunnable[counter=38, running=false, delay=1, minWasted=0, maxWasted=1867779000, totalWasted=9963711000]
runnables[4]: WastedTimeLockRunnable[counter=15, running=false, delay=1, minWasted=0, maxWasted=1832521000, totalWasted=9988460000]
runnables[5]: WastedTimeLockRunnable[counter=16, running=false, delay=1, minWasted=0, maxWasted=1832510000, totalWasted=9988727000]
runnables[6]: WastedTimeLockRunnable[counter=17, running=false, delay=1, minWasted=0, maxWasted=1976649000, totalWasted=9988772000]
runnables[7]: WastedTimeLockRunnable[counter=38, running=false, delay=1, minWasted=0, maxWasted=2767358000, totalWasted=9967229000]
runnables[8]: WastedTimeLockRunnable[counter=16, running=false, delay=1, minWasted=0, maxWasted=2732013000, totalWasted=9989731000]
runnables[9]: WastedTimeLockRunnable[counter=31, running=false, delay=1, minWasted=0, maxWasted=2078606000, totalWasted=9974291000]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=8495, running=false, delay=1]
runnables[1]: WastedTimeLockRunnable[counter=8400, running=false, delay=1, minWasted=0, maxWasted=24380000, totalWasted=1290865000]
runnables[2]: WastedTimeLockRunnable[counter=8371, running=false, delay=1, minWasted=0, maxWasted=21195000, totalWasted=1304971000]
runnables[3]: WastedTimeLockRunnable[counter=8387, running=false, delay=1, minWasted=0, maxWasted=27723000, totalWasted=1282987000]
runnables[4]: WastedTimeLockRunnable[counter=8367, running=false, delay=1, minWasted=0, maxWasted=22515000, totalWasted=1304042000]
runnables[5]: WastedTimeLockRunnable[counter=8350, running=false, delay=1, minWasted=0, maxWasted=37975000, totalWasted=1333368000]
runnables[6]: WastedTimeLockRunnable[counter=8389, running=false, delay=1, minWasted=0, maxWasted=29434000, totalWasted=1294695000]
runnables[7]: WastedTimeLockRunnable[counter=8397, running=false, delay=1, minWasted=0, maxWasted=27503000, totalWasted=1271671000]
runnables[8]: WastedTimeLockRunnable[counter=8387, running=false, delay=1, minWasted=0, maxWasted=27743000, totalWasted=1293263000]
runnables[9]: WastedTimeLockRunnable[counter=8372, running=false, delay=1, minWasted=0, maxWasted=27691000, totalWasted=1295582000]
##########################################

Environment:
delay = 10
java.runtime.name = Java(TM) SE Runtime Environment
java.runtime.version = 1.6.0_07-b06-153
java.vendor = Apple Inc.
java.version = 1.6.0_07
java.vm.name = Java HotSpot(TM) 64-Bit Server VM
java.vm.info = mixed mode
os.name = Mac OS X
os.version = 10.5.7
os.arch = x86_64
##########################################
About to execute usingSynchronized...
Results for usingSynchronized:
runnables[0]: SynchronizedRunnable[counter=997, running=false, delay=10]
runnables[1]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9991803000, totalWasted=9991803000]
runnables[2]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9991705000, totalWasted=9991705000]
runnables[3]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9991312000, totalWasted=9991312000]
runnables[4]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9991104000, totalWasted=9991104000]
runnables[5]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990939000, totalWasted=9990939000]
runnables[6]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990764000, totalWasted=9990764000]
runnables[7]: WastedTimeSynchronizedRunnable[counter=1, running=false, delay=10, minWasted=0, maxWasted=9990540000, totalWasted=9990540000]
runnables[8]: WastedTimeSynchronizedRunnable[counter=11, running=false, delay=10, minWasted=0, maxWasted=7930974000, totalWasted=9890440000]
runnables[9]: WastedTimeSynchronizedRunnable[counter=6, running=false, delay=10, minWasted=0, maxWasted=7592208000, totalWasted=9940592000]
##########################################
About to execute usingUnfairLock...
Results for usingUnfairLock:
runnables[0]: LockRunnable[counter=996, running=false, delay=10]
runnables[1]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9757274000, totalWasted=9981028000]
runnables[2]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9757284000, totalWasted=9982310000]
runnables[3]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9757167000, totalWasted=9982302000]
runnables[4]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9757132000, totalWasted=9982360000]
runnables[5]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9757062000, totalWasted=9982479000]
runnables[6]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9756981000, totalWasted=9982632000]
runnables[7]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9756907000, totalWasted=9982735000]
runnables[8]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9756817000, totalWasted=9983801000]
runnables[9]: WastedTimeLockRunnable[counter=2, running=false, delay=10, minWasted=0, maxWasted=9756758000, totalWasted=9985104000]
##########################################
About to execute usingFairLock...
Results for usingFairLock:
runnables[0]: LockRunnable[counter=981, running=false, delay=10]
runnables[1]: WastedTimeLockRunnable[counter=968, running=false, delay=10, minWasted=0, maxWasted=12768000, totalWasted=282213000]
runnables[2]: WastedTimeLockRunnable[counter=964, running=false, delay=10, minWasted=0, maxWasted=12802000, totalWasted=321530000]
runnables[3]: WastedTimeLockRunnable[counter=967, running=false, delay=10, minWasted=0, maxWasted=12773000, totalWasted=293688000]
runnables[4]: WastedTimeLockRunnable[counter=962, running=false, delay=10, minWasted=0, maxWasted=11011000, totalWasted=330372000]
runnables[5]: WastedTimeLockRunnable[counter=963, running=false, delay=10, minWasted=0, maxWasted=22772000, totalWasted=330998000]
runnables[6]: WastedTimeLockRunnable[counter=968, running=false, delay=10, minWasted=0, maxWasted=10242000, totalWasted=269637000]
runnables[7]: WastedTimeLockRunnable[counter=966, running=false, delay=10, minWasted=0, maxWasted=10276000, totalWasted=290630000]
runnables[8]: WastedTimeLockRunnable[counter=963, running=false, delay=10, minWasted=0, maxWasted=22807000, totalWasted=333009000]
runnables[9]: WastedTimeLockRunnable[counter=968, running=false, delay=10, minWasted=0, maxWasted=12816000, totalWasted=282952000]
##########################################

[ Permlink | « Hide ]

Joern Huxhorn - 14/Jun/09 05:22 AM - edited

Wasted time, taking arbitrary sample:
10ms delay:
9940592000 synchronized
9985104000 unfair lock
282952000 fair lock

1ms delay:
9988193000 synchronized
9974291000 unfair lock
1295582000 fair lock

Even more relevant than the total amount of wasted time are the maxWasted values... they illustrate that a big amount of time is spent waiting in starvation instead of performing some work.

1ms delay:
2959018000 synchronized
2078606000 unfair lock
27691000 fair lock

10ms delay:
7592208000 synchronized
9756758000 unfair lock
12816000 fair lock

As can be seen, much more actual "work" can be done in case of a fair lock. Work, in this case, would be the added up counter values of all threads.

So the penalty of the fair lock isn't relevant anymore since the system, as a whole, is performing much better concerning concurrency.

This is, of course, another pretty hardcore example... Nevertheless, it mimics the effect of a slow(ish) appender pretty well, I think/hope.

[ Permlink | « Hide ]

Ceki Gulcu - 15/Jun/09 09:28 PM

Joern,

It seems to me that your latest examples shows that if the overall system is prone to starvation, then threads starve. In particular since the total amount of wasted time is often very close to that of the maxWasted value. I am still looking into this.

[ Permlink | « Hide ]

Ceki Gulcu - 15/Jun/09 10:11 PM

If you run NoLockThroughput found in the ch.qos.logback.core.issue on Sun's JDK 1.6 on Linux. You get:

=== NoLockThroughput ===
java.runtime.version = 1.6.0_11-b03
java.vendor = Sun Microsystems Inc.
java.version = 1.6.0_11
os.name = Linux
os.version = 2.6.25-gentoo-r6

The application never returns. Translation: any single line of code with "for(;;) {}" will bring your application to its knees (on Sun's JDK on a multicore 64 bit Linux).

The same test under IBM's JDK 1.6 yields:

=== NoLockThroughput ===
java.runtime.version = jvmxa6460-20081105_25433
java.vendor = IBM Corporation
java.version = 1.6.0
os.name = Linux
os.version = 2.6.25-gentoo-r6
SelectiveLockRunnable NOLOCK count=220890254
SelectiveLockRunnable NOLOCK count=244948022
SelectiveLockRunnable NOLOCK count=312806143
No lock: total of 778644419 operations, or 389322 operations per millisecond

Thus, IBMs JDK does reasonable time slicing. Perhaps Sun's does not want Java to run well on 64bit Linux. As for the fair locks, I don't believe its logback's job to fix the JDK.

[ Permlink | « Hide ]

Joern Huxhorn - 16/Jun/09 12:36 PM

Hi Ceki.

Please check if the NoLockThroughput.patch fixes behavior on Linux.

My guess is that the threads are never notified of the change of the done boolean.

http://www.cs.umd.edu/~pugh/java/memoryModel/jsr-133-faq.html

[ Permlink | « Hide ]

Ceki Gulcu - 26/Jun/09 08:43 PM

I filed a bug report on this a while ago. It just got accepted the link is:

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6855216

[ Permlink | « Hide ]

Joern Huxhorn - 26/Jun/09 10:56 PM

But this *is* a core-issue. AppenderBase is part of logback-core... I'm confused :p

[ Permlink | « Hide ]

Ceki Gulcu - 26/Jun/09 11:05 PM - edited

Yes, you are right. However, it makes life easier for me (for testing purposes) if the bug is considered to be part of logback-classic. For example, it allows me to group LBCLASSIC-139 as sub-task of LBCLASSIC-135.

I would not read too much into this re-organization.

[ Permlink | « Hide ]

Ceki Gulcu - 22/Mar/10 10:03 PM

The issue relates to how Sun's JVM does time slicing more than anything else.

[ Permlink | « Hide ]

Rü - 23/Mar/10 09:02 AM

@Ceki: Your bug report at oracle is marked as related to bug 4985566 [1] which is marked as "won't fix" for the following reason:
---------------------
When multiple threads contend for a monitor, strict alternation of the threads is "fair" (when viewed from a short time-frame), but also results in excessive context switch costs. It's usually the case that large commercial applications show higher throughput with an unfair "competive" succession policy as compared to fair alternation.

If you need truly fair locking with strict alternation then you might consider Doug Lea's dl.util.concurrent packages.
---------------------

Thread starvation is not a VM bug but a valuable feature, so maybe you should consider reopening this bug.

[1] http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4985566

[ Permlink | « Hide ]

Joern Huxhorn - 23/Mar/10 09:55 AM

Since the behavior of the Sun JVM, while extreme, is correct anyway according to the documentation, I still think that this should be fixed in Logback.

I'm not personally involved anymore because I switched my implementation over to UnsynchronizedAppenderBase + fair locks, though.

My opinion is that a logging framework simply must not cause such a starvation effect even if will perform slower because of the counter measures. This will get a bigger and bigger problem with increasing amount of CPU cores so I'm pretty sure that this will show up again, anyway.