I have 10-12 applications deployed in jboss . recently added one more application .

after adding the new appliation lot many times (not always) . the services not get UP. analysized the PID using jconsole and found the deadlock .as given below.

 

the new application has follwing logger jars  without logback.xml . My observation is added one logback.xmllogback.xml and after that its working fine.  Please suggest is logback.xml is the solution any thing else .

logback-core-1.1.11.jar 

logback-classic-1.1.11.jar

log4j-over-slf4j-1.7.25.jar

jul-to-slf4j-1.7.25.jar

jcl-over-slf4j-1.7.25.jar

slf4j-api-1.7.25.jar

 

DEADLOCK LOG AS per JConsole

1 >>>>

Name: RMI TCP Connection(1)-10.96.36.21
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@4ec68cb4 owned by: MSC service thread 1-1
Total blocked: 0 Total waited: 1

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
ch.qos.logback.classic.Logger.log(Logger.java:765)
org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.Logger.logRaw(Logger.java:850)
org.jboss.logmanager.Logger.log(Logger.java:802)
org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.java:53)
org.jboss.logging.Logger.logf(Logger.java:2431)
org.jboss.as.controller.logging.ControllerLogger_$logger.operationFailed(ControllerLogger_$logger.java:175)
org.jboss.as.controller.AbstractOperationContext.executeStep(AbstractOperationContext.java:893)
org.jboss.as.controller.AbstractOperationContext.processStages(AbstractOperationContext.java:659)
org.jboss.as.controller.AbstractOperationContext.executeOperation(AbstractOperationContext.java:370)
org.jboss.as.controller.OperationContextImpl.executeOperation(OperationContextImpl.java:1329)
org.jboss.as.controller.ModelControllerImpl.internalExecute(ModelControllerImpl.java:400)
org.jboss.as.controller.ModelControllerImpl.execute(ModelControllerImpl.java:208)
org.jboss.as.jmx.model.ResourceAccessControlUtil.getResourceAccess(ResourceAccessControlUtil.java:85)
org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:51)
org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
org.jboss.as.jmx.model.RootResourceIterator.doIterate(RootResourceIterator.java:61)
org.jboss.as.jmx.model.RootResourceIterator.iterate(RootResourceIterator.java:43)
org.jboss.as.jmx.model.ModelControllerMBeanHelper.queryNames(ModelControllerMBeanHelper.java:196)
org.jboss.as.jmx.model.ModelControllerMBeanServerPlugin.queryNames(ModelControllerMBeanServerPlugin.java:216)
org.jboss.as.jmx.PluggableMBeanServerImpl.queryNames(PluggableMBeanServerImpl.java:838)
com.sun.jmx.remote.internal.ArrayNotificationBuffer$3.run(ArrayNotificationBuffer.java:700)
com.sun.jmx.remote.internal.ArrayNotificationBuffer$3.run(ArrayNotificationBuffer.java:698)
java.security.AccessController.doPrivileged(Native Method)
com.sun.jmx.remote.internal.ArrayNotificationBuffer.queryNames(ArrayNotificationBuffer.java:704)
com.sun.jmx.remote.internal.ArrayNotificationBuffer.createListeners(ArrayNotificationBuffer.java:622)
com.sun.jmx.remote.internal.ArrayNotificationBuffer.getNotificationBuffer(ArrayNotificationBuffer.java:153)
javax.management.remote.rmi.RMIServerImpl.getNotifBuffer(RMIServerImpl.java:526)

• locked javax.management.remote.rmi.RMIJRMPServerImpl@2e65774
  javax.management.remote.rmi.RMIConnectionImpl.getServerNotifFwd(RMIConnectionImpl.java:187)
• locked javax.management.remote.rmi.RMIConnectionImpl@7e4ca4ff
  javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1504)
  javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
  javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
  javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
  javax.management.remote.rmi.RMIConnectionImpl.addNotificationListeners(RMIConnectionImpl.java:1004)
  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  java.lang.reflect.Method.invoke(Method.java:498)
  sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:324)
  sun.rmi.transport.Transport$1.run(Transport.java:200)
  sun.rmi.transport.Transport$1.run(Transport.java:197)
  java.security.AccessController.doPrivileged(Native Method)
  sun.rmi.transport.Transport.serviceCall(Transport.java:196)
  sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568)
  sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826)
  sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683)
  sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$$Lambda$108/544213880.run(Unknown Source)
  java.security.AccessController.doPrivileged(Native Method)
  sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  java.lang.Thread.run(Thread.java:745)

 

2 >>>>

 

Name: MSC service thread 1-1
State: BLOCKED on java.io.PrintStream@5f820eee owned by: ServerService Thread Pool – 61
Total blocked: 223 Total waited: 231

Stack trace:
java.io.PrintStream.write(PrintStream.java:478)
org.jboss.stdio.StdioContext$DelegatingPrintStream.write(StdioContext.java:264)
java.io.FilterOutputStream.write(FilterOutputStream.java:97)
ch.qos.logback.core.joran.spi.ConsoleTarget$1.write(ConsoleTarget.java:37)
ch.qos.logback.core.encoder.LayoutWrappingEncoder.doEncode(LayoutWrappingEncoder.java:131)
ch.qos.logback.core.OutputStreamAppender.writeOut(OutputStreamAppender.java:187)
ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:212)
ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
ch.qos.logback.classic.Logger.log(Logger.java:765)
org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.Logger.logRaw(Logger.java:850)
org.jboss.logmanager.Logger.log(Logger.java:802)
org.jboss.logging.JBossLogManagerLogger.doLogf(JBossLogManagerLogger.java:53)
org.jboss.logging.Logger.logf(Logger.java:2414)
org.jboss.as.connector.logging.ConnectorLogger_$logger.boundJca(ConnectorLogger_$logger.java:49)
org.jboss.as.connector.services.resourceadapters.deployment.AbstractResourceAdapterDeploymentService$AbstractWildFlyRaDeployer$1.transition(AbstractResourceAdapterDeploymentService.java:432)
org.jboss.msc.service.ServiceControllerImpl.invokeListener(ServiceControllerImpl.java:1533)
org.jboss.msc.service.ServiceControllerImpl.access$2800(ServiceControllerImpl.java:51)
org.jboss.msc.service.ServiceControllerImpl$ListenerTask.run(ServiceControllerImpl.java:2099)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
java.lang.Thread.run(Thread.java:745)

 

3 >>>>

Name: ServerService Thread Pool – 61
State: WAITING on java.util.concurrent.locks.ReentrantLock$NonfairSync@4ec68cb4 owned by: MSC service thread 1-1
Total blocked: 2 Total waited: 4

Stack trace:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:209)
java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:285)
ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:210)
ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:100)
ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
ch.qos.logback.classic.Logger.log(Logger.java:765)
org.slf4j.bridge.SLF4JBridgeHandler.callLocationAwareLogger(SLF4JBridgeHandler.java:221)
org.slf4j.bridge.SLF4JBridgeHandler.publish(SLF4JBridgeHandler.java:303)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:314)
org.jboss.logmanager.LoggerNode.publish(LoggerNode.java:322)
org.jboss.logmanager.Logger.logRaw(Logger.java:850)
org.jboss.logmanager.Logger.log(Logger.java:596)
org.jboss.stdio.AbstractLoggingWriter.write(AbstractLoggingWriter.java:71)

• locked java.lang.StringBuilder@43132c5b
  org.jboss.stdio.WriterOutputStream.finish(WriterOutputStream.java:143)
  org.jboss.stdio.WriterOutputStream.flush(WriterOutputStream.java:164)
• locked sun.nio.cs.SingleByte$Decoder@712fd96d
  java.io.PrintStream.write(PrintStream.java:482)
• locked java.io.PrintStream@5f820eee
  sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
  sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:291)
  sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:104)
• locked java.io.OutputStreamWriter@6d0596de
  java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:185)
  java.io.PrintStream.newLine(PrintStream.java:546)
• locked java.io.PrintStream@5f820eee
  java.io.PrintStream.println(PrintStream.java:807)
• locked java.io.PrintStream@5f820eee
  org.jboss.stdio.StdioContext$DelegatingPrintStream.println(StdioContext.java:474)
  org.springframework.boot.SpringBootBanner.printBanner(SpringBootBanner.java:49)
  org.springframework.boot.SpringApplicationBannerPrinter.print(SpringApplicationBannerPrinter.java:71)
  org.springframework.boot.SpringApplication.printBanner(SpringApplication.java:505)
  org.springframework.boot.SpringApplication.run(SpringApplication.java:298)
  org.springframework.boot.web.support.SpringBootServletInitializer.run(SpringBootServletInitializer.java:151)
  org.springframework.boot.web.support.SpringBootServletInitializer.createRootApplicationContext(SpringBootServletInitializer.java:131)
  org.springframework.boot.web.support.SpringBootServletInitializer.onStartup(SpringBootServletInitializer.java:86)
  org.springframework.web.SpringServletContainerInitializer.onStartup(SpringServletContainerInitializer.java:169)
  io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:186)
  io.undertow.servlet.core.DeploymentManagerImpl$1.call(DeploymentManagerImpl.java:171)
  io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:42)
  io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
  io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
  io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
  io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
  io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
  io.undertow.servlet.api.LegacyThreadSetupActionWrapper$1.call(LegacyThreadSetupActionWrapper.java:44)
  io.undertow.servlet.core.DeploymentManagerImpl.deploy(DeploymentManagerImpl.java:234)
  org.wildfly.extension.undertow.deployment.UndertowDeploymentService.startContext(UndertowDeploymentService.java:100)
  org.wildfly.extension.undertow.deployment.UndertowDeploymentService$1.run(UndertowDeploymentService.java:82)
  java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
  java.util.concurrent.FutureTask.run(FutureTask.java:266)
  java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  java.lang.Thread.run(Thread.java:745)
  org.jboss.threads.JBossThread.run(JBossThread.java:320)