Deadlock in SubProtocolWebSocketHandler on shutdown with Undertow [SPR-16488] #21031
Comments
|
Mathieu Carbou commented Note: switching to next Spring Boot version (1.5.9) which depends on Spring FW 4.3.13 might not fix the issue. |
|
Juergen Hoeller commented It turns out that our proactive notifications of all active WebSocket sessions on shutdown shouldn't happen within the lifecycle lock there since they could interfere with a write lock held by the underlying engine (Undertow in this case). We still trigger them within |
|
Juergen Hoeller commented This fix is available in the latest |
|
Mathieu Carbou commented This issue is not fixed. Here is a stack below showing the blocked thread: "XNIO-2 I/O-14" #58 prio=5 os_prio=0 tid=0x00007f700460c000 nid=0xf6fe waiting for monitor entry [0x00007f70a2eec000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.isRunning(SubProtocolWebSocketHandler.java:286)
- waiting to lock <0x00000005c1e97308> (a java.lang.Object)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.checkSessions(SubProtocolWebSocketHandler.java:462)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:318)more details: 2018-03-15 02:39:34
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.162-b12 mixed mode):
"Attach Listener" #7618 daemon prio=9 os_prio=0 tid=0x00007f7058002000 nid=0xd5da waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"SIGTERM handler" #965 daemon prio=9 os_prio=0 tid=0x00007f7058004000 nid=0x32ed waiting for monitor entry [0x00007f7091edd000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.Shutdown.exit(Shutdown.java:212)
- waiting to lock <0x00000000806fc660> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
"Thread-13" #78 prio=5 os_prio=0 tid=0x00007f6db0001000 nid=0x10090 in Object.wait() [0x00007f70924e2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005c2243668> (a java.lang.Object)
at java.lang.Object.wait(Object.java:502)
at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:281)
- locked <0x00000005c2243668> (a java.lang.Object)
at org.xnio.channels.Channels.flushBlocking(Channels.java:64)
at io.undertow.websockets.jsr.WebSocketSessionRemoteEndpoint$BasicWebSocketSessionRemoteEndpoint.sendText(WebSocketSessionRemoteEndpoint.java:287)
at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:216)
at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:100)
at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.writeFrameInternal(WebSocketServerSockJsSession.java:224)
at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.close(AbstractSockJsSession.java:210)
at org.springframework.web.socket.handler.WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:160)
at org.springframework.web.socket.handler.ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:213)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.stop(SubProtocolWebSocketHandler.java:265)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.stop(SubProtocolWebSocketHandler.java:278)
- locked <0x00000005c1e97308> (a java.lang.Object)
at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:236)
at org.springframework.context.support.DefaultLifecycleProcessor.access$300(DefaultLifecycleProcessor.java:52)
at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:373)
at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:209)
at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:127)
at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1005)
at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:933)
- locked <0x000000008021d4e8> (a java.lang.Object)
"SIGTERM handler" #215 daemon prio=9 os_prio=0 tid=0x00007f7058003800 nid=0x1008f in Object.wait() [0x00007f70925e4000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000005c1e96c68> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1252)
- locked <0x00000005c1e96c68> (a org.springframework.context.support.AbstractApplicationContext$1)
at java.lang.Thread.join(Thread.java:1326)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000000806fc660> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:748)
"XNIO-2 I/O-14" #58 prio=5 os_prio=0 tid=0x00007f700460c000 nid=0xf6fe waiting for monitor entry [0x00007f70a2eec000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.isRunning(SubProtocolWebSocketHandler.java:286)
- waiting to lock <0x00000005c1e97308> (a java.lang.Object)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.checkSessions(SubProtocolWebSocketHandler.java:462)
at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:318)
at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleMessage(WebSocketHandlerDecorator.java:75)
at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleMessage(LoggingWebSocketHandlerDecorator.java:56)
at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleMessage(ExceptionWebSocketHandlerDecorator.java:58)
at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.delegateMessages(AbstractSockJsSession.java:386)
at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.handleMessage(WebSocketServerSockJsSession.java:195)
at org.springframework.web.socket.sockjs.transport.handler.SockJsWebSocketHandler.handleTextMessage(SockJsWebSocketHandler.java:93)
at org.springframework.web.socket.handler.AbstractWebSocketHandler.handleMessage(AbstractWebSocketHandler.java:43)
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.handleTextMessage(StandardWebSocketHandlerAdapter.java:113)
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.access$000(StandardWebSocketHandlerAdapter.java:42)
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:84)
at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:81)
at io.undertow.websockets.jsr.FrameHandler$7.run(FrameHandler.java:286)
at io.undertow.websockets.jsr.ServerWebSocketContainer$1.call(ServerWebSocketContainer.java:162)
at io.undertow.websockets.jsr.ServerWebSocketContainer$1.call(ServerWebSocketContainer.java:159)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.websockets.jsr.ServerWebSocketContainer.invokeEndpointMethod(ServerWebSocketContainer.java:575)
at io.undertow.websockets.jsr.ServerWebSocketContainer.invokeEndpointMethod(ServerWebSocketContainer.java:565)
at io.undertow.websockets.jsr.FrameHandler.invokeTextHandler(FrameHandler.java:266)
at io.undertow.websockets.jsr.FrameHandler.onFullTextMessage(FrameHandler.java:317)
at io.undertow.websockets.core.AbstractReceiveListener$2.complete(AbstractReceiveListener.java:156)
at io.undertow.websockets.core.AbstractReceiveListener$2.complete(AbstractReceiveListener.java:152)
at io.undertow.websockets.core.BufferedTextMessage.read(BufferedTextMessage.java:105)
at io.undertow.websockets.core.AbstractReceiveListener.readBufferedText(AbstractReceiveListener.java:152)
at io.undertow.websockets.core.AbstractReceiveListener.bufferFullMessage(AbstractReceiveListener.java:90)
at io.undertow.websockets.jsr.FrameHandler.onText(FrameHandler.java:182)
at io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:44)
at io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:33)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:942)
at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:923)
at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:88)
at org.xnio.nio.WorkerThread.run(WorkerThread.java:561)
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007f71380ac000 nid=0xf187 in Object.wait() [0x00007f70b22e1000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000080209418> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
- locked <0x0000000080209418> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:212)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007f71380a7800 nid=0xf186 in Object.wait() [0x00007f70b23e2000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000080209648> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153) |
|
Juergen Hoeller commented It looks like our enforced lock in I'll let you know once this is available in a snapshot. Please prepare for giving it a try before the 5.0.5 / 4.3.15 release, making sure that the issue is gone for good now. |
|
Juergen Hoeller commented
|
|
Mathieu Carbou commented Hi, UPDATE: |
|
Juergen Hoeller commented Alright, I'm considering this one as resolved since I don't see any remaining deadlock potential that we could address on our end. Your thread dump indicates that Undertow itself possibly uses several locks within its processing arrangement which implies some deadlock potential on that end as well, but that's outside of our powers. Please note: #21137 has yet to be backported to the 4.3.x branch; I'll include it in my next round of backports early next week. So for the time being, the above-mentioned refinement is only available in 5.0.5 snapshots. |
|
Mathieu Carbou commented Hi, |
|
Juergen Hoeller commented Thanks for the feedback! That's great to hear. |
spring-projects-issues
added
type: bug
Mathieu Carbou opened SPR-16488 and commented
Probably affects also 4.3.13 because the change log does not seem to fix this issue (https://jira.spring.io/browse/SPR/fixforversion/16354/?selectedTab=com.atlassian.jira.plugins.jira-development-integration-plugin:release-report-tabpanel).
We are using:
Thread dump extract:
Affects: 4.3.12
Issue Links:
Backported to: 4.3.15
The text was updated successfully, but these errors were encountered: