WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39469

konnik · 2024-02-08T14:18:14Z

Description of the problem

Incoming http-requests during graceful shutdown throws exceptions and 500 responses to the client.
Expected behavior is "connection refused" or 503 with no exceptions logged.

I'm using:

Spring Boot 3.2.2
Spring Webflux
Kotlin

Example program

The application is configured with:

server.shutdown=graceful
management.endpoint.health.probes.enabled=true
management.health.livenessState.enabled=true
management.health.readinessState.enabled=true

Application code:

@SpringBootApplication
class GracefulShutdownLabbApplication

fun main(args: Array<String>) {
    runApplication<GracefulShutdownLabbApplication>(*args)
}


@RestController
class MyController {

    @GetMapping("/hello")
    suspend fun hello(): String {
        println("Enter /hello")
        delay(1.seconds)
        println("Leave /hello")
        return "Hello my friend, the current time is " + LocalTime.now()
    }
}

/**
 * Simulate a component with a slow shutdown.
 */
@Component
class MySlowComponent(private var running: Boolean = false) : SmartLifecycle {
    override fun start() {
        running = true
    }

    override fun stop() {
        println("5 seconds delay before stopping MySlowComponent..")
        Thread.sleep(5000)
        println("Stopping MySlowComponent!")
        running = false
    }

    override fun isRunning(): Boolean {
        return running
    }
}

Reproduce

Start application
Generate traffic watch -n 0.5 curl -s -v http://localhost:8080/hello
Stop application with SIGINT

The application responds with 500 internal server errors and logs exceptions.

Logs:

2024-02-08T14:59:59.361+01:00  INFO 3484426 --- [  restartedMain] n.g.GracefulShutdownLabbApplicationKt    : Starting GracefulShutdownLabbApplicationKt using Java 21 with PID 3484426 (/home/nikkon/devel/graceful-shutdown-labb/build/classes/kotlin/main started by nikkon in /home/nikkon/devel/graceful-shutdown-labb)
2024-02-08T14:59:59.363+01:00  INFO 3484426 --- [  restartedMain] n.g.GracefulShutdownLabbApplicationKt    : No active profile set, falling back to 1 default profile: "default"
2024-02-08T14:59:59.397+01:00  INFO 3484426 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : Devtools property defaults active! Set 'spring.devtools.add-properties' to 'false' to disable
2024-02-08T14:59:59.397+01:00  INFO 3484426 --- [  restartedMain] .e.DevToolsPropertyDefaultsPostProcessor : For additional web related logging consider setting the 'logging.level.web' property to 'DEBUG'
2024-02-08T15:00:00.216+01:00  INFO 3484426 --- [  restartedMain] o.s.b.a.e.web.EndpointLinksResolver      : Exposing 1 endpoint(s) beneath base path '/actuator'
2024-02-08T15:00:00.375+01:00  INFO 3484426 --- [  restartedMain] o.s.b.d.a.OptionalLiveReloadServer       : LiveReload server is running on port 35729
2024-02-08T15:00:00.445+01:00  INFO 3484426 --- [  restartedMain] o.s.b.web.embedded.netty.NettyWebServer  : Netty started on port 8080
2024-02-08T15:00:00.452+01:00  INFO 3484426 --- [  restartedMain] n.g.GracefulShutdownLabbApplicationKt    : Started GracefulShutdownLabbApplicationKt in 1.292 seconds (process running for 1.621)
Enter /hello
Leave /hello
Enter /hello
Leave /hello
Enter /hello
Leave /hello
Enter /hello
Leave /hello
Enter /hello
Leave /hello
Enter /hello
Leave /hello
5 seconds delay before stopping MySlowComponent..
Exception in thread "reactor-http-epoll-2" org.springframework.core.task.TaskRejectedException: ExecutorService in shutdown state did not accept task: false
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:365)
	at reactor.core.scheduler.ExecutorScheduler$ExecutorSchedulerWorker.schedule(ExecutorScheduler.java:252)
	at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63)
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165)
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79)
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74)
	at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259)
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180)
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571)
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139)
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339)
	at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367)
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241)
	at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70)
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201)
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53)
	at reactor.core.publisher.Mono.subscribe(Mono.java:4512)
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265)
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51)
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76)
	at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55)
	at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1169)
	at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710)
	at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481)
	at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:652)
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:238)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436)
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318)
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509)
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.util.concurrent.RejectedExecutionException: Task false rejected from org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1@354c4a7d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2081)
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376)
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1.execute(ThreadPoolTaskExecutor.java:269)
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:362)
	... 60 more
2024-02-08T15:00:09.761+01:00 ERROR 3484426 --- [or-http-epoll-2] a.w.r.e.AbstractErrorWebExceptionHandler : [2e34d503-49]  500 Server Error for HTTP GET "/hello"

reactor.core.Exceptions$ReactorRejectedExecutionException: Scheduler unavailable
	at reactor.core.Exceptions.failWithRejected(Exceptions.java:283) ~[reactor-core-3.6.2.jar:3.6.2]
	Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException: 
Error has been observed at the following site(s):
	*__checkpoint ⇢ HTTP GET "/hello" [ExceptionHandlingWebHandler]
Original Stack Trace:
		at reactor.core.Exceptions.failWithRejected(Exceptions.java:283) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.scheduler.ExecutorScheduler$ExecutorSchedulerWorker.schedule(ExecutorScheduler.java:257) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.Mono.subscribe(Mono.java:4512) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.6.2.jar:3.6.2]
		at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1169) ~[reactor-netty-http-1.1.15.jar:1.1.15]
		at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710) ~[reactor-netty-core-1.1.15.jar:1.1.15]
		at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481) ~[reactor-netty-core-1.1.15.jar:1.1.15]
		at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:652) ~[reactor-netty-http-1.1.15.jar:1.1.15]
		at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114) ~[reactor-netty-core-1.1.15.jar:1.1.15]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:238) ~[reactor-netty-http-1.1.15.jar:1.1.15]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
		at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
		at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: org.springframework.core.task.TaskRejectedException: ExecutorService in shutdown state did not accept task: false
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:365) ~[spring-context-6.1.3.jar:6.1.3]
	at reactor.core.scheduler.ExecutorScheduler$ExecutorSchedulerWorker.schedule(ExecutorScheduler.java:252) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoSubscribeOn.subscribeOrReturn(MonoSubscribeOn.java:55) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:63) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:165) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onNext(FluxOnErrorResume.java:79) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxSwitchIfEmpty$SwitchIfEmptySubscriber.onNext(FluxSwitchIfEmpty.java:74) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoNext$NextSubscriber.onNext(MonoNext.java:82) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.innerNext(FluxConcatMapNoPrefetch.java:259) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxConcatMap$ConcatMapInner.onNext(FluxConcatMap.java:865) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.onNext(FluxMapFuseable.java:129) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.onNext(MonoPeekTerminal.java:180) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoPeekTerminal$MonoTerminalPeekSubscriber.request(MonoPeekTerminal.java:139) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxMapFuseable$MapFuseableSubscriber.request(FluxMapFuseable.java:171) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.request(Operators.java:2331) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.request(FluxConcatMapNoPrefetch.java:339) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoNext$NextSubscriber.request(MonoNext.java:108) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.onSubscribe(Operators.java:2241) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoNext$NextSubscriber.onSubscribe(MonoNext.java:70) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxConcatMapNoPrefetch$FluxConcatMapNoPrefetchSubscriber.onSubscribe(FluxConcatMapNoPrefetch.java:164) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:201) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.FluxIterable.subscribe(FluxIterable.java:83) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoDefer.subscribe(MonoDefer.java:53) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.Mono.subscribe(Mono.java:4512) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoIgnoreThen$ThenIgnoreMain.subscribeNext(MonoIgnoreThen.java:265) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoIgnoreThen.subscribe(MonoIgnoreThen.java:51) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.InternalMonoOperator.subscribe(InternalMonoOperator.java:76) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.core.publisher.MonoDeferContextual.subscribe(MonoDeferContextual.java:55) ~[reactor-core-3.6.2.jar:3.6.2]
	at reactor.netty.http.server.HttpServer$HttpServerHandle.onStateChange(HttpServer.java:1169) ~[reactor-netty-http-1.1.15.jar:1.1.15]
	at reactor.netty.ReactorNetty$CompositeConnectionObserver.onStateChange(ReactorNetty.java:710) ~[reactor-netty-core-1.1.15.jar:1.1.15]
	at reactor.netty.transport.ServerTransport$ChildObserver.onStateChange(ServerTransport.java:481) ~[reactor-netty-core-1.1.15.jar:1.1.15]
	at reactor.netty.http.server.HttpServerOperations.onInboundNext(HttpServerOperations.java:652) ~[reactor-netty-http-1.1.15.jar:1.1.15]
	at reactor.netty.channel.ChannelOperationsHandler.channelRead(ChannelOperationsHandler.java:114) ~[reactor-netty-core-1.1.15.jar:1.1.15]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at reactor.netty.http.server.HttpTrafficHandler.channelRead(HttpTrafficHandler.java:238) ~[reactor-netty-http-1.1.15.jar:1.1.15]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) ~[netty-codec-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) ~[netty-codec-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[netty-transport-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:509) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:407) ~[netty-transport-classes-epoll-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-common-4.1.105.Final.jar:4.1.105.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: java.util.concurrent.RejectedExecutionException: Task false rejected from org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1@354c4a7d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 6]
	at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2081) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841) ~[na:na]
	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1376) ~[na:na]
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor$1.execute(ThreadPoolTaskExecutor.java:269) ~[spring-context-6.1.3.jar:6.1.3]
	at org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor.execute(ThreadPoolTaskExecutor.java:362) ~[spring-context-6.1.3.jar:6.1.3]
	... 60 common frames omitted

<repeated errors removed> 

Stopping MySlowComponent!
2024-02-08T15:00:14.265+01:00  INFO 3484426 --- [ionShutdownHook] o.s.b.w.embedded.netty.GracefulShutdown  : Commencing graceful shutdown. Waiting for active requests to complete
2024-02-08T15:00:14.266+01:00  INFO 3484426 --- [ netty-shutdown] o.s.b.w.embedded.netty.GracefulShutdown  : Graceful shutdown complete

Process finished with exit code 130 (interrupted by signal 2:SIGINT)

The text was updated successfully, but these errors were encountered:

wilkinsona · 2024-02-08T15:47:50Z

It appears that ThreadPoolTaskExecutor$1@354c4a7d has been stopped before graceful shutdown has begun. I think this is due to this change in Spring Framework where the executor will start rejecting tasks when it receives the ContextClosedEvent. This event is published before graceful shutdown begins so the executor is stopped before requests will be rejected. Even without graceful shutdown, there is still a window where this problem could occur as the executor will be stopped before the HTTP connector is stopped. We'll discuss it with the Framework team.

wilkinsona · 2024-02-09T08:11:42Z

There are a few things going on here that are contributing to the problem:

Framework is rejecting tasks as soon as the context has been closed (ThreadPoolTaskExecutor rejecting tasks as soon as the context has been closed can lead to exceptions during web app shutdown spring-framework#32226)
Framework is treating your coroutine as blocking so it's unnecessarily being dispatched onto a separate thread (Add Coroutines support to NonReactiveHandlerMethodPredicate spring-framework#32227)
Boot is configuring Framework to use the application's task executor for blocking operations in all cases when the intent was that this would only happen when virtual threads are enabled

Any one of these should fix the problem. We'll use this issue to address 3.

…ot-starter-parent to v3.2.3 This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [org.springframework.boot:spring-boot-starter-parent](https://spring.io/projects/spring-boot) ([source](https://github.com/spring-projects/spring-boot)) | parent | patch | `3.2.2` -> `3.2.3` | --- ### ⚠ Dependency Lookup Warnings ⚠ Warnings were logged while processing this repo. Please check the logs for more information. --- ### Release Notes <details> <summary>spring-projects/spring-boot (org.springframework.boot:spring-boot-starter-parent)</summary> ### [`v3.2.3`](https://github.com/spring-projects/spring-boot/releases/tag/v3.2.3) [Compare Source](spring-projects/spring-boot@v3.2.2...v3.2.3) #### ⚠️ Noteworthy - This release upgrades to Hibernate 6.4.4.Final. While it contains a number of valuable bug fixes, it does not work correctly in a native image. It you are using GraalVM, Hibernate should be temporarily downgraded to 6.4.2.Final using the `hibernate.version` property. #### 🐞 Bug Fixes - Nested jar URLs can not be resolved if the path contains spaces [#39675](spring-projects/spring-boot#39675) - Image building runs for a long time when a long image name is used and the tag contains an illegal character [#39638](spring-projects/spring-boot#39638) - Banner printing doesn't respect set charset [#39621](spring-projects/spring-boot#39621) - "micrometer.observations.\*" configuration properties should be "management.observations.\*" [#39600](spring-projects/spring-boot#39600) - Metadata reading during configuration class parsing uses the default resource loader rather than the application's resource loader [#39598](spring-projects/spring-boot#39598) - Several gson properties, including spring.gson.disable-html-escaping, do not behave correctly when set to false [#39524](spring-projects/spring-boot#39524) - Property placeholders aren't resolved when configuration property binding creates a Map from a property value using a converter [#39515](spring-projects/spring-boot#39515) - Gradle plugin allows the use of Gradle 7.4 but the documented and tested minimum is 7.5 [#39513](spring-projects/spring-boot#39513) - WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled [#39469](spring-projects/spring-boot#39469) - TestcontainersPropertySource assertion has typo [#39449](spring-projects/spring-boot#39449) - Webflux actuator endpoints respond with 500 when a parameter is missing [#39444](spring-projects/spring-boot#39444) - NoSuchMethod error when using the non-shaded Pu...

konnik changed the title ~~ReactorRejectedExecutionException: Scheduler unavailable during graceful shutdown~~ "ReactorRejectedExecutionException: Scheduler unavailable" when handling requests during graceful shutdown Feb 8, 2024

spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged label Feb 8, 2024

wilkinsona mentioned this issue Feb 8, 2024

ThreadPoolTaskExecutor rejecting tasks as soon as the context has been closed can lead to exceptions during web app shutdown spring-projects/spring-framework#32226

Closed

wilkinsona changed the title ~~"ReactorRejectedExecutionException: Scheduler unavailable" when handling requests during graceful shutdown~~ WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled Feb 9, 2024

wilkinsona added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged labels Feb 9, 2024

wilkinsona added this to the 3.2.x milestone Feb 9, 2024

emindeniz99 mentioned this issue Feb 9, 2024

Redis Connection Factory Earlier Graceful Shutdown after Spring 3.2 Upgrade #39501

Closed

wilkinsona self-assigned this Feb 9, 2024

emindeniz99 mentioned this issue Feb 12, 2024

Redis Connection Factory Earlier Graceful Shutdown after Spring 3.2 Upgrade - LettuceConnectionFactory SmartLifecycle Backward Compatibility spring-projects/spring-data-redis#2843

Closed

wilkinsona mentioned this issue Feb 14, 2024

WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39542

Closed

wilkinsona modified the milestones: 3.2.x, 3.2.3 Feb 14, 2024

wilkinsona closed this as completed in 256f9fe Feb 14, 2024

Netyyyy mentioned this issue Feb 29, 2024

External dependencies upgrade - Spring Boot 3.2.3 and Spring Cloud 2023.0.0 Azure/azure-sdk-for-java#39017

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39469

WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39469

konnik commented Feb 8, 2024

wilkinsona commented Feb 8, 2024

wilkinsona commented Feb 9, 2024

WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39469

WebFlux auto-configuration should only configure the blocking executor when virtual threads are enabled #39469

Comments

konnik commented Feb 8, 2024

Description of the problem

Example program

Reproduce

wilkinsona commented Feb 8, 2024

wilkinsona commented Feb 9, 2024