这是具体报错的堆栈信息，这种错误有没有办法避免 2022-10-10 17:28:42.890 [RID:] [traceId:] [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR c.a.nacos.common.notify.NotifyCenter [DefaultPublisher.java:210] Event callback exception: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@3bb61a6e rejected from java.util.concurrent.ScheduledThreadPoolExecutor@52260039[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 31] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.dubbo.registry.RegistryNotifier.notify(RegistryNotifier.java:77) at org.apache.dubbo.registry.nacos.NacosRegistry$RegistryChildListenerImpl.onEvent(NacosRegistry.java:715) at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:124) at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:40) at com.alibaba.nacos.common.notify.DefaultPublisher.lambda$notifySubscriber$0(DefaultPublisher.java:201) at com.alibaba.nacos.common.notify.DefaultPublisher.notifySubscriber(DefaultPublisher.java:208) at com.alibaba.nacos.common.notify.DefaultPublisher.receiveEvent(DefaultPublisher.java:192) at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:118) at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)
[apache/dubbo]dubbo 使用 QOS 下线服务会偶现 InstancesChangeEvent callback exception
回答
这个报错是在 offline 的过程中出现的还是进程 shutdown 的过程中出现的。另外使用的是 qos 的下线方式是通过 offline 命令还是 shutdown 命令。
这个报错是在 offline 的过程中出现的还是进程 shutdown 的过程中出现的。另外使用的是 qos 的下线方式是通过
offline命令还是shutdown命令。
用的 offline 命令，应该是在执行 SpringApplicationShutdownHook 过程中出现的，补充一下上下文日志
2022-10-10 17:28:26.145 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.r.GlobalResourcesRepository [GlobalResourcesRepository.java:120] [DUBBO] Dubbo is completely destroyed, dubbo version: 3.0.11, current host: 172.xx.xxx.xx
2022-10-10 17:28:26.146 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.t.m.FrameworkExecutorRepository [FrameworkExecutorRepository.java:175] [DUBBO] destroying framework executor repository .., dubbo version: 3.0.11, current host: 172.xx.xxx.xx
2022-10-10 17:28:26.147 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO o.a.d.c.s.c.DubboSpringInitializer [DubboSpringInitializer.java:78] [DUBBO] Unbind Dubbo Module[1.1.1] from spring container: org.springframework.beans.factory.support.DefaultListableBeanFactory@5c83ae01, dubbo version: 3.0.11, current host: 172.xx.xxx.xx [INFO] [10/10/2022 17:28:26.251] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-88] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Shutting down remote daemon. [INFO] [10/10/2022 17:28:26.254] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-88] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Remote daemon shut down; proceeding with flushing remote transports. [INFO] [10/10/2022 17:28:26.302] [172018251086_1_79792-akka.actor.default-dispatcher-303] [akka.remote.Remoting] Remoting shut down [INFO] [10/10/2022 17:28:26.304] [172018251086_1_79792-akka.remote.remote-dispatcher-thread8-16] [akka.tcp://172018251086_1_79792@172.18.251.86:40127/system/remoting-terminator] Remoting shut down.
2022-10-10 17:28:42.890 [RID:] [traceId:] [nacos.publisher-com.alibaba.nacos.client.naming.event.InstancesChangeEvent] ERROR c.a.nacos.common.notify.NotifyCenter [DefaultPublisher.java:210] Event callback exception: java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@197de0ca rejected from java.util.concurrent.ScheduledThreadPoolExecutor@5ba4f606[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 21] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:326) at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:533) at java.util.concurrent.ScheduledThreadPoolExecutor.submit(ScheduledThreadPoolExecutor.java:632) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.dubbo.registry.RegistryNotifier.notify(RegistryNotifier.java:77) at org.apache.dubbo.registry.nacos.NacosRegistry$RegistryChildListenerImpl.onEvent(NacosRegistry.java:715) at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:124) at com.alibaba.nacos.client.naming.event.InstancesChangeNotifier.onEvent(InstancesChangeNotifier.java:40) at com.alibaba.nacos.common.notify.DefaultPublisher.lambda$notifySubscriber$0(DefaultPublisher.java:201) at com.alibaba.nacos.common.notify.DefaultPublisher.notifySubscriber(DefaultPublisher.java:208) at com.alibaba.nacos.common.notify.DefaultPublisher.receiveEvent(DefaultPublisher.java:192) at com.alibaba.nacos.common.notify.DefaultPublisher.openEventHandler(DefaultPublisher.java:118) at com.alibaba.nacos.common.notify.DefaultPublisher.run(DefaultPublisher.java:95)
2022-10-10 17:28:58.314 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.b.d.d.DynamicRoutingDataSource [DynamicRoutingDataSource.java:205] dynamic-datasource start closing ....
2022-10-10 17:28:58.318 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.alibaba.druid.pool.DruidDataSource [DruidDataSource.java:2029] {dataSource-3} closing ...
2022-10-10 17:28:58.321 [RID:] [traceId:] [SpringApplicationShutdownHook] INFO c.alibaba.druid.pool.DruidDataSource [DruidDataSource.java:2101] {dataSource-3} closed
这个问题复现频率有多高，看起来是 Dubbo 已经反订阅后销毁了线程池了，Nacos 还是推送了数据
频率的话蛮高的，4个pod滚动部署有2个pod会出现这个报错，理论上 offline 指令发送后应该不会推送过来才对，毕竟我看到 dubbo 入口的流量已经没有了
这个在本地能搭个复现的场景嘛
本地的话不太好模拟滚动部署，还有没有别的方式或者你这边还需要啥额外的信息
能不能把从 shutdownhook 开始触发的到结束的所有日志发一下，数据多的话也可以直接发我邮箱
已经发送，请查收，部分敏感信息手动做了处理