[xuxueli/xxl-job]内存溢出,运行一个月左右,会内存溢出并重启!

2023-12-16 299 views
0
您使用的是 XXL-JOB 的哪个版本?

v2.1.0

预期行为

正常运行

实际行为

运行一段时间后,会发生内存溢出,并重新启动容器

重现行为的步骤

报错日志如下:

Caused by: java.lang.OutOfMemoryError: Java heap space 21:04:52.545 logback [http-nio-80-exec-33] ERROR c.x.r.r.p.XxlRpcProviderFactory - xxl-rpc provider invokeService error. java.lang.reflect.InvocationTargetException: null at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.xxl.rpc.remoting.provider.XxlRpcProviderFactory.invokeService(XxlRpcProviderFactory.java:219) at com.xxl.job.admin.core.conf.ServletServerHandler1.handle(ServletServerHandler1.java:87) at com.xxl.job.admin.core.conf.XxlJobScheduler.invokeAdminService(XxlJobScheduler.java:122) at com.xxl.job.admin.controller.JobApiController.api(JobApiController.java:41) at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765) at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HttpPutFormContentFilter.doFilterInternal(HttpPutFormContentFilter.java:109) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:197) at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343) at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.OutOfMemoryError: Java heap space at sun.nio.ch.EPollArrayWrapper.<init>(EPollArrayWrapper.java:120) at sun.nio.ch.EPollSelectorImpl.<init>(EPollSelectorImpl.java:69) at sun.nio.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:36) at io.netty.channel.nio.NioEventLoop.openSelector(NioEventLoop.java:167) at io.netty.channel.nio.NioEventLoop.<init>(NioEventLoop.java:143) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:127) at io.netty.channel.nio.NioEventLoopGroup.newChild(NioEventLoopGroup.java:36) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:84) at io.netty.util.concurrent.MultithreadEventExecutorGroup.<init>(MultithreadEventExecutorGroup.java:58) at io.netty.channel.MultithreadEventLoopGroup.<init>(MultithreadEventLoopGroup.java:52) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:87) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:82) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:63) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:51) at io.netty.channel.nio.NioEventLoopGroup.<init>(NioEventLoopGroup.java:43) at com.xxl.job.admin.service.impl.AdminBizImpl.checkAddress(AdminBizImpl.java:206) at com.xxl.job.admin.service.impl.AdminBizImpl.registry(AdminBizImpl.java:268) at sun.reflect.GeneratedMethodAccessor241.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.xxl.rpc.remoting.provider.XxlRpcProviderFactory.invokeService(XxlRpcProviderFactory.java:219) at com.xxl.job.admin.core.conf.ServletServerHandler1.handle(ServletServerHandler1.java:87) at com.xxl.job.admin.core.conf.XxlJobScheduler.invokeAdminService(XxlJobScheduler.java:122) at com.xxl.job.admin.controller.JobApiController.api(JobApiController.java:41) at sun.reflect.GeneratedMethodAccessor85.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:854) at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:765) Exception in thread "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-220612474" java.lang.OutOfMemoryError: Java heap space Exception in thread "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-132000778" java.lang.OutOfMemoryError: Java heap space Exception in thread "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-407339717" java.lang.OutOfMemoryError: Java heap space Exception in thread "xxl-job, admin JobTriggerPoolHelper-fastTriggerPool-1121755797" java.lang.OutOfMemoryError: Java heap space

其他信息

应用运行在k8s上,机器配置:2c8G

jvm参数:

-XX:MaxGCPauseMillis=200 -XX:MetaspaceSize=512m -XX:MaxMetaspaceSize=512m -Xss300k -Xmx5120m -Xms5120m -XX:NewSize=3072M -XX:MaxNewSize=3072M -XX:MaxTenuringThreshold=15 -XX:MaxRAMFraction=1 -XX:+UseConcMarkSweepGC -XX:+PrintGCDateStamps -verbose:gc -XX:GCLogFileSize=20M -XX:NumberOfGCLogFiles=5 -Xloggc:/opt/app/gc-%t.log -XX:MaxTenuringThreshold=15 -XX:MaxRAMFraction=1 -XX:NativeMemoryTracking=detail -XX:+PrintGCCause"

求高人解决。。。。。。

回答

7

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/opt/apps/xx-service 添加jvm参数,dump文件拿到分析一下

1

你好,建议升级最新版本。