系统变慢调整小结

时间：2014-07-25 00:04:44 阅读：379 评论：0 收藏：0 [点我收藏+]

标签：style http java color os strong 数据 io

客户反映系统越用越慢，重启服务器后段时间内系统恢复正常

老年代垃圾回收不了。 full gc 每分钟执行40次左右。

开始以为是内存泄漏，导出jvm内存快照

快照中可以看到线程池里有问题。但是看不出具体的问题原因

导出线程快照没有发现可疑线程信息。。。郁闷了。。（当前时间19:21已经没有客户在操作系统了）

第二天继续今天的问题。上午客户打电话说又有问题了。

赶紧导出线程快照

$jstack pid

"Running HTTPHandler-26" prio=10 tid=0x00007f2140001000 nid=0x3c9b waiting for monitor entry [0x00007f21f4d50000]

java.lang.Thread.State: BLOCKED (on object monitor)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.repeatJudgeMethod(RepeatJudgeManager.java:266)

- waiting to lock <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager$$FastClassByCGLIB$$971a2e6a.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager$$EnhancerByCGLIB$$1805e0ee.repeatJudgeMethod(<generated>)

at com.sinosoft.xf.petition.petitionaccept.web.RepeatJudgeAction.repeatJudgeMethod(RepeatJudgeAction.java:385)

at com.sinosoft.xf.petition.petitionaccept.web.RepeatJudgeAction$$FastClassByCGLIB$$720d47c0.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

java.lang.Thread.State: BLOCKED (on object monitor)：线程阻塞

- waiting to lock <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)：线程在等待<0x00000007a994d600>这个锁。因为无法获取到所以发生阻塞。

#cat statck.txt | grep ‘java.lang.Thread.State‘ | sort | uniq -c | sort -n > 1.txt

18 java.lang.Thread.State: TIMED_WAITING (sleeping)

18 java.lang.Thread.State: WAITING (parking)

45 java.lang.Thread.State: TIMED_WAITING (parking)

63 java.lang.Thread.State: WAITING (on object monitor)

132 java.lang.Thread.State: RUNNABLE

135 java.lang.Thread.State: BLOCKED (on object monitor)

300 java.lang.Thread.State: TIMED_WAITING (on object monitor)

发现竟然有135个线程发生了阻塞

通过导出线程信息定位到具体的代码位置。。

public synchronized String repeatJudgeMethod() {
    ...
}

根据业务需求拆分，调整该方法。

之后有又观察一两天时间发现系统还是很慢，继续导出线程快照

发现该方法执行了一个批量更新，而且这边更新过程比较慢，锁住了一大批的数据导致

"Running HTTPHandler-10" prio=10 tid=0x00007f21b0004800 nid=0x48c1 runnable [0x00007f21f5afa000]

java.lang.Thread.State: RUNNABLE

at java.net.SocketInputStream.socketRead0(Native Method)

at java.net.SocketInputStream.read(SocketInputStream.java:129)

at com.ibm.db2.jcc.t4.ab.b(ab.java:193)

at com.ibm.db2.jcc.t4.ab.c(ab.java:237)

at com.ibm.db2.jcc.t4.ab.c(ab.java:348)

at com.ibm.db2.jcc.t4.ab.v(ab.java:1133)

at com.ibm.db2.jcc.t4.eb.c(eb.java:30)

at com.ibm.db2.jcc.t4.u.a(u.java:32)

at com.ibm.db2.jcc.t4.j.Xb(j.java:245)

at com.ibm.db2.jcc.b.jk.X(jk.java:3412)

at com.ibm.db2.jcc.t4.d.lb(d.java:1928)

at com.ibm.db2.jcc.t4.d.n(d.java:2115)

at com.ibm.db2.jcc.t4.d.o(d.java:2127)

at com.ibm.db2.jcc.t4.d.a(d.java:163)

at com.ibm.db2.jcc.t4.d.b(d.java:478)

at com.ibm.db2.jcc.b.bc.a(bc.java:213)

at com.ibm.db2.jcc.b.jk.c(jk.java:296)

at com.ibm.db2.jcc.b.jk.next(jk.java:258)

- locked <0x00000007b02094a8> (a com.ibm.db2.jcc.t4.b)

at com.apusic.jdbc.adapter.ResultSetHandle.next(Unknown Source)

at org.springframework.jdbc.core.RowMapperResultSetExtractor.extractData(RowMapperResultSetExtractor.java:91)

at org.springframework.jdbc.core.JdbcTemplate$1QueryStatementCallback.doInStatement(JdbcTemplate.java:448)

at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:396)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:458)

at org.springframework.jdbc.core.JdbcTemplate.query(JdbcTemplate.java:466)

at org.springframework.jdbc.core.JdbcTemplate.queryForList(JdbcTemplate.java:497)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao.getRepeatInfoTreeList(RepeatJudgeDao.java:429)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao$$FastClassByCGLIB$$f0ee3a4a.invoke(<generated>)

at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)

at org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:700)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)

at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)

at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)

at org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:635)

at com.sinosoft.xf.petition.petitionaccept.dao.RepeatJudgeDao$$EnhancerByCGLIB$$e0e3d2b0.getRepeatInfoTreeList(<generated>)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.editRepeatInfoRepeatNum(RepeatJudgeManager.java:476)

at com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager.repeatJudgeMethod(RepeatJudgeManager.java:370)

- locked <0x00000007a994d600> (a com.sinosoft.xf.petition.petitionaccept.manager.RepeatJudgeManager)