一、异常日志

最近发现pms系统异常历史日志中出现如下，由com.mchange.v2.async.ThreadPoolAsynchronousRunner.processReplacedThreads()抛出线程池响应最大延时60000ms

2016-01-17 16:23:20@b@[WARN]-[Thread: Timer-0]-[com.mchange.v2.async.ThreadPoolAsynchronousRunner$Dead@b@lockDetector.run()]: com.mchange.v2.async.ThreadPoolAsynchronousRunner$DeadlockD@b@etector@b1d72a -- APPARENT DEADLOCK!!! Complete Status:@b@        Managed Threads: 3@b@        Active Threads: 3@b@        Active Tasks:@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@269963@b@ (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1)@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@2062df@b@ (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0)@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@3b3b42@b@ (com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2)@b@        Pending Tasks:@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@af9648@b@@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b3774@b@6@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@15bbc8@b@4@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1b59fa@b@1@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@1033b4@b@c@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@12a12b@b@0@b@                com.mchange.v2.resourcepool.BasicResourcePool$1DestroyResourceTa@b@sk@1312102@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask@6e3600@b@@b@Pool thread stack traces:@b@        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1,5@b@,main]@b@                java.net.PlainSocketImpl.socketConnect(Native Method)@b@                java.net.PlainSocketImpl.doConnect(Unknown Source)@b@                java.net.PlainSocketImpl.connectToAddress(Unknown Source)@b@                java.net.PlainSocketImpl.connect(Unknown Source)@b@                java.net.SocksSocketImpl.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFacto@b@ry.java:257)@b@                com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:298)@b@                com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:23@b@91)@b@                com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.j@b@ava:2428)@b@                com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:22@b@13)@b@                com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:797)@b@                com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)@b@                sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown S@b@ource)@b@                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknow@b@n Source)@b@                java.lang.reflect.Constructor.newInstance(Unknown Source)@b@                com.mysql.jdbc.Util.handleNewInstance(Util.java:411)@b@                com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:38@b@9)@b@                com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver@b@.java:305)@b@                com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(Driver@b@ManagerDataSource.java:134)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:182)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:171)@b@                com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnect@b@ionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)@b@                com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicRes@b@ourcePool.java:1014)@b@                com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicRe@b@sourcePool.java:32)@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(Ba@b@sicResourcePool.java:1810)@b@                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run@b@(ThreadPoolAsynchronousRunner.java:547)@b@        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0,5@b@,main]@b@                java.net.PlainSocketImpl.socketConnect(Native Method)@b@                java.net.PlainSocketImpl.doConnect(Unknown Source)@b@                java.net.PlainSocketImpl.connectToAddress(Unknown Source)@b@                java.net.PlainSocketImpl.connect(Unknown Source)@b@                java.net.SocksSocketImpl.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFacto@b@ry.java:257)@b@                com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:298)@b@                com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:23@b@91)@b@                com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.j@b@ava:2428)@b@                com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:22@b@13)@b@                com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:797)@b@                com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)@b@                sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown S@b@ource)@b@                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknow@b@n Source)@b@                java.lang.reflect.Constructor.newInstance(Unknown Source)@b@                com.mysql.jdbc.Util.handleNewInstance(Util.java:411)@b@                com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:38@b@9)@b@                com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver@b@.java:305)@b@                com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(Driver@b@ManagerDataSource.java:134)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:182)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:171)@b@                com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnect@b@ionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)@b@                com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicRes@b@ourcePool.java:1014)@b@                com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicRe@b@sourcePool.java:32)@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(Ba@b@sicResourcePool.java:1810)@b@                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run@b@(ThreadPoolAsynchronousRunner.java:547)@b@        Thread[com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2,5@b@,main]@b@                java.net.PlainSocketImpl.socketConnect(Native Method)@b@                java.net.PlainSocketImpl.doConnect(Unknown Source)@b@                java.net.PlainSocketImpl.connectToAddress(Unknown Source)@b@                java.net.PlainSocketImpl.connect(Unknown Source)@b@                java.net.SocksSocketImpl.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.connect(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                java.net.Socket.<init>(Unknown Source)@b@                com.mysql.jdbc.StandardSocketFactory.connect(StandardSocketFacto@b@ry.java:257)@b@                com.mysql.jdbc.MysqlIO.<init>(MysqlIO.java:298)@b@                com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:23@b@91)@b@                com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.j@b@ava:2428)@b@                com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:22@b@13)@b@                com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:797)@b@                com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)@b@                sun.reflect.GeneratedConstructorAccessor61.newInstance(Unknown S@b@ource)@b@                sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknow@b@n Source)@b@                java.lang.reflect.Constructor.newInstance(Unknown Source)@b@                com.mysql.jdbc.Util.handleNewInstance(Util.java:411)@b@                com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:38@b@9)@b@                com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver@b@.java:305)@b@                com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(Driver@b@ManagerDataSource.java:134)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:182)@b@                com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledCon@b@nection(WrapperConnectionPoolDataSource.java:171)@b@                com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnect@b@ionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:137)@b@                com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicRes@b@ourcePool.java:1014)@b@                com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicRe@b@sourcePool.java:32)@b@                com.mchange.v2.resourcepool.BasicResourcePool$AcquireTask.run(Ba@b@sicResourcePool.java:1810)@b@                com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run@b@(ThreadPoolAsynchronousRunner.java:547)@b@@b@@b@@b@2016-01-17 16:24:35@b@[WARN]-[Thread: Timer-0]-[com.mchange.v2.async.ThreadPoolAsynchronousRunner.proc@b@essReplacedThreads()]: Task com.mchange.v2.resourcepool.BasicResourcePool$Acquir@b@eTask@269963 (in deadlocked PoolThread) failed to complete in maximum time 60000@b@ms. Trying interrupt().

二、解决步骤

出现上述异常，完全由于最近服务器断网造成了，远程数据库连接不上，超过了做大连接”<property name="maxIdleTime" value="60" />“延时设置（更多连接池配置说明），将服务器网络连接恢复，重启pms服务应用，一切正常。