In this Document
APPLIES TO:
Oracle WebLogic Server - Version 12.1.2.0.0 and later
Information in this document applies to any platform.
SYMPTOMS
In a lightly loaded staging environment, the maximum open sockets was reached as below.
06-Dec-2013 20:03:03 o‘clock GMT Warning Server rds09470app145 rdms1-restgp2b DynamicListenThread[Default] WLS Kernel 1386360183611 BEA-002631 Max threshold reached for open sockets [total open sockets: 1,000, allowed max: 1,000]. No more connections allowed.
In WLS 11g and 8.1 domains running on the same hardware with 1000 sockets, this issue was not observed.
CAUSE
Using NIOSocketMuxer (the default socket muxer in 12.1.2), the socket closure depends on the finalizer mechanism. If the finalizer thread does not kick in (e.g., no GC), the socket closure will not happen and you will reach the number of open sockets configured.
A second issue was that disabling the MAX-OPEN-SOCK-COUNT setting requires a reboot, although the console says that a reboot is unnecessary.
A third issue was that although the first issue is solved, a number of sockets in CLOSE_WAIT were still found, leading to the server ceasing to process requests.
Observations:
- For the first issue, when using JFR (that regularly kicks GC), the issue does not happen. The source code analysis confirms this.
- For the second issue, this is shown in console and while reproducing the first issue.
- For the third issue, one could watch, only on the customer instances, this phenomenon, although first issue was fixed! The same symptom occurred (leak of CLOSE_WAIT sockets) but in this case, stopping JFR does not make the issue disappear.
SOLUTION
Solution retained by this customer is to drop MAX-OPEN-SOCK-COUNT and to apply the known workaround against the deadlock. The other solution is to install the following patches on WLS 12.1.2:
REFERENCES
BUG:18396023 - NIO: MANY NATIVE SOCKETS IN CLOSE_WAIT, NO JAVA SOCKET LEAKS
BUG:18289179 - SETTING MAX-OPEN-SOCK-COUNT REQUIRES ENTITY REBOOT
