While I was onsite at a customer to decommission an old storage system, one of my very first tasks was to unmount and detach some old datastores. No big deal, until I saw that one after one ESXi hosts went to “not responding”. Time for a heart attack but hey: Why should a host ran into a PDL/ APD, while I was dismounting datastores on the vSphere layer? The LUNs were still there and accessible. The hosts came back quickly and from that point, I watched the hosts flapping between “connected” and “not responding”. Time for an investigation. My first thought was that it must have something to do with the network. But the network was okay, no problems with interfaces, (M/R)STP or similar. Then I checked the logs and found this
2014-09-18 10:53:42,072 [Timer-13] ERROR com.vmware.vim.sms.provider.vasa.event.EventDispatcher - Error occurred while polling events
com.vmware.vim.sms.fault.VasaServiceException: org.apache.axis2.AxisFault: No buffer space available (maximum connections reached?): JVM_Bind
2014-09-18 10:53:42,946 [Timer-15] DEBUG com.vmware.vim.sms.provider.vasa.event.EventDispatcher - [pollEvents] Last event ID = -1
2014-09-18 10:53:42,946 [Timer-13] INFO org.apache.commons.httpclient.HttpMethodDirector - I/O exception (java.net.SocketException) caught when processing request: No buffer space available (maximum connections reached?): JVM_Bind
in the sms.log on the vCenter server. This messages was logged multiple times per second in the Windows Server eventlog:
Transport authentication failed.
ClientIdentity: CN=SMS-120913144508527, O=VMware; C595AE5B1DF15E922569E6E3BDB4D512C1366353
MessageSecurityException: The HTTP request with client authentication scheme 'Anonymous' failed with 'Forbidden' status.
A short search in the Microsoft Knowledgebase pointed me to KB2577795. Checking the number of current connections showed, that the server opens hundreds and thousands of connections to itself. Unfortunately, the installation of KB2577795 did not solved the problem. My second attempted was to rise the number of MaxUserPort which is used to limit the number of dynamic ports available to applications. Open the registry editor and locate the key:
Create a new DWORD wamed MaxUserPort and add the value 65534 to it.
Value Name: MaxUserPort
Value Type: DWORD
Value data: 65534
After a reboot of the vCenter server, the host state flapping has stopped and and did not recur. This is NOT the solution, it’s only a dirty workaround. The customer and I decided to reinstall the virtual vCenter Server and update it to vCenter 5.5 U2. The reinstallation wasn’t a big deal due to an external database and a backup of the SSL certificates.
As I already mentioned: I don’t have a clue what happened to the vCenter server. I was able to create a workaround but I wasn’t able to solve to root cause. But the customer used this situation to make a clear cut and to start with a new vCenter server.
Feel free to follow him on Twitter and/ or leave a comment.
Latest posts by Patrick Terlisten (see all)
- VCAP6.5-DCV Design – Objective 2.4 Build manageability requirements into a vSphere 6.x logical design - December 31, 2019
- VCAP6.5-DCV Design – Objective 2.3 Build availability requirements into a vSphere 6.x logical design - December 23, 2019
- Why we need a vSAN licensing for SMB customers - December 10, 2019