This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in Exchange 2013 CU6.
I published this blog post in July 2015 and it is still relevant. The feedback for this blog post was incredible, and I’m not joking when I say: I saved many admins weekends. ;) It has shown, that this error still occurs with Exchange 2016 and even 2019. Maybe not because of the same, with Exchange 2013 CU6 fixed bug, but maybe for other reasons. And the solution below still applies to it. Because of this I have decided to re-publish this blog post with a modified title and this little preamble.
Feel free to leave a comment if this blog post worked for you. :)
I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:
An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.
If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.
C:\windows\system32>netsh http show sslcert
SSL Certificate bindings:
-------------------------
IP:port : 0.0.0.0:443
Certificate Hash : 1ec7413b4fb1782b4b40868d967161d29154fd7f
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 0.0.0.0:444
Certificate Hash : a80c9de605a1525cd252c250495b459f06ed2ec1
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 0.0.0.0:8172
Certificate Hash : 09093ca95154929df92f1bee395b2670a1036a06
Application ID : {00000000-0000-0000-0000-000000000000}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 127.0.0.1:443
Certificate Hash : 1ec7413b4fb1782b4b40868d967161d29154fd7f
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.
This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.
skylarvision/ pixabay.com/ Creative Commons CC0
When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.
Update or replace?
There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate includes the unbinding of the old, and binding the new certificate.
Replacing a certificate
The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.
You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.
This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
A few days ago, I ran into a very nasty problem. Fortunately, it was in my lab. Some months ago, I replaced the certificates of my vCenter Server Appliance (VCSA), and I’ve chosen to use the VMware Certificate Authority (VMCA) as a subordinate of my AD-based enterprise CA. The VMCA was used as intermediate CA. The certificates were replaced using the vSphere 6.0 Certificate Manager (/usr/lib/vmware-vmca/bin/certificate-manager), and I followed the instructions of KB2112016 (Configuring VMware vSphere 6.0 VMware Certificate Authority as a subordinate Certificate Authority).
The VCSA was migrated from vSphere 5.5, and with vSphere 5.5 I was also using custom certificates. These certificates were also issued by my AD-based enterprise CA, and these certificates were migration during the vSphere 5.5 > 6.0 migration. So at the end, I replaced custom certificates with VMCA (as an intermediate CA) certificates.
Everything was fine, until a power outage. After powering-on my VMs, I noticed several errors. After logging into the vSphere Web Client, I got an error message at the top of the page:
Error occurred while processing request. Check vSphere WebClient logs for details.
While searching for the cause, I checked the URL of the Platform Services Controller (https://vcsa1.lab.local/psc/login) and got this:
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server - null
type Status report
message An error occurred while sending an authentication request to the PSC Single Sign-On server - null
description The request sent by the client was syntactically incorrect.
This error led me to KB2144086 (Updating certificates using certificate manager on vCenter Server or PSC 6.0 Update 1b fails), but was able to proof, that I have used different subject names for the different solution user certificates.
While digging in the PSC logs, I found this error in the /var/log/vmware/psc-client/psc-client.log:
Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint doesn't match
at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:217)
at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(Unknown Source)
... 71 more
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
This was the original custom certificate, issued by my AD-based enterprise CA, and installed on my vSphere 5.5 VCSA.
Aaron also offered the solution by referencing KB2118939 (Replacing the Lookup Service SSL certificate on a Platform Services Controller 6.0). I followed the instructions in KB2118939 and replaced the certificate of the Lookup Service with a certificate of the VMCA.
Take care of your certificates
With vSphere 6.0, the Lookup Service should be accessed through the HTTP Reverse Proxy. This proxy uses the machine certificate. Therefore, an expired Lookup Certificate is not obvious. If you connect directly to the Lookup Service using port 7444, you will see the expired certificate. The Lookup Service certificate is not replaced with a custom certificate, if you replace the different solution user certificates.
If you have a vSphere 6.0 VCSA, which was migrated from vSphere 5.5, and you have replaced the certificates on that vSphere 5.5 VCSA with custom certificates, you should check your Lookup Service certificate immidiately! Follow KB2118939 for further instructions.
This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Generating a certificate signing request (CSR) is the first step towards a signed certificate. The requests is generated with the applicants private key and consists of the public key, a name and optional attributes.
To generate a CSR, you can use tools like OpenSSL on a Linux box, or sometimes the application itself can generate a CSR. But if you have a Windows box, you don’t have OpenSSL by default. And it’s unhandy to install something just for a single CSR. You can use certreq.exe to create a CSR. This tool is mostly unknown, but it’s included since Server 2000. The syntax slightly differs between the version, so I focus on the version that is shipped with Server 2008/ Windows Vista and newer.
To generate a CSR, you have to create a configuration file. This file specifies the key length, the common name, if the private key is exportable etc. This is a configuration file which includes additional names (subject alternative names, SAN).
The csr-server1.req file can be used to create a CA signed certificate. The result is a signed certificate, based on the issued CSR. Very handy, especially in VMware Horizon View deployments in which you do not have access to a Windows-based Enterprise CA.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in CU6.
I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:
An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.
If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.
C:\windows\system32>netsh http show sslcert
SSL Certificate bindings:
-------------------------
IP:port : 0.0.0.0:443
Certificate Hash : 1ec7413b4fb1782b4b40868d967161d29154fd7f
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 0.0.0.0:444
Certificate Hash : a80c9de605a1525cd252c250495b459f06ed2ec1
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 0.0.0.0:8172
Certificate Hash : 09093ca95154929df92f1bee395b2670a1036a06
Application ID : {00000000-0000-0000-0000-000000000000}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
IP:port : 127.0.0.1:443
Certificate Hash : 1ec7413b4fb1782b4b40868d967161d29154fd7f
Application ID : {4dc3e181-e14b-4a21-b022-59fc669b0914}
Certificate Store Name : MY
Verify Client Certificate Revocation : Enabled
Verify Revocation Using Cached Client Certificate Only : Disabled
Usage Check : Enabled
Revocation Freshness Time : 0
URL Retrieval Timeout : 0
Ctl Identifier : (null)
Ctl Store Name : (null)
DS Mapper Usage : Disabled
Negotiate Client Certificate : Disabled
Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.
This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
It’s a common practice to replace self-signed certificates, that are used in several VMware products, with CA signed certificates. I did this in my lab for my vCenter Server Appliance and my VMware Update Manager. While I was working with vRealize Orchestrator I noticed, that it is also using self-signed certificates (what else?). For completeness, I decided to replace the self-signed certificates with CA signed.
My lab environment
VMware vSphere 5.5 environment running a vCenter Server appliance (already using CA signed certificates)
vRealize Orchestrator Appliance 5.5.2 (not version 5.5.2.1, because I had problems with this release)
Microsoft Windows CA running on a Windows 2012 R2 Standard server
You don’t need a Microsoft Windows CA. You can use any other CA. There is no need to use a special vendor. I use a windows-based CA in my lab, so the screenshots reflect this fact. The way how certificates are replaced differs between vRealize Orchestrator Appliance and the windows-based standalone or vCenter Server embedded version. If you use the in the vCenter Server embedded or Standalone Orchestrator check Derek SeamansVMware vSphere 5.5 SSL Toolkit. I used the Orchestrator appliance.
I will only highlight the necessary steps to replace the certificates. I assume that you have a running Orchestrator appliance.
Create the package signing certificate
This certificate is used to sign packages. This certificate is NOT used with HTTPS.
1. Log into the Orchestrator Configuration website using the username “vmware” and click “Server Certificate” on the left navigation page. On the right side appears the server package signing certificate.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
2. Create a new certificate. Otherwise, if you directly export the CSR, the CSR would include the organization, common name, OU etc. from the self-signed certificate. Choose the fourth option “Create a certificate database and self-signed server certificate”.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
3. Enter at least the common name (FQDN of your Orchestrator appliance) and click “Create” (on the right at the end of the page).
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
4. Now the CSR can be exported. The CSR is saved into a file called “vCO_SigningRequest.csr”.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
5. Take the CSR and submit a certificate request at your CA. In my case I took the content of the file and copied it into the corresponding text box of my CA. Make sure that you only use the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–“. I used a customized certificate template (check Derek Seamans blog for more information about VMware and SSL certificates!).
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
6. Download the Base 64 encoded certificate and give it a meaningful name (certnew.cer is NOT meaningful…).
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
Import the CA certificate
1. Now we have to import the CA certificate. Otherwise we would get an error message when we try to import the CA signed certificate. If you use a Microsoft CA, you can get the CA certificate from the “Active Directory Certificate Services” website. Simply click “Download a CA certificate, certificate chain, or CRL” from the “Select a task:” list. Then save the Base 64 encoded certificate file by choosing “Download CA certificate”. Give the file a meaningful name.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
2. Start the Orchestrator Client and login with an account, that has administrator privileges. In my case this is my domain-admin account (Administrator@lab.local) which is member of the Orchestrator administrator group.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
3. Select “Tools” > “Certificate manager…” from the right top of the Orchestrator client.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
4. Click “Import certificate…”, choose the certificate file you saves some seconds ago and import it.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
That’s it. Now we can move forward and replace the package signing certificate.
Replace the package signing certificate
1. Switch back to the Orchestrator Configuration website and choose the third option: “Import a certificate signing request signed by a certificate authority”.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
2. Choose the saved certificate for your Orchestrator appliance and click “Import” (on the right at the end of the page).
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
That’s it! The package signing certificate is now replaced by a CA signed one.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
As I already wrote: This certificate is not used to secure HTTPS. To get rid of the certificate warning when using the Orchestrator Client or the Orchestrator Configuration website, we need some additional steps.
Replace the client certificate
This certificate is used to for HTTPS. After replacing this certificate, the certificate warning for the Orchestrator configuration page (port 8283), the application page (port 8281) and the appliance management page (port 5480) should disappear.
These steps can’t be done using the Orchestrator Configuration, or the appliance management website. Let’s start a SSH session to the Orchestrator appliance.
1. Use SSH, connect to the Orchestrator appliance and login with root credentials. Change to the directory /etc/vco/app-server/security and take a backup of the Java Keystore (JKS).
vco:~ # cd /etc/vco/app-server/security
vco:/etc/vco/app-server/security # cp -a jssecacerts jssecacerts.old
2. Stop the Orchestrator service
vco:/etc/vco/app-server/security # service vco-server stop
Stopping tcServer
Instance is running as PID=3454, shutting down...
Instance is running PID=3454, sleeping for up to 60 seconds waiting for shutdown
Instance shut down gracefully
3. The utility “keytool” is used to manage the Java Keystore. The certificate we want to replace has the alias “dunes”. The password for the Java Keystore is “dunesdunes”. This password is valid for every Orchestrator installation! Before we can create a new keypair and export the CSR, the old key needs to be removed from the Java Keystore.
vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -genkey -alias dunes -keyalg RSA -sigalg SHA512withRSA
What is your first and last name?
[Unknown]: vco.lab.local
What is the name of your organizational unit?
[Unknown]: Lab
What is the name of your organization?
[Unknown]: vcloudnine.de
What is the name of your City or Locality?
[Unknown]:
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]: DE
Is CN=vco.lab.local, OU=Lab, O=vcloudnine.de, L=Unknown, ST=Unknown, C=DE correct?
[no]: yes
Enter key password for <dunes>
(RETURN if same as keystore password):
Make sure that you hit RETURN keytool asks for the password! Just accept, that the same password is used as for the Java Keystore. btw: “dunes” is a hint to the company who originally developed the Orchestrator. This compay was bought by VMware some years ago.
You can copy the file to your CA by using SCP. Otherwise use a simple cat and copy the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–” directly into the corresponding text box of the CA.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
7. Download the Base 64 encloded certificate.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
8. Copy the certificate (using SCP) to the Orchestrator appliance, e.g. to /root or /etc/vco/app-server/security. Depending on the path, you have to change the “-file” parameter! I’ve copied the certificate to /etc/vco/app-server/security.
vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -importcert -alias dunes -file vco-client-cert.cer
Certificate reply was installed in keystore
Please note that you also have to import the CA certificate into the Java Keystore! In my case, the CA certificate was already imported during the initial certificate import from my vCenter Server Appliance, where I also use CA signed certificates. You can import the CA certificate using the “SSL Tab” on the Orchestrator Configuration website.
9. Start the Orchestrator service.
service vco-server start
10. Navigate to the Orchestrator website and check the success of the certificate import.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
I still got a certificate warning when starting the Orchestrator client. But I am sure that this behavior is due to Java, because Java doesn’t know the CA.
Replace the appliance management website certificate
The appliance management (port 5480) is also secured with HTTPS. By default the certificate and private key are stored in a PEM file (the file is not protected by a passphrase), which is located at /opt/vmware/etc/lighttpd/server.pem. The PEM file includes the certificate AND the private key. It’s a bit tricky to export a PEM file with the private key from the Java Keystore.
1. First of all: Backup the old PEM file. I assume that you are still logged in on the Orchestrator appliance and still located at /etc/vco/app-server/security.
vco:/etc/vco/app-server/security # cp -a /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.old
2. Export the dunes key from the Java Keystore to a PKCS#12 store.
vco:/etc/vco/app-server/security # service vami-lighttp restart
Shutting down vami-lighttpd:done.
Starting vami-lighttpd:2014-11-30 14:57:09: (/build/mts/release/bora-1191928/vadk/src/vami/apps/lighttpd/1.4.29/src/network.c.239) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
done.
You can safly ignore the warning. Check the state of the daemon using this command:
vco:/etc/vco/app-server/security # service vami-lighttp status
Checking vami-lighttpd status: 5838 ? 00:00:00 vami-lighttpd
Lighttpd is running.
6. Check the status of the appliance management website.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
Congratulations! The certificate is working.
Final words
As always, working with certificates is challenging. My first attempts have cost me an entire Sunday, especially because the documentation didn’t cover all aspects. I hope this blog post helps you to get through the certificate jungle. Feel free to provide feedback!
This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
During a vSphere 5.0 > 5.5 upgrade I got this message:
The SSL certificate for this product is expired. See Knowledge Base article kb.vmware.com/kb/1009092.
The customer hasn’t installed CA-signed certificats, so the expired certificates are the out-of-the-box self-signed certificates. The certificates are valid for two (VirtualCenter 2.5) respectively 10 years (since vCenter 4.x), depending on the Version. The only way to continue the installation is to renew the certificates. After renewing the certificates, you can simply continue the setup due the fact, that the vCenter service is stopped at this point of the setup and it loads the new certificates during startup. It’s the setup which checks the validity of the certificates. KB1009092 describes in great detail what to do, so I will not repeat what is already written there. You should note, that you can’t use the ESXi busybox to renew the certificates. The necessary OpenSSL binary isn’t included. The KB articles recommends OpenSSL for Windows. I simply used my Linux root server. But you can also use a small Linux VM. After renewing the certificates for vCenter, Inventory server and Web Client I simply continued the setup and it ran without problems by. The deployment of CA-signed certifcates is planned.
I recommend to use CA-signed certificates. You need a CA (this can be your own CA) and the vCenter Certificate Automation Tool, which makes the deployment of your own certificates much more easy! There are a couple of excellent posts on this topic. Derek Seaman wrote an awesome four-part series about the usage of the vCenter Certificate Automation Tool. This posting of Craig Kilborn is also a good reference. Craig refers to other sources, like Michael Webster.
Dealing with certificates can bedifficult for unexperienced administrators. They have to clearly understand how certificates work, what the job of a CA is and how all works together. Don’t be a beginner and quickly deploy a CA, just because you need NOW CA-signed certificates. Just use the self-signed certificates for a couple of weeks and work out a CA design that satisfies the customers requirements. Maybe the customer can use a CA for other purposes. Be a trusted advisor, not Mr. Quick ‘n Dirty. ;)
To change your privacy setting, e.g. granting or withdrawing consent, click here:
Settings
