Tag Archives: ssl

Microsoft Exchange 2013/ 2016/ 2019 shows blank ECP & OWA after changes to SSL certificates

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in Exchange 2013 CU6.

I published this blog post in July 2015 and it is still relevant. The feedback for this blog post was incredible, and I’m not joking when I say: I saved many admins weekends. ;) It has shown, that this error still occurs with Exchange 2016 and even 2019. Maybe not because of the same, with Exchange 2013 CU6 fixed bug, but maybe for other reasons. And the solution below still applies to it. Because of this I have decided to re-publish this blog post with a modified title and this little preamble.

Feel free to leave a comment if this blog post worked for you. :)

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

C:\windows\system32>netsh http show sslcert

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:444
    Certificate Hash             : a80c9de605a1525cd252c250495b459f06ed2ec1
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:8172
    Certificate Hash             : 09093ca95154929df92f1bee395b2670a1036a06
    Application ID               : {00000000-0000-0000-0000-000000000000}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 127.0.0.1:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

C:\windows\system32>netsh http delete sslcert ipport=0.0.0.0:444

SSL Certificate successfully deleted

Now add it again with the correct certificate hash and application ID.

C:\windows\system32>netsh http add sslcert ipport=0.0.0.0:444 certhash=1ec7413b4fb1782b4b40868d967161d29154fd7f appid="{4dc3e181-e14b-4a21-b022-59fc669b0914}"

SSL Certificate successfully added

That’s it. Reboot the Exchange server and everything should be up and running again.

Replace SSL certificates on Citrix NetScaler using the CLI

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.

skylarvision/ pixabay.com/ Creative Commons CC0

When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.

Update or replace?

There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate  includes the unbinding of the old, and binding the new certificate.

Replacing a certificate

The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.

add ssl certKey wildcard_exp_20190712 -cert wildcard_exp_20190712.pem -key wildcard_exp_20190712.pem -passcrypt random -expiryMonitor ENABLED -notificationPeriod 90

Do yourself a favor and add the expiration date to the name of the certificate/ key pair.

Now you can unbind the old, and bind the new certificate. Please note, that this causes a short outage of your service!

unbind ssl vserver lb_vs_exchange_owa_https_443 -certkeyName exchange2016_exp_20180713
bind ssl vserver lb_vs_exchange_owa_https_443 -certkeyName wildcard_exp_20190712

SSL Cert Unbind Causing NetScaler Crash

You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.

Replacing an expired lookup service SSL certificate on a vSphere PSC

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A few days ago, I ran into a very nasty problem. Fortunately, it was in my lab. Some months ago, I replaced the certificates of my vCenter Server Appliance (VCSA), and I’ve chosen to use the VMware Certificate Authority (VMCA) as a subordinate of my AD-based enterprise CA. The VMCA was used as intermediate CA. The certificates were replaced using the  vSphere 6.0 Certificate Manager (/usr/lib/vmware-vmca/bin/certificate-manager), and I followed the instructions of KB2112016 (Configuring VMware vSphere 6.0 VMware Certificate Authority as a subordinate Certificate Authority).

The VCSA was migrated from vSphere 5.5, and with vSphere 5.5 I was also using custom certificates. These certificates were also issued by my AD-based enterprise CA, and these certificates were migration during the vSphere 5.5 > 6.0 migration. So at the end, I replaced custom certificates with VMCA (as an intermediate CA) certificates.

Everything was fine, until a power outage. After powering-on my VMs, I noticed several errors. After logging into the vSphere Web Client, I got an error message at the top of the page:

Error occurred while processing request. Check vSphere WebClient logs for details.

While searching for the cause, I checked the URL of the Platform Services Controller (https://vcsa1.lab.local/psc/login) and got this:

psc_error_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

HTTP Status 400 - An error occurred while sending an authentication request to the PSC Single Sign-On server - null

type Status report
message An error occurred while sending an authentication request to the PSC Single Sign-On server - null
description The request sent by the client was syntactically incorrect.

This error led me to KB2144086 (Updating certificates using certificate manager on vCenter Server or PSC 6.0 Update 1b fails), but was able to proof, that I have used different subject names for the different solution user certificates.

While digging in the PSC logs, I found this error in the /var/log/vmware/psc-client/psc-client.log:

Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint doesn't match
        at com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:217)
        at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(Unknown Source)

        ... 71 more

Finally, I found Aaron Smiths blog post “Troubleshooting Expired PSC Certificates with vSphere 6“, who had the same problem. I checked the certificate of the Lookup Service and there it was:

psc_error_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This was the original custom certificate, issued by my AD-based enterprise CA, and installed on my vSphere 5.5 VCSA.

Aaron also offered the solution by referencing KB2118939 (Replacing the Lookup Service SSL certificate on a Platform Services Controller 6.0). I followed the instructions in KB2118939 and replaced the certificate of the Lookup Service with a certificate of the VMCA.

Take care of your certificates

With vSphere 6.0, the Lookup Service should be accessed through the HTTP Reverse Proxy. This proxy uses the machine certificate. Therefore, an expired Lookup Certificate is not obvious. If you connect directly to the Lookup Service using port 7444, you will see the expired certificate. The Lookup Service certificate is not replaced with a custom certificate, if you replace the different solution user certificates.

If you have a vSphere 6.0 VCSA, which was migrated from vSphere 5.5, and you have replaced the certificates on that vSphere 5.5 VCSA with custom certificates, you should check your Lookup Service certificate immidiately! Follow KB2118939 for further instructions.

Credit to Aaron Smith for this blog post. Thank you!

Using Microsoft certreq.exe to generate a certificate signing request (CSR)

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Generating a certificate signing request (CSR) is the first step towards a signed certificate. The requests is generated with the applicants private key and consists of the public key, a name and optional attributes.

To generate a CSR, you can use tools like OpenSSL on a Linux box, or sometimes the application itself can generate a CSR. But if you have a Windows box, you don’t have OpenSSL by default. And it’s unhandy to install something just for a single CSR. You can use certreq.exe to create a CSR. This tool is mostly unknown, but it’s included since Server 2000. The syntax slightly differs between the version, so I focus on the version that is shipped with Server 2008/ Windows Vista and newer.

To generate a CSR, you have to create a configuration file. This file specifies the key length, the common name, if the private key is exportable etc. This is a configuration file which includes additional names (subject alternative names, SAN).

[Version]

Signature= $Windows NT$

[NewRequest]

Subject = "CN=server1.lab.local, OU=Lab, O=vcloudnine.de, L=Cologne, S=NRW, C=DE"
KeySpec = 1
KeyLength = 4096
Hashalgorithm = sha256
Exportable = TRUE
FriendlyName = server1
MachineKeySet = TRUE
SMIME = False
PrivateKeyArchive = FALSE
UserProtected = FALSE
UseExistingKeySet = FALSE
ProviderName = Microsoft RSA SChannel Cryptographic Provider
ProviderType = 12
RequestType = PKCS10
KeyUsage = 0xa0

[EnhancedKeyUsageExtension]

OID=1.3.6.1.5.5.7.3.1

[Extensions]

2.5.29.17 = "{text}"
_continue_ = "dns=server1.vcloudnine.de&"
_continue_ = "dns=app.terlisten-consulting.de&"
_continue_ = "dns=app.blazilla.de&"

This CSR includes three subject alternative names, which are listed below the [Extension] section. The syntax of this file is very important!

To create a CSR, open a CMD and change to the directory where the CSR is stored:

C:\Users\Patrick\Downloads>certreq -new request.inf csr-server1.req

CertReq: Anforderung erstellt

The csr-server1.req file can be used to create a CA signed certificate. The result is a signed certificate, based on the issued CSR. Very handy, especially in VMware Horizon View deployments in which you do not have access to a Windows-based Enterprise CA.

Microsoft Exchange 2013 shows blank ECP & OWA after changes to SSL certificates

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in CU6.

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

C:\windows\system32>netsh http show sslcert

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:444
    Certificate Hash             : a80c9de605a1525cd252c250495b459f06ed2ec1
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:8172
    Certificate Hash             : 09093ca95154929df92f1bee395b2670a1036a06
    Application ID               : {00000000-0000-0000-0000-000000000000}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 127.0.0.1:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

C:\windows\system32>netsh http delete sslcert ipport=0.0.0.0:444

SSL Certificate successfully deleted

Now add it again with the correct certificate hash and application ID.

C:\windows\system32>netsh http add sslcert ipport=0.0.0.0:444 certhash=1ec7413b4fb1782b4b40868d967161d29154fd7f appid="{4dc3e181-e14b-4a21-b022-59fc669b0914}"

SSL Certificate successfully added

That’s it. Reboot the Exchange 2013 server and everything should be up and running again.

Replacing SSL certificates for vRealize Orchestrator Appliance

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

It’s a common practice to replace self-signed certificates, that are used in several VMware products, with CA signed certificates. I did this in my lab for my vCenter Server Appliance and my VMware Update Manager. While I was working with vRealize Orchestrator I noticed, that it is also using self-signed certificates (what else?). For completeness, I decided to replace the self-signed certificates with CA signed.

My lab environment

  1. VMware vSphere 5.5 environment running a vCenter Server appliance (already using CA signed certificates)
  2. vRealize Orchestrator Appliance 5.5.2 (not version 5.5.2.1,  because I had problems with this release)
  3. Microsoft Windows CA running on a Windows 2012 R2 Standard server

You don’t need a Microsoft Windows CA. You can use any other CA. There is no need to use a special vendor. I use a windows-based CA in my lab, so the screenshots reflect this fact. The way how certificates are replaced differs between vRealize Orchestrator Appliance and the windows-based standalone or vCenter Server embedded version. If you use the in the vCenter Server embedded or Standalone Orchestrator check Derek Seamans VMware vSphere 5.5 SSL Toolkit. I used the Orchestrator appliance.

I will only highlight the necessary steps to replace the certificates. I assume that you have a running Orchestrator appliance.

Create the package signing certificate

This certificate is used to sign packages. This certificate is NOT used with HTTPS.

1. Log into the Orchestrator Configuration website using the username “vmware” and click “Server Certificate” on the left navigation page. On the right side appears the server package signing certificate.

vco_package_sign_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Create a new certificate. Otherwise, if you directly export the CSR, the CSR would include the organization, common name, OU etc. from the self-signed certificate. Choose the fourth option “Create a certificate database and self-signed server certificate”.

vco_package_sign_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

3. Enter at least the common name (FQDN of your Orchestrator appliance) and click “Create” (on the right at the end of the page).

vco_package_sign_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

4. Now the CSR can be exported. The CSR is saved into a file called “vCO_SigningRequest.csr”.

vco_package_sign_certificate_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

5. Take the CSR and submit a certificate request at your CA. In my case I took the content of the file and copied it into the corresponding text box of my CA. Make sure that you only use the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–“. I used a customized certificate template (check Derek Seamans blog for more information about VMware and SSL certificates!).

vco_package_sign_certificate_05

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

6. Download the Base 64 encoded certificate and give it a meaningful name (certnew.cer is NOT meaningful…).

vco_package_sign_certificate_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Import the CA certificate

1. Now we have to import the CA certificate. Otherwise we would get an error message when we try to import the CA signed certificate. If you use a Microsoft CA, you can get the CA certificate from the “Active Directory Certificate Services” website. Simply click “Download a CA certificate, certificate chain, or CRL” from the “Select a task:” list. Then save the Base 64 encoded certificate file by choosing “Download CA certificate”. Give the file a meaningful name.

import_ca_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Start the Orchestrator Client and login with an account, that has administrator privileges. In my case this is my domain-admin account (Administrator@lab.local) which is member of the Orchestrator administrator group.

import_ca_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

3. Select “Tools” > “Certificate manager…” from the right top of the Orchestrator client.

import_ca_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

4. Click “Import certificate…”, choose the certificate file you saves some seconds ago and import it.

import_ca_certificate_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it. Now we can move forward and replace the package signing certificate.

Replace the package signing certificate

1. Switch back to the Orchestrator Configuration website and choose the third option: “Import a certificate signing request signed by a certificate authority”.

vco_package_sign_certificate_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Choose the saved certificate for your Orchestrator appliance and click “Import” (on the right at the end of the page).

vco_package_sign_certificate_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it! The package signing certificate is now replaced by a CA signed one.

vco_package_sign_certificate_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

As I already wrote: This certificate is not used to secure HTTPS. To get rid of the certificate warning when using the Orchestrator Client or the Orchestrator Configuration website, we need some additional steps.

Replace the client certificate

This certificate is used to for HTTPS. After replacing this certificate, the certificate warning for the Orchestrator configuration page (port 8283), the application page (port 8281) and the appliance management page (port 5480) should disappear.

These steps can’t be done using the Orchestrator Configuration, or the appliance management website. Let’s start a SSH session to the Orchestrator appliance.

1. Use SSH, connect to the Orchestrator appliance and login with root credentials. Change to the directory /etc/vco/app-server/security and take a backup of the Java Keystore (JKS).

vco:~ # cd /etc/vco/app-server/security
vco:/etc/vco/app-server/security # cp -a jssecacerts jssecacerts.old

2. Stop the Orchestrator service

vco:/etc/vco/app-server/security # service vco-server stop
Stopping tcServer
Instance is running as PID=3454, shutting down...
Instance is running PID=3454, sleeping for up to 60 seconds waiting for shutdown
Instance shut down gracefully

3. The utility “keytool” is used to manage the Java Keystore. The certificate we want to replace has the alias “dunes”. The password for the Java Keystore is “dunesdunes”. This password is valid for every Orchestrator installation! Before we can create a new keypair and export the CSR, the old key needs to be removed from the Java Keystore.

vco:/etc/vco/app-server/security # keytool -delete -alias dunes -keystore jssecacerts -storepass dunesdunes

4. Now a new keypair must be created.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -genkey -alias dunes -keyalg RSA -sigalg SHA512withRSA
What is your first and last name?
[Unknown]: vco.lab.local
What is the name of your organizational unit?
[Unknown]: Lab
What is the name of your organization?
[Unknown]: vcloudnine.de
What is the name of your City or Locality?
[Unknown]:
What is the name of your State or Province?
[Unknown]:
What is the two-letter country code for this unit?
[Unknown]: DE
Is CN=vco.lab.local, OU=Lab, O=vcloudnine.de, L=Unknown, ST=Unknown, C=DE correct?
[no]: yes

Enter key password for <dunes>
(RETURN if same as keystore password):

Make sure that you hit RETURN keytool asks for the password! Just accept, that the same password is used as for the Java Keystore. btw: “dunes” is a hint to the company who originally developed the Orchestrator. This compay was bought by VMware some years ago.

5. Export the CSR to a file.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -certreq -alias dunes -file vco-dunes.csr

You can copy the file to your CA by using SCP. Otherwise use a simple cat and copy the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–” directly into the corresponding text box of the CA.

vco:/etc/vco/app-server/security # cat vco-dunes.csr
-----BEGIN NEW CERTIFICATE REQUEST-----
MIICpDCCAmICAQAwbzELMAkGA1UEBhMCREUxEDAOBgNVBAgTB1Vua25vd24xEDAOBgNVBAcTB1Vu
a25vd24xFjAUBgNVBAoTDXZjbG91ZG5pbmUuZGUxDDAKBgNVBAsTA0xhYjEWMBQGA1UEAxMNdmNv
---snip---
WU/mcQcQgYC0SRZxI+hMKBYTt88JMozIpuE8FnqLVHyNKOCjrh4rs6Z1kW6jfwv6ITVi8ftiegEk
O8yk8b6oUZCJqIPf4VrlnwaSi2ZegHtVJWQBTDv+z0kqA4GFAAKBgQCU1o/X0gdOU7AECclXb7FM
WJlxNSNs2mJfvvsXjh+hinY+SNA3k1QnZ2oLdlW/BM81KlQgO3i4tS1R08WC9UJ0VeEDPhNWkD0V
LRPPCfjT1jTdo+UFsWIJz/XAX0ATiVyDVnktToRTXuFaKUYTz7eU80H0Wp67pr1L0oV5mr5Q5aAw
MC4GCSqGSIb3DQEJDjEhMB8wHQYDVR0OBBYEFMWW1WR7ropcYC5jzBxicMi/LL3lMAsGByqGSM44
BAMFAAMvADAsAhRJYkO/JB7xnjYGlC9b5TYJJIQDXAIUCpCX4p/MV85glncQpc2deDlUfyY=
-----END NEW CERTIFICATE REQUEST-----

6. Use the CSR to issue a new certificate.

vco_client_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

7. Download the Base 64 encloded certificate.

vco_client_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

8. Copy the certificate (using SCP) to the Orchestrator appliance, e.g. to /root or /etc/vco/app-server/security. Depending on the path, you have to change the “-file” parameter! I’ve copied the certificate to /etc/vco/app-server/security.

vco:/etc/vco/app-server/security # keytool -keystore jssecacerts -storepass dunesdunes -importcert -alias dunes -file vco-client-cert.cer
Certificate reply was installed in keystore

Please note that you also have to import the CA certificate into the Java Keystore! In my case, the CA certificate was already imported during the initial certificate import from my vCenter Server Appliance, where I also use CA signed certificates. You can import the CA certificate using the “SSL Tab” on the Orchestrator Configuration website.

9. Start the Orchestrator service.

service vco-server start

10. Navigate to the Orchestrator website and check the success of the certificate import.

vco_client_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I still got a certificate warning when starting the Orchestrator client. But I am sure that this behavior is due to Java, because Java doesn’t know the CA.

Replace the appliance management website certificate

The appliance management (port 5480) is also secured with HTTPS. By default the certificate and private key are stored in a PEM file (the file is not protected by a passphrase), which is located at /opt/vmware/etc/lighttpd/server.pem. The PEM file includes the certificate AND the private key. It’s a bit tricky to export a PEM file with the private key from the Java Keystore.

1. First of all: Backup the old PEM file. I assume that you are still logged in on the Orchestrator appliance and still located at /etc/vco/app-server/security.

vco:/etc/vco/app-server/security # cp -a /opt/vmware/etc/lighttpd/server.pem /opt/vmware/etc/lighttpd/server.pem.old

2. Export the dunes key from the Java Keystore to a PKCS#12 store.

vco:/etc/vco/app-server/security # keytool -importkeystore -srckeystore jssecacerts -destkeystore dunes.p12 -deststoretype PKCS12 -srcalias dunes

3. Export a PEM file from the PCKS12 keystore. Make sure that you add the “-nodes” parameter.

vco:/etc/vco/app-server/security # openssl pkcs12 -in dunes.p12 -passin pass:dunesdunes -out server.pem -nodes
MAC verified OK

4. Copy the PEM file to /opt/vmware/etc/lighttpd/server.pem.

vco:/etc/vco/app-server/security # cp server.pem /opt/vmware/etc/lighttpd/server.pem

5. Restart the lighttpd.

vco:/etc/vco/app-server/security # service vami-lighttp restart
Shutting down vami-lighttpd:done.
Starting vami-lighttpd:2014-11-30 14:57:09: (/build/mts/release/bora-1191928/vadk/src/vami/apps/lighttpd/1.4.29/src/network.c.239) warning: please use server.use-ipv6 only for hostnames, not without server.bind / empty address; your config will break if the kernel default for IPV6_V6ONLY changes
done.

You can safly ignore the warning. Check the state of the daemon using this command:

vco:/etc/vco/app-server/security # service vami-lighttp status
Checking vami-lighttpd status: 5838 ? 00:00:00 vami-lighttpd

Lighttpd is running.

6. Check the status of the appliance management website.

vco_mgmt_cert_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Congratulations! The certificate is working.

Final words

As always, working with certificates is challenging. My first attempts have cost me an entire Sunday, especially because the documentation didn’t cover all aspects. I hope this blog post helps you to get through the certificate jungle. Feel free to provide feedback!

Regenerating expired vCenter SSL certificates

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

During a vSphere 5.0 > 5.5 upgrade I got this message:

The SSL certificate for this product is expired. See Knowledge Base article kb.vmware.com/kb/1009092.

The customer hasn’t installed CA-signed certificats, so the expired certificates are the out-of-the-box self-signed certificates. The certificates are valid for two (VirtualCenter 2.5) respectively 10 years (since vCenter 4.x), depending on the Version. The only way to continue the installation is to renew the certificates. After renewing the certificates, you can simply continue the setup due the fact, that the vCenter service is stopped at this point of the setup and it loads the new certificates during startup. It’s the setup which checks the validity of the certificates. KB1009092 describes in great detail what to do, so I will not repeat what is already written there. You should note, that you can’t use the ESXi busybox to renew the certificates. The necessary OpenSSL binary isn’t included. The KB articles recommends OpenSSL for Windows. I simply used my Linux root server. But you can also use a small Linux VM. After renewing the certificates for vCenter, Inventory server and Web Client I simply continued the setup and it ran without problems by. The deployment of CA-signed certifcates is planned.

I recommend to use CA-signed certificates. You need a CA (this can be your own CA) and the vCenter Certificate Automation Tool, which makes the deployment of your own certificates much more easy! There are a couple of excellent posts on this topic. Derek Seaman wrote an awesome four-part series about the usage of the vCenter Certificate Automation Tool. This posting of Craig Kilborn is also a good reference. Craig refers to other sources, like Michael Webster.

Dealing with certificates can bedifficult for unexperienced administrators. They have to clearly understand how certificates work, what the job of a CA is and how all works together. Don’t be a beginner and quickly deploy a CA, just because you need NOW CA-signed certificates. Just use the self-signed certificates for a couple of weeks and work out a CA design that satisfies the customers requirements. Maybe the customer can use a CA for other purposes. Be a trusted advisor, not Mr. Quick ‘n Dirty. ;)