Tag Archives: ssl

Replace SSL certificates on Citrix NetScaler using the CLI

Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.

skylarvision/ pixabay.com/ Creative Commons CC0

When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.

Update or replace?

There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate  includes the unbinding of the old, and binding the new certificate.

Replacing a certificate

The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.

Do yourself a favor and add the expiration date to the name of the certificate/ key pair.

Now you can unbind the old, and bind the new certificate. Please note, that this causes a short outage of your service!

SSL Cert Unbind Causing NetScaler Crash

You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.

Replacing an expired lookup service SSL certificate on a vSphere PSC

A few days ago, I ran into a very nasty problem. Fortunately, it was in my lab. Some months ago, I replaced the certificates of my vCenter Server Appliance (VCSA), and I’ve chosen to use the VMware Certificate Authority (VMCA) as a subordinate of my AD-based enterprise CA. The VMCA was used as intermediate CA. The certificates were replaced using the  vSphere 6.0 Certificate Manager (/usr/lib/vmware-vmca/bin/certificate-manager), and I followed the instructions of KB2112016 (Configuring VMware vSphere 6.0 VMware Certificate Authority as a subordinate Certificate Authority).

The VCSA was migrated from vSphere 5.5, and with vSphere 5.5 I was also using custom certificates. These certificates were also issued by my AD-based enterprise CA, and these certificates were migration during the vSphere 5.5 > 6.0 migration. So at the end, I replaced custom certificates with VMCA (as an intermediate CA) certificates.

Everything was fine, until a power outage. After powering-on my VMs, I noticed several errors. After logging into the vSphere Web Client, I got an error message at the top of the page:

While searching for the cause, I checked the URL of the Platform Services Controller (https://vcsa1.lab.local/psc/login) and got this:

psc_error_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0


This error led me to KB2144086 (Updating certificates using certificate manager on vCenter Server or PSC 6.0 Update 1b fails), but was able to proof, that I have used different subject names for the different solution user certificates.

While digging in the PSC logs, I found this error in the /var/log/vmware/psc-client/psc-client.log:

Finally, I found Aaron Smiths blog post “Troubleshooting Expired PSC Certificates with vSphere 6“, who had the same problem. I checked the certificate of the Lookup Service and there it was:

psc_error_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This was the original custom certificate, issued by my AD-based enterprise CA, and installed on my vSphere 5.5 VCSA.

Aaron also offered the solution by referencing KB2118939 (Replacing the Lookup Service SSL certificate on a Platform Services Controller 6.0). I followed the instructions in KB2118939 and replaced the certificate of the Lookup Service with a certificate of the VMCA.

Take care of your certificates

With vSphere 6.0, the Lookup Service should be accessed through the HTTP Reverse Proxy. This proxy uses the machine certificate. Therefore, an expired Lookup Certificate is not obvious. If you connect directly to the Lookup Service using port 7444, you will see the expired certificate. The Lookup Service certificate is not replaced with a custom certificate, if you replace the different solution user certificates.

If you have a vSphere 6.0 VCSA, which was migrated from vSphere 5.5, and you have replaced the certificates on that vSphere 5.5 VCSA with custom certificates, you should check your Lookup Service certificate immidiately! Follow KB2118939 for further instructions.

Credit to Aaron Smith for this blog post. Thank you!

Using Microsoft certreq.exe to generate a certificate signing request (CSR)

Generating a certificate signing request (CSR) is the first step towards a signed certificate. The requests is generated with the applicants private key and consists of the public key, a name and optional attributes.

To generate a CSR, you can use tools like OpenSSL on a Linux box, or sometimes the application itself can generate a CSR. But if you have a Windows box, you don’t have OpenSSL by default. And it’s unhandy to install something just for a single CSR. You can use certreq.exe to create a CSR. This tool is mostly unknown, but it’s included since Server 2000. The syntax slightly differs between the version, so I focus on the version that is shipped with Server 2008/ Windows Vista and newer.

To generate a CSR, you have to create a configuration file. This file specifies the key length, the common name, if the private key is exportable etc. This is a configuration file which includes additional names (subject alternative names, SAN).

This CSR includes three subject alternative names, which are listed below the [Extension] section. The syntax of this file is very important!

To create a CSR, open a CMD and change to the directory where the CSR is stored:

The csr-server1.req file can be used to create a CA signed certificate. The result is a signed certificate, based on the issued CSR. Very handy, especially in VMware Horizon View deployments in which you do not have access to a Windows-based Enterprise CA.

Microsoft Exchange 2013 shows blank ECP & OWA after changes to SSL certificates

EDIT
This issue is described in KB2971270 and is fixed in CU6.

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

Now add it again with the correct certificate hash and application ID.

That’s it. Reboot the Exchange 2013 server and everything should be up and running again.

Replacing SSL certificates for vRealize Orchestrator Appliance

It’s a common practice to replace self-signed certificates, that are used in several VMware products, with CA signed certificates. I did this in my lab for my vCenter Server Appliance and my VMware Update Manager. While I was working with vRealize Orchestrator I noticed, that it is also using self-signed certificates (what else?). For completeness, I decided to replace the self-signed certificates with CA signed.

My lab environment

  1. VMware vSphere 5.5 environment running a vCenter Server appliance (already using CA signed certificates)
  2. vRealize Orchestrator Appliance 5.5.2 (not version 5.5.2.1,  because I had problems with this release)
  3. Microsoft Windows CA running on a Windows 2012 R2 Standard server

You don’t need a Microsoft Windows CA. You can use any other CA. There is no need to use a special vendor. I use a windows-based CA in my lab, so the screenshots reflect this fact. The way how certificates are replaced differs between vRealize Orchestrator Appliance and the windows-based standalone or vCenter Server embedded version. If you use the in the vCenter Server embedded or Standalone Orchestrator check Derek Seamans VMware vSphere 5.5 SSL Toolkit. I used the Orchestrator appliance.

I will only highlight the necessary steps to replace the certificates. I assume that you have a running Orchestrator appliance.

Create the package signing certificate

This certificate is used to sign packages. This certificate is NOT used with HTTPS.

1. Log into the Orchestrator Configuration website using the username “vmware” and click “Server Certificate” on the left navigation page. On the right side appears the server package signing certificate.

vco_package_sign_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Create a new certificate. Otherwise, if you directly export the CSR, the CSR would include the organization, common name, OU etc. from the self-signed certificate. Choose the fourth option “Create a certificate database and self-signed server certificate”.

vco_package_sign_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

3. Enter at least the common name (FQDN of your Orchestrator appliance) and click “Create” (on the right at the end of the page).

vco_package_sign_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

4. Now the CSR can be exported. The CSR is saved into a file called “vCO_SigningRequest.csr”.

vco_package_sign_certificate_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

5. Take the CSR and submit a certificate request at your CA. In my case I took the content of the file and copied it into the corresponding text box of my CA. Make sure that you only use the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–“. I used a customized certificate template (check Derek Seamans blog for more information about VMware and SSL certificates!).

vco_package_sign_certificate_05

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

6. Download the Base 64 encoded certificate and give it a meaningful name (certnew.cer is NOT meaningful…).

vco_package_sign_certificate_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Import the CA certificate

1. Now we have to import the CA certificate. Otherwise we would get an error message when we try to import the CA signed certificate. If you use a Microsoft CA, you can get the CA certificate from the “Active Directory Certificate Services” website. Simply click “Download a CA certificate, certificate chain, or CRL” from the “Select a task:” list. Then save the Base 64 encoded certificate file by choosing “Download CA certificate”. Give the file a meaningful name.

import_ca_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Start the Orchestrator Client and login with an account, that has administrator privileges. In my case this is my domain-admin account (Administrator@lab.local) which is member of the Orchestrator administrator group.

import_ca_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

3. Select “Tools” > “Certificate manager…” from the right top of the Orchestrator client.

import_ca_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

4. Click “Import certificate…”, choose the certificate file you saves some seconds ago and import it.

import_ca_certificate_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it. Now we can move forward and replace the package signing certificate.

Replace the package signing certificate

1. Switch back to the Orchestrator Configuration website and choose the third option: “Import a certificate signing request signed by a certificate authority”.

vco_package_sign_certificate_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

2. Choose the saved certificate for your Orchestrator appliance and click “Import” (on the right at the end of the page).

vco_package_sign_certificate_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

That’s it! The package signing certificate is now replaced by a CA signed one.

vco_package_sign_certificate_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

As I already wrote: This certificate is not used to secure HTTPS. To get rid of the certificate warning when using the Orchestrator Client or the Orchestrator Configuration website, we need some additional steps.

Replace the client certificate

This certificate is used to for HTTPS. After replacing this certificate, the certificate warning for the Orchestrator configuration page (port 8283), the application page (port 8281) and the appliance management page (port 5480) should disappear.

These steps can’t be done using the Orchestrator Configuration, or the appliance management website. Let’s start a SSH session to the Orchestrator appliance.

1. Use SSH, connect to the Orchestrator appliance and login with root credentials. Change to the directory /etc/vco/app-server/security and take a backup of the Java Keystore (JKS).

2. Stop the Orchestrator service

3. The utility “keytool” is used to manage the Java Keystore. The certificate we want to replace has the alias “dunes”. The password for the Java Keystore is “dunesdunes”. This password is valid for every Orchestrator installation! Before we can create a new keypair and export the CSR, the old key needs to be removed from the Java Keystore.

4. Now a new keypair must be created.

Make sure that you hit RETURN keytool asks for the password! Just accept, that the same password is used as for the Java Keystore. btw: “dunes” is a hint to the company who originally developed the Orchestrator. This compay was bought by VMware some years ago.

5. Export the CSR to a file.

You can copy the file to your CA by using SCP. Otherwise use a simple cat and copy the content between “—–BEGIN NEW CERTIFICATE REQUEST—–” and “—–END NEW CERTIFICATE REQUEST—–” directly into the corresponding text box of the CA.

6. Use the CSR to issue a new certificate.

vco_client_certificate_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

7. Download the Base 64 encloded certificate.

vco_client_certificate_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

8. Copy the certificate (using SCP) to the Orchestrator appliance, e.g. to /root or /etc/vco/app-server/security. Depending on the path, you have to change the “-file” parameter! I’ve copied the certificate to /etc/vco/app-server/security.

Please note that you also have to import the CA certificate into the Java Keystore! In my case, the CA certificate was already imported during the initial certificate import from my vCenter Server Appliance, where I also use CA signed certificates. You can import the CA certificate using the “SSL Tab” on the Orchestrator Configuration website.

9. Start the Orchestrator service.

10. Navigate to the Orchestrator website and check the success of the certificate import.

vco_client_certificate_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I still got a certificate warning when starting the Orchestrator client. But I am sure that this behavior is due to Java, because Java doesn’t know the CA.

Replace the appliance management website certificate

The appliance management (port 5480) is also secured with HTTPS. By default the certificate and private key are stored in a PEM file (the file is not protected by a passphrase), which is located at /opt/vmware/etc/lighttpd/server.pem. The PEM file includes the certificate AND the private key. It’s a bit tricky to export a PEM file with the private key from the Java Keystore.

1. First of all: Backup the old PEM file. I assume that you are still logged in on the Orchestrator appliance and still located at /etc/vco/app-server/security.

2. Export the dunes key from the Java Keystore to a PKCS#12 store.

3. Export a PEM file from the PCKS12 keystore. Make sure that you add the “-nodes” parameter.

4. Copy the PEM file to /opt/vmware/etc/lighttpd/server.pem.

5. Restart the lighttpd.

You can safly ignore the warning. Check the state of the daemon using this command:

Lighttpd is running.

6. Check the status of the appliance management website.

vco_mgmt_cert_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Congratulations! The certificate is working.

Final words

As always, working with certificates is challenging. My first attempts have cost me an entire Sunday, especially because the documentation didn’t cover all aspects. I hope this blog post helps you to get through the certificate jungle. Feel free to provide feedback!

Regenerating expired vCenter SSL certificates

During a vSphere 5.0 > 5.5 upgrade I got this message:

The customer hasn’t installed CA-signed certificats, so the expired certificates are the out-of-the-box self-signed certificates. The certificates are valid for two (VirtualCenter 2.5) respectively 10 years (since vCenter 4.x), depending on the Version. The only way to continue the installation is to renew the certificates. After renewing the certificates, you can simply continue the setup due the fact, that the vCenter service is stopped at this point of the setup and it loads the new certificates during startup. It’s the setup which checks the validity of the certificates. KB1009092 describes in great detail what to do, so I will not repeat what is already written there. You should note, that you can’t use the ESXi busybox to renew the certificates. The necessary OpenSSL binary isn’t included. The KB articles recommends OpenSSL for Windows. I simply used my Linux root server. But you can also use a small Linux VM. After renewing the certificates for vCenter, Inventory server and Web Client I simply continued the setup and it ran without problems by. The deployment of CA-signed certifcates is planned.

I recommend to use CA-signed certificates. You need a CA (this can be your own CA) and the vCenter Certificate Automation Tool, which makes the deployment of your own certificates much more easy! There are a couple of excellent posts on this topic. Derek Seaman wrote an awesome four-part series about the usage of the vCenter Certificate Automation Tool. This posting of Craig Kilborn is also a good reference. Craig refers to other sources, like Michael Webster.

Dealing with certificates can bedifficult for unexperienced administrators. They have to clearly understand how certificates work, what the job of a CA is and how all works together. Don’t be a beginner and quickly deploy a CA, just because you need NOW CA-signed certificates. Just use the self-signed certificates for a couple of weeks and work out a CA design that satisfies the customers requirements. Maybe the customer can use a CA for other purposes. Be a trusted advisor, not Mr. Quick ‘n Dirty. ;)