Category Archives: Microsoft

Office365/ Exchange OAuth errors after replacing TLS certificate

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer of mine asked for help to analyse a weird OAuth error. They are using a Microsoft Dynamics 365 Outlook plugin, which came up with an error:

“Can’t connect to Exchange”

In addition to this, they also faced an issueaccessing shared calendars of Exchange Online mailboxes.

Clearly an OAuth error. So we ran the Hybrid Connection Wizard again, which finished without any errors. But the errors persisted. Next stop: OAuth configuration.

We logged into one of the Exchange servers, started an Exchange Management Shell and checked the current OAuth configuration:

[PS] C:\Windows\system32>Get-AuthConfig
RunspaceId : e7c560cd-8316-4d9c-b97d-f4358d665e7d
CurrentCertificateThumbprint : CA420022ACC542D3AF7598456615715E98FFE986
PreviousCertificateThumbprint : E7B56BEAF4704BCBBF8C297F1D8D000215679C6F
NextCertificateThumbprint :
NextCertificateEffectiveDate :
ServiceName : 00000002-0000-0ff1-ce00-000000000000
Realm :
DeploymentId :
IssuerIdentifier :
Name : Auth Configuration
AdminDisplayName :
ExchangeVersion : 0.20 (15.0.0.0)
DistinguishedName : CN=Auth Configuration,CN=EXORG,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=tld
Identity : Auth Configuration
Guid : 911b6246-6248-4278-91be-1b7b4943d2f1
ObjectCategory : mlnetwork.local/Configuration/Schema/ms-Exch-Auth-Auth-Config
ObjectClass : {top, container, msExchContainer, msExchAuthAuthConfig}
WhenChanged : 11.04.2020 12:33:12
WhenCreated : 27.02.2015 14:24:37
WhenChangedUTC : 11.04.2020 10:33:12
WhenCreatedUTC : 27.02.2015 13:24:37
OrganizationId :
Id : Auth Configuration
OriginatingServer : DC01.domain.tld
IsValid : True
ObjectState : Unchanged

The CurrentCertificateThumbprint was the thumbprint of the old certificate. So we had to update it.

Set-AuthConfig -NewCertificateThumbprint 45CCEEDFAAB00F3A4312FF56774FA3B3412B9C (get-date)
Set-AuthConfig -PublishCertificate
iisreset /noforce
Set-AuthConfig -ClearPreviousCertificate

After the iisreset, the Dynamics plugin and the Exchange Online calendars started to work again.

Once in a year: How to update TLS certificates on ADFS server and proxies

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

You might got this news some days ago: Starting with September 1, 2020, browsers and devices from Apple, Google, and Mozilla will show errors for new TLS certificates that have a lifespan greater than 398 days. Due to this move from Apple, Google and Mozilla, you have to deal with the replacement of certificates much more often. And we all know: Replacing certificates can be a real PITA!

Bild von skylarvision auf Pixabay

Replacing TLS certificates used for ADFS and Office 365 can be a challenging task, and this blog post will cover the neccessary steps.

ADFS Server

The first service, for which we will replace the certificate, is the ADFS server, or the ADFS server farm. At this point it is important to understand that we are dealing with two different points to which the certificate is bound:

  • the ADFS service communications certificate, and
  • the ADFS SSL certificate

The first step is to replace the service communication certificate. After importing the certificate with private key, you need to assign “read” permission to the ADFS service account. Right click on the certificate, then “All Tasks” > “Manage Private Keys”.

Make sure to import the certificate on all farm servers! Next step: Start the ADFS management console on the primary node. Select “Certificates” and then “Select service communication certificate” on the right window pane.

Now we have successfully replaced the service communication certificate. But we are no finished yet! Now we have to set the ADFS SSL certificate. Depending on your OS, you have to run the PowerShell command on the primary node. If your are running Windows Server 2012 R2 or older, you have to run the PowerShell command on EVERY ADFS farm server!

You can get the certificate thumbprint using the Get-AdfsSslCertificate command. Set the ADFS SSL certificate with

Set-AdfsSslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then restart the ADFS service.

Restart-Service adfssrv

ADFS Proxies

In most cases you will have one or more ADFS proxies in your DMZ. The ADFS proxy is nothing more than a Web Application Proxy (WAP) and therefore the PowerShell commands for WAP will be used.

First of all: Import the new certificate with the private key on all ADFS proxies, and then get the certificate hash of the new certificate. Then open an elevated PowerShell on each proxy.

Set-WebApplicationProxySslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then we have to re-establish the trust between the proxies and the primary ADFS farm server. You will need the local (!) administrator account of the primary farm server.

$cred = Get-Credential
Install-WebApplicationProxy -FederationServiceTrustCredential $cred -CertificateThumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125 -FederationServiceName adfs.domain.tld

The last step is to update thefederated trust with Office 365.

Update the federated trust with Office 365

To update the federated trust with Office 365, you will need the Windows Azure Active Direcotry Module for Windows PowerShell and an elevated PowerShell. Connect to Office 365 and update the federated trust:

$cred = Get-Credential
Connect-MsolService -Credential $cred
Update-MSOLFederatedDomain –DomainName domain.tld

That’s it! Bookmark this page and set a calendar entry on today +12 months. :)

Passed Microsoft exam AZ-103 – Azure Administrator Associate

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Six weeks ago, I passed the Microsoft AZ-103 exam and earned the Azure Administrator Associate. A last minute pass, because AZ-104 was already launched. But better late than never. I had to re-schedule the exam a couple of times because the test center was closed due to COVID19.

The Azure Administrator Associate is a Administrator-role certification and it is all about implementing, managing and monitoring the Azure identity, governance, storage, compute, and virtual network solutions.

The exam covers a couple of topics and you should have knowledge and hands-on experience in administering Azure services using the Azure Portal, PowerShell, Azure CLI, and Azure Resource Manager templates.

Your knowledge is tested over a broad band of topics. These topics are:

  • Manage Azure identities and governance
  • Implement and manage storage
  • Deploy and manage Azure compute resources
  • Configure and manage virtual networking
  • Monitor and back up Azure resources

How to prepare for the exam

Fortunately I have a monthly Azure credit which I can use to gain new skills. I used these Azure credit together with the Microsoft Learning Path for AZ-103 (now 104).

It is pretty important no only to focus on VMs, storage or networking. Web Apps was one of my blind spots, and I had to get my head around it. Azure identities and governance is not so hard, if you are already familiar with Office 365.

I learned a lot from the Microsoft Documentation for Azure, and I was really impressed how much I was able to find, read and learn from there.

Next stop: Microsoft Certified: Azure Solutions Architect Expert

Microsoft has announced to retire all remaining exams associated with Microsoft Certified Solutions Associate (MCSA), Microsoft Certified Solutions Developer (MCSD), Microsoft Certified Solutions Expert (MCSE) on January 31, 2021, so the role-based certifications introduced in September 2018 are the way to go.

I’m currently holding a MCSE for Core Infrastructure and one for Productivity. Based on this, the Azure Solutions Architect Expert is the next step for me.

Missing Microsoft Teams calendar tab with on-premise Exchange

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Microsoft Teams got a big push due to the current COVID19 crisis and many of my customers deployed it in the past weeks. At ML Network, we are using Microsoft Teams for more than a year, and we don’t want to miss it anymore.

Source: Microsoft

We are running Exchange 2016 on-premises, currently CU16. We were missing the calendar tab in Teams since we started with Microsoft Teams. when you do some research about this issue, you will find many threads and blog posts, but these are the two key facts:

  • it is supported with on-premises hybrid Exchange deployments
  • it works flawless with Exchange Online

Our Exchange is configured as full-hybrid mode deployment. I did this as we deployed Office 365 at our organization.

Let’s summarize:

  • Exchange 2016 CU16
  • Hybrid Deployment
  • Office 365 with Teams enabled
  • no calendar tab when the Exchange mailbox is hosted on-premises

OAuth FTW!

While doing an Exchange Hybrid deployment for one of my customers some weeks ago, I’ve stumbled over an OAuth error message at the end of the Hybric Connection Wizard. The message was HCW8064

“HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration”

We were not able to fix this. Microsoft offers two solutions:

Yesterday I did the upgrade from CU15 to CU16 on our Exchange server and while watching the progress bar I did some research on this issue again. I found strong evidence that Microsoft Teams needs working OAuth to display the calendar tab and access the on-premises hosted mailbox. So I gave it a try and used the latest version of the HCW wizard.

What should I say? No OAuth configuration error and after a restart of Microsoft Teams, the calendar tab appeared.

Lessons Learned:

  • always use the latest CU für Exchange
  • always use the latest HCW Wizard

Connecting to Exchange Online with PowerShell

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The task was simple: Change the alias and the primary SMTP address of a Microsoft Teams team. This can be done by changing the alias and the SMTP address of the underlaying Office 365 group. But how? All you need is a PowerShell connection to Exchange Online.

All you need is a PowerShell on your local computer and Office 365 credentials with the necessary privileges.

First we need to provide the necessary credentials.

 $cred = Get-Credential

A windows will come up and you must enter your Office365 credentials.

The next step is to create a PowerShell remote session with Exchange Online.

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $cred -Authentication Basic -AllowRedirection

Please note that basic auth will be disabled in October 2020!

To connect to this remote session, use Import-PSSession.

Import-PSSession $Session -DisableNameChecking

When you finished your work, make sure to remove the remote session with Remove-PSSession!

Remove-PSSession $Session 

Load balancing ADFS and ADFS Proxy using Citrix ADC

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week I had to setup a small Active Directory Federation Services (ADFS) farm that will be used to allow Single Sign-On (SSO) with Office 365.

Active Directory Federation Services (ADFS) is a solution developed by Microsoft to provide users an authenticated access to applications, that are not capable of using Integrated Windows Authentication (IWA).

Required by the customer was a two node ADFS farm located on the internal network, and a two node ADFS Proxy farm located at the DMZ.

An ADFS Proxyserver acts as a reverse proxy and it is typically located in your organizations perimeter network (DMZ).

This picture shows a typical ADFS/ ADFS Proxy setup:

ADFS/ WAP Design/ Citrix/ citrix.com

My customer has decided to use Citrix ADC (former NetScaler) to load balance the requests for the ADFS farm and the ADFS Proxy farm. In addition to load balancing, this offers high availability in case of a failed ADFS server or ADFS Proxy server. Please note that Citrix ADC can act as a ADFS Proxy, but this requires the Advanced Edition license. My customer “only” had a Standard License, so we had to setup dedicated ADFS Proxy servers on the DMZ network.

Citrix ADC setup

The ADFS service name is typically something like adfs.customer.tld. This farm name has to be the same for internal and external access. For internal access, the ADFS service name must be resolved to the VIP of the Citrix ADC. The same applies to external accesss. So you have to setup split DNS.

ADFS uses HTTP and HTTP, so my first attempt was to use this Citrix ADC Content Switch based setup:

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add cs vserver cs_vsrv_adfs SSL x.x.x.x 443 -cltTimeout 180 -caseSensitive OFF
add lb vserver lb_vsrv_adfs SSL 0.0.0.0 0 -persistenceType SSLSESSION -cltTimeout 180

add cs action cs_action_adfs -targetLBVserver lb_vsrv_adfs
add cs policy cs_pol_adfs -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"adfs.customer.tld\")" -action cs_action_adfs
bind cs vserver cs_vsrv_adfsL -policyName cs_pol_adfs -priority 100

add serviceGroup svcgrp_adfs SSL -maxClient 0 -maxReq 0 -cip ENABLED X-MS-Forwarded-Client-IP -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES -appflowLog DISABLED

add lb monitor mon_adfs HTTP-ECV -send "GET /federationmetadata/2007-06/federationmetadata.xml" -recv "adfs.customer.tld/adfs/services/trust" -LRTM ENABLED -secure YES

bind serviceGroup svcgrp_adfs srv_gk-adfs1 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs srv_gk-adfs2 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs -monitorName mon_adfs

bind lb vserver lb_vsrv_adfs svcgrp_adfs

bind ssl vserver lb_vsrv_adfs -certkeyName cert-key-pair
bind ssl vserver cs_vsrv_adfs -certkeyName cert-key-pair

set ssl vserver lb_vsrv_adfs -ssl3 DISABLED
set ssl vserver cs_vsrv_adfs -ssl3 DISABLED

This is a pretty common setup for HTTP/ HTTPS based services. But it doesn’t work… Mainly because the monitor was not getting the required response. So the monitored service was down for the ADC, and therefore the service group, the load balancing virtual server and the content switch won’t came up.

The reason for this is Server Name Indication (SNI), an extension to Transport Layer Security (TLS). SNI is enabled and required since ADFS 3.0. The monitor tries to access the URL http://x.x.x.x/federationmetadata/2007-06/federationmetadata.xml, but the ADFS service won’t answer to those requests, because it includes the ip address, and not the ADFS service name.

But there is a workaround for everything on the Internet! You can change the binding on the ADFS server nodes using netsh.

netsh http add sslcert ipport=<IPAddress:port> certhash=<certhash> appid=<appid> certstorename=MY

I will not add the necessary options to this command, because: DON’T DO THIS!

Yes, the service group, the load balancing virtual server and the content switch will come up after this change. But you will not be able to enable a trust between your ADFS Proxy servers and the ADFS farm.

Microsofts requirements on Load Balancing ADFS

Microsoft offers a nice overview about the requirements when deploying ADFS. There is a section about the Network requirements. Below this, Microsoft clearly documents the requirements when load balancing ADFS servers and ADFS Proxy servers.

The load balancer MUST NOT terminate SSL. AD FS supports multiple use cases with certificate authentication which will break when terminating SSL. Terminating SSL at the load balancer is not supported for any use case.

Requirements for deploying AD FS/ microsoft.com

Okay, with this in mind, the you can’t use a ADC Content Switch as described above. Because it will terminate SSL. You have to switch to a load balancing virtual server and a service group with SSL bridge . Citrix describes SSL bridge as follows:

A SSL bridge configured on the NetScaler appliance enables the appliance to bridge all secure traffic between the SSL client and the SSL server. The appliance does not offload or accelerate the bridged traffic, nor does it perform encryption or decryption. Only load balancing is done by the appliance. The SSL server must handle all SSL-related processing. Features such as content switching, SureConnect, and cache redirection do not work, because the traffic passing through the appliance is encrypted.

But there is a second, very interesting statement:

It is recommended to use the HTTP (not HTTPS) health probe endpoints to perform load balancer health checks for routing traffic. This avoids any issues relating to SNI. The response to these probe endpoints is an HTTP 200 OK and is served locally with no dependence on back-end services. The HTTP probe can be accessed over HTTP using the path ‘/adfs/probe’http://<Web Application Proxy name>/adfs/probe
http://<ADFS server name>/adfs/probe
http://<Web Application Proxy IP address>/adfs/probe
http://<ADFS IP address>/adfs/probe

Requirements for deploying AD FS/ microsoft.com

This is pretty interesting, because it addresses the above described issue with the monitor. The solution to this is a HTTP-ECV monitor with on port 80, a GET to “/adfs/probe” and the check for a HTTP/200.

A working Citrix ADC setup

This setup is divided into two parts: One for the ADFS farm, and a second one for the ADFS Proxy farm. It uses SSL bridge and HTTP for the service monitor.

Load balancing the ADFS farm

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add serviceGroup svcgrp_adfs SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfs SSL_BRIDGE x.x.x.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfs srv_adfs1 443
bind serviceGroup svcgrp_adfs srv_adfs2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_http
bind lb vserver lb_vsrv_adfs svcgrp_adfs
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

Load balancing the ADFS Proxy farm

add server srv_adfsproxy1 y.y.y.y
add server srv_adfsproxy2 y.y.y.x

add serviceGroup svcgrp_adfsproxy SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfsproxy SSL_BRIDGE y.y.y.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_proxy_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfsproxy srv_adfsproxy1 443
bind serviceGroup svcgrp_adfsproxy srv_adfsproxy2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_proxy_http
bind lb vserver lb_vsrv_adfsproxy svcgrp_adfsproxy
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

I have implemented it on a NetScaler 12.1 with a Standard license. If you have feedback or questions, please leave a comment. :)

Supported Active Directory environments for Microsoft Exchange

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

It is time for some words of wisdom, in regard to Exchange and the supported Active Directory environments. It is the same as with the supported. NET Framework releases: Latest release does not automatically mean “supported”.

To be honest: I nearly nuked a customer environment with ~ 300 users yesterday by preparing the domain for the first Windows Server 2019 Domain Controller.

First things first: Everything is fine! I did not prepared to forest schema for Windows Server 2019.

The support for Windows Server 2008 R2 comes to an end and some customers are still running it. Like my customer yesterday. Some application servers are still on 2008 R2… and the Domain Controllers. The customer is also running Exchange 2013 on Windows Server 2012 R2.

The customer has decided to go to Windows Server 2019 wherever possible. This includes file servers, application servers, and the Domain Controllers. On of the first steps was the deployment of Active Directory-Based Activation. The AD schema needs to be prepared for this and I decided to prepare the schema for Windows Server 2019. I already copied the adprep folder from the Server 2019 ISO and openened a PowerShell. And then I paused. Something felt odd. I wanted to take a look at the Exchange Server supportability matrix.

Exchange 2013 does NOT supported Windows Server 2019 Domain Controllers! Uhh… that was unexpected.

Lessons learned

Always check the Exchange Server supportability matrix. Always! Regardless if it’s because of .NET Framework, Active Directory, Outlook Clients etc. Just check it every time you plan to change something in your environment.

Especially in regard to Microsoft Exchange “newer” does not automatically mean “supported”. Most times the opposite is true.

Microsoft Exchange 2013/ 2016/ 2019 shows blank ECP & OWA after changes to SSL certificates

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in Exchange 2013 CU6.

I published this blog post in July 2015 and it is still relevant. The feedback for this blog post was incredible, and I’m not joking when I say: I saved many admins weekends. ;) It has shown, that this error still occurs with Exchange 2016 and even 2019. Maybe not because of the same, with Exchange 2013 CU6 fixed bug, but maybe for other reasons. And the solution below still applies to it. Because of this I have decided to re-publish this blog post with a modified title and this little preamble.

Feel free to leave a comment if this blog post worked for you. :)

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

C:\windows\system32>netsh http show sslcert

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:444
    Certificate Hash             : a80c9de605a1525cd252c250495b459f06ed2ec1
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:8172
    Certificate Hash             : 09093ca95154929df92f1bee395b2670a1036a06
    Application ID               : {00000000-0000-0000-0000-000000000000}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 127.0.0.1:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

C:\windows\system32>netsh http delete sslcert ipport=0.0.0.0:444

SSL Certificate successfully deleted

Now add it again with the correct certificate hash and application ID.

C:\windows\system32>netsh http add sslcert ipport=0.0.0.0:444 certhash=1ec7413b4fb1782b4b40868d967161d29154fd7f appid="{4dc3e181-e14b-4a21-b022-59fc669b0914}"

SSL Certificate successfully added

That’s it. Reboot the Exchange server and everything should be up and running again.

Notes for a 2-Tier Microsoft Windows PKI

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Implementing a public key infrastructure (PKI) is a recurring task for me. More and more customers tend to implement a PKI in their environment. Mostly not to increase security, rather then to get rid of browser warnings because of self-signed certificates, to secure intra-org email communication with S/MIME, or to sign Microsoft Office macros.

tumbledore / pixabay.com/ Pixybay License

What is a 2-tier PKI?

Why is a multi-tier PKI hierarchy a good idea? Such a hierarchy typically consits of a root Certificate Authority (CA), and an issuing CA. Sometimes you see a 3-tier hierarchy, in which a root CA, a sub CA and an issuing CA are tied together in a chain of trust.

A root CA issues, stores and signs the digital certificates for sub CA. A sub CA issues, stores and signs the digital certificates for issuing CA. Only an issuing CA issues, stores and signs the digital certificates for users and devices.

In a 2-tier hierarchy, a root CA issues the certificate for an issuing CA.

In case of security breach, in which the issuing CA might become compromised, only the CA certificate for the issuing CA needs to be revoked. But what of the root CA becomes compromised? Because of this, a root CA is typically installed on a secured, and powered-off (offline) VM or computer. It will only be powered-on to publish new Certificate Revocation Lists (CRL), or to sign/ renew a new sub or issuing CA certificate.

Lessons learned

Think about the processes! Creating a PKI is more than provisioning a couple of VMs. You need to think about processes to

  • request
  • sign, and
  • revoke

Be aware of what a digital certificate is. You, or your CA, confirms the identity of a party by handing out a digital certificate. Make sure that no one can issue certificates without a proof of his identity.

Think about lifetimes of certificates! Customers tend to create root CA certificates with lifetimes of 10, 20 or even 40 years. Think about the typical lifetime of a VM or server, which is necessary to run an offline root CA. Typically the server OS has a lifetime of 10 to 12 years. This should determine the lifetime of a root CA certificate. IMHO 10 years is a good compromise.

For a sub or issuing CA, a lifespan of 5 years is a good compromise. Using the same lifetime as for a root CA is not a good idea, because an issued certificate can’t be longer valid than the lifetime of the CA certificate of the issuing CA.

A lifespan of 1 to 3 years for thinks like computer or web server certificates is okay. If a certificate is used for S/MIME or code signing, you should go for a lifetime of 1 year.

But to be honest: At the end of the day, YOU decide how long your certificates will be valid.

Publish CRLs and make them accessable! You can’t know if a certificate is revoked by a CA. But you can use a CRL to check if a certificate is revoked. Because of this, the CA must publish CRLs regulary. Use split DNS to use the same URL for internal and external requests. Make sure that the CRL is available for external users.

This applies not only to certificates for users or computers, but also for sub and issuing CAs. So there must be a CRL from each of your CAs!

I recommend to publish CRLs to a webserver and make this webserver reachable over HTTP. An issued certificate includes the URL or path to the CRL of the CA, that has issued the certificate.

Make sure that the CRL has a meaningful validity period. Of an offline root CA, which issues only a few certificates of its lifetime, this can be 1 year or more. For an issuing CA, the validity period should only a few days.

Publish AIA (Authority Information Access) information and make them accessable! AIA is an certificate extension that is used to offer two types of information :

  • How to get the certificate of the issuing or upper CAs, and
  • who is the OCSP responder from where revocation of this certificate can be checked

I tend to use the same place for the AIA as for the CDP. Make sure that you configure the AIA extension before you issue the first certificates, especially configure the AIA and CDP extension before you issue intermediate and issuing CA certificates.

Use a secure hash algorithm and key length! Please stop using SHA1! I recommend at least SHA256 and 4096 bit key length. Depending on the used CPUs, SHA512 can be faster than SHA256.

Create a CApolicy.inf! The CApolicy.inf is located uder C:\Windows and will will be used during the creation of the CA certificate. I often use this CApolicy.inf files.

For the root CA:

[Certsrv_Server]
RenewalKeyLength = 4096
RenewalValidityPeriod = Years
RenewalValidityPeriodUnits = 10
AlternateSignatureAlgorithm = 0
CNGHashAlgorithm = SHA512
CRLPeriod = years
CRLPeriodUnits = 1

For the issuing CA:

[Certsrv_Server]
RenewalKeyLength = 4096
RenewalValidityPeriod = Years
RenewalValidityPeriodUnits = 5
AlternateSignatureAlgorithm = 0
CNGHashAlgorithm = SHA512
CRLPeriod = Days
CRLPeriodUnits = 7
CRLDeltaPeriod = Hours
CRLDeltaPeriodUnits = 12
[CRLDistributionPoint]
URL = http://crl.domain.tld/crld/RootCA.crl

Final words

I do not claim that this is blog post covers all necessary aspects of such an complex thing like an PKI. But I hope that I have mentioned some of the important parts. And at least: I have a reference from which I can copy and paste the CApolicy.inf files. :D

Windows NPS – Authentication failed with error code 16

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, a customer called me and reported, on the first sight, a pretty weired error: Only Windows clients were unable to login into a WPA2-Enterprise wireless network. The setup itself was pretty simple: Cisco Meraki WiFi access points, a Windows Network Protection Server (NPS) on a Windows Server 2016 Domain Controller, and a Sophos SG 125 was acting as DHCP for different WiFi networks.

Pixybay / pixabay.com/ Pixabay License

Windows clients failed to authenticate, but Apple iOS, Android, and even Windows 10 Tablets had no problem.

The following error was logged into the Windows Security event log.

Authentication Details:
Connection Request Policy Name: Use Windows authentication for all users
Network Policy Name: Wireless Users
Authentication Provider: Windows
Authentication Server: domaincontroller.domain.tld
Authentication Type: PEAP
EAP Type: -
Account Session Identifier: -
Logging Results: Accounting information was written to the local log file.
Reason Code: 16
Reason: Authentication failed due to a user credentials mismatch. Either the user name provided does not map to an existing user account or the password was incorrect.

The credentials were definitely correct, the customer and I tried different user and password combinations.

I also checked the NPS network policy. When choosing PEAP as authentication type, the NPS needs a valid server certificate. This is necessary, because the EAP session is protected by a TLS tunnel. A valid certificate was given, in this case a wildcard certificate. A second certificate was also in place, this was a certificate for the domain controller from the internal enterprise CA.

It was an educated guess, but I disabled the server certificate check for the WPA2-Enterprise conntection, and the client was able to login into the WiFi. This clearly showed, that the certificate was the problem. But it was valid, all necessary CA certificates were in place and there was no reason, why the certificate was the cause.

The customer told me, that they installed updates on friday (today is monday), and a reboot of the domain controller was issued. This also restarted the NPS service, and with this restart, the Wildcard certificate was used for client connections.

I switched to the domain controller certificate, restarted the NPS, and all Windows clients were again able to connect to the WiFi.

Lessons learned

Try to avoid Wildcard certificates, or at least check the certificate that is used by the NPS, if you get authentication error with reason code 16.