Tag Archives: microsoft

Exchange Control Panel /ecp broken after certificate replacement

As part of an ongoing Exchange 2010 to 2016 migration, I had to replace the self-signed certificate with a certificate from the customers PKI. Everything went fine, the customer had a suitable template, we’ve added the necessary hostnames and bound IIS and SMTP to the certificate. The mess started with an iisreset /noforce

Bild von Oskars Zvejs auf Pixabay 

The iisreset took longer than expected. After that, I tried to login into the ECP, entered username and password and got an error.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="MSExchange Front End HTTP Proxy" />
  <EventID Qualifiers="49152">1003</EventID>
  <Level>2</Level>
  <Task>1</Task>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2020-10-22T12:16:38.934123400Z" />
  <EventRecordID>368718</EventRecordID>
  <Channel>Application</Channel>
  <Computer>server.domain.tld</Computer>
  <Security />
</System>
<EventData>
  <Data>Owa</Data>
  <Data>System.NullReferenceException: Object reference not set to an instance of an object. at 
Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.ProxyModule.<>c__DisplayClass16_0.<OnBeginRequest>b__0() at 
Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
</Data>
</EventData>
</Event>

Pretty strange. We switched back to the self-singned certificate, did an iisreset and everyting was fine again.So it was pretty obvious that the error was related to the certificate, or to be more clear, to the certificate template.

A short research confirmed this. The template was a modified v3 web server template from an Enterprise CA running Windows Server 2008 R2.

With Windows Server 2008, Microsoft introduced a new cryptographic API called Cryptography Next Generation (CNG), which separates cryptographic providers (algorithm implementation) from key storage providers (create, delete, export, import, open and store keys). The older CryptoAPI does not differ between this and implements cryptographic algorithms and key storage.

The modified template used CNG instead of CryptoAPI. We noticed this when we checked the certificate with certutil -store my <thumbprint>.

If the listed provider for the certificate is Microsoft Software Key Storage Provider, then you will have to re-import the certificate. If Microsoft RSA SChannel Cryptographic Provider is used, everything is fine.

You have to remove the certificate, then re-import it using

certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx <CertificateFilename>

You need a PKCS#12 file (PFX) and the password. Re-import it and then you can use the certificate for Exchange. Bind services to it and restart the IIS.

Moving a small on-prem environment to Azure/ O365 – Part 1

It was a bit quiet here due to the current COVID 19 pandemic. But now I’m back with a pretty interesting story on how my colleagues and I moved most of our on-prem server stuff to Microsoft Azure and Office 365.

Microsoft 2020 (c)

It all started with the COVID19 lockdown in Germany in March 2020. We moved into our home offices after setting up a small VMware Horizon View deployment to access our PCs using physical View Agents and manual desktop pools. Most projects were stopped, and we did most of our work remote. No lay-offs or short-time work.

We were running a small VMware vSphere cluster for a couple of years. Nothing fancy: Two HPE ProLiants, vCenter, two DCs, File-/ Printserver, WSUS, Exchange, Linux maschines for web services, Sophos UTM, a pfSense, View Connection Server, UAG, ADFS/ ADFS Proxy, PKI etc. In sum 18 VMs on two hosts, some VLANs with firewalls in between etc. We were running Exchange 2016, AzureAD Sync, Exchange Hybrid, but we only used Microsoft Teams from our Office 365 deployment. Veeam Backup & Replication was used for backups, a backup copy to a NAS and some Robocopy jobs that moved Veeam Backups to USB drives for DR. Everything was pretty simple and designed to work without much operations. Our focus is on our customers, not on our internal IT. It was stable, secure and pretty slick.

In March 2020 we asked “What if?”. What if we lose our offices due to a fire (we are located in a bigger office building and we had a couple of fire alarms this year due to remodeling work). How can we work if your DSL line is cut? How can we get our backups offsite? How can we modernize our IT withtout big invests? Money, that we don’t spend on our internal IT can given to our employees. ;) (By the way, that’s the same reason why we try to drive smaller and more efficient cars…).

We developed a couple of ideas, including new servers, storage etc. and put that stuff into a datacenter. But in the end, we decided to move most of our stuff to Microsoft Azure and Office 365.

I want to share some of the things we have learned on the road to Azure.

Initial assessment

We used the Azure Migrate Server Assessment tool to assess our vSphere environment. We wanted to get a ball park on how we had to size the VMs. We knew, that we wont need to migrate all VMs. For example our virtual pfSense firewall, the vCenter, the Sophos UTM or our ADFS setup were not planned to migrate.

After the first assessment, we started to play around the the Azure pricing calculator. Just to get an idea on how different VM sizes affect the costs.

Subscription

As a Microsoft partner, we were able to use our internal user rights (IUR) for Microsoft Office 365 and Azure. Microsoft offers us 25 Office 365 E3 plans and a 6000 US-$ budget for Azure (Azure Sponsorship Subscription). Our plan was to stretch the Azure budget over 12 months, so that we don’t have additional costs until we re-apply for our Microsoft partnership. Starting with 6000 US-$ Azure budget, it makes ~16 US-$ per day for our complete Azure deployment.

Sizing

Now, as we knew that we have 16 US-$ per day, we planned our Azure deployment. First of all, we planned the number of VMs. We had 18 VMs on-prem, and we managed to get down to 9 VMs.

  • two Domain Controller
  • PKI
  • Fileserver
  • Management Server
  • Remote Desktop Host (all-in-one Deployment)
  • Ticket System
  • SQL/ App server
  • Exchange 2016 for Hybrid Deployment

The View Connection Servers and UAG are still running on-prem. Our virtual pfSense will be moved to a WatchGuard Firebox soon. Sophos UTM and ADFS are gone. A dedicated WSUS server is not necessary any more, we moved back to simple Windows Update and Delivery Optimization.

Instead of D-Series VMs, we decided to go for B-Series VMs. The main reason for this were costs, but today I can say: The performance is quite good. I can’t see any reason for us to move to D-Series.

To connect to our Azure deployment, we had to setup Site-2-Site VPN. We deployed a simple Gen1 Basic SKU VPN Gateway. We had no need for more than 100 Mbit (we’re using a 50/10 VDSL at our office location), BGP or zone redundancy.

Instead of Standard SSD or Premium SSD storage, most of our VMs are using simple Standard HDD storage.

Backups are kept in a Recovery Services Vault with pretty simple polices. Either a VM needs to be current, in this case we keep 7 restore points, or we might need to keep more restore points. In this case we keep 7 daily, 5 weekly, 12 monthly and 3 yearly restore points. And this is only the case for our fileserver.

Additional cost savings

But with this setup we would not get under 16 US-$ a day. :( So we took another approach to break the mark: We shut down VMs at night and at the weekends! It took a bit until my colleagues and I get used to this. Nobody wants to shut down servers without a good reason.

But: We are currently at 18 US-$ per workday, and 10 US-$ for saturday and sunday. Everything, except domain controllers and ticket system, is shutdown at night and on the weekends.

We are using an Automation Account with some simple scripts and schedules to shutdown VMs and start them again.

What’s next?

The next blog post will be around how we planned the usage of Office 365, and how we started with Azure.

Access to on-premise hosted Public Folders using Exchange Online mailboxes

Public Folders are still a thing. And while companies are moving their stuff into the cloud, Public Folders still need to be accessed by cloud-located mailboxes.

Allowing the access from Exchange Online mailboxes to on-premise hosted Public Folders is well documented by Microsoft, but there are also some fuzz. I had to deal with this during a Office 365 transition project at one of my customers.

The background

The customer is running a single Exchange 2016 server in a Windows Server 2012 R2 forest. AzureAD Sync is running and its syncing on-premise identities to AzureAD. The customer uses Office 365 E5 plans and he wants to move to Exchange Online, aside other O365 services like SharePoint Online, Teams etc.

Something was missing

After setting up the Exchange Hybrid, the customer and I where able to migrate the first mailboxes to Exchange Online.

To our surprise the on-premise Public Folders were not visible from the migrated Exchange Online mailboxes. We had still things to do…

In order to get the access to the Public Folders working, the Public Folder mailbox object needs to be synced to AzureAD. This is not complicated, because all you need to make sure is, that the user object is synced. If you are using an OU filter for the AzureAD sync, make sure that the OU with the Public Folder mailbox user object is included into the sync.

But there is also a second requirement: You also need to sync the Microsoft Exchange System Objects container! This is pretty important and it is not mentioned in the docs (Public folders in Microsoft 365, Office 365, and Exchange Online).

Check if the user is synced by using the Exchange Online PowerShell. This is how the Public Folder mailbox user looks like from the Exchange Online perspective:

PS C:\Users\p.terlisten> Get-MailUser Mailbox1

Name                                     RecipientType
----                                     -------------
Mailbox1                                 MailUser

This is the Public Folder mailbox that is hosted on-premise.

PS C:\Users\p.terlisten> Get-Mailbox -PublicFolder

Name                      Alias                ServerName     ProhibitSendQuota
----                      -----                ----------     -----------------
Mailbox1                  Mailbox1             EX             Unlimited

As long as you can’t see the MailUser in Exchange Online, you have no chance to configure the Public Folder access.

The next step is to synchronize the mail-enabled Public Folder objects to Exchange Online. For this, you have to download two scripts from Microsoft.

  • Sync-ModernMailPublicFolders.ps1
  • Sync-ModernMailPublicFolders.psd1

Run the Sync-ModernMailPublicFolders.ps1 script on your on-premise Exchange server. You will need your Office 365 admin credentials for this task.

PS C:\Users\p.terlisten\Download> .\Sync-ModernMailPublicFolders.ps1 -Credential (Get-Credential) -CsvSummaryFile:sync_summary.csv

The last step is to enable the Public Folder access using the Set-OrganizationConfig cmdlet in the Exchange Online PowerShell session.

PS C:\Users\p.terlisten> Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes Mailbox1

Please note that some of these steps need some time to get active! It will take some time for the background tasks to get some things sorted.

Controlled Connections to Public Folders in Outlook

It is worth mentioning that after enabling the access to Public Folders all Exchange online users can see the on-premise hosted Public Folders. If you need to enable the access only for some Exchange Online users, Microsoft has a solution for you: Controlled Connections to Public Folders.

First, you need to enable the Public Folder access for the users you have selected.

Set-CASMailbox "John Doe" -PublicFolderClientAccess $true

Then you enable the feature globally using the Exchange Online PowerShell.

Set-OrganizationConfig -PublicFolderShowClientControl $true

This setting applies only to Exchange Online hosted mailboxes!

Office365/ Exchange OAuth errors after replacing TLS certificate

A customer of mine asked for help to analyse a weird OAuth error. They are using a Microsoft Dynamics 365 Outlook plugin, which came up with an error:

“Can’t connect to Exchange”

In addition to this, they also faced an issueaccessing shared calendars of Exchange Online mailboxes.

Clearly an OAuth error. So we ran the Hybrid Connection Wizard again, which finished without any errors. But the errors persisted. Next stop: OAuth configuration.

We logged into one of the Exchange servers, started an Exchange Management Shell and checked the current OAuth configuration:

[PS] C:\Windows\system32>Get-AuthConfig
RunspaceId : e7c560cd-8316-4d9c-b97d-f4358d665e7d
CurrentCertificateThumbprint : CA420022ACC542D3AF7598456615715E98FFE986
PreviousCertificateThumbprint : E7B56BEAF4704BCBBF8C297F1D8D000215679C6F
NextCertificateThumbprint :
NextCertificateEffectiveDate :
ServiceName : 00000002-0000-0ff1-ce00-000000000000
Realm :
DeploymentId :
IssuerIdentifier :
Name : Auth Configuration
AdminDisplayName :
ExchangeVersion : 0.20 (15.0.0.0)
DistinguishedName : CN=Auth Configuration,CN=EXORG,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=tld
Identity : Auth Configuration
Guid : 911b6246-6248-4278-91be-1b7b4943d2f1
ObjectCategory : mlnetwork.local/Configuration/Schema/ms-Exch-Auth-Auth-Config
ObjectClass : {top, container, msExchContainer, msExchAuthAuthConfig}
WhenChanged : 11.04.2020 12:33:12
WhenCreated : 27.02.2015 14:24:37
WhenChangedUTC : 11.04.2020 10:33:12
WhenCreatedUTC : 27.02.2015 13:24:37
OrganizationId :
Id : Auth Configuration
OriginatingServer : DC01.domain.tld
IsValid : True
ObjectState : Unchanged

The CurrentCertificateThumbprint was the thumbprint of the old certificate. So we had to update it.

Set-AuthConfig -NewCertificateThumbprint 45CCEEDFAAB00F3A4312FF56774FA3B3412B9C (get-date)
Set-AuthConfig -PublishCertificate
iisreset /noforce
Set-AuthConfig -ClearPreviousCertificate

After the iisreset, the Dynamics plugin and the Exchange Online calendars started to work again.

Once in a year: How to update TLS certificates on ADFS server and proxies

You might got this news some days ago: Starting with September 1, 2020, browsers and devices from Apple, Google, and Mozilla will show errors for new TLS certificates that have a lifespan greater than 398 days. Due to this move from Apple, Google and Mozilla, you have to deal with the replacement of certificates much more often. And we all know: Replacing certificates can be a real PITA!

Bild von skylarvision auf Pixabay

Replacing TLS certificates used for ADFS and Office 365 can be a challenging task, and this blog post will cover the neccessary steps.

ADFS Server

The first service, for which we will replace the certificate, is the ADFS server, or the ADFS server farm. At this point it is important to understand that we are dealing with two different points to which the certificate is bound:

  • the ADFS service communications certificate, and
  • the ADFS SSL certificate

The first step is to replace the service communication certificate. After importing the certificate with private key, you need to assign “read” permission to the ADFS service account. Right click on the certificate, then “All Tasks” > “Manage Private Keys”.

Make sure to import the certificate on all farm servers! Next step: Start the ADFS management console on the primary node. Select “Certificates” and then “Select service communication certificate” on the right window pane.

Now we have successfully replaced the service communication certificate. But we are no finished yet! Now we have to set the ADFS SSL certificate. Depending on your OS, you have to run the PowerShell command on the primary node. If your are running Windows Server 2012 R2 or older, you have to run the PowerShell command on EVERY ADFS farm server!

You can get the certificate thumbprint using the Get-AdfsSslCertificate command. Set the ADFS SSL certificate with

Set-AdfsSslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then restart the ADFS service.

Restart-Service adfssrv

ADFS Proxies

In most cases you will have one or more ADFS proxies in your DMZ. The ADFS proxy is nothing more than a Web Application Proxy (WAP) and therefore the PowerShell commands for WAP will be used.

First of all: Import the new certificate with the private key on all ADFS proxies, and then get the certificate hash of the new certificate. Then open an elevated PowerShell on each proxy.

Set-WebApplicationProxySslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then we have to re-establish the trust between the proxies and the primary ADFS farm server. You will need the local (!) administrator account of the primary farm server.

$cred = Get-Credential
Install-WebApplicationProxy -FederationServiceTrustCredential $cred -CertificateThumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125 -FederationServiceName adfs.domain.tld

The last step is to update thefederated trust with Office 365.

Update the federated trust with Office 365

To update the federated trust with Office 365, you will need the Windows Azure Active Direcotry Module for Windows PowerShell and an elevated PowerShell. Connect to Office 365 and update the federated trust:

$cred = Get-Credential
Connect-MsolService -Credential $cred
Update-MSOLFederatedDomain –DomainName domain.tld

That’s it! Bookmark this page and set a calendar entry on today +12 months. :)

Passed Microsoft exam AZ-103 – Azure Administrator Associate

Six weeks ago, I passed the Microsoft AZ-103 exam and earned the Azure Administrator Associate. A last minute pass, because AZ-104 was already launched. But better late than never. I had to re-schedule the exam a couple of times because the test center was closed due to COVID19.

The Azure Administrator Associate is a Administrator-role certification and it is all about implementing, managing and monitoring the Azure identity, governance, storage, compute, and virtual network solutions.

The exam covers a couple of topics and you should have knowledge and hands-on experience in administering Azure services using the Azure Portal, PowerShell, Azure CLI, and Azure Resource Manager templates.

Your knowledge is tested over a broad band of topics. These topics are:

  • Manage Azure identities and governance
  • Implement and manage storage
  • Deploy and manage Azure compute resources
  • Configure and manage virtual networking
  • Monitor and back up Azure resources

How to prepare for the exam

Fortunately I have a monthly Azure credit which I can use to gain new skills. I used these Azure credit together with the Microsoft Learning Path for AZ-103 (now 104).

It is pretty important no only to focus on VMs, storage or networking. Web Apps was one of my blind spots, and I had to get my head around it. Azure identities and governance is not so hard, if you are already familiar with Office 365.

I learned a lot from the Microsoft Documentation for Azure, and I was really impressed how much I was able to find, read and learn from there.

Next stop: Microsoft Certified: Azure Solutions Architect Expert

Microsoft has announced to retire all remaining exams associated with Microsoft Certified Solutions Associate (MCSA), Microsoft Certified Solutions Developer (MCSD), Microsoft Certified Solutions Expert (MCSE) on January 31, 2021, so the role-based certifications introduced in September 2018 are the way to go.

I’m currently holding a MCSE for Core Infrastructure and one for Productivity. Based on this, the Azure Solutions Architect Expert is the next step for me.

Missing Microsoft Teams calendar tab with on-premise Exchange

Microsoft Teams got a big push due to the current COVID19 crisis and many of my customers deployed it in the past weeks. At ML Network, we are using Microsoft Teams for more than a year, and we don’t want to miss it anymore.

Source: Microsoft

We are running Exchange 2016 on-premises, currently CU16. We were missing the calendar tab in Teams since we started with Microsoft Teams. when you do some research about this issue, you will find many threads and blog posts, but these are the two key facts:

  • it is supported with on-premises hybrid Exchange deployments
  • it works flawless with Exchange Online

Our Exchange is configured as full-hybrid mode deployment. I did this as we deployed Office 365 at our organization.

Let’s summarize:

  • Exchange 2016 CU16
  • Hybrid Deployment
  • Office 365 with Teams enabled
  • no calendar tab when the Exchange mailbox is hosted on-premises

OAuth FTW!

While doing an Exchange Hybrid deployment for one of my customers some weeks ago, I’ve stumbled over an OAuth error message at the end of the Hybric Connection Wizard. The message was HCW8064

“HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration”

We were not able to fix this. Microsoft offers two solutions:

Yesterday I did the upgrade from CU15 to CU16 on our Exchange server and while watching the progress bar I did some research on this issue again. I found strong evidence that Microsoft Teams needs working OAuth to display the calendar tab and access the on-premises hosted mailbox. So I gave it a try and used the latest version of the HCW wizard.

What should I say? No OAuth configuration error and after a restart of Microsoft Teams, the calendar tab appeared.

Lessons Learned:

  • always use the latest CU für Exchange
  • always use the latest HCW Wizard

Connecting to Exchange Online with PowerShell

The task was simple: Change the alias and the primary SMTP address of a Microsoft Teams team. This can be done by changing the alias and the SMTP address of the underlaying Office 365 group. But how? All you need is a PowerShell connection to Exchange Online.

All you need is a PowerShell on your local computer and Office 365 credentials with the necessary privileges.

First we need to provide the necessary credentials.

 $cred = Get-Credential

A windows will come up and you must enter your Office365 credentials.

The next step is to create a PowerShell remote session with Exchange Online.

$Session = New-PSSession -ConfigurationName Microsoft.Exchange -ConnectionUri https://outlook.office365.com/powershell-liveid/ -Credential $cred -Authentication Basic -AllowRedirection

Please note that basic auth will be disabled in October 2020!

To connect to this remote session, use Import-PSSession.

Import-PSSession $Session -DisableNameChecking

When you finished your work, make sure to remove the remote session with Remove-PSSession!

Remove-PSSession $Session 

Load balancing ADFS and ADFS Proxy using Citrix ADC

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week I had to setup a small Active Directory Federation Services (ADFS) farm that will be used to allow Single Sign-On (SSO) with Office 365.

Active Directory Federation Services (ADFS) is a solution developed by Microsoft to provide users an authenticated access to applications, that are not capable of using Integrated Windows Authentication (IWA).

Required by the customer was a two node ADFS farm located on the internal network, and a two node ADFS Proxy farm located at the DMZ.

An ADFS Proxyserver acts as a reverse proxy and it is typically located in your organizations perimeter network (DMZ).

This picture shows a typical ADFS/ ADFS Proxy setup:

ADFS/ WAP Design/ Citrix/ citrix.com

My customer has decided to use Citrix ADC (former NetScaler) to load balance the requests for the ADFS farm and the ADFS Proxy farm. In addition to load balancing, this offers high availability in case of a failed ADFS server or ADFS Proxy server. Please note that Citrix ADC can act as a ADFS Proxy, but this requires the Advanced Edition license. My customer “only” had a Standard License, so we had to setup dedicated ADFS Proxy servers on the DMZ network.

Citrix ADC setup

The ADFS service name is typically something like adfs.customer.tld. This farm name has to be the same for internal and external access. For internal access, the ADFS service name must be resolved to the VIP of the Citrix ADC. The same applies to external accesss. So you have to setup split DNS.

ADFS uses HTTP and HTTP, so my first attempt was to use this Citrix ADC Content Switch based setup:

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add cs vserver cs_vsrv_adfs SSL x.x.x.x 443 -cltTimeout 180 -caseSensitive OFF
add lb vserver lb_vsrv_adfs SSL 0.0.0.0 0 -persistenceType SSLSESSION -cltTimeout 180

add cs action cs_action_adfs -targetLBVserver lb_vsrv_adfs
add cs policy cs_pol_adfs -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"adfs.customer.tld\")" -action cs_action_adfs
bind cs vserver cs_vsrv_adfsL -policyName cs_pol_adfs -priority 100

add serviceGroup svcgrp_adfs SSL -maxClient 0 -maxReq 0 -cip ENABLED X-MS-Forwarded-Client-IP -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES -appflowLog DISABLED

add lb monitor mon_adfs HTTP-ECV -send "GET /federationmetadata/2007-06/federationmetadata.xml" -recv "adfs.customer.tld/adfs/services/trust" -LRTM ENABLED -secure YES

bind serviceGroup svcgrp_adfs srv_gk-adfs1 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs srv_gk-adfs2 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs -monitorName mon_adfs

bind lb vserver lb_vsrv_adfs svcgrp_adfs

bind ssl vserver lb_vsrv_adfs -certkeyName cert-key-pair
bind ssl vserver cs_vsrv_adfs -certkeyName cert-key-pair

set ssl vserver lb_vsrv_adfs -ssl3 DISABLED
set ssl vserver cs_vsrv_adfs -ssl3 DISABLED

This is a pretty common setup for HTTP/ HTTPS based services. But it doesn’t work… Mainly because the monitor was not getting the required response. So the monitored service was down for the ADC, and therefore the service group, the load balancing virtual server and the content switch won’t came up.

The reason for this is Server Name Indication (SNI), an extension to Transport Layer Security (TLS). SNI is enabled and required since ADFS 3.0. The monitor tries to access the URL http://x.x.x.x/federationmetadata/2007-06/federationmetadata.xml, but the ADFS service won’t answer to those requests, because it includes the ip address, and not the ADFS service name.

But there is a workaround for everything on the Internet! You can change the binding on the ADFS server nodes using netsh.

netsh http add sslcert ipport=<IPAddress:port> certhash=<certhash> appid=<appid> certstorename=MY

I will not add the necessary options to this command, because: DON’T DO THIS!

Yes, the service group, the load balancing virtual server and the content switch will come up after this change. But you will not be able to enable a trust between your ADFS Proxy servers and the ADFS farm.

Microsofts requirements on Load Balancing ADFS

Microsoft offers a nice overview about the requirements when deploying ADFS. There is a section about the Network requirements. Below this, Microsoft clearly documents the requirements when load balancing ADFS servers and ADFS Proxy servers.

The load balancer MUST NOT terminate SSL. AD FS supports multiple use cases with certificate authentication which will break when terminating SSL. Terminating SSL at the load balancer is not supported for any use case.

Requirements for deploying AD FS/ microsoft.com

Okay, with this in mind, the you can’t use a ADC Content Switch as described above. Because it will terminate SSL. You have to switch to a load balancing virtual server and a service group with SSL bridge . Citrix describes SSL bridge as follows:

A SSL bridge configured on the NetScaler appliance enables the appliance to bridge all secure traffic between the SSL client and the SSL server. The appliance does not offload or accelerate the bridged traffic, nor does it perform encryption or decryption. Only load balancing is done by the appliance. The SSL server must handle all SSL-related processing. Features such as content switching, SureConnect, and cache redirection do not work, because the traffic passing through the appliance is encrypted.

But there is a second, very interesting statement:

It is recommended to use the HTTP (not HTTPS) health probe endpoints to perform load balancer health checks for routing traffic. This avoids any issues relating to SNI. The response to these probe endpoints is an HTTP 200 OK and is served locally with no dependence on back-end services. The HTTP probe can be accessed over HTTP using the path ‘/adfs/probe’http://<Web Application Proxy name>/adfs/probe
http://<ADFS server name>/adfs/probe
http://<Web Application Proxy IP address>/adfs/probe
http://<ADFS IP address>/adfs/probe

Requirements for deploying AD FS/ microsoft.com

This is pretty interesting, because it addresses the above described issue with the monitor. The solution to this is a HTTP-ECV monitor with on port 80, a GET to “/adfs/probe” and the check for a HTTP/200.

A working Citrix ADC setup

This setup is divided into two parts: One for the ADFS farm, and a second one for the ADFS Proxy farm. It uses SSL bridge and HTTP for the service monitor.

Load balancing the ADFS farm

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add serviceGroup svcgrp_adfs SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfs SSL_BRIDGE x.x.x.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfs srv_adfs1 443
bind serviceGroup svcgrp_adfs srv_adfs2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_http
bind lb vserver lb_vsrv_adfs svcgrp_adfs
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

Load balancing the ADFS Proxy farm

add server srv_adfsproxy1 y.y.y.y
add server srv_adfsproxy2 y.y.y.x

add serviceGroup svcgrp_adfsproxy SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfsproxy SSL_BRIDGE y.y.y.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_proxy_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfsproxy srv_adfsproxy1 443
bind serviceGroup svcgrp_adfsproxy srv_adfsproxy2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_proxy_http
bind lb vserver lb_vsrv_adfsproxy svcgrp_adfsproxy
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

I have implemented it on a NetScaler 12.1 with a Standard license. If you have feedback or questions, please leave a comment. :)

Supported Active Directory environments for Microsoft Exchange

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

It is time for some words of wisdom, in regard to Exchange and the supported Active Directory environments. It is the same as with the supported. NET Framework releases: Latest release does not automatically mean “supported”.

To be honest: I nearly nuked a customer environment with ~ 300 users yesterday by preparing the domain for the first Windows Server 2019 Domain Controller.

First things first: Everything is fine! I did not prepared to forest schema for Windows Server 2019.

The support for Windows Server 2008 R2 comes to an end and some customers are still running it. Like my customer yesterday. Some application servers are still on 2008 R2… and the Domain Controllers. The customer is also running Exchange 2013 on Windows Server 2012 R2.

The customer has decided to go to Windows Server 2019 wherever possible. This includes file servers, application servers, and the Domain Controllers. On of the first steps was the deployment of Active Directory-Based Activation. The AD schema needs to be prepared for this and I decided to prepare the schema for Windows Server 2019. I already copied the adprep folder from the Server 2019 ISO and openened a PowerShell. And then I paused. Something felt odd. I wanted to take a look at the Exchange Server supportability matrix.

Exchange 2013 does NOT supported Windows Server 2019 Domain Controllers! Uhh… that was unexpected.

Lessons learned

Always check the Exchange Server supportability matrix. Always! Regardless if it’s because of .NET Framework, Active Directory, Outlook Clients etc. Just check it every time you plan to change something in your environment.

Especially in regard to Microsoft Exchange “newer” does not automatically mean “supported”. Most times the opposite is true.