Tag Archives: exchange 2016

Exchange Control Panel /ecp broken after certificate replacement

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of an ongoing Exchange 2010 to 2016 migration, I had to replace the self-signed certificate with a certificate from the customers PKI. Everything went fine, the customer had a suitable template, we’ve added the necessary hostnames and bound IIS and SMTP to the certificate. The mess started with an iisreset /noforce…

The iisreset took longer than expected. After that, I tried to login into the ECP, entered username and password and got an error.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="MSExchange Front End HTTP Proxy" />
  <EventID Qualifiers="49152">1003</EventID>
  <Level>2</Level>
  <Task>1</Task>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2020-10-22T12:16:38.934123400Z" />
  <EventRecordID>368718</EventRecordID>
  <Channel>Application</Channel>
  <Computer>server.domain.tld</Computer>
  <Security />
</System>
<EventData>
  <Data>Owa</Data>
  <Data>System.NullReferenceException: Object reference not set to an instance of an object. at 
Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.ProxyModule.<>c__DisplayClass16_0.<OnBeginRequest>b__0() at 
Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
</Data>
</EventData>
</Event>

Pretty strange. We switched back to the self-singned certificate, did an iisreset and everyting was fine again.So it was pretty obvious that the error was related to the certificate, or to be more clear, to the certificate template.

A short research confirmed this. The template was a modified v3 web server template from an Enterprise CA running Windows Server 2008 R2.

With Windows Server 2008, Microsoft introduced a new cryptographic API called Cryptography Next Generation (CNG), which separates cryptographic providers (algorithm implementation) from key storage providers (create, delete, export, import, open and store keys). The older CryptoAPI does not differ between this and implements cryptographic algorithms and key storage.

The modified template used CNG instead of CryptoAPI. We noticed this when we checked the certificate with certutil -store my <thumbprint>.

If the listed provider for the certificate is Microsoft Software Key Storage Provider, then you will have to re-import the certificate. If Microsoft RSA SChannel Cryptographic Provider is used, everything is fine.

You have to remove the certificate, then re-import it using

certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx <CertificateFilename>

You need a PKCS#12 file (PFX) and the password. Re-import it and then you can use the certificate for Exchange. Bind services to it and restart the IIS.

Citrix NetScaler and Exchange: Case-sensitivity of internal and external URLs

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Exchange has known the concept of internal and external URLs for the different services (Outlook Web Access, OAB, EWS, ActiveSync etc) since Exchange 2007. And it’s still confusing people. The internal URL is the URL, that is used to access the desired service from the intranet. The external URL represents the URL that is used to access the service from the internet. Best practice is to use the same URL (the external) for both, use a certificate from a public CA, and use split DNS to access the external domain from the inside of your network.

People tend to imply, that URLs are not case-sensitive. This seems to be true in most cases. The World Wide Web Consortium (W3C) states:

URLs in general are case-sensitive (with the exception of machine names). There may be URLs, or parts of URLs, where case doesn’t matter, but identifying these may not be easy. Users should always consider that URLs are case-sensitive.
Source W3C

Citrix NetScaler and URLs

Citrix NetScaler handles URLs as case-sensitive.

A frequently used concept to load balance Microsoft Exchange with a NetScaler is Content Switching. Policies are used to identify traffic, and actions are used to take action on the traffic that matches the policies. The NetScaler uses the advanced policy engine to create expressions for the Content Switching Policies. When creating a Content Switching policy by creating an expression that uses the CONTAINS operator, you might notice that the results are case-sensitive.

This can be a problem in case of Microsoft Exchange, because /Autodiscover/Autodiscover.xml and /autodiscover/autodiscover.xml, or /ews/exchange.asmx and /EWS/Exchange.asmx are handled different.

Solution

To make sure that different cases are handled, you should add SET_TEXT_MODE(IGNORECASE) to you policy expression. Citrix describes this in CTX115528.

I’ve changed my NetScaler setup script for Exchange to handle this behavior.

Data Protector Exchange GRE and IP-less Exchange DAG

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When dealing with Microsoft Exchange restore requests, you will come across three different restore situations:

a database
a single mailbox
a single mailbox item (mail, calendar entry etc.)

Restoring a complete database is not a complicated task, but restoring a single mailbox, or a single mailbox item, is. First, you need to restore the mailbox, that includes the desired mailbox, into a recovery database. Then you can restore the mailbox, or the mailbox items, from the recovery database. Some of the tasks can only be done with the Exchange Management Shell.

The HPE Data Protector Granular Recovery Extension (GRE) for Microsoft Exchange helps you to simplify the necessary steps to recover a single mailbox, or mailbox items. But the GRE can only assist you during the restore. It hids the above described tasks behind a nice GUI. The backup of Microsoft Exchange is still something you have to do with HPE Data Protector.

Database Availability Group without an Administrative Access Point

With Exchange 2013 SP1, Microsoft introduced the IP-less Database Availability Group (DAG). This type of DAG does not need a Cluster Name Object (CNO), and therefore has no IP address. With Exchange 2016, the IP-less DAG is the default DAG configuration.

But how to backup a DAG, that has no IP address and no name? It is easier than imagined. You have to create a DNS A-Record that includes all IP addresses of the cluster nodes, resulting in a DNS round-robin A-Record. You also have to install the Data Protector Disk Agent and On-line Extension on all cluster nodes. After that, you simply import the DAG by using the DNS A-Record into Data Protector. Then you can proceed with the creation and configuration of a backup job, that uses the newly imported cluster.

Backup runs fine, but the GRE fails

During the test phase of a new Exchange 2016 cluster, a customer of mine discovered a strange error, when he tried to restore a mailbox, or mailbox item, using the Exchange GRE.

Either parsing of command builder output failed or no databases were backed up.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The customer and I double-checked the installation of the GRE on both nodes. Everything was fine. We also found out, that Data Protector was able to list the backup objects. This is a shortened output of the command.

Object Name                                                     Object type
===============================================================================
exchange-2010.domain.tld:/25e0d4ac-5a63-4035-b718-09d07bbbba47/DB3 E2010
dag-backup.domain.tld:/c7982a7e-76cd-4759-9d03-019c0c410957/DB03 E2010
dag-backup.domain.tld:/5d3faa67-6fd9-493e-bf56-5044f75620a8/DB01 E2010

As you can see, dag-backup.domain.tld is the DNS A-Record, that was created to backup the DAG with Data Protector.

Connection between A-Record and DAG name

It took some time to get this sorted, but at the end, a new A-Record was the key. The DAG has a name, e.g. customer-dag1.domain.tld. But there is no matching A-Record, and the DAG has no IP address.

When the GRE searches for available database backups, it stumbles over the mismatch between the DAG name, that is reported by the Exchange organization, and the name of the Data Protector client that was used to backup the databases.

The key to success was to change the DNS A-Record from dag-backup.domain.tld to customer-dag1.domain.tld. Latter is the name of the DAG, that is given during DAG creation. After removing the Data Protector client, the re-import of the DAG with the new A-Record, and a successful backup, the customer was able to restore mailboxes and mailbox items using the GRE for Microsoft Exchange.

This process is not described in detail in the Data Protector documentation. All you find is this foot note in the Data Protector Platform Integration Matrix (page 12, foot note 19):

Microsoft Exchange Server DAG configured without a Cluster Administrator Access Point is supported with Round Robin DNS mapping of DAG name to all the node IPs.

Make sure that the DNS round-robin A-Record matches your DAG name.

Get-MailboxDatabase doesn’t show last backup timestamp

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes you have to check when the last backup of an Exchange mailbox database was taken. This is pretty simple, because the timestamps of the last full, incremental and differential backup is stored for each mailbox database. You can check these attributes using the Exchange Control Panel (ECP), or you can use the Get-MailboxDatabase cmdlet.

Backup successful, but no timestamp?

Take a look at this output. As you can see, there’s no timestamp for the last full, incremental and differential backup. But this database was successfully backuped some minutes before.

[PS] C:\Windows\system32>get-mailboxdatabase | select name, *backup*


Name                           : DB01
BackupInProgress               :
SnapshotLastFullBackup         :
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 :
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

Missing -status switch

The solution is easy: The -status switch was missing.

[PS] C:\Windows\system32>Get-MailboxDatabase -status | fl name, *backup*


Name                           : DB01
BackupInProgress               : True
SnapshotLastFullBackup         : True
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 : 19.07.2016 15:22:24
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

[PS] C:\Windows\system32>Get-MailboxDatabase -status | fl name, *backup*


Name                           : DB01
BackupInProgress               : False
SnapshotLastFullBackup         : True
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 : 16.08.2016 15:04:02
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

After adding the -status switch to the Get-MailboxDatabase command, the timestamps were added to the output. If you run the command during a running backup session, this is also added to the output (BackupInProgress).

Changes to supported .NET Frameworks for Exchange 2013/2016

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

EDIT: If you have already installed .NET 4.6.1, check this blog post on how to remove it (You Had Me At EHLO…)

Microsoft Exchange heavily relies on Microsoft .NET Framework. Because of this, Microsoft provides a matrix for the supported Microsoft .NET Frameworks. Mostly unknown is the fact, that Exchange doesn’t support the every Microsoft .NET Framework, and this is causing trouble sometimes. Some admins simply install the latest .NET releases because “it doesn’t hurt”. Well… it hurts!

Changes for .NET Framework 4.6.1

Microsoft has changed the support policy for .NET Framework 4.6.1 with the release of Exchange 2013 CU13 and Exchange 2016 CU2. Up to this versions, only .NET Framework 4.5.2 is supported. Starting with Exchange 2013 CU13 and Exchange 2016 CU2, Microsoft supports .NET Framework 4.6.1 together with a hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2). If you wish to install .NET Framework 4.6.1, make sure to install Exchange 2013 CU13 or 2016 CU2 first.

.NET Framework/ Microsoft Exchange	Exchange 2007 SP3	Exchange 2010 SP3	Exchange 2013 CU13 and later	Exchange 2016 CU2 and later
.NET Framework 3.5.1	X	X
.NET Framework 4.0		X¹
.NET Framework 4.5		X¹	X
.NET Framework 4.5.1		X¹	X
.NET Framework 4.5.2			X	X
.NET Framework 4.6.1			X²	X²
.NET Framework 4.6.2

¹ .NET 3.5 or 3.5.1 must be installed

² Supported with hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2)

Other .NET Framework versions

Microsoft .NET Framework 4.6.2 isn’t supported for any version of Microsoft Exchange. Other example: If you’re running Exchange 2010 SP3, don’t install anything above .NET Framework 4.5, not even 4.5.1. Check the Exchange Server Supportability Matrix for the supported .NET Framework for the Exchange version you’re running.

Side notes

Microsoft PowerShell is part of the Windows Management Framework (WMF). Microsoft Exchange only supports the WMF built into the underlying Windows Server version.

This also applies to Microsoft Outlook. I wonder how many Exchange projects fail because the Microsoft Outlook version, that is used by the customer, isn’t supported.

Receive Connector role not selectable in Exchange 2016 CU2

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Another bug in Exchange 2016 CU2. The Role of a new receive connector is greyed out. You can select “Front-End-Transport”. This is a screenshot from a german Exchange 2016 CU2.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Solution

Use the Exchange Management Shell to create a new receive connector. Afterwards, you can modify it with the Exchange Control Panel (ECP).

[PS] C:\Windows\system32>New-Receiveconnector -Name "Client Frontend Dummy" -RemoteIPRange ("192.168.200.99") -TransportRo
le "FrontendTransport" -Bindings ("0.0.0.0:25") -usage "Custom" -Server "exchange1"

Identity                                Bindings                                Enabled
--------                                --------                                -------
EXCHANGE1\Client Frontend Dummy         {0.0.0.0:25}                            True

Microsoft has confirmed, that this is a bug in Exchange 2016 CU2.

Exchange 2013 Offline Address Book visible after Exchange 2016 deployment?!

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

After deploying a new Microsoft Exchange organization with Exchange 2016, or after deploying a Microsoft Exchange 2016 into an existing organization, you might notice a strange behaviour regarding the Offline Address Books (OAB).

[PS] C:\Windows\system32>Get-OfflineAddressBook

Name                                    Versions                                AddressLists
----                                    --------                                ------------
Standard-Offlineadressliste             {Version2, Version3, Version4}          {\Globale Standardadressliste}
Standard-Offlineadressliste (Ex2013)    {Version4}                              {\Globale Standardadressliste}

Huh?! Where does this Exchange 2013 OAB come from? As you can see in the cmdlet output, there’s no Exchange 2013 in this organization.

[PS] C:\Windows\system32>Get-ExchangeServer | select Name, AdminDisplayVersion

Name                                                 AdminDisplayVersion
----                                                 -------------------
MAIL-1                                               Version 14.3 (Build 123.4)
MAIL-2                                               Version 15.1 (Build 466.34)
MAIL-3                                               Version 15.1 (Build 466.34)

There is no Exchange 2013 server in this organization. Only Exchange 2010 (Build 14.3) and 2016 (Build 15.1).

This is nothing to worry about. Microsoft has confirmed that this is a bug. The OAB simply has the wrong name and Microsoft will fix it in an upcoming cumulative update. It’s not fixed in the latest CU2 for Exchange 2016! There’s also no need to change the name of the OAB.

So don’t panic, if you deploy a Microsoft Exchange 2016 CU2 and you see “Ex2013” in the OAB name. Ignore it.