Tag Archives: exchange

Meltdown & Spectre: What about Microsoft Exchange?

On January 18, 2018, Microsoft has published KB4074871 which has the title “Exchange Server guidance to protect against speculative execution side-channel vulnerabilities”. As you might guess, Exchange is affected by Meltdown & Spectre – like any other software. Microsoft explains in KB4074871:

Because these are hardware-level attacks that target x64-based and x86-based processor systems, all supported versions of Microsoft Exchange Server are affected by this issue.

Like Citrix, Microsoft does not offer any updates to address this issue, because there is nothing to fix in Microsoft Exchange. Instead of this, Microsoft recommends to run the lates Exchange Server cumulative update and any required security updates. On top, Microsoft recommends to check software before it is deployed into production. If Exchange is running in a VM, Microsoft recommends to follow the instructions offered by the cloud or hypervisor vendor.

Citrix NetScaler and Exchange: Case-sensitivity of internal and external URLs

Exchange has known the concept of internal and external URLs for the different services (Outlook Web Access, OAB, EWS, ActiveSync etc) since Exchange 2007. And it’s still confusing people. The internal URL is the URL, that is used to access the desired service from the intranet. The external URL represents the URL that is used to access the service from the internet. Best practice is to use the same URL (the external) for both, use a certificate from a public CA, and use split DNS to access the external domain from the inside of your network.

People tend to imply, that URLs are not case-sensitive. This seems to be true in most cases. The World Wide Web Consortium (W3C) states:

URLs in general are case-sensitive (with the exception of machine names). There may be URLs, or parts of URLs, where case doesn’t matter, but identifying these may not be easy. Users should always consider that URLs are case-sensitive.

Source W3C

Citrix NetScaler and URLs

Citrix NetScaler handles URLs as case-sensitive.

A frequently used concept to load balance Microsoft Exchange with a NetScaler is Content Switching. Policies are used to identify traffic, and actions are used to take action on the traffic that matches the policies. The NetScaler uses the advanced policy engine to create expressions for the Content Switching Policies. When creating a Content Switching policy by creating an expression that uses the CONTAINS operator, you might notice that the results are case-sensitive.

This can be a problem in case of Microsoft Exchange, because /Autodiscover/Autodiscover.xml and /autodiscover/autodiscover.xml, or /ews/exchange.asmx and /EWS/Exchange.asmx are handled different.

Solution

To make sure that different cases are handled, you should add  SET_TEXT_MODE(IGNORECASE)  to you policy expression. Citrix describes this in CTX115528.

I’ve changed my NetScaler setup script for Exchange to handle this behavior.

Exchange DAG member dies during snapshot creation

Yesterday, a customer called me and told me about a scary observation on one of his Exchange 2016 DAG (Database Availability Groups) nodes.

In preparation of a security check, my customer created a snapshot of a Exchange 2016 DAG node. This node is part of a two node Windows Server 2012 R2/ Exchange 2016 CU7 cluster.

That something went wrong was instantly clear, after the first alarm messages were received. My customer opened a console windows and saw, that the VM was booting.

What went wrong?

Nothing. Something worked as designed, except the fact, that the observed behaviour was not intended.

That a snapshot was created was clearly visible in the logs. Interesting was the amount of time, that the snapshot creation took. It took 5 minutes from the start of the snapshot creation until the task finished. During this time, pretty much data was written to the disks.

The server eventlog contained an entry, that pointed me to to the right direction.

Event Type: Information
Event ID: 1001
Source: BugCheck
Description: The computer has rebooted from a bugcheck. The bugcheck was 0x0000009E (0xffffe0001eccf900, 0x000000000000003c, 0x000000000000000a, 0x0000000000000000).

The Ask the Core Team wrote a nice blog post about this STOP error. In short: The failvoer clustering service incorporates a detection mechanism that may detect unresponsive user-mode processes. If an unresponsive user-mode process is detected, a HangRecoveryAction is called. Since Windows Server 2008, a STOP error (Bugcheck) is caused on the cluster node.

Most likely hypothesis

My explanation of the observed behaviour is, that my customer accidentally created a snapshot that has contained the VM memory. Because the Exchange server has 32 GB memory, the snapshot creation took some time and the VM became unresponsive. As the VM was responding again, the HangRecoveryAction did its dirty job.

Check if the checkbox for the VM memory is disabled, before you create a snapshot. Otherwise the bugcheck will do its job. Please note, that you might see this behaviour in all Microsoft Windows Failover Clusters, not only with Microsoft Exchange.

Exchange receive connector rejects incoming connections

As part of a bigger Microsoft Exchange migration, one of my customers moved the in- and outbound mailflow to a newly installed mail relay cluster. We modified MX records to move the mailflow to the new mail relay, because the customer also switched the ISP. While changing the MX records for ~40 domains, and therefore more and more mails received through the new mail relay cluster, we noticed events from MSExchangeTransport (event id 1021):

192.168.xxx.xxx is the mail relay cluster, which is used for the in- and outbound mailflow.

This event indicates that the remote server has reached the maximum number of simultaneous incoming connections to the receive connector. This value is specified by the MaxInboundConnectionPerSource  parameter, and the default value is 20. You can easily increase the value using the  Set-ReceiveConnector  cmdlet.

Microsoft has decreased this value over time. It was 100 in Exchange 2007, but 20 since Exchange 2010.

Data Protector Exchange GRE and IP-less Exchange DAG

When dealing with Microsoft Exchange restore requests, you will come across three different restore situations:

  • a database
  • a single mailbox
  • a single mailbox item (mail, calendar entry etc.)

Restoring a complete database is not a complicated task, but restoring a single mailbox, or a single mailbox item, is. First, you need to restore the mailbox, that includes the desired mailbox, into a recovery database. Then you can restore the mailbox, or the mailbox items, from the recovery database. Some of the tasks can only be done with the Exchange Management Shell.

The HPE Data Protector Granular Recovery Extension (GRE) for Microsoft Exchange helps you to simplify the necessary steps to recover a single mailbox, or mailbox items. But the GRE can only assist you during the restore. It hids the above described tasks behind a nice GUI. The backup of Microsoft Exchange is still something you have to do with HPE Data Protector.

Database Availability Group without an Administrative Access Point

With Exchange 2013 SP1, Microsoft introduced the IP-less Database Availability Group (DAG). This type of DAG does not need a Cluster Name Object (CNO), and therefore has no IP address. With Exchange 2016, the IP-less DAG is the default DAG configuration.

But how to backup a DAG, that has no IP address and no name? It is easier than imagined. You have to create a DNS A-Record that includes all IP addresses of the cluster nodes, resulting in a DNS round-robin A-Record. You also have to install the Data Protector Disk Agent and On-line Extension on all cluster nodes. After that, you simply import the DAG by using the DNS A-Record into Data Protector. Then you can proceed with the creation and configuration of a backup job, that uses the newly imported cluster.

Backup runs fine, but the GRE fails

During the test phase of a new Exchange 2016 cluster, a customer of mine discovered a strange error, when he tried to restore a mailbox, or mailbox item, using the Exchange GRE.

The customer and I double-checked the installation of the GRE on both nodes. Everything was fine. We also found out, that Data Protector was able to list the backup objects. This is a shortened output of the command.

As you can see, dag-backup.domain.tld is the DNS A-Record, that was created to backup the DAG with Data Protector.

Connection between A-Record and DAG name

It took some time to get this sorted, but at the end, a new A-Record was the key. The DAG has a name, e.g. customer-dag1.domain.tld. But there is no matching A-Record, and the DAG has no IP address.

When the GRE searches for available database backups, it stumbles over the mismatch between the DAG name, that is reported by the Exchange organization, and the name of the Data Protector client that was used to backup the databases.

The key to success was to change the DNS A-Record from dag-backup.domain.tld to customer-dag1.domain.tld. Latter is the name of the DAG, that is given during DAG creation. After removing the Data Protector client, the re-import of the DAG with the new A-Record, and a successful backup, the customer was able to restore mailboxes and mailbox items using the GRE for Microsoft Exchange.

This process is not described in detail in the Data Protector documentation. All you find is this foot note in the Data Protector Platform Integration Matrix (page 12, foot note 19):

Microsoft Exchange Server DAG configured without a Cluster Administrator Access Point is supported with Round Robin DNS mapping of DAG name to all the node IPs.

Make sure that the DNS round-robin A-Record matches your DAG name.

Surprise, surprise: Enable/ disable circular logging without downtime

As part of a troubleshooting process, I had to disable circular logging on a Microsoft Exchange 2013 mailbox database, that was part of a Database Availability Group (DAG).

What is circular logging? Microsoft Exchange uses log files that record all transactions and modifications to a mailbox database. In this case, Exchange works like MS SQL or Oracle. These log files are used, to bring the database back to a consistent state after a crash, or to recover a database to a specific point-in-time. Because everything is written to log files, the number of log files increases over time. Log files are truncated, as soon as a successful full backup of the database is taken.

If you don’t need the capability to recover to a specific point-in-time, or if you are running low on disk space, circular logging can be an alternative. Especially if you are running low on disk space, because your backup isn’t working as expected. With circular logging enabled, Microsoft Exchange maintains only the minimum number of log files that is necessary, to keep the Exchange server running. As soon as a transaction was written to the database, the log file will be overwritten.

I rarely use circular logging. But this time I had to. As already mentioned, I had a mailbox database with enabled circular logging. This database was part of a DAG and I had to disable circular logging. Usually, you need to dismount and re-mount the database after enabling or disabling circular logging.

You can enable or disable circular logging using the Microsoft Exchange Control Panel (ECP), or with the Microsoft Exchange Management Shell. I have used the PowerShell.

To my surprise, dismounting and re-mounting the database was not necessary. The circular logging was disabled without downtime. I’ve checked the TechNet, and the observed behaviour was confirmed.

JET vs. CRCL

A non-replicated mailbox databases will use JET circular logging. If the database is part of a DAG, the database will use continuous replication circular logging (CRCL). A benefit of CRCL is, that it can be enabled and disabled without the need of dismounting and re-mounting the mailbox database.

To get this clear: This only works if you are using replicated mailbox databases, because only databases that belong to a DAG are using CRCL. If you are using standalone mailbox databases, you have to dismount and re-mount the database, after enabling or disabling circular logging.

As I mentioned earlier: I really don’t use circular logging often, but that was very handy today!

Get-MailboxDatabase doesn’t show last backup timestamp

Sometimes you have to check when the last backup of an Exchange mailbox database was taken. This is pretty simple, because the timestamps of the last full, incremental and differential backup is stored for each mailbox database. You can check these attributes using the Exchange Control Panel (ECP), or you can use the Get-MailboxDatabase cmdlet.

Backup successful, but no timestamp?

Take a look at this output. As you can see, there’s no timestamp for the last full, incremental and differential backup. But this database was successfully backuped some minutes before.

Missing -status switch

The solution is easy: The -status  switch was missing.

After adding the -status  switch to the Get-MailboxDatabase command, the timestamps were added to the output. If you run the command during a running backup session, this is also added to the output (BackupInProgress).

Disable Outlook cached mode for shared mailboxes

When you use Microsoft Outlook in cached mode, what I always recommend, and you add additional mailboxes to your outlook profile, you will notice that the OST file will grow. Outlook will download the mailbox items (mails, calendar entries, contacts etc.), and store them in the OST file. This is the default behaviour since Microsoft Outlook 2010. If you want to disable this behaviour, you have two options:

  • Edit the registry
  • Use a group policy object (GPO)

Edit the Windows registry

The easiest way is to use a reg file. Copy this text into a file and save it as disablecachedmode.reg. Then double click the file and confirm, that you want to import the registry file.

Please note the version number after “Office”.

OutlookVersion
Outlook 201616.0
Outlook 201315.0
Outlook 201014.0

Make sure that you use the appropriate version number for your Outlook! Otherwise this setting is applied, but not working.

Group Policy

If you want to apply this setting on a bunch of clients, you should use a GPO. Before you can use a GPO, you have to install the necessary template files for Microsoft Office/ Outlook. These ADMX files are part of the “Office 2013 Administrative Template files (ADMX/ADML)” or “Office 2016 Administrative Template files (ADMX/ADML)” package.

Copy the Outlk16.admx or Outlk15.admx files to the PolicyDefinitions folder (either C:\Windows or Central Store), and the Outlk16.adml or Outlk15.adml to the corresponding language folder.

Then you can create a new GPO. The desired setting can be found under User Configuration >> Administrative Templates >> Microsoft Outlook 201x >> Outlook Options >> Delegates >> Disable shared mail folder caching. Set this to “enable” and apply the GPO.

Still using Microsoft Outlook 2010?

Please note, that the GPO template for Microsoft Outlook 2010 doesn’t contain the necessary setting that controls this functionality!

Changes to supported .NET Frameworks for Exchange 2013/2016

EDIT: If you have already installed .NET 4.6.1, check this blog post on how to remove it (You Had Me At EHLO…)

Microsoft Exchange heavily relies on Microsoft .NET Framework. Because of this, Microsoft provides a matrix for the supported Microsoft .NET Frameworks. Mostly unknown is the fact, that Exchange doesn’t support the every Microsoft .NET Framework, and this is causing trouble sometimes. Some admins simply install the latest .NET releases because “it doesn’t hurt”. Well… it hurts!

Changes for .NET Framework 4.6.1

Microsoft has changed the support policy for .NET Framework 4.6.1 with the release of Exchange 2013 CU13 and Exchange 2016 CU2. Up to this versions, only .NET Framework 4.5.2 is supported. Starting with Exchange 2013 CU13 and Exchange 2016 CU2, Microsoft supports .NET Framework 4.6.1 together with a hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2). If you wish to install .NET Framework 4.6.1, make sure to install Exchange 2013 CU13 or 2016 CU2 first.

.NET Framework/ Microsoft ExchangeExchange 2007 SP3Exchange 2010 SP3Exchange 2013 CU13 and laterExchange 2016 CU2 and later
.NET Framework 3.5.1XX
.NET Framework 4.0
.NET Framework 4.5X
.NET Framework 4.5.1X
.NET Framework 4.5.2XX
.NET Framework 4.6.1
.NET Framework 4.6.2

¹ .NET 3.5 or 3.5.1 must be installed

² Supported with hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2)

Other .NET Framework versions

Microsoft .NET Framework 4.6.2 isn’t supported for any version of Microsoft Exchange. Other example: If you’re running Exchange 2010 SP3, don’t install anything above .NET Framework 4.5, not even 4.5.1. Check the Exchange Server Supportability Matrix for the supported .NET Framework for the Exchange version you’re running.

Side notes

Microsoft PowerShell is part of the Windows Management Framework (WMF). Microsoft Exchange only supports the WMF built into the underlying Windows Server version.

This also applies to Microsoft Outlook. I wonder how many Exchange projects fail because the Microsoft Outlook version, that is used by the customer, isn’t supported.

Receive Connector role not selectable in Exchange 2016 CU2

Another bug in Exchange 2016 CU2. The Role of a new receive connector is greyed out. You can select “Front-End-Transport”. This is a screenshot from a german Exchange 2016 CU2.

receive_connect_role_not_selectable

Solution

Use the Exchange Management Shell to create a new receive connector. Afterwards, you can modify it with the Exchange Control Panel (ECP).

Microsoft has confirmed, that this is a bug in Exchange 2016 CU2.