Tag Archives: exchange

Out-of-Office replies are dropped due to empty MAIL FROM

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I had an interesting support call. A customer noticed that Out-of-Office replies were not received by recipients, even though the OoO option were enabled for internal and external recipients. Internal recipients got the OoO reply, but none of the external recipients.

cattu/ pixabay.com/ Creative Commons CC0

The Message Tracking Log is a good point to start. I quickly discovered that the Exchange server was unable to send the OoO mails. You can use the eventid FAIL to get a list of all failed messages.

Very interesting was the RecipientStatus of a failed mail.

RecipientStatus         : {[{LED=550 Requested action not taken: mailbox unavailable};{MSG=};{FQDN=mailrelay-out.xxxx.de};{IP=213.xxx.xxx.xxx};{LRT=20.12.2018 10:22:39}]}

550 Requested action not taken: mailbox unavailable is a pretty interesting error when sending mails over a mail relay of your ISP. Especially when other mails were successfully sent over the same mail relay.

Next stop: Protcol log of the send connector

I enabled the logging on the send connector using the EAC. This option is disabled by default. Depending on the amount of mails sent over the connector, you should make sure to disable the logging after your troubleshooting session. To enable the logging, follow these steps:

Open the EAC and navigate to
Mail flow > Send connectors
Select the connector you want to configure, and then click Edit
On the General tab in the Protocol logging level section, select the Verbose option
When you’re finished, click Save

The protocol log can be found under %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend.

After enabling the logging and another test mail, the log contained the necessary details to find the root cause. This is the interesting part of the SMTP communication:

2018-12-20T10:22:39.313Z,Relay,08D640AAC0AD8811,3,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,220 mailrelay-out.xxx.de ESMTP Postfix (Debian/GNU),
2018-12-20T10:22:39.313Z,Relay,08D640AAC0AD8811,4,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,EHLO mail.domain.local,
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,5,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,250  mailrelay-out2.xxx.de SIZE 52428800 8BITMIME OK,
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,6,192.168.0.212:49986,213.xxx.xxx.xxx:25,*,,sending message with RecordId 22471268892695 and InternetMessageId <b9613be791c141e3b76828228bd6cdb3@exchange.domain.local>
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,7,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,MAIL FROM:<> SIZE=4758,
2018-12-20T10:22:39.331Z,Relay,08D640AAC0AD8811,8,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,550 Requested action not taken: mailbox unavailable,
2018-12-20T10:22:39.332Z,Relay,08D640AAC0AD8811,9,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,QUIT,

The error occured right after the exchange server issued MAIL FROM:<> . But why is the MAIL FROM empty?

RFC 2298 is the key

An Out-of-Office reply is a Delivery Status Notification message. And RFC 2298 clearly states:

The envelope sender address (i.e., SMTP MAIL FROM) of the MDN MUST be
null (<>), specifying that no Delivery Status Notification messages
or other messages indicating successful or unsuccessful delivery are
to be sent in response to an MDN.

So the empty MAIL FROM is something that a mail relay should expect. In case of my customer that mail relay seems to act different. Maybe some kind of spam protection.

Database Availability Group (DAG) witness is in a failed state

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a maintenance job I had to update a 2-node Exchange Database Availability Group and a file-share witness server.

After the installation of Windows updates on the witness server and the obligatory reboot, the witness left in a failed state.

[PS] C:\Windows\system32>Get-DatabaseAvailabilityGroup -Identity DAG1 -Status | fl *wit*
WARNING: Database availability group ‘DAG01’ witness is in a failed state. The database
availability group requires the witness server to maintain quorum. Please use the
Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and the directory.

WitnessServer : fsw.domain.local
WitnessDirectory : C:\DAGFileShareWitnesses\DAG1.domain.local
AlternateWitnessServer :
AlternateWitnessDirectory :
WitnessShareInUse : InvalidConfiguration
DxStoreWitnessServers :

In my opinion, the re-creation of the witness server and the witness directory cannot be the correct way to solve this. There must be another way to solve this. In addition to this: The server was not dead. Only a reboot occured.

Check the basics

Both DAG nodes were online and working. A good starting point is a check of the cluster resources using the PowerShell.

In my case the cluster resource for the File Share Witness was in a failed state. A simple Start-ClusterResource solved my issue immediately.

[PS] C:\Windows\system32>Get-ClusterResource

Name                                              State                                             OwnerGroup                                        ResourceType
----                                              -----                                             ----------                                        ------------
File Share Witness (\\fsw.domain.local            Failed                                            Cluster Group                                     File Share Witness


[PS] C:\Windows\system32>Get-ClusterResource | Start-ClusterResource

Name                                              State                                             OwnerGroup                                        ResourceType
----                                              -----                                             ----------                                        ------------
File Share Witness (\\fsw.domain.local            Online                                            Cluster Group                                     File Share Witness

In this case, it seems that the the cluster has marked the file share witness as unreliable, thus the resource was not started after the file share witness was back online again. In this case, I managed it to manually bring it back online by running Start-ClusterResource on one of the DAG members.

Office 365 – Outlook keeps prompting for password

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This is only a short blog post to document a solution for a very annoying problem. After the automatic update of my Outlook to the latest Office 365 build (version 1809), it has started to prompting for credentials. I’m using Outlook to access a Microsoft Exchange 2016 server (on-premises), without any hybrid configuration. A pretty simple and plain Exchange 2016 on-prem deployment.

I knew, that it has to be related to Office 365, because the Outlook 2016 on my PC at the office was not affected. Only the two Office 365 deployments on my ThinkPad T480s and ThinkPad X250.

To make this long story short: ExcludeExplicitO365Endpoint is the key! You have to add a DWORD under HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\AutoDiscover.

HKEY_CURRENT_USER\SOFTWARE\Policies\Microsoft\office\16.0\outlook\autodiscover
DWORD: ExcludeExplicitO365Endpoint
Value = 1

Restart your computer and the annoying credentials prompts are gone.

Meltdown & Spectre: What about Microsoft Exchange?

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

On January 18, 2018, Microsoft has published KB4074871 which has the title “Exchange Server guidance to protect against speculative execution side-channel vulnerabilities”. As you might guess, Exchange is affected by Meltdown & Spectre – like any other software. Microsoft explains in KB4074871:

Because these are hardware-level attacks that target x64-based and x86-based processor systems, all supported versions of Microsoft Exchange Server are affected by this issue.

Like Citrix, Microsoft does not offer any updates to address this issue, because there is nothing to fix in Microsoft Exchange. Instead of this, Microsoft recommends to run the lates Exchange Server cumulative update and any required security updates. On top, Microsoft recommends to check software before it is deployed into production. If Exchange is running in a VM, Microsoft recommends to follow the instructions offered by the cloud or hypervisor vendor.

Citrix NetScaler and Exchange: Case-sensitivity of internal and external URLs

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Exchange has known the concept of internal and external URLs for the different services (Outlook Web Access, OAB, EWS, ActiveSync etc) since Exchange 2007. And it’s still confusing people. The internal URL is the URL, that is used to access the desired service from the intranet. The external URL represents the URL that is used to access the service from the internet. Best practice is to use the same URL (the external) for both, use a certificate from a public CA, and use split DNS to access the external domain from the inside of your network.

People tend to imply, that URLs are not case-sensitive. This seems to be true in most cases. The World Wide Web Consortium (W3C) states:

URLs in general are case-sensitive (with the exception of machine names). There may be URLs, or parts of URLs, where case doesn’t matter, but identifying these may not be easy. Users should always consider that URLs are case-sensitive.
Source W3C

Citrix NetScaler and URLs

Citrix NetScaler handles URLs as case-sensitive.

A frequently used concept to load balance Microsoft Exchange with a NetScaler is Content Switching. Policies are used to identify traffic, and actions are used to take action on the traffic that matches the policies. The NetScaler uses the advanced policy engine to create expressions for the Content Switching Policies. When creating a Content Switching policy by creating an expression that uses the CONTAINS operator, you might notice that the results are case-sensitive.

This can be a problem in case of Microsoft Exchange, because /Autodiscover/Autodiscover.xml and /autodiscover/autodiscover.xml, or /ews/exchange.asmx and /EWS/Exchange.asmx are handled different.

Solution

To make sure that different cases are handled, you should add SET_TEXT_MODE(IGNORECASE) to you policy expression. Citrix describes this in CTX115528.

I’ve changed my NetScaler setup script for Exchange to handle this behavior.

Exchange DAG member dies during snapshot creation

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Yesterday, a customer called me and told me about a scary observation on one of his Exchange 2016 DAG (Database Availability Groups) nodes.

In preparation of a security check, my customer created a snapshot of a Exchange 2016 DAG node. This node is part of a two node Windows Server 2012 R2/ Exchange 2016 CU7 cluster.

That something went wrong was instantly clear, after the first alarm messages were received. My customer opened a console windows and saw, that the VM was booting.

What went wrong?

Nothing. Something worked as designed, except the fact, that the observed behaviour was not intended.

That a snapshot was created was clearly visible in the logs. Interesting was the amount of time, that the snapshot creation took. It took 5 minutes from the start of the snapshot creation until the task finished. During this time, pretty much data was written to the disks.

VMware vSphere Throughput Snapshot Creation

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The server eventlog contained an entry, that pointed me to to the right direction.

Event Type: Information
Event ID: 1001
Source: BugCheck
Description: The computer has rebooted from a bugcheck. The bugcheck was 0x0000009E (0xffffe0001eccf900, 0x000000000000003c, 0x000000000000000a, 0x0000000000000000).

The Ask the Core Team wrote a nice blog post about this STOP error. In short: The failvoer clustering service incorporates a detection mechanism that may detect unresponsive user-mode processes. If an unresponsive user-mode process is detected, a HangRecoveryAction is called. Since Windows Server 2008, a STOP error (Bugcheck) is caused on the cluster node.

Most likely hypothesis

My explanation of the observed behaviour is, that my customer accidentally created a snapshot that has contained the VM memory. Because the Exchange server has 32 GB memory, the snapshot creation took some time and the VM became unresponsive. As the VM was responding again, the HangRecoveryAction did its dirty job.

Check if the checkbox for the VM memory is disabled, before you create a snapshot. Otherwise the bugcheck will do its job. Please note, that you might see this behaviour in all Microsoft Windows Failover Clusters, not only with Microsoft Exchange.

Exchange receive connector rejects incoming connections

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a bigger Microsoft Exchange migration, one of my customers moved the in- and outbound mailflow to a newly installed mail relay cluster. We modified MX records to move the mailflow to the new mail relay, because the customer also switched the ISP. While changing the MX records for ~40 domains, and therefore more and more mails received through the new mail relay cluster, we noticed events from MSExchangeTransport (event id 1021):

Receive connector Default Frontend EXCHANGE rejected an incoming connection
from IP address 192.168.xxx.xxx. The maximum number of connections per source
20) for this connector has been reached by this source IP address.

192.168.xxx.xxx is the mail relay cluster, which is used for the in- and outbound mailflow.

This event indicates that the remote server has reached the maximum number of simultaneous incoming connections to the receive connector. This value is specified by the MaxInboundConnectionPerSource parameter, and the default value is 20. You can easily increase the value using the Set-ReceiveConnector cmdlet.

Set-ReceiveConnector - identity "Default Frontend EXCHANGE" -MaxInboundconnectionPersource 100

Microsoft has decreased this value over time. It was 100 in Exchange 2007, but 20 since Exchange 2010.

Data Protector Exchange GRE and IP-less Exchange DAG

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When dealing with Microsoft Exchange restore requests, you will come across three different restore situations:

a database
a single mailbox
a single mailbox item (mail, calendar entry etc.)

Restoring a complete database is not a complicated task, but restoring a single mailbox, or a single mailbox item, is. First, you need to restore the mailbox, that includes the desired mailbox, into a recovery database. Then you can restore the mailbox, or the mailbox items, from the recovery database. Some of the tasks can only be done with the Exchange Management Shell.

The HPE Data Protector Granular Recovery Extension (GRE) for Microsoft Exchange helps you to simplify the necessary steps to recover a single mailbox, or mailbox items. But the GRE can only assist you during the restore. It hids the above described tasks behind a nice GUI. The backup of Microsoft Exchange is still something you have to do with HPE Data Protector.

Database Availability Group without an Administrative Access Point

With Exchange 2013 SP1, Microsoft introduced the IP-less Database Availability Group (DAG). This type of DAG does not need a Cluster Name Object (CNO), and therefore has no IP address. With Exchange 2016, the IP-less DAG is the default DAG configuration.

But how to backup a DAG, that has no IP address and no name? It is easier than imagined. You have to create a DNS A-Record that includes all IP addresses of the cluster nodes, resulting in a DNS round-robin A-Record. You also have to install the Data Protector Disk Agent and On-line Extension on all cluster nodes. After that, you simply import the DAG by using the DNS A-Record into Data Protector. Then you can proceed with the creation and configuration of a backup job, that uses the newly imported cluster.

Backup runs fine, but the GRE fails

During the test phase of a new Exchange 2016 cluster, a customer of mine discovered a strange error, when he tried to restore a mailbox, or mailbox item, using the Exchange GRE.

Either parsing of command builder output failed or no databases were backed up.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The customer and I double-checked the installation of the GRE on both nodes. Everything was fine. We also found out, that Data Protector was able to list the backup objects. This is a shortened output of the command.

Object Name                                                     Object type
===============================================================================
exchange-2010.domain.tld:/25e0d4ac-5a63-4035-b718-09d07bbbba47/DB3 E2010
dag-backup.domain.tld:/c7982a7e-76cd-4759-9d03-019c0c410957/DB03 E2010
dag-backup.domain.tld:/5d3faa67-6fd9-493e-bf56-5044f75620a8/DB01 E2010

As you can see, dag-backup.domain.tld is the DNS A-Record, that was created to backup the DAG with Data Protector.

Connection between A-Record and DAG name

It took some time to get this sorted, but at the end, a new A-Record was the key. The DAG has a name, e.g. customer-dag1.domain.tld. But there is no matching A-Record, and the DAG has no IP address.

When the GRE searches for available database backups, it stumbles over the mismatch between the DAG name, that is reported by the Exchange organization, and the name of the Data Protector client that was used to backup the databases.

The key to success was to change the DNS A-Record from dag-backup.domain.tld to customer-dag1.domain.tld. Latter is the name of the DAG, that is given during DAG creation. After removing the Data Protector client, the re-import of the DAG with the new A-Record, and a successful backup, the customer was able to restore mailboxes and mailbox items using the GRE for Microsoft Exchange.

This process is not described in detail in the Data Protector documentation. All you find is this foot note in the Data Protector Platform Integration Matrix (page 12, foot note 19):

Microsoft Exchange Server DAG configured without a Cluster Administrator Access Point is supported with Round Robin DNS mapping of DAG name to all the node IPs.

Make sure that the DNS round-robin A-Record matches your DAG name.

Surprise, surprise: Enable/ disable circular logging without downtime

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a troubleshooting process, I had to disable circular logging on a Microsoft Exchange 2013 mailbox database, that was part of a Database Availability Group (DAG).

What is circular logging? Microsoft Exchange uses log files that record all transactions and modifications to a mailbox database. In this case, Exchange works like MS SQL or Oracle. These log files are used, to bring the database back to a consistent state after a crash, or to recover a database to a specific point-in-time. Because everything is written to log files, the number of log files increases over time. Log files are truncated, as soon as a successful full backup of the database is taken.

If you don’t need the capability to recover to a specific point-in-time, or if you are running low on disk space, circular logging can be an alternative. Especially if you are running low on disk space, because your backup isn’t working as expected. With circular logging enabled, Microsoft Exchange maintains only the minimum number of log files that is necessary, to keep the Exchange server running. As soon as a transaction was written to the database, the log file will be overwritten.

I rarely use circular logging. But this time I had to. As already mentioned, I had a mailbox database with enabled circular logging. This database was part of a DAG and I had to disable circular logging. Usually, you need to dismount and re-mount the database after enabling or disabling circular logging.

[PS] C:\Windows\system32>Get-MailboxDatabase | ? {$_.CircularLoggingEnabled -eq $true}  | select name,server

Name                                              Server
----                                              ------
DAG1_DB_AH1                                       MBX4

You can enable or disable circular logging using the Microsoft Exchange Control Panel (ECP), or with the Microsoft Exchange Management Shell. I have used the PowerShell.

[PS] C:\Windows\system32>Set-MailboxDatabase DAG1_DB_AH1 -CircularLoggingEnabled $False

To my surprise, dismounting and re-mounting the database was not necessary. The circular logging was disabled without downtime. I’ve checked the TechNet, and the observed behaviour was confirmed.

JET vs. CRCL

A non-replicated mailbox databases will use JET circular logging. If the database is part of a DAG, the database will use continuous replication circular logging (CRCL). A benefit of CRCL is, that it can be enabled and disabled without the need of dismounting and re-mounting the mailbox database.

To get this clear: This only works if you are using replicated mailbox databases, because only databases that belong to a DAG are using CRCL. If you are using standalone mailbox databases, you have to dismount and re-mount the database, after enabling or disabling circular logging.

As I mentioned earlier: I really don’t use circular logging often, but that was very handy today!

Get-MailboxDatabase doesn’t show last backup timestamp

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes you have to check when the last backup of an Exchange mailbox database was taken. This is pretty simple, because the timestamps of the last full, incremental and differential backup is stored for each mailbox database. You can check these attributes using the Exchange Control Panel (ECP), or you can use the Get-MailboxDatabase cmdlet.

Backup successful, but no timestamp?

Take a look at this output. As you can see, there’s no timestamp for the last full, incremental and differential backup. But this database was successfully backuped some minutes before.

[PS] C:\Windows\system32>get-mailboxdatabase | select name, *backup*


Name                           : DB01
BackupInProgress               :
SnapshotLastFullBackup         :
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 :
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

Missing -status switch

The solution is easy: The -status switch was missing.

[PS] C:\Windows\system32>Get-MailboxDatabase -status | fl name, *backup*


Name                           : DB01
BackupInProgress               : True
SnapshotLastFullBackup         : True
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 : 19.07.2016 15:22:24
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

[PS] C:\Windows\system32>Get-MailboxDatabase -status | fl name, *backup*


Name                           : DB01
BackupInProgress               : False
SnapshotLastFullBackup         : True
SnapshotLastIncrementalBackup  :
SnapshotLastDifferentialBackup :
SnapshotLastCopyBackup         :
LastFullBackup                 : 16.08.2016 15:04:02
LastIncrementalBackup          :
LastDifferentialBackup         :
LastCopyBackup                 :
RetainDeletedItemsUntilBackup  : False

After adding the -status switch to the Get-MailboxDatabase command, the timestamps were added to the output. If you run the command during a running backup session, this is also added to the output (BackupInProgress).