Tag Archives: exchange

Exchange HCW8078 – Migration Endpoint could not be created

While migrating a customer from Exchange 2010 to Exchange 2016, I had to create an Exchange Hybrid Deployment, because the customer wants to use Microsoft Teams. Nothing fancy and I’ve did this a couple of times.

Unfortunantely the Hybrid Connection Wizard failed to create the migration endpoint. A quick check of the logs showed this error:

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'mail.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure

The customer had not plans to move mailboxes to Exchange Online, so we didn’t care about this error. But the Calendar tab in Teams was not visible, and Teams logs stated that Teams was unable to discover the mailbox. A typical sign of a not working EWS connection.

It’s always TLS… or DNS… or NTP

The customer used a certificate from its own PKI, so it was not trusted by Microsoft. In addition, the Exchange was located behind a Sophos XG which was running Webserver Protection (Reverse Proxy). But this was not the main cause for the problems.

The root cause was the certificate from the customers PKI.

And therefore you should make sure to use a proper certificate from a 3rd CA for Exchange Hybrid Deployments. I really please every customer to stop using self-signed certificates, or certificates from their own PKI for external connections.

The customer has switched to a Let’s Encrypt certificate for testing purposes and the problems went away, without running the HCW again. He will now purchase a certificate from a 3rd party CA.

Exchange Control Panel /ecp broken after certificate replacement

As part of an ongoing Exchange 2010 to 2016 migration, I had to replace the self-signed certificate with a certificate from the customers PKI. Everything went fine, the customer had a suitable template, we’ve added the necessary hostnames and bound IIS and SMTP to the certificate. The mess started with an iisreset /noforce

Bild von Oskars Zvejs auf Pixabay 

The iisreset took longer than expected. After that, I tried to login into the ECP, entered username and password and got an error.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="MSExchange Front End HTTP Proxy" />
  <EventID Qualifiers="49152">1003</EventID>
  <Level>2</Level>
  <Task>1</Task>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2020-10-22T12:16:38.934123400Z" />
  <EventRecordID>368718</EventRecordID>
  <Channel>Application</Channel>
  <Computer>server.domain.tld</Computer>
  <Security />
</System>
<EventData>
  <Data>Owa</Data>
  <Data>System.NullReferenceException: Object reference not set to an instance of an object. at 
Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.ProxyModule.<>c__DisplayClass16_0.<OnBeginRequest>b__0() at 
Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
</Data>
</EventData>
</Event>

Pretty strange. We switched back to the self-singned certificate, did an iisreset and everyting was fine again.So it was pretty obvious that the error was related to the certificate, or to be more clear, to the certificate template.

A short research confirmed this. The template was a modified v3 web server template from an Enterprise CA running Windows Server 2008 R2.

With Windows Server 2008, Microsoft introduced a new cryptographic API called Cryptography Next Generation (CNG), which separates cryptographic providers (algorithm implementation) from key storage providers (create, delete, export, import, open and store keys). The older CryptoAPI does not differ between this and implements cryptographic algorithms and key storage.

The modified template used CNG instead of CryptoAPI. We noticed this when we checked the certificate with certutil -store my <thumbprint>.

If the listed provider for the certificate is Microsoft Software Key Storage Provider, then you will have to re-import the certificate. If Microsoft RSA SChannel Cryptographic Provider is used, everything is fine.

You have to remove the certificate, then re-import it using

certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx <CertificateFilename>

You need a PKCS#12 file (PFX) and the password. Re-import it and then you can use the certificate for Exchange. Bind services to it and restart the IIS.

Access to on-premise hosted Public Folders using Exchange Online mailboxes

Public Folders are still a thing. And while companies are moving their stuff into the cloud, Public Folders still need to be accessed by cloud-located mailboxes.

Allowing the access from Exchange Online mailboxes to on-premise hosted Public Folders is well documented by Microsoft, but there are also some fuzz. I had to deal with this during a Office 365 transition project at one of my customers.

The background

The customer is running a single Exchange 2016 server in a Windows Server 2012 R2 forest. AzureAD Sync is running and its syncing on-premise identities to AzureAD. The customer uses Office 365 E5 plans and he wants to move to Exchange Online, aside other O365 services like SharePoint Online, Teams etc.

Something was missing

After setting up the Exchange Hybrid, the customer and I where able to migrate the first mailboxes to Exchange Online.

To our surprise the on-premise Public Folders were not visible from the migrated Exchange Online mailboxes. We had still things to do…

In order to get the access to the Public Folders working, the Public Folder mailbox object needs to be synced to AzureAD. This is not complicated, because all you need to make sure is, that the user object is synced. If you are using an OU filter for the AzureAD sync, make sure that the OU with the Public Folder mailbox user object is included into the sync.

But there is also a second requirement: You also need to sync the Microsoft Exchange System Objects container! This is pretty important and it is not mentioned in the docs (Public folders in Microsoft 365, Office 365, and Exchange Online).

Check if the user is synced by using the Exchange Online PowerShell. This is how the Public Folder mailbox user looks like from the Exchange Online perspective:

PS C:\Users\p.terlisten> Get-MailUser Mailbox1

Name                                     RecipientType
----                                     -------------
Mailbox1                                 MailUser

This is the Public Folder mailbox that is hosted on-premise.

PS C:\Users\p.terlisten> Get-Mailbox -PublicFolder

Name                      Alias                ServerName     ProhibitSendQuota
----                      -----                ----------     -----------------
Mailbox1                  Mailbox1             EX             Unlimited

As long as you can’t see the MailUser in Exchange Online, you have no chance to configure the Public Folder access.

The next step is to synchronize the mail-enabled Public Folder objects to Exchange Online. For this, you have to download two scripts from Microsoft.

  • Sync-ModernMailPublicFolders.ps1
  • Sync-ModernMailPublicFolders.psd1

Run the Sync-ModernMailPublicFolders.ps1 script on your on-premise Exchange server. You will need your Office 365 admin credentials for this task.

PS C:\Users\p.terlisten\Download> .\Sync-ModernMailPublicFolders.ps1 -Credential (Get-Credential) -CsvSummaryFile:sync_summary.csv

The last step is to enable the Public Folder access using the Set-OrganizationConfig cmdlet in the Exchange Online PowerShell session.

PS C:\Users\p.terlisten> Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes Mailbox1

Please note that some of these steps need some time to get active! It will take some time for the background tasks to get some things sorted.

Controlled Connections to Public Folders in Outlook

It is worth mentioning that after enabling the access to Public Folders all Exchange online users can see the on-premise hosted Public Folders. If you need to enable the access only for some Exchange Online users, Microsoft has a solution for you: Controlled Connections to Public Folders.

First, you need to enable the Public Folder access for the users you have selected.

Set-CASMailbox "John Doe" -PublicFolderClientAccess $true

Then you enable the feature globally using the Exchange Online PowerShell.

Set-OrganizationConfig -PublicFolderShowClientControl $true

This setting applies only to Exchange Online hosted mailboxes!

Office365/ Exchange OAuth errors after replacing TLS certificate

A customer of mine asked for help to analyse a weird OAuth error. They are using a Microsoft Dynamics 365 Outlook plugin, which came up with an error:

“Can’t connect to Exchange”

In addition to this, they also faced an issueaccessing shared calendars of Exchange Online mailboxes.

Clearly an OAuth error. So we ran the Hybrid Connection Wizard again, which finished without any errors. But the errors persisted. Next stop: OAuth configuration.

We logged into one of the Exchange servers, started an Exchange Management Shell and checked the current OAuth configuration:

[PS] C:\Windows\system32>Get-AuthConfig
RunspaceId : e7c560cd-8316-4d9c-b97d-f4358d665e7d
CurrentCertificateThumbprint : CA420022ACC542D3AF7598456615715E98FFE986
PreviousCertificateThumbprint : E7B56BEAF4704BCBBF8C297F1D8D000215679C6F
NextCertificateThumbprint :
NextCertificateEffectiveDate :
ServiceName : 00000002-0000-0ff1-ce00-000000000000
Realm :
DeploymentId :
IssuerIdentifier :
Name : Auth Configuration
AdminDisplayName :
ExchangeVersion : 0.20 (15.0.0.0)
DistinguishedName : CN=Auth Configuration,CN=EXORG,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=tld
Identity : Auth Configuration
Guid : 911b6246-6248-4278-91be-1b7b4943d2f1
ObjectCategory : mlnetwork.local/Configuration/Schema/ms-Exch-Auth-Auth-Config
ObjectClass : {top, container, msExchContainer, msExchAuthAuthConfig}
WhenChanged : 11.04.2020 12:33:12
WhenCreated : 27.02.2015 14:24:37
WhenChangedUTC : 11.04.2020 10:33:12
WhenCreatedUTC : 27.02.2015 13:24:37
OrganizationId :
Id : Auth Configuration
OriginatingServer : DC01.domain.tld
IsValid : True
ObjectState : Unchanged

The CurrentCertificateThumbprint was the thumbprint of the old certificate. So we had to update it.

Set-AuthConfig -NewCertificateThumbprint 45CCEEDFAAB00F3A4312FF56774FA3B3412B9C (get-date)
Set-AuthConfig -PublishCertificate
iisreset /noforce
Set-AuthConfig -ClearPreviousCertificate

After the iisreset, the Dynamics plugin and the Exchange Online calendars started to work again.

Missing Microsoft Teams calendar tab with on-premise Exchange

Microsoft Teams got a big push due to the current COVID19 crisis and many of my customers deployed it in the past weeks. At ML Network, we are using Microsoft Teams for more than a year, and we don’t want to miss it anymore.

Source: Microsoft

We are running Exchange 2016 on-premises, currently CU16. We were missing the calendar tab in Teams since we started with Microsoft Teams. when you do some research about this issue, you will find many threads and blog posts, but these are the two key facts:

  • it is supported with on-premises hybrid Exchange deployments
  • it works flawless with Exchange Online

Our Exchange is configured as full-hybrid mode deployment. I did this as we deployed Office 365 at our organization.

Let’s summarize:

  • Exchange 2016 CU16
  • Hybrid Deployment
  • Office 365 with Teams enabled
  • no calendar tab when the Exchange mailbox is hosted on-premises

OAuth FTW!

While doing an Exchange Hybrid deployment for one of my customers some weeks ago, I’ve stumbled over an OAuth error message at the end of the Hybric Connection Wizard. The message was HCW8064

“HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration”

We were not able to fix this. Microsoft offers two solutions:

Yesterday I did the upgrade from CU15 to CU16 on our Exchange server and while watching the progress bar I did some research on this issue again. I found strong evidence that Microsoft Teams needs working OAuth to display the calendar tab and access the on-premises hosted mailbox. So I gave it a try and used the latest version of the HCW wizard.

What should I say? No OAuth configuration error and after a restart of Microsoft Teams, the calendar tab appeared.

Lessons Learned:

  • always use the latest CU für Exchange
  • always use the latest HCW Wizard

Supported Active Directory environments for Microsoft Exchange

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

It is time for some words of wisdom, in regard to Exchange and the supported Active Directory environments. It is the same as with the supported. NET Framework releases: Latest release does not automatically mean “supported”.

To be honest: I nearly nuked a customer environment with ~ 300 users yesterday by preparing the domain for the first Windows Server 2019 Domain Controller.

First things first: Everything is fine! I did not prepared to forest schema for Windows Server 2019.

The support for Windows Server 2008 R2 comes to an end and some customers are still running it. Like my customer yesterday. Some application servers are still on 2008 R2… and the Domain Controllers. The customer is also running Exchange 2013 on Windows Server 2012 R2.

The customer has decided to go to Windows Server 2019 wherever possible. This includes file servers, application servers, and the Domain Controllers. On of the first steps was the deployment of Active Directory-Based Activation. The AD schema needs to be prepared for this and I decided to prepare the schema for Windows Server 2019. I already copied the adprep folder from the Server 2019 ISO and openened a PowerShell. And then I paused. Something felt odd. I wanted to take a look at the Exchange Server supportability matrix.

Exchange 2013 does NOT supported Windows Server 2019 Domain Controllers! Uhh… that was unexpected.

Lessons learned

Always check the Exchange Server supportability matrix. Always! Regardless if it’s because of .NET Framework, Active Directory, Outlook Clients etc. Just check it every time you plan to change something in your environment.

Especially in regard to Microsoft Exchange “newer” does not automatically mean “supported”. Most times the opposite is true.

Microsoft Exchange 2013/ 2016/ 2019 shows blank ECP & OWA after changes to SSL certificates

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.
EDIT
This issue is described in KB2971270 and is fixed in Exchange 2013 CU6.

I published this blog post in July 2015 and it is still relevant. The feedback for this blog post was incredible, and I’m not joking when I say: I saved many admins weekends. ;) It has shown, that this error still occurs with Exchange 2016 and even 2019. Maybe not because of the same, with Exchange 2013 CU6 fixed bug, but maybe for other reasons. And the solution below still applies to it. Because of this I have decided to re-publish this blog post with a modified title and this little preamble.

Feel free to leave a comment if this blog post worked for you. :)

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

An error occurred while using SSL configuration for endpoint 0.0.0.0:444. The error status code is contained within the returned data.

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

C:\windows\system32>netsh http show sslcert

SSL Certificate bindings:
-------------------------

    IP:port                      : 0.0.0.0:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:444
    Certificate Hash             : a80c9de605a1525cd252c250495b459f06ed2ec1
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 0.0.0.0:8172
    Certificate Hash             : 09093ca95154929df92f1bee395b2670a1036a06
    Application ID               : {00000000-0000-0000-0000-000000000000}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

    IP:port                      : 127.0.0.1:443
    Certificate Hash             : 1ec7413b4fb1782b4b40868d967161d29154fd7f
    Application ID               : {4dc3e181-e14b-4a21-b022-59fc669b0914}
    Certificate Store Name       : MY
    Verify Client Certificate Revocation : Enabled
    Verify Revocation Using Cached Client Certificate Only : Disabled
    Usage Check                  : Enabled
    Revocation Freshness Time    : 0
    URL Retrieval Timeout        : 0
    Ctl Identifier               : (null)
    Ctl Store Name               : (null)
    DS Mapper Usage              : Disabled
    Negotiate Client Certificate : Disabled

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

C:\windows\system32>netsh http delete sslcert ipport=0.0.0.0:444

SSL Certificate successfully deleted

Now add it again with the correct certificate hash and application ID.

C:\windows\system32>netsh http add sslcert ipport=0.0.0.0:444 certhash=1ec7413b4fb1782b4b40868d967161d29154fd7f appid="{4dc3e181-e14b-4a21-b022-59fc669b0914}"

SSL Certificate successfully added

That’s it. Reboot the Exchange server and everything should be up and running again.

Client-specific message size limits – or the reason why iOS won’t sent emails

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week, a customer complained that he could not send emails with pictures with the native iOS email app. He attached three, four or five pictures to an emails, pushed the send button and instantly an error was displayed.

We checked the different connectors as well as the organizational limit for messages. The test mails were between 10 to 20 MB, and the message size limit was much higher.

geralt / pixabay.com/ Creative Commons CC0

The cross-check with Outlook Web Access indicated, that the issue was not a configured limit on one of the Exchange connectors. Instead, a quick search directed us towards the client-specific message size limits. Especially this statement caught our attention:

For any message size limit, you need to set a value that’s larger than the actual size you want enforced. This accounts for the Base64 encoding of attachments and other binary data. Base64 encoding increases the size of the message by approximately 33%, so the value you specify should be approximately 33% larger than the actual message size you want enforced. For example, if you specify a maximum message size value of 64 MB, you can expect a realistic maximum message size of approximately 48 MB.

The message size limit for Active Sync is 10 MB (Source). This is a server limit which can’t configured using the Exchange Admin Center. Taking the 33% Base64 overhead into account, the message size limit is ~ 6,5 MB.  My customer and I were able to proof this assumption. A 10 MB mail stuck in the outbox, a 6 MB mail was sent.

How to change client-specific message size limits?

In this case, my customer and I only changed the Active Sync limit. You can use the commands below to change the limit. This will rise the limit to ~ 67 MB. Without the Base64 overhead, this values allow messages sizes up to 50 MB. You have to run these commands from an administrative CMD.

%windir%\system32\inetsrv\appcmd.exe set config "Default Web Site/Microsoft-Server-ActiveSync/" -section:system.webServer/security/requestFiltering /requestLimits.maxAllowedContentLength:69730304
%windir%\system32\inetsrv\appcmd.exe set config "Default Web Site/Microsoft-Server-ActiveSync/" -section:system.web/httpRuntime /maxRequestLength:68096
%windir%\system32\inetsrv\appcmd.exe set config "Exchange Back End/Microsoft-Server-ActiveSync/" -section:system.webServer/security/requestFiltering /requestLimits.maxAllowedContentLength:69730304
%windir%\system32\inetsrv\appcmd.exe set config "Exchange Back End/Microsoft-Server-ActiveSync/" -section:system.web/httpRuntime /maxRequestLength:68096
%windir%\system32\inetsrv\appcmd.exe set config "Exchange Back End/Microsoft-Server-ActiveSync/" -section:appSettings /[key='MaxDocumentDataSize'].value:69730304

Make sure that you restart the IIS after the changes. Run iisreset from an administrative CMD.

Please note, that you have to run these commands after you installed an Exchange Server Cumulative Update (CU), because the files, in which the changes are made, will be overwritten by the CU. This statement is from the Microsoft:

Any customized Exchange or Internet Information Server (IIS) settings that you made in Exchange XML application configuration files on the Exchange server (for example, web.config files or the EdgeTransport.exe.config file) will be overwritten when you install an Exchange CU. Be sure save this information so you can easily re-apply the settings after the install. After you install the Exchange CU, you need to re-configure these settings.

The maximum size for a message sent by Exchange Web Services clients is 64 MB, which is much more that the 10 MB for Active Sync. This might explain why customers, that use Outlook for iOS app, might not recognize this issue.

EDIT: Today I found a blog post written by Frank Zöchling in June 2018, which addresses this topic.

Out-of-Office replies are dropped due to empty MAIL FROM

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I had an interesting support call. A customer noticed that Out-of-Office replies were not received by recipients, even though the OoO option were enabled for internal and external recipients. Internal recipients got the OoO reply, but none of the external recipients.

cattu/ pixabay.com/ Creative Commons CC0

The Message Tracking Log is a good point to start. I quickly discovered that the Exchange server was unable to send the OoO mails. You can use the eventid FAIL to get a list of all failed messages.

Very interesting was the RecipientStatus of a failed mail.

RecipientStatus         : {[{LED=550 Requested action not taken: mailbox unavailable};{MSG=};{FQDN=mailrelay-out.xxxx.de};{IP=213.xxx.xxx.xxx};{LRT=20.12.2018 10:22:39}]}

550 Requested action not taken: mailbox unavailable  is a pretty interesting error when sending mails over a mail relay of your ISP. Especially when other mails were successfully sent over the same mail relay.

Next stop: Protcol log of the send connector

I enabled the logging on the send connector using the EAC. This option is disabled by default. Depending on the amount of mails sent over the connector, you should make sure to disable the logging after your troubleshooting session. To enable the logging, follow these steps:

  • Open the EAC and navigate to
  • Mail flow > Send connectors
  • Select the connector you want to configure, and then click Edit
  • On the General tab in the Protocol logging level section, select the Verbose option
  • When you’re finished, click Save

The protocol log can be found under %ExchangeInstallPath%TransportRoles\Logs\Hub\ProtocolLog\SmtpSend.

After enabling the logging and another test mail, the log contained the necessary details to find the root cause. This is the interesting part of the SMTP communication:

2018-12-20T10:22:39.313Z,Relay,08D640AAC0AD8811,3,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,220 mailrelay-out.xxx.de ESMTP Postfix (Debian/GNU),
2018-12-20T10:22:39.313Z,Relay,08D640AAC0AD8811,4,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,EHLO mail.domain.local,
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,5,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,250  mailrelay-out2.xxx.de SIZE 52428800 8BITMIME OK,
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,6,192.168.0.212:49986,213.xxx.xxx.xxx:25,*,,sending message with RecordId 22471268892695 and InternetMessageId <b9613be791c141e3b76828228bd6cdb3@exchange.domain.local>
2018-12-20T10:22:39.330Z,Relay,08D640AAC0AD8811,7,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,MAIL FROM:<> SIZE=4758,
2018-12-20T10:22:39.331Z,Relay,08D640AAC0AD8811,8,192.168.0.212:49986,213.xxx.xxx.xxx:25,<,550 Requested action not taken: mailbox unavailable,
2018-12-20T10:22:39.332Z,Relay,08D640AAC0AD8811,9,192.168.0.212:49986,213.xxx.xxx.xxx:25,>,QUIT,

The error occured right after the exchange server issued MAIL FROM:<> . But why is the MAIL FROM empty?

RFC 2298 is the key

An Out-of-Office reply is a Delivery Status Notification message. And RFC 2298 clearly states:

The envelope sender address (i.e., SMTP MAIL FROM) of the MDN MUST be
null (<>), specifying that no Delivery Status Notification messages
or other messages indicating successful or unsuccessful delivery are
to be sent in response to an MDN.

So the empty MAIL FROM is something that a mail relay should expect. In case of my customer that mail relay seems to act different. Maybe some kind of spam protection.

Database Availability Group (DAG) witness is in a failed state

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a maintenance job I had to update a 2-node Exchange Database Availability Group and a file-share witness server.

After the installation of Windows updates on the witness server and the obligatory reboot, the witness left in a failed state.

[PS] C:\Windows\system32>Get-DatabaseAvailabilityGroup -Identity DAG1 -Status | fl *wit*
WARNING: Database availability group ‘DAG01’ witness is in a failed state. The database
availability group requires the witness server to maintain quorum. Please use the
Set-DatabaseAvailabilityGroup cmdlet to re-create the witness server and the directory.

WitnessServer : fsw.domain.local
WitnessDirectory : C:\DAGFileShareWitnesses\DAG1.domain.local
AlternateWitnessServer :
AlternateWitnessDirectory :
WitnessShareInUse : InvalidConfiguration
DxStoreWitnessServers :

In my opinion, the re-creation of the witness server and the witness directory cannot be the correct way to solve this. There must be another way to solve this. In addition to this: The server was not dead. Only a reboot occured.

Check the basics

Both DAG nodes were online and working. A good starting point is a check of the cluster resources using the PowerShell.

In my case the cluster resource for the File Share Witness was in a failed state. A simple Start-ClusterResource  solved my issue immediately.

[PS] C:\Windows\system32>Get-ClusterResource

Name                                              State                                             OwnerGroup                                        ResourceType
----                                              -----                                             ----------                                        ------------
File Share Witness (\\fsw.domain.local            Failed                                            Cluster Group                                     File Share Witness


[PS] C:\Windows\system32>Get-ClusterResource | Start-ClusterResource

Name                                              State                                             OwnerGroup                                        ResourceType
----                                              -----                                             ----------                                        ------------
File Share Witness (\\fsw.domain.local            Online                                            Cluster Group                                     File Share Witness

In this case, it seems that the the cluster has marked the file share witness as unreliable, thus the resource was not started after the file share witness was back online again. In this case, I managed it to manually bring it back online by running Start-ClusterResource  on one of the DAG members.