Tag Archives: exchange 2013

Citrix NetScaler and Exchange: Case-sensitivity of internal and external URLs

Exchange has known the concept of internal and external URLs for the different services (Outlook Web Access, OAB, EWS, ActiveSync etc) since Exchange 2007. And it’s still confusing people. The internal URL is the URL, that is used to access the desired service from the intranet. The external URL represents the URL that is used to access the service from the internet. Best practice is to use the same URL (the external) for both, use a certificate from a public CA, and use split DNS to access the external domain from the inside of your network.

People tend to imply, that URLs are not case-sensitive. This seems to be true in most cases. The World Wide Web Consortium (W3C) states:

URLs in general are case-sensitive (with the exception of machine names). There may be URLs, or parts of URLs, where case doesn’t matter, but identifying these may not be easy. Users should always consider that URLs are case-sensitive.

Source W3C

Citrix NetScaler and URLs

Citrix NetScaler handles URLs as case-sensitive.

A frequently used concept to load balance Microsoft Exchange with a NetScaler is Content Switching. Policies are used to identify traffic, and actions are used to take action on the traffic that matches the policies. The NetScaler uses the advanced policy engine to create expressions for the Content Switching Policies. When creating a Content Switching policy by creating an expression that uses the CONTAINS operator, you might notice that the results are case-sensitive.

This can be a problem in case of Microsoft Exchange, because /Autodiscover/Autodiscover.xml and /autodiscover/autodiscover.xml, or /ews/exchange.asmx and /EWS/Exchange.asmx are handled different.

Solution

To make sure that different cases are handled, you should add  SET_TEXT_MODE(IGNORECASE)  to you policy expression. Citrix describes this in CTX115528.

I’ve changed my NetScaler setup script for Exchange to handle this behavior.

Surprise, surprise: Enable/ disable circular logging without downtime

As part of a troubleshooting process, I had to disable circular logging on a Microsoft Exchange 2013 mailbox database, that was part of a Database Availability Group (DAG).

What is circular logging? Microsoft Exchange uses log files that record all transactions and modifications to a mailbox database. In this case, Exchange works like MS SQL or Oracle. These log files are used, to bring the database back to a consistent state after a crash, or to recover a database to a specific point-in-time. Because everything is written to log files, the number of log files increases over time. Log files are truncated, as soon as a successful full backup of the database is taken.

If you don’t need the capability to recover to a specific point-in-time, or if you are running low on disk space, circular logging can be an alternative. Especially if you are running low on disk space, because your backup isn’t working as expected. With circular logging enabled, Microsoft Exchange maintains only the minimum number of log files that is necessary, to keep the Exchange server running. As soon as a transaction was written to the database, the log file will be overwritten.

I rarely use circular logging. But this time I had to. As already mentioned, I had a mailbox database with enabled circular logging. This database was part of a DAG and I had to disable circular logging. Usually, you need to dismount and re-mount the database after enabling or disabling circular logging.

You can enable or disable circular logging using the Microsoft Exchange Control Panel (ECP), or with the Microsoft Exchange Management Shell. I have used the PowerShell.

To my surprise, dismounting and re-mounting the database was not necessary. The circular logging was disabled without downtime. I’ve checked the TechNet, and the observed behaviour was confirmed.

JET vs. CRCL

A non-replicated mailbox databases will use JET circular logging. If the database is part of a DAG, the database will use continuous replication circular logging (CRCL). A benefit of CRCL is, that it can be enabled and disabled without the need of dismounting and re-mounting the mailbox database.

To get this clear: This only works if you are using replicated mailbox databases, because only databases that belong to a DAG are using CRCL. If you are using standalone mailbox databases, you have to dismount and re-mount the database, after enabling or disabling circular logging.

As I mentioned earlier: I really don’t use circular logging often, but that was very handy today!

Changes to supported .NET Frameworks for Exchange 2013/2016

EDIT: If you have already installed .NET 4.6.1, check this blog post on how to remove it (You Had Me At EHLO…)

Microsoft Exchange heavily relies on Microsoft .NET Framework. Because of this, Microsoft provides a matrix for the supported Microsoft .NET Frameworks. Mostly unknown is the fact, that Exchange doesn’t support the every Microsoft .NET Framework, and this is causing trouble sometimes. Some admins simply install the latest .NET releases because “it doesn’t hurt”. Well… it hurts!

Changes for .NET Framework 4.6.1

Microsoft has changed the support policy for .NET Framework 4.6.1 with the release of Exchange 2013 CU13 and Exchange 2016 CU2. Up to this versions, only .NET Framework 4.5.2 is supported. Starting with Exchange 2013 CU13 and Exchange 2016 CU2, Microsoft supports .NET Framework 4.6.1 together with a hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2). If you wish to install .NET Framework 4.6.1, make sure to install Exchange 2013 CU13 or 2016 CU2 first.

.NET Framework/ Microsoft ExchangeExchange 2007 SP3Exchange 2010 SP3Exchange 2013 CU13 and laterExchange 2016 CU2 and later
.NET Framework 3.5.1XX
.NET Framework 4.0
.NET Framework 4.5X
.NET Framework 4.5.1X
.NET Framework 4.5.2XX
.NET Framework 4.6.1
.NET Framework 4.6.2

¹ .NET 3.5 or 3.5.1 must be installed

² Supported with hotfix rollup (KB3146715 for Server 2012 R2, KB3146714 for Server 2012 and KB3146716 for Server 2008 R2)

Other .NET Framework versions

Microsoft .NET Framework 4.6.2 isn’t supported for any version of Microsoft Exchange. Other example: If you’re running Exchange 2010 SP3, don’t install anything above .NET Framework 4.5, not even 4.5.1. Check the Exchange Server Supportability Matrix for the supported .NET Framework for the Exchange version you’re running.

Side notes

Microsoft PowerShell is part of the Windows Management Framework (WMF). Microsoft Exchange only supports the WMF built into the underlying Windows Server version.

This also applies to Microsoft Outlook. I wonder how many Exchange projects fail because the Microsoft Outlook version, that is used by the customer, isn’t supported.

Exchange Management Shell (EMS) and new PowerShell releases

Some day ago, I installed a new Exchange 2013 CU11 for some test ins my lab. Nothing fancy, just a single server deployment on a Windows Server 2012 R2 VM. I deployed this Windows Server from a template, which was updated with the latest Windows Patches and WMF some days ago. The Exchange setup went smooth. I updated the SSL certificates and the internal and external URLs for the virtual directories. Then I started the Exchange Management Shell (EMS), to update the Autodiscover URL in the service connection point (SCP) of the Active Directory.

Well… that doesn’t look successful. I quickly switched to a PowerShell windows and imported the Exchange snap-in manually.

Looks better, isn’t it?

I compared my lab setup to a running Exchange 2013 single server deployment and I stumbled over the PowerShell version. In addition, I found the Windows Management Framework 5 Production Preview (KB3066437) on my freshly deployed Windows Server 2012 R2 VM.

After checking the Exchange Server Supportability Matrix, it was clear what had happened: WMF 5 is not supported (Source). Not supported with Exchange 2013, and also not supported with Exchange 2016.

exchange_supported_wmf

After I had removed KB3066437 from my Exchange server, the EMS loaded successfully.

You should ALWAYS check if installed applications are supported with newer version of PowerShell/ WMF! Currentyl, no Exchange version is supported with PowerShell 5/ WMF 5.

Publishing Outlook Web Access with Microsoft Web Application Proxy (WAP)

Microsoft has introduced the Web Application Proxy (WAP) with Windows Server 2012 R2 and has it positioned as a replacement for Microsoft User Access Gateway (UAG), Thread Management Gateway (TMG) and IIS Application Request Routung (ARR). WAP ist tightly bound to the Active Directory Federation Services (AD FS) role. WAP can be used

  • pre-authenticate access to published web applications, and
  • it can function as an AD FS proxy

The AD FS proxy role was removed in Windows Server 2012 R2 and it’s replaced by the WAP role. Because WAP stores its configuration in the AD FS, you must deploy AD FS in your organization. The server, that hosts the WAP, has no local configuration. This allows you to deploy additional WAP servers to create a cluster deployment. The additional servers get their configuration from the AD FS.

The deployment of WAP can be split into two parts:

  • deployment of the AD FS role
  • deployment of the WAP role

The AD FS deployment

You can deploy the AD FS role on a domain controller or on a separate server. AD FS  acts as an identity provider. This means, that it authenticates users and provides security tokens to applications, that trust the AD FS instance. On the other hand it can act as a federation provider. This means, that it can use tokens from other identity providers and can provide security tokens to applications that trust AD FS. The AD FS role can be deployed onto a domain controller or a AD member server.

The first step is to install the AD FS role onto a AD member server or domain controller. I used the DC in my lab. Depending on your needs, this can be different. I used the PowerShell to install the AD FS role.

A reboot is not necessary. The next step is to configure the AD FS role. This process is supported by configuration wizard. Before you can start, it’s necessary to deploy the group Managed Service Account (GMSA). Open a PowerShell console and execute the following commands:

Then you can start the configuration wizard. If this is the first first AD FS server, select the first option “Create the first federation server in a federation server farm”.

adfs_setup_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

To perform the configuration, you need an account with domain administrator permissions. In my case, I simply used the Administrator account.

adfs_setup_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You need to enroll an SSL certificate that is used for AD FS. This SSL certificate must include the DNS name for the AD FS server and also the Subject Alternative Names enterpriseregistration and enterpriseregistration.yourdomainname.tld. This screenshot includes the values that I used in my lab deployment. I entered this values into the “Attributes” box:

adfs_setup_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Create the certificate and export it with the private key as pfx file. You must import the certificate into the “Personal” store of the local computer, that acts as AD FS server. You also need two DNS entries for the names, that are included in the certificate.

adfs_setup_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If the certificate import was successful, you can select the certificate in the wizard. Add the Federation Service Name and the Display Name.

adfs_setup_05

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The Service Account can be a existing domain user account or a Managed Service Account. I used my Administrator account for simplicity.

adfs_setup_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you deploy a single server, you can use the Windows Internal Database. If you plan to depliy multiple AD FS servers, you have ot use a SQL server database.

adfs_setup_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Review the options and continue with the pre-requisite checks. If everything went well, you can proceed with the installation. Finish the setup and close the wizard.

adfs_setup_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open a browser and enter the AD FS URL into the address bar. In my case this URL looks like this:

https://adfs.terlisten-consulting.de/adfs/ls/idpinitiatedsignon.htm

adfs_setup_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you get a screen like this, everything’s fine and AD FS works as expected. Check the Windows Server 2012 R2 AD FS Deployment Guide for more information. Now it’s time to deploy the Web Application Proxy.

The WAP deployment

To install the WAP role, simply open a PowerShell and run the Install-WindowsFeature cmdlet.

Then you can run the WAP configuration wizard. This wizard guides you through the configuration of the WAP role.

wap_setup_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

First you have to connect to the AD FS server. Enter the Federation service name you used to deploy the AD FS instance, and provide the necessary user credentials.

wap_setup_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

At this point you have to select the certificate, that is used by the AD FS proxy. You can use the same certificate you used for the AD FS server. But you can also create a new certificate. The certificate must be imported into the “Personal” store of the WAP server.

wap_setup_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Confirm the settings and click “Configure”. At this point, the wizard executes the shown PowerShell command.

wap_setup_04

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Close the wizard and open the management console of the Web Application Proxy to check the operational status. At this point, the WAP only acts as a AD FS proxy.

wap_setup_06

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

To test the functionality, I decided to publish Outlook Web Access (OWA). Use the “Publish New Application Wizard” to publish a new application.

wap_setup_07

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

To publish OWA, select “Pass-through” as pre-authentication method.

wap_setup_08

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Now it’s getting interesting. When you enter the external URL, the backend server URL is automatically filled. External and Backend URL have to be the same URL. Because of this, you need split DNS (see “Configure the Web Application Proxy Infrastructure” and “AD FS Requirements” at the Microsoft TechNet Library). You also need a valid external certificate, that matches the FQDN used in the external URL.

wap_setup_09

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check the settings and click “Publish”. The wizard executes the shown PowerShell command.

wap_setup_10

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Close the wizard and check the functionality of the published application. This screenshot shows the access to OWA from one of my management VMs (MGMTWKS1):

wap_test_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This drawing shows my lab setup. I’ve used two subnets (192.168.200.64/27 and 192.168.200.96/27) to simulate internal and external access, as well as split DNS.

wap-adfs

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The host dc.vcloudlab.local (192.168.200.97) has the AD FS role installed and resolves cas.terlisten-consulting.de to 192.168.200.103 (HAProxy). MGMTWKS1 resolves the same FQDN to 192.168.200.109 (WAP1 – my WAP server).

Final words

This is only a very, very basic setup and I deployed it in my lab. The installation was not very difficult and I was quickly able to set up a working environment. Before you start to deploy AD FS/ WAP, I recommend to take a look into the TechNet Library:

Users on Exchange 2013 can’t open public folders or shared mailboxes on an Exchange 2007/ 2010

When moving users to Exchange 2013 it can happen, that they can’t access public folders housed on the old Exchange 2010 or 2007 server. The same can happen to shared mailboxes (mailboxes with Full Access permissions). The users are constantly prompted for credentials or they get this message:

This can be a huge problem during a migration. Microsoft described this in KB2834139. This error is caused by a misconfigured Outlook security setting, called “Logon network security”. If you experience this issue, check the “Logon network security” setting. If it’s set to “Anonymous Authentication”, then you experience the in KB2834139 described problem. Otherwise you have another problem. Check the “Logon network security” settings in your Outlook client. I took this screenshots from a Outlook 2013, but it looks the same in Outlook 2010. With this setting you will have a problem:

outlook_settings_login_security_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

With one of these settings, it will work.

outlook_settings_login_security_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

outlook_settings_login_security_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can change the setting and try the access to a public folder or shared mailbox. If you can access the public folder or shared mailbox, then you have to change some settings on the Exchange Server 2013 Client Access Server (CAS).

Open an Exchange Management Shell:

I would recommend to execute “iisreset” after changing the settings. Please note, that this interrupts the client access for a short moment! After a restart of the Outlook client or during the next Autodiscover, the client should get the correct settings and the access to the public folders and shared mailboxes should work.

Importance of client-side proxy settings in Exchange 2013 environments

There is an advantage, if you solves problems: You can learn something. I’m currently migrate a small Exchange 2007 environment to Exchange 2013. The first thing I learnt was, that IT staff still uses their own accounts for administration, and sometimes they assign administrator rights to users for testing and troubleshooting purposes. This can be a problem, as I described in my last posting. Today I learnt something different: Sometimes it’s the little things that bring you to despair.

After moving a mailbox from Exchange 2007 to 2013, Outlook must change the server for the client access. Nothing fancy and the user normally doesn’t notices it. If Outlook is online during the mailbox migration, the user gets a message, that he has to restart Outlook. Please note, that you need at least Outlook 2007 SP3, when you wish to migrate to Exchange 2013. This is because of an important change with Exchange 2013: The abolition of direct MAPI connections. Exchange 2013 only supports RPC-over-HTTP (aka Outlook Anywhere), even for LAN connections. RPC-over-HTTP has several advantages, e.g. the CAS role has to deal with only one protocol or easier load balancing and high-availability.

The problem

After moving a mailbox from Exchange 2007 to 2013, the Outlook 2010 client wasn’t able to connect to the Exchange server. The server was correctly changed, as I was able to see in the Outlook profile, but everytime I tried to start Outlook 2010, I got this error:

outlook_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0


Moving the mailbox back to the Exchange 2007 solved the problem. Moving the mailbox again to the Exchange 2013 resulted in the same error. Outlook Web Access was working fine (internal and external) on Exchange 2007 and 2013. The Microsoft Office Outlook Connectivity Test completed successful, after moving the mailbox to the Exchange 2013. Even Enterprise Active Sync (EAS) worked on the Exchange 2013 server. Only Outlook 2010 wasn’t able to connect to the mailbox, when it was on the Exchange 2013. The customer and I were puzzled… While testing this and that, we got an error while accessing Outlook Web Access. On another client, with another user OWA worked fine. BÄÄÄM! A possible cause popped in my mind.

The solution

I checked immediately  the proxy settings and there it was: A big proxy bypass list in the Internet Explorer with several entries, but the new Exchange server was missing. I added the server to the proxy bypass list and Outlook started without any problems. To be honest: It was a bit more complex, because the used proxy wasn’t an internal system. A solution provider operates it and the proxy settings were managed by a GPO, that wasn’t working correctly. In addition to that, an AD group membership was used to allow users to pass a web filter. But at the core it was the missing entry for the new Exchange server, that caused the problem.

The explanation

Exchange 2013 only supports RPC-over-HTTP and it uses the system-wide proxy settings. Therefore, HTTP(S) traffic is sent to the proxy server (regardless if the destination is internal or external), unless there is an entry in the proxy bypass list for the destionation (in this case the Exchange server). If the proxy can’t handle the traffic, Outlook will not be able to connect to the Exchange server. With MAPI, the proxy isn’t a problem, because MAPI traffic isn’t sent to the proxy. This explains, why Outlook was able to connect to the Exchange server, if the mailbox was moved back to the Exchange 2007. With Exchange 2007, Outlook uses MAPI for the connection. With Exchange 2013 RPC-over-HTTP is used.

So if you experience connection problems after moving a mailbox to Exchange 2013, check your proxy settings. This also applies when using Outlook Anywhere, because Outlook Anywhere uses also RPC-over-HTTP.

Active Directory property ‎’homeMDB‎’ is not writeable on recipient

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

How to enable permissions inheritance?

Start the Active Directory Users and Computers MMC. Click “View” and enable “Advanced Features”. Otherwise the needed tabs will not shown.

permissions_inheritance_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the user that can’t be migrated. Switch to the “Security” tab. Click “Advanced” and enable the “Include inheritable permissions from this object’s parent” check box. Click “OK” and confirm popups regarding added ACL entries.

permissions_inheritance_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check if the Exchange Trusted Subsystem group is now part of the ACL. If yes, try the migration of the mailbox again.

Why is  permissions inheritance disabled?

Good question, the customer asked me the same question. But the answer is easy. As above mentioned, is a ACL with disabled permission inheritance called a protected ACL. When should a ACL be protected? If the ACL of the associated object shouldn’t be changed, when a permission from a parent object is inherited. A good example for such an object is the Administrator account. Active Directory protects members of the following groups

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

and the following users

  • Administrator
  • krbtgt

with three simple steps:

  1. Set the value of the AdminCount attribute to 1.
  2. Disable permission inheritance
  3. Apply permissions from the AdminSDHolder template

This process runs once an hour on a Active Directory Domain Controller and recovers the ACL of the two user accounts and the member accounts of the mentioned groups to a protected state and well-known permissions. What happens if you add an user account to the Domain Admins group? AdminCount is set to 1, the permission inheritance is disabled and the permissions specified in the AdminSDHolder template are applied. If you remove the user account from the Domain Admins group, the changes are not rolled back and so  permission inheritance stays disabled! Changes to parent ACLs are not applied to the user account and, in the case of my customer, the Exchange Trusted Subsystem group wasn’t added to the user objects ACL.

You can search for accounts with AdminCount=1 easily with the PowerShell:

Why is this a problem? With AdminCount=1 and disabled permission inheritance you will face a lot of problems:

  • Changes to an object ACLs are rolled back after an hour
  • Send-As doesn’t work
  • ActiveSync doesn’t work
  • Language settings in Outlook Web Access isn’t saved
  • ..and some other nasty effects

I have seen this error so often. Mostly at customers, that use normal work accounts for administrative tasks, or that add normal user accounts to admin groups during the setup of new profiles or clients. The solution is easy: Don’t do it!

Exchange 2013: Event ID 2937 MSExchange ADAccess after public folder migration

I got a couple of warnings (source MSExchange ADAccess, Event ID 2937) after removing a Exchange 2007 server at the end of a Exchange 2007 > 2013 migration. The details of the warning told me, that there was a faulty value set to a attribute of the mailbox database object. Because the public folder migration was part of the migration, the error message seemed plausible.

A quick check with ADSI Edit confirmed the message. To be honest: I made a mistake and searched for the attribute PublicFolderDatabase in the database object, but in the end I found the wrong entry as a value of the msExchHomePublicMDB attribute in the database object. It must be set to the distinguished name of the mailbox database that houses the public folder mailboxes. If you don’t have any public folders in your Exchange 2013 org, then you have to clear the value!

The solution

Start ADSI Edit (Start > Run > adsiedit.msc) and right click the “ADSI Edit”. Select “Connect to…”.

event_2937_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the configuration context.

event_2937_4

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Navigate to the database object, which is mentioned in the warning.

event_2937_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Right click the database object and select “Properties” from the context menu. You can limit the number of attributes if you hit the “Filter” button and select “Show only attributed that have values”. Scroll down to the msExchHomePublicMDB attribute. Double click it and enter the complete CN of the mailbox database, that houses the public folder mailboxes (this is new with Exchange 2013!). If you have not public folders in your Exchange org, than you can clear the value!

event_2937_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Click “OK” and restart the Exchange Information Store service. After the restart, no new 2937 events should be logged.

EDIT

If you get the message “Your Administrator has made a change and requires you to restart Outlook” (or in german “Der Microsoft Exchange Administrator hat eine Änderung durchgeführt, die einen Neustart von Outlook erfordert”), check if event id 2937 is logged on the Exchange servers. I had several migrations where a removed public folder DB caused Outlook to throw the error message.