Tag Archives: exchange

Load Balancing Microsoft Exchange 2013 with HAProxy

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Since Exchange 2007 client connections are handled by the Client Access Server role. With Exchange 2010, Microsoft has introduced the concept of the Client Access Server Array (CAS Array). A CAS Array is required, when internal and external client connections should be load balanced over multiple client access servers. Many client access protocols in Exchange 2010 require session affinity. This means, that the connection between the client and a particular client access server must persist. This requires application-level load balancing for Exchange 2010 and Microsoft recommends this explicitly. Microsoft dropped the concept of the CAS Array in Exchange 2013 and implemented much more logic into the Exchange 2013 Client Access Server role. There is no more need for session affinity in any client access protocol used in Microsoft Exchange 2013. Connections to a Exchange 2013 client access servers can be directed to an available server. A simple DNS round-robin works, but if a server fails, DNS would not handle this.You can use Windows Network Load Balancing (WNLB), but it has several limitations and downsides. I blogged about one of them in my blog post Flooded network due HP Networking Switches & Windows NLB. The other point is, that you can’t use it when you build a two server CAS/ DAG Exchange 2013 environment: You can’t use WNLB on servers that have the Microsoft Failover Cluster role installed. At this point HAProxy comes into play.

HAProxy is a small and reliable TCP/ HTTP Load Balancer. HAproxy is Open Source and supports in its current release everything you need, e.g. support for SSL, IPv6, keep-alive etc. Sometimes there is no need for cost intensive or complex Load Balancers, e.g. for lab setups. HAProxy is small and easy to set up. All you need is your favorite Linux distribution in its current release. I like CentOS and I decied to use CentOS 7 to setup a small HAProxy deployment in my lab.

Installation

I have installed a minimal installation of CentOS 7 in a VM (2 GB memory, 1x vCPU, 1x VMXNET3 adapter). You can easily install HAProxy using YUM.

[[email protected] ~]# yum install haproxy
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: ftp.hosteurope.de
 * extras: centos.intergenia.de
 * updates: centos.intergenia.de
Resolving Dependencies
--> Running transaction check
---> Package haproxy.x86_64 0:1.5.2-3.el7_0 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

==============================================================================
 Package          Arch          Version          Repository          Size
==============================================================================
Installing:
 haproxy          x86_64        1.5.2-3.el7_0    updates             812 k

Transaction Summary
==============================================================================
Install  1 Package

Total download size: 812 k
Installed size: 2.5 M
Is this ok [y/d/N]: y
Downloading packages:
haproxy-1.5.2-3.el7_0.x86_64.rpm                                                                                                                           | 812 kB  00:00:03
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : haproxy-1.5.2-3.el7_0.x86_64                                                                                                                                   1/1
  Verifying  : haproxy-1.5.2-3.el7_0.x86_64                                                                                                                                   1/1

Installed:
  haproxy.x86_64 0:1.5.2-3.el7_0

Complete!

That’s it. There’s nothing more to do. Now let’s configure HAproxy.

Configuration

Before you edit the configuration file, take a backup of the /etc/haproxy/haproxy.conf. My haproxy.conf looks like this.

defaults
    log global
    option tcplog
    option dontlognull
    option redispatch
    retries 3
    timeout http-request 10s
    timeout http-keep-alive 10s
    timeout check 10s
    timeout server 10s
    timeout connect 10s
    timeout client 10s

listen stats 192.168.200.103:4711
    mode http
    stats enable
    stats hide-version
    stats uri /

listen e2k13 192.168.200.103:443
    mode tcp
    option ssl-hello-chk
    option http-keep-alive
    balance roundrobin
    stick-table type ip size 20k expire 15m
    stick on src
    timeout server 1m
    timeout connect 1m
    timeout client 5m
    server exchange1 192.168.200.100:443 check
    server exchange2 192.168.200.102:443 check

Nothing fancy. 192.168.200.103 is the IP of my CentOS 7 VM. 192.168.200.100 and 192.168.200.102 are two Exchange 2013 servers (CAS & Mailbox). To get some stats, I added the listen stats section to my config. Please note, that this config passes HTTPS traffic to the backend servers! SSL traffic is not terminated at the HAProxy itself. Therefore you need valid certificates on all of your client access servers. When you finished your config, you can start the HAProxy and check the success with netstat.

[[email protected] ~]# systemctl start haproxy.service
[[email protected] ~]# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          15431      1309/master
tcp        0      0 192.168.200.103:443     0.0.0.0:*               LISTEN      0          29484      1979/haproxy
tcp        0      0 192.168.200.103:4711    0.0.0.0:*               LISTEN      0          29482      1979/haproxy
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          14752      812/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      0          15432      1309/master
tcp6       0      0 :::22                   :::*                    LISTEN      0          14754      812/sshd
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           70         13708      541/avahi-daemon: r
udp        0      0 0.0.0.0:42748           0.0.0.0:*                           70         13709      541/avahi-daemon: r

You need to create DNS A-Records that points to the IP address of the HAProxy. Then add this A-Records as internal and external hostnames for the Exchange 2013 virtual directories. Here’s an example for Outlook Anywhere:

[PS] C:\windows\system32>Get-OutlookAnywhere | select servername, *hostname

ServerName                              ExternalHostname                        InternalHostname
----------                              ----------------                        ----------------
EXCHANGE1                               cas.terlisten-consulting.de             mail.vcloudlab.local
EXCHANGE2                               cas.terlisten-consulting.de             mail.vcloudlab.local

A change to the Outlook Anywhere config can take up to 15 minutes, until clients discover the change. As you can see, the client in my lab uses the internal hostname.

haproxy_outlook_connection

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Both Exchange servers receive requests. This is screenshot is taken from the HAProxy stats website (click to enlarge).

haproxy_stats

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Final words

I really like HAProxy. It’s perfect for lab environments or small deployments, not only to load balance HTTP/ HTTPS requests for Microsoft Exchange 2013.

Data Protector: Exchange 2010 database recovery from copy session fails

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

The recovery of an Exchange mailbox using a recovery database is usually no big deal. Simply restore the database, create a recovery database and recover the mailbox or items from the mailbox. Sometimes you have the luck that the customer has licensed the Data Protector Exchange 2010 Granular Recovery for Exchange (GRE). This was unfortunately not true in my case. Okay, so let’s do it the old way. The needed tape was available in the library and luckily it was a full backup. So I quickly added a disk to the VM and started the recovery of the database to a temporary location. At this point, the disaster took its course…

Constraints

This isn’t a general problem of Data Protector, it’s a bug. The following constraints met in my case:

  • Recovery source is a copy session (e.g. post-backup object copy to tape)
  • Data Protector 7.03
  • Microsoft Exchange 2010 SP2
  • Recovery to temporary location

This is the output of “omnidb -session 2014/08/01-6 -detail” for the wanted session. As you can see, this is a copy of session 2014/08/01-3, which was without protection and therefore was removed by Data Protector.

Object name : exchangeserver.domain.tld:/30ae2b6b-df08-4ed1-a030-a7
0dbae354a6/DB2
Object type : E2010
Object status : Completed
Started : Freitag, 1. August 2014, 15:49:39
Finished : Samstag, 2. August 2014, 03:31:22
Object size : 0 KB
Backup type : Full
Protection : Protected for 50 weeks
Catalog retention : Same as data protection.
Version type : Normal
Access : Public
Number of warnings : 0
Number of errors : 0
Device name : HP:Ultrium 4-SCSI_1
Backup ID : 2014/08/01-3
Copy ID : 81705 (Orig)
Encrypted : No
DiskAgent ID : 1406901317

When trying to recover the DB to a temporary location, I ran into this error:

[Normal] From: RSM@backupserver.domain.tld "" Time: 21.08.2014 09:26:21
Restore session 2014/08/21-47 started.

[Normal] From: OB2BAR_E2010_BAR@exchangeserver.domain.tld "MS Exchange 2010 Server" Time: 21.08.2014 09:26:21
Analyzing MS Exchange Server 2010 environment.

[Normal] From: OB2BAR_E2010_BAR@exchangeserver.domain.tld "MS Exchange 2010 Server" Time: 21.08.2014 09:26:30
Restoring database DB2 :
Restore mode : Restore files to a temporary location
Session ID : 2014/08/01-3
Restore only this backup : TRUE
Target client/s : exchangeserver.domain.tld
Restore location : R:\RDB
Restore options : Perform database recovery - FALSE
Restore database file only - TRUE

[Minor] From: OB2BAR_E2010_BAR@exchangeserver.domain.tld "MS Exchange 2010 Server" Time: 21.08.2014 09:26:30
Getting the restore information from IDB and creating the restore chain failed for database DB2(30ae2b6b-df08-4ed1-a030-a70dbae354a6).

[Critical] From: OB2BAR_E2010_BAR@exchangeserver.domain.tld "MS Exchange 2010 Server" Time: 21.08.2014 09:26:30
No mailbox database copy can be selected for restore/instant recovery.

[Normal] From: RSM@backupserver.domain.tld "" Time: 21.08.2014 09:27:09
OB2BAR application on "exchangeserver.domain.tld" disconnected.

============================================================================
Session failed!
============================================================================

The session I tried to recover was 2014/08/01-06, the session that was chosen by Data Protector for recovery was 2014/08/01-03. To make the long story short: You can fix it with a site specific path for Data Protector 7.03. Log a call at the HP Support and ask for SSPNT700_038. Plase note that you need a valid HP Software Support contract to get this patch! The patch delivers fixes for the three defects QCCR2A51280, QCCR2A53555 and QCCR2A46724. The patch is delivered as a ZIP file and contains binaries and libraries which has to be installed on the Cell Manager and the Exchange server. The patch contains files for Windows on x64, HP-UX 11.23, 11.31 on IA64, HP-UX 11.23, 11.31 on PA RISC and Linux on x64. When running Data Protector on Windows, you have to replace the following files:

On the Exchange server

e2010_bar.exe
PSHAPI.dll
vssbar.exe

On the Data Protector Cell Manager

asm.exe
bsm.exe
csm.exe
dbsm.exe
libob2eadm.dll
libob2ecdb.dll
libob2ecmn.dll
libob2emmdb.dll
mmd.exe
msm.exe
rsm.exe

You have to stop the Data Protector services on the Cell Manager and the Data Protector Inet service on the Exchange server before you can replace the files. Make a backup of the files before you replace them. After the file replacement start the services and try the restore again. In my case the restore didn’t worked after applying the patch: It failed with the same error. I opened a case at HP and after a few day I got the notice, that lab engineering was involved in the case. Short after that notice, the support sent me two files (QCCR2A54842_TM1) that I had to replace on the Data Protector Cell Manager (libob2ecdb.dll & libob2ecmn.dll). Both files were part of SSPNT700_038, so you can interprete this as a patch for a patch. ;) This patch did the trick and the restore was successful. The root cause was, that a database query returns the wrong session for the recovery. So if you face the same problems, ask HP for the site-specific patch. If the problem remains, ask for QCCR2A54842_TM1.

Users on Exchange 2013 can’t open public folders or shared mailboxes on an Exchange 2007/ 2010

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When moving users to Exchange 2013 it can happen, that they can’t access public folders housed on the old Exchange 2010 or 2007 server. The same can happen to shared mailboxes (mailboxes with Full Access permissions). The users are constantly prompted for credentials or they get this message:

Cannot expand the folder. Microsoft Exchange is not available. Either there are network problems or the Exchange server is down for maintenance.

This can be a huge problem during a migration. Microsoft described this in KB2834139. This error is caused by a misconfigured Outlook security setting, called “Logon network security”. If you experience this issue, check the “Logon network security” setting. If it’s set to “Anonymous Authentication”, then you experience the in KB2834139 described problem. Otherwise you have another problem. Check the “Logon network security” settings in your Outlook client. I took this screenshots from a Outlook 2013, but it looks the same in Outlook 2010. With this setting you will have a problem:

outlook_settings_login_security_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

With one of these settings, it will work.

outlook_settings_login_security_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

outlook_settings_login_security_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

You can change the setting and try the access to a public folder or shared mailbox. If you can access the public folder or shared mailbox, then you have to change some settings on the Exchange Server 2013 Client Access Server (CAS).

Open an Exchange Management Shell:

[PS] C:\windows\system32>Get-OutlookAnywhere -Server exchange2.testing.local | select *external*,*internal*

ExternalHostname : mail.terlisten-consulting.de
ExternalClientAuthenticationMethod : Negotiate
ExternalClientsRequireSsl : False
InternalHostname : exchange2.testing.local
InternalClientAuthenticationMethod : Negotiate
InternalClientsRequireSsl : False

[PS] C:\windows\system32>Get-OutlookAnywhere -Server exchange2.testing.local | Set-OutlookAnywhere -ExternalClientAuthenticationMethod NTLM
[PS] C:\windows\system32>Get-OutlookAnywhere -Server exchange2.testing.local | Set-OutlookAnywhere -InternalClientAuthenticationMethod NTLM

[PS] C:\windows\system32>Get-OutlookAnywhere -Server exchange2.testing.local | select *external*,*internal*

ExternalHostname : mail.terlisten-consulting.de
ExternalClientAuthenticationMethod : NTLM
ExternalClientsRequireSsl : False
InternalHostname : exchange2.testing.local
InternalClientAuthenticationMethod : NTLM
InternalClientsRequireSsl : False

I would recommend to execute “iisreset” after changing the settings. Please note, that this interrupts the client access for a short moment! After a restart of the Outlook client or during the next Autodiscover, the client should get the correct settings and the access to the public folders and shared mailboxes should work.

Importance of client-side proxy settings in Exchange 2013 environments

This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

There is an advantage, if you solves problems: You can learn something. I’m currently migrate a small Exchange 2007 environment to Exchange 2013. The first thing I learnt was, that IT staff still uses their own accounts for administration, and sometimes they assign administrator rights to users for testing and troubleshooting purposes. This can be a problem, as I described in my last posting. Today I learnt something different: Sometimes it’s the little things that bring you to despair.

After moving a mailbox from Exchange 2007 to 2013, Outlook must change the server for the client access. Nothing fancy and the user normally doesn’t notices it. If Outlook is online during the mailbox migration, the user gets a message, that he has to restart Outlook. Please note, that you need at least Outlook 2007 SP3, when you wish to migrate to Exchange 2013. This is because of an important change with Exchange 2013: The abolition of direct MAPI connections. Exchange 2013 only supports RPC-over-HTTP (aka Outlook Anywhere), even for LAN connections. RPC-over-HTTP has several advantages, e.g. the CAS role has to deal with only one protocol or easier load balancing and high-availability.

The problem

After moving a mailbox from Exchange 2007 to 2013, the Outlook 2010 client wasn’t able to connect to the Exchange server. The server was correctly changed, as I was able to see in the Outlook profile, but everytime I tried to start Outlook 2010, I got this error:

outlook_error

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The name cannot be resolved. The Connection to microsoft Exchange is unavailable. Outlook must be online or connected to complete this action.

Moving the mailbox back to the Exchange 2007 solved the problem. Moving the mailbox again to the Exchange 2013 resulted in the same error. Outlook Web Access was working fine (internal and external) on Exchange 2007 and 2013. The Microsoft Office Outlook Connectivity Test completed successful, after moving the mailbox to the Exchange 2013. Even Enterprise Active Sync (EAS) worked on the Exchange 2013 server. Only Outlook 2010 wasn’t able to connect to the mailbox, when it was on the Exchange 2013. The customer and I were puzzled… While testing this and that, we got an error while accessing Outlook Web Access. On another client, with another user OWA worked fine. BÄÄÄM! A possible cause popped in my mind.

The solution

I checked immediately  the proxy settings and there it was: A big proxy bypass list in the Internet Explorer with several entries, but the new Exchange server was missing. I added the server to the proxy bypass list and Outlook started without any problems. To be honest: It was a bit more complex, because the used proxy wasn’t an internal system. A solution provider operates it and the proxy settings were managed by a GPO, that wasn’t working correctly. In addition to that, an AD group membership was used to allow users to pass a web filter. But at the core it was the missing entry for the new Exchange server, that caused the problem.

The explanation

Exchange 2013 only supports RPC-over-HTTP and it uses the system-wide proxy settings. Therefore, HTTP(S) traffic is sent to the proxy server (regardless if the destination is internal or external), unless there is an entry in the proxy bypass list for the destionation (in this case the Exchange server). If the proxy can’t handle the traffic, Outlook will not be able to connect to the Exchange server. With MAPI, the proxy isn’t a problem, because MAPI traffic isn’t sent to the proxy. This explains, why Outlook was able to connect to the Exchange server, if the mailbox was moved back to the Exchange 2007. With Exchange 2007, Outlook uses MAPI for the connection. With Exchange 2013 RPC-over-HTTP is used.

So if you experience connection problems after moving a mailbox to Exchange 2013, check your proxy settings. This also applies when using Outlook Anywhere, because Outlook Anywhere uses also RPC-over-HTTP.