Tag Archives: server

Using WP fail2ban with the CloudFlare API to protect your website

The downside of using WordPress is that many people use it. That makes WordPress a perfect target for attacks. I have some trouble with attacks, and one of the consequences is, that my web server crashes under load. The easiest way to solve this issue would be to ban those IP addresses. I use Fail2ban to protect some other services. So the idea of using Fail2ban to ban IP addresses, that are used for attacks, was obvious.

From the Fail2ban wiki:

Fail2ban scans log files (e.g. /var/log/apache/error_log) and bans IPs that show the malicious signs — too many password failures, seeking for exploits, etc. Generally Fail2Ban is then used to update firewall rules to reject the IP addresses for a specified amount of time, although any arbitrary other action (e.g. sending an email) could also be configured. Out of the box Fail2Ban comes with filters for various services (apache, courier, ssh, etc).

That works for services, like IMAP, very good. Unfortunately, this does not work out of the box for WordPress. But adding the WordPress plugin WP fail2ban brings us closer to the solution. For performance and security reasons, vcloudnine.de can only be accessed through a content delivery network (CDN), in this case CloudFlare. Because CloudFlare acts as a reverse proxy, I can not see “the real” IP address. Furthermore, I can not log the IP addresses because of the German data protection law. This makes the Fail2ban and the WordPress Fail2ban plugin nearly useless, because all I would ban with iptables, would be the CloudFlare CND IP ranges. But CloudFlare offers a firewall service. CloudFlare would be the right place to block IP addresses.

So, how can I stick Fail2ban, the WP Fail2ban plugin and CloudFlares firewall service together?

APIs FTW!

APIs are the solution for nearly every problem. Like others, CloudFlare offers an API that can be used to automate tasks. In this case, I use the API to add entries to the CloudFlare firewall. Or honestly: Someone wrote a Fail2ban action that do this for me.

First of all, you have to install the WP Fail2ban plugin. That is easy. Simply install the plugin. Then copy the wordpress-hard.conf from the plugin directory to the filters.d directory of Fail2ban.

Then edit the /etc/fail2ban/jail.conf and add the necessary entries for WordPress.

Please note, that in my case, the plugin logs to /var/log/messages. The action is “cloudflare”. To allow Fail2ban to work with the CloudFlare API, you need the CloudFlare API Key. This key is uniqe for every CloudFlare account. You can get this key from you CloudFlare user profile. Go to the user settings and scroll down.

Cloudflare Global API Key

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the /etc/fail2ban/action.d/cloudflare.conf and scroll to the end of the file. Add the token and your CloudFlare login name (e-mail address) to the file.

Last step is to tell the WP Fail2ban plugin which IPs should be trusted. We have to add subnets of the CloudFlare CDN. Edit you wp-config.php and add this line at the end:

The reason for this can be found in the FAQ of the WP Fail2ban plugin. The IP ranges used by CloudFlare can be found at CloudFlare.

Does it work?

Seems so… This is an example from /var/log/messages.

And this is a screenshot from the CloudFlare firewall section.

Cloudflare Firewall Blocked Websites

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Another short test with curl has also worked. I will monitor the firewall section of CloudFlare. Let’s see who’s added next…

Important note for those, who use SELinux: Make sure that you install the policycoreutils-python package, and create a custom policy for Fail2Ban!

A strong indicator are errors like this in /var/log/messages:

You will find corresponding audit messages in the /var/log/audit.log:

Make sure that you create a custom policy for Fail2Ban, and that you load the policy.

HPE ProLiant PowerShell SDK

Some days ago, my colleague Claudia and I started to work on a new project: A greenfield deployment consisting of some well known building blocks: HPE ProLiant, HPE MSA, HPE Networking (now Aruba) and VMware vSphere. Nothing new for us, because we did this a couple times together. But this led us to the idea, to automate some tasks. Especially the configuration of the HPE ProLiants: Changing BIOS settings and configuring the iLO.

Do not automate what you have not fully understood

Some of the wisest words I have ever said to a customer. Modifying the BIOS and iLO settings is a well understood task. But if you have to deploy a bunch of ProLiants, this is a monotonous, and therefore error prone process. Perfect for automation!

Scripting Tools for Windows PowerShell

To support the automation of HPE ProLiant deployments, HPE offers the Scripting Tools for Windows PowerShell. HPE offers the PowerShell modules free for charge. There are three different downloads:

  • iLO cmdlets
  • BIOS cmdlets
  • Onboard Administrator (OA) cmdlets

The iLO cmdlets include PowerShell cmdlets to configure and manage iLO on HPE ProLiant G7, Gen8 or Gen9 servers. The BIOS cmdlets does not support G7 servers, so you can only configure and manage legacy and UEFI BIOS for Gen8 (except DL580) and all Gen9 models. The OA cmdlets support the configuration and management of the HPE Onboard Administrator, which is used with HPEs well known ProLiant BL blade servers. The OA cmdlets need at least  OA v3.11, whereby v4.60 is the latest version available.  All you need to get started are

  • Microsoft .NET Framework 4.5, and
  • Windows Management Framework 3.0 or later

If you are using Windows 8 or 10, you already have PowerShell 4 respectively PowerShell 5.

Support for HPE ProLiant Gen9 iLO RESTful API

If you have ever seen a HPE ProLiant Gen9 booting up, you might have noticed the iLO RESTful API icon down right. Depending on the server model, the BIOS cmdlets utilize the ILO4 RESTful API. But the iLO RESTful API ecosystem is it worth to be presented in an own blog post. Stay tuned.

Documentation and examples

HPE offers a simple documentation for the BIOS, iLO and OA cmdlets. You can find the documentation in HPEs Information Library. Documentation is important, but sometimes example code is necessary to quickly ramp up code. Check HPEs PowerShell SDK GitHub repository for examples.

Time to code

I’m keen on it and curious to automate some of my regular deployment tasks with these PowerShell modules. Some of these tasks are always the same:

  • change the power management and other BIOS settings
  • change the network settings of the iLO
  • change the initial password of the iLO administrator account and create additional iLO user accounts

Further automation tasks are not necessarily related to the HPE ProLiant PowerShell SDK, but to PowerShell, respectively VMware PowerCLI. PowerShell is great to automate the different aspects and modules of an infrastructure deployment. You can use it to build your own tool box.

How to monitor ESXi host hardware with SNMP

The Simple Network Management Protocol (SNMP) is a protocol for monitoring and configuration of network-attached devices. SNMP exposes data in the form of variables and values. These variables can then be queried or set. A query retrieves the value of a variable, a set operation assigns a value to a variable. The variables are organized in a hierarchy and each variable is identified by an object identifiers (OID). The management information base (MIB ) describes this hierarchy. MIB files (simple text files) contain metadata for each OID. These are necessary for the translation of a numeric OID into a human-readable format.  SNMP knows two devices types:

  • the managed device which runs the SNMP agent
  • the network management station (NMS) which runs the management software

The NMS queries the SNMP agent with GET requests. Configuration changes are made using SET requests. The SNMP agent can inform the NMS about state changes using a SNMP trap message. The easiest way for authentication is the SNMP community string.

SNMP is pretty handy and it’s still used, especially for monitoring and managing networking components. SNMP has the benefit, that it’s very lightweight. Monitoring a system with WBEM or using an API can cause slightly more load, compared to SNMP. Furthermore, SNMP is a internet-protocol standard. Nearly every device supports SNMP.

Monitoring host hardware with SNMP

Why should I monitor my ESXi host hardware with SNMP? The vCenter Server can trigger an alarm and most customers use applications like VMware vRealize Operations, Microsoft System Center Operations Manager, or HPE Systems Insight Manager (SIM). There are better ways to monitor the overall health of an ESXi host. But sometimes you want to get some stats about the network interfaces (throughput), or you have a script that should do something, if a NIC goes down or something else happens. Again, SNMP is very resource-friendly and widely supported.

Configure SNMP on ESXi

I focus on ESXi 5.1 and beyond. The ESXi host is called “the SNMP Agent”. We don’t configure traps or trap destinations. We just want to poll the SNMP agent using SNMP GET requests. The configuration is done using esxcli . First of all, we need to set a community string and enable SNMP.

That’s it! The necessary firewall ports and services are opened and started automatically.

Querying the SNMP agent

I use a CentOS VM to show you some queries. The Net-SNMP package contains the tools snmpwalk  and  snmpget. To install the Net-SNMP utils, simply use yum .

Download the VMware SNMP MIB files, extract the ZIP file, and copy the content to to /usr/share/snmp/mibs.

Now we can use snmpwalk  to “walk down the hierarchy “. This is only a small part of the complete output. The complete snmpwalk  output has more than 4000 lines!

Now we can search for interesting parts. If you want to monitor the link status of the NICs, try this:

As you can see, I used a subtree of the whole hierarchy (IF-MIB::ifDescr). This is the “translated” OID. To get the numeric OID, you have to add the option  -O fn to snmpwalk .

You can use snmptranslate  to translate an OID.

So far, we have only the description of the interfaces. With a little searching, we find the status of the interfaces (I stripped the output).

ifOperStatus.1  corresponds with ifDescr.1 , ifOperStatus.2  corresponds with ifDescr.2  and so on. The ifOperStatus corresponds  with the status of the NICs in the vSphere Web Client.

nic_status_web_client

If you want to monitor the fans or power supplies, use these these OIDs.

Many possibilities

SNMP offers a simple and lightweight way to monitor a managed device. It’s not a replacement for vCenter, vROps or SCOM. But it can be an addition, especially because SNMP is an internet-protocol standard.

HPE Hyper Converged 380 – A look under the hood

In March 2016, HPE CEO Meg Whitman announced a ProLiant-based HCI solution, that should be easier to use and cheaper than Nutanix.

This isn’t HPEs first dance on this floor. In August 2015, HP launched the Hyper Converged 250 System (HC250), which is based on the Apollo server platform. The HW design of the HC250 comes close to a Nutanix Block, because the Apollo platform supports up to four nodes in 2U. Let me say this clear: The Hyper Converged 380 (HC380) is not a replacement for the HC250! And before the HC250, HPE offered the Converged System 200-HC StoreVirtual and 200-HC EVO:RAIL (different models).

The HC380 is based on the ProLiant DL380 Gen9 platform. The DL380 Gen9 is one of the, if not the best selling x86 server on the market. Instead of developing everything from scratch, HPE build their new HC380 from different already available HPE products. With one exception: HPE OneView User Experience (UX). IT was developed from scratch and consolidates all management and monitoring tasks into a single console. The use of already available components was the reason for the low time-to-market (TTM) of the HC380.

Currently, the HC380 can only run VMware vSphere (HPE CloudSystem uses VMware vSphere). Support for Microsoft Hyper-V and Citrix XenServer will be added later. If you wish to run Microsoft Hyper-V, check the HC250 or wait until it’s supported with the HC380.

What flavor would you like?

The HC380 is available in three editions (use cases):

  • HC380 (Virtualization)
  • HC380 (HPE CloudSystem)
  • HC380 (VDI)

All three use cases are orderable using a single SKU and include two DL380 Gen9 nodes (2U). You can add up to 14 expansion nodes, so that you can have up to 16 dual-socket DL380 Gen9.

Each node comes with two Intel Xeon E5 CPUs. The exact CPU model has to be selected before ordering. The same applies to the memory (128 GB or 256 GB per node, up to 1,5 TB) and disk groups (up to three disk groups, each with 4,5 to 8 TB usable capacity per block, 8 drives either SSD/ HDD or all HDD with a maximum of 25 TB usable per node). The memory and disk group configuration depends on the specific use case (virtualization, CloudSystem, VDI). The same applies to the number of network ports (something between 8x 1 GbE and 6x 10 GbE plus 4x 1 GbE). For VDI, customers can add NVIDIA GRID K1, GRID K2 or Telsa M60 cards.

VMware vSphere 6 Enterprise or Enterprise Plus are pre-installed and licences can be bought from HPE. Interesting note from the QuickSpecs:

NOTE: HPE Hyper Converged 380 for VMware vSphere requires valid VMware vSphere Enterprise or higher, and vCenter licenses. VMware licenses can only be removed from the order if it is confirmed that the end-customer has a valid licenses in place (Enterprise License Agreement (ELA), vCloud Air Partner or unused Enterprise Purchasing Program tokens).

Hewlett Packard Enterprise supports VMware vSphere Enterprise, vSphere Enterprise Plus and Horizon on the HPE Hyper Converged 380.

No support for vSphere Standard or Essentials (Plus)! Let’s see how HPE will react on the fact, that VMware will phase out vSphere Enterprise licenses.

The server includes 3y/ 3y/ 3y onsite support with next business day response. Nevertheless, at least 3-year HPE Hyper Converged 380 solution support is requires according to the latest QuickSpecs.

What’s under the hood?

As I already mentioned, the HC380 was built from well known HPE products. Only HPE OneView User Experience (UX) was developed from scratch. OneView User Experience (UX) consolidates the following tasks into a single console (source QuickSpecs):

  • Virtual machine (VM) vending (create, edit, delete)
  • Hardware/driver and appliance UI frictionless updates
  • Advanced capacity and performance analytics (optional)
  • Backup and restore of appliance configuration details
  • Role-based access
  • Integration with existing LDAP or Active Directory
  • Physical and virtual hardware monitoring

Pretty cool fact: HPE OneView User Experience (UX) will be available for the HC250 later this year. Part of a 2-node cluster are not only the two DL380 Gen9 servers, but also three VMs:

  • HC380 Management VM
  • HC380 OneView VM
  • HC380 Management UI VM

The Management VM is used for VMware vCenter (local install) and HPE OneView for vCenter. You can use a remote vCenter (or a vCenter Server Appliance), but you have to make sure that the remote vCenter has HPE Oneview for vCenter integrated. The OneView VM running HPE OneView for for HW/ SW management. The Management UI VM is running HPE OneView User Experience.

The shared storage is provided by HPE StoreVirtual VSA. A VSA is running on each node. As you might know, StoreVirtual VSA comes with an all-inclusive license. No need to buy additional licenses. You can have it all: Snapshots, Remote Copy, Clustering, Thin Provisioning, Tiering etc. The StoreVirtual VSA delivers sustainable performance, a good VMware vSphere integration and added value, for example support for Veeam Storage Snapshots.

When dealing with a 2-node cluster, the 25 TB usable capacity per node means in fact 25 TB usable for the whole 2-node cluster. This is because of the Network RAID 1 between the two StoreVirtual VSA. The data is mirrored between the VSAs. When adding more nodes, the data is striped accross the nodes in the cluster (Network RAID 10+2).

Also important in case of the 2-node cluster: The quorum. At least two StoreVirtual VSA build a cluster. As in every cluster, you need some kind of quorum. StoreVirtual 12.5 added support for a NFSv3 based quorum witness. This is in fact a NFS file share, which has to be available for both nodes. This is only supported in 2-node clusters and I highly recommend to use this. I have a customer that uses a Raspberry Pi for this…

Start the engine

You have to meet some requirements before you can start.

  • 1 GbE connections for each nodes iLO and 1 GbE ports
  • 1 GbE or 10 GbE connections for each node FlexLOM ports
  • Windows-based computer directly connected to a node (MacOS X or Linux should also work)
  • VMware vSphere Enterprise or Enterprise Plus licenses
  • enough IP addresses and VLANs (depending on the use case)

For general purpose server virtualization, you need at least three subnets and three VLANs:

  • Management
  • vMotion
  • Storage (iSCSI)

Although you have the choice between a flat (untagged) and a VLAN-tagged network design, I would always recomment a VLAN-tagged approach. It’s highly recommended to use multiple VLANs to get the traffic seperated. The installation guide includes worksheets and examples to help you planning the deployment. For a 2-node cluster you need at least:

  • 5 IP addresses for the management network
  • 2 IP addresses for the vMotion network
  • 8 IP addresses for the iSCSI storage network

You should leave space for expansion nodes. A proper planning saves you later trouble.

HP OneView InstantOn is used for the automated deployment. It guides you through the necessary configuration steps. HPE says that the deployment requires less than 60 minutes and all you need to enter are

  • IP addresses
  • credentials
  • VMware licenses

After the deployment, you have to install the StoreVirtual VSA licenses. Then you can create datastores and, finally, VMs.

hpehc380_ux

HPE/ hpw.com

Summary

Hyper-Converged has nothing to do with the form factor. Despite the fact that a 2-node cluster comes in 4U, the HC380 has everything you would expect from a HCIA. The customers will decide if HPE held promise. The argument for the HC380 shouldn’t be the lower price compared to Nutanix or other HCI players. Especially, HPE should not repeat the mistake of the HC200 EVO:RAIL: To buggy and to expensive. The HC380 combines known and mature products (ProLiant DL380 Gen9, StoreVirtual VSA, OneView). It’s now up to HPE.

I have several small and mid-sized customers that are running two to six nodes VMware vSphere environments. Also the HC380 for VDI can be very interesting.

HP ProLiant BL460c Gen9: MicroSD card missing during ESXi 5.5 setup

Today, I was at a customer to prepare a two node vSphere cluster for some MS SQL server tests. Nothing fancy, just two HP ProLiant BL460c Gen9 blades and two virtual volumes from a HP 3PAR. Each blade had two 400 GB SSDs, two 64 GB M.2 SSDs and a 1 GB MicroSD card. Usually, I install ESXi to a SD card. In this case, a MicroSD card. The SSDs were dedicated for PernixData FVP. Although I saw the MicroSD card in the boot menu, ESXi doesn’t showed it as a installation target.

bl460_no_sdcard

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I’ve read a lot about similar observations of other users, but no solution seemed to be plausible. It’s not a solution to switch from UEFI to legacy BIOS or play with power management settings. BIOS and ILO firmware were up to date. But disabling USB3 support seemed to be a possible solution.

You can disable USB3 support in the RBSU.

bl460_rbsu

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After a reboot, the MicroSD card appeared as installation target during the ESXi setup.

bl460_with_sdcard

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I never noticed this behaviour with ProLiant DL360 Gen9, 380 Gen9, 560 Gen9 or 580 Gen9. I only saw this with BL460c Gen9. The affected blade servers had System ROM I36 05/06/2015 and I36 09/24/2015, as well as ILO4 2.30 installed.

Reset the HP iLO Administrator password with hponcfg on ESXi

Sometimes you need to reset the ILO Administrator password. Sure, you can reboot the server, press F8 and then reset the Administrator password. If you have installed a HP customized ESXi image, then there is a much better way to reset the password: HPONCFG.

Check the /opt/hp/tools directory. You will find a binary called hponcfg.

All you need is a simple XML file. You can use the VI editor or you can copy the necessary file with WinSCP to the root home directory on your ESXi host. I prefer VI. Change the directory to /opt/hp/tools. Then open the pwreset.xml.

Press i to switch to the insert mode. Then paste this content into the file. You don’t have to know the current password!

Press ESC and then :wq<ENTER> to save the file and leave the VI. Now use HPONCFG together with the XML file to reset the password.

That’s it! You can now login with “Administrator” and “password”.

HP Service Pack for ProLiant 2015.04

Some weeks ago, HP has published an updated version of their HP Service Pack for ProLiant (SPP). The SPP 2015.04.0 has added support for

  • new HP ProLiant servers and options,
  • support for Red Had Enterprise Linux 6.6, SUSE Linux Enterprise Server 12, VMware vSphere 5.5 U2 and (of course) VMware vSphere 6.0,
  • HP Smart Update Manager v7.2.0 was added,
  • the HP USB Key Utility for Windows v2.0.0.0 can now handle downloads greater than 4GB (important, because this release may not fit on a standard DVD media…)
  • select Linux firmware components is now available in rpm format

In addition, the SPP covers two important customer advisories:

  • ProLiant Gen9 Servers – SYSTEM ROM UPDATE REQUIRED to Prevent Memory Subsystem Anomalies on Servers With DDR4 Memory Installed Due to Intel Processor BIOS Upgrades (c04542689)
  • HP Virtual Connect (VC) – Some VC Flex-10/10D Modules for c-Class BladeSystem May Shut Down When Running VC Firmware Version 4.20 or 4.30 Due to an Erroneous High Temperature Reading (c04459474)

Two CAs fixed, but another CA arised (and it’s an ugly one..):

  • HP OneView 1.20 – Upgrading Virtual Connect Version 4.40 with Service Pack for ProLiant (SPP) 2015.04.0 Will Result in a Configuration Error and an Inability to Manage the HP Virtual Connect 8Gb 24-Port Fibre Channel Module (c04638459)

If you are using HP OneView >1.10 and 1.20.04, you will be unable to manage HP Virtual Connect 8Gb 24-port Fibre Channel Modules after updating the module to firmware version 3.00 or later. This is also the case, if you use the smart components from Virtual Connect  firmware version 4.40! After the update, the VC module will enter a “Configuration Error” state. Currently there is no fix. The only workaround is not to update to HP Virtual Connect 8Gb 24-port Fibre Channel Module firmware version 3.00. This will be fixed in a future HP OneView 1.20 patch.

Important to know: With this release, the SPP may not fit on standard DVD media! But to be honest: I’ve never burned the SPP to DVD, I always used USB media.

Check the release notes for more information about this SPP release. You can download the latest SPP version from the HP website. You need an active warranty or HP support agreement to download the SPP.

Is Nutanix the perfect fit for SMBs?

There’s a world below clouds and enterprise environments with thousands of VMs and hundered or thousands of hosts. A world that consists of maximal three hosts. I’m working with quite a few customers, that are using VMware vSphere Essentials Plus. Those environments consist typically of two or three hosts and something between 10 and 100 VMs. Just to mention it: I don’t have any VMware vSphere Essentials customer. I can’t see any benefit for buying these license. Most of these environments are designed for a lifeime of three to four years. After that time, I come again and replace it with new gear. I can’t remember any customer that upgraded his VMware vSphere Essentials Plus. Even if the demands to the IT infrastructure increases, the license stays the same. The hosts and storage gets bigger, but the requirements stays the same: HA, vMotion, sometimes vSphere Replication, often (vSphere API for) Data Protection. Maybe this is a german thing and customers outside of german are growing faster and invest more in their IT.

Hyperconverged, scale-out IT infrastructure for SMBs?

Think enterprise and break it down to smaller customers. That is said easily, but we saw so many technologies coming from the enterprise down to the SMBs over the last years. Think about SAN. 15 years ago, no SMB even thought about it. Today it’s standard.

I’ve taken this statement from the Nutanix webseite.

Nutanix simplifies datacenter infrastructure by integrating server and storage resources into a turnkey appliance that is deployed in just 30 to 60 minutes, and runs any application at any scale.

When working with SMBs, most of them have to deal with a tight budget. This means that they use the maximum principle, to get most hardware, software and service for their money. Customers do not like long implementation phases. Long implementation phases means, that lots of money can’t invested in hardware or software. Every single Euro/ Dollar/ $CURRENCY invested for service can’t be invested in hardware and software.

Another important requirement for the most SMBs is simple operation. I know a lot customers with only one, two or three people, that doing all that stuff around helpdesk, server, networking etc. IT infrastructure, or IT in general, isn’t the main focus for many of them. It should just work. Every day. Until it’s replaced.This applies not only to the area of server virtualization, it applies to IT in general. This often requires lean and simple designs, designs that follow the principle of error prevention. Because of this, it’s a good practice to reduce the components used in a design and automate where it’s useful and valuable. And if a solution is robust, then this can only be an advantage.

Why Nutanix?

In my opinion, simplicity is the key to sucess. If you see Nutanix for the first time, you will be surprised how easy it is to manage. Deployment, operation, updates. It’s slick, it’s simple, it’s lightweight. Everything the customer needs, is combined on 2U. The same applies to the support. I’ve followed the discussion on Twitter between Nutanix and VMware on who may/ can/ is allowed to provide support for VMware. It was started by a blog post of Chuck Hollis (10 Reasons why VMware is leading the hyperconverged industry). To make it short: I don’t share his opinion. In my opinion, Nutanix focus on customer experience is the key.

Simplicity and the ability to change

I don’t think that pre-configured systems like Fujitsu Cluster-in-a-boxVCE vBlocks or HP ConvergedSystems are the answer to simplified IT infrastructure for SMBs. They are not hyperconverged. They are pre-configured. That’s an important difference. Pre-configured doesn’t mean that it’s easy to manage or fast and easy to implement. SMBs want hyperconverged platforms to simplify their IT infrastructure. Okay, so why not buy any other offered hyperconverged platform on the market, like SimpliVity OmniCubeHP ConvergedSystems HC or VMware EVO:RAIL? Because these offerings are focused on VMware. The question was: Why Nutanix? Because you can run KVM, Microsoft Hyper-V and VMware ESXi on it. That’s an unique selling point (USP). You can offer the customer a hyperconverged platform, that allows him to change to another hypervisor later. I think we all agree that VMware is the market leader. But Microsoft is catching up. All features of the Essentials Plus kit can be delivered with Microsoft Hyper-V (and much more if you add SCVMM). Remeber: I talk about the typical Essentials Plus customer. VMware vSphere Essentials Plus includes all what a customer needs: Failover, live migration, data protection, and if needed, replication. In my experience, DRS, Host Profiles and vSphere Distributed Switches are nice, but SMBs can’t take advantage of it (exceptions are not excluded…). Add the Microsofts SCVMM and the gap between VMware vSphere and Microsoft Hyper-V is even smaller. The licensing of Microsoft Windows Server makes it interesting for customers to take a look at Microsoft Hyper-V, especially if you take the licensing costs into account. Sure, it’s not all about CAPEX (capital expenditure), OPEX (operational expenditures) is also important. Don’t get me wrong, I love VMware. But it’s important to be prepared. If the customer decides to change to Microsoft Hyper-V, you should be able to deliver it.

How can it look like?

Depending on the computing and storage needs, take a closer look at the Nutanix NX-1000 or NX-3000 series. I think a NX-1350 or NX-3350/ 3360 block is a good catch. Add a VMware vSphere Essentials Plus kit (or some Microsoft Windows Server 2012 R2 licenses… maybe also System Center 2012), Veeam Backup Essentials, something to store the backups on, like a HP StoreOnce 2700, and your favorite switches for 10 GbE networking connectivity (for example two HP 2920 switches in a stack with 10 GbE modules). A complete datacenter on 5U. This is only an example, but I think this should fit for most SMB customers (depending how you define SMB…).

Famous last words

Is Nutanix the perfect fit for SMBs? Yes! Easy to implement, easy to manage and robust. Nutanix stands out with its platform independence. This allows customers to have a choice in regard of the used hypervisor. Investment protection is a valuable asset, if you constantly have to fight for budgets.

Logon problems after demoting a branch office Domain Controller

A customer of mine is currently refreshing his branch office server infrastructure. A part of this project is to demote the Active Directory Domain Controllers, that are currently running in each branch office. The customer has multiple branch offices and each branch office has an Active Directory Domain Controller which is acting as file-/ print- and DHCP server. Each branch office has its own Active Directory site. The Domain Controller and the used IP subnets are assigned to the corresponding AD site. Due to this configuration the clients at the branch office choose the site-local Domain Controller as logon server. This works totally flawless since a couple of years. Over the year bandwidth of site connection has increased and even small branch offices have a redundant MPLS connection to the HQ. And no one likes single domain AD forests with 20 or more Domain Controllers…

j4mib

After demoting the Domain Controller in the first branch office I visited, my colleague and I discovered an interesting behaviour: The removal of the Domain Controller was flawless. Everything went fine. But when we tried to logon at a client, we got no GPOs and no network drives mapped. The name resolution was fine, so this was not our problem. I checked the content of the % LOGONSERVER% variable and discovered that it contained the name of the (now) demoted Domain Controller. After another logout and login, everything was fine. The client had chosen a new domain controller, now from the HQ AD site. This was correct and an expected behaviour. The branch office IP subnets were changed at the same time and the new IP subnets were assigned to the HQ AD site days before we demoted the DC.

Assumption: The client (Windows 7 Enterprise) used cached credentials to logon. These credentials included the old Domain Controller. During the second logon, a new Domain Controller is discovered based in the AD site.

To the lab!

Because I had to do the same in other branch offices, I searched for a solution. I used a couple of VMs to create a similar situation.

  • 2 sites
  • Each site has its own port group
  • 1 IP subnet per site
  • Router VM routes traffic between IP subnets
  • Subnets were assigned to AD sites
  • 1 DC per AD site
  • Client gets IP address from a DHCP in the site
  • Client is moved from one site to another site by switching port group
  • Client uses “last” logonserver (from the old site)
  • After the second logon, the site-local DC is chosen
  • nltest shows the correct DC for the site
  • If usage of cached credentials is disabled, client uses the site-local DC at every logon

I took some screenshots to clarify this. Logon as [email protected] on client2 in site 2.

client_ad_site_logon_server_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I powered off the VM, switched the port group and powered on the VM. I logged on as [email protected] on client2 in site 1. You can see, that the client still uses DC2 as logon server. The client got an IP from site 1.

client_ad_site_logon_server_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Logout and logon as [email protected] on client2 in site 1. Now the client uses DC als logon server.

client_ad_site_logon_server_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I tried this several times. The behaviour was always the same. Then I disabled cached credentials using a GPO (“Interactive logon: Number of previous logons to cache” set to 0). Now the client always chose the site-local DC on the first attempt.

Solution?

I don’t know if this is willed behavior. It’s reproducible and I don’t think that this is the result of a misconfiguration or a bug. If you demote a Domain Controller in a branch office, and it’s the only Domain Controller, the clients will try to reach it on the next logon. If the Domain Controller is still available, maybe because you moved it to another site, everything’s fine. But if it’s gone, you will get in trouble due to cached credentials.

In my case, the customer and I decided to assign all used IP subnets from the branch offices to the HQ AD site. Even if the branch offices still have a Domain Controller, the clients now chose the Domain Controller from the HQ. The Domain Controller in the branch offices now acts as only as DHCP and DNS until they are demoted.

Load Balancing inbound SMTP connection with HAProxy

In my last blog post I have highlighted how HAProxy can be used to distribute client connections to two or more servers with Exchange 2013 CAS role. But there is another common use case for load balancers in a Exchange environment: SMTP. Let’s take a look at this drawing:

mailflow

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The inbound SMTP connections are distributed to two Mail Transfer Agents (often a cluster of appliances, like Cisco IronPort or Symantec Messaging Gateway) and the MTAs forward the e-mails to the Exchange servers. Sometimes the e-mails are not directly forwarded to the Exchange servers, but to mail security appliances instead (like Zertificon Z1 SecureMail Gateway). After the e-mails have been processed by the mail security appliances, they are forwarded to the Exchange backend. Such setups are quite common. If a load balancer isn’t used, the MX records often point to the public IP address of a specific MTA. In this case, two or more MX records have to be set to ensure that e-mails can be received, even if a MTA fails.

A setup with a load balancers allows you to have a single MX record in your DNS, but two or more servers that can handle inbound SMTP connections. This makes maintenance easier und allows you to scale without having to fumble on the DNS. It’s without saying that your Load Balancer should be highly available, if you decide to realize such a setup.

It’s not hard to persuade HAProxy to distribute inbound SMTP connections. All you have to do is to add this to your haproxy.conf. To get the full config, check my last blog post about HAProxy.

The “send-proxy” parameter ensures, that the incoming IP address is forwarded to the servers behind the load balancer. This is important if you use Greylisting or real-time blacklists on your MTA or mail server. When running Postfix 2.10 or later, please make sure that you add this line to your main.cf:

This option add support for the PROXY protocol. Incoming requests are distributed alternating to the servers behind the load balancer. The “balance roundrobin” parameter ensures this. Please make sure that the MTA, that is running on your Linux host, doesn’t listen on the external IP. In my case, Postfix listens only on 127.0.0.1.

The statistics page can be used to verify the success of the configuration (click the picture to enlarge).

haproxy_smtp_roundrobin

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Alternatively you can use Telnet to connect to the load balancer on port 25/tcp. As you can see in the screenshot, using the FQDN mailin.vcloudlab.local resulted in an alternating connection to the backend servers.

haproxy_smtp_roundrobin_check_png

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0