Tag Archives: migration

HP Comware: Forwarding subnet-directed broadcasts for Wake-on-LAN

Last week, my colleague Claudia and I have ported a HP ProVision configuration to HP Comware. Unexpectedly, it wasn’t routing or VLANs or OSPF that caused headaches, it was a Wake-on-LAN (WoL). Depending on the used tool, the magic packet (which wakes up the computer) is a broadcast (255.255.255.255) or a subnet-directed broadcast (e.g. 192.168.200.255). So it was important to know what tool the customer used.

This is how HP ProVision implements subnet-directed broadcasts:

The first two commands are issued globally. The “ip forward-protocol” statement has to be entered in the source VLAN (from which the magic packets are sent). The “ip forward-protocol” statement includes the protocol (udp), the destination (the broadcast address of the subnet) and the udp port (WoL uses port 9). Pretty simple, right? But please note that this config works only for sunet-directed broadcasts. And it only works for WoL. If you need another port (e.g. udp port 7), you have to add an additional “ip forward-protocol” statement.

If you use HP Comware, the configuration differs in some points. You can enable the reception of subnet-directed broadcasts by entering “ip forward-broadcast” globally in the system-view. This is mandatory.

The next step is to tell the switch, to which destination it should forward subnet-directed broadcasts. This can be done by entering “ip forward-protocol” in the vlan-interface context.

The downside: All subnet-directed broadcasts will be forwarded, regardless of source, destination or protocol. To avoid this, you have to create a ACL and add this to the “ip forward-broadcast” statement. To create an ACL enter:

You have to bound the ACL to the source VLAN interface. The subnet-directed broadcast will be forwarded to the VLAN interface which is directly connected to the destination subnet, or if there is no directly connected interface, to a router which knows the way to the destination. If you have applied packet filter ACLs to VLAN interfaces, make sure that your forwarded subnet-directed broadcasts aren’t filtered!

How to migrate from VMware vCOps to vROps – Part 3

I wrote about what’s new in vROps 6 and about the deployment of the virtual appliance. I also described how to migrate the data from the old vCOps vApp. Part 3 covers the decommission of the old vApp.

Enter the IP or FQDN of your UI VM into the browser. Login as admin into the administration UI.

remove_old_vcops_01

Before the vApp can be removed, the vCOps needs to be unregistered from the vCenter. Click “Unregister”.

remove_old_vcops_02

A confirmation pop-up appears. Click “Yes”.

remove_old_vcops_03

The process can take some time, depending on your environment. In my case the unregistration took about 5 minutes.

remove_old_vcops_04

That’s it. After the successful unregistration, the vApp can be shutdown and removed from the vCenter. Say good bye and enjoy your new vROps 6.0!

EDIT

Some users complained about the absence of the Health Widget in the vSphere Web Client, after the removal of the old vCOps. Michael White (@mwVme) posted the solution: vSphere Web Client Health State Widget has errors after vR Ops Migration? Thanks to Michael for sharing this!

How to migrate from VMware vCOps to vROps – Part 2

Part 1 of this series has covered a short overview over vRealize Operations Manager 6.0 and the initial deployment of the virtual appliance. Now it’s time to bring it to life.

Open a browser and enter the IP of your newly deployed vROps appliance. You will get this nice initial setup screen. “New Installation” is always a good start. Click “New Installation”.

migrate_vrops_01

Within five little steps, the configuration of vROps will be done. You may have noticed the “Migrate Data” icon on the right of the screenshot. This will be important later. Start with clicking “Next”.

migrate_vrops_02

Enter the password for the build-in administrator account and click “Next”.

migrate_vrops_03

I really liked part: Choose Certificate. No command line, no complex knowledge base articles. Simply choose “Install a certificate” and point the installer to a valid certificate. You can replace the certificate later. Because of this, I’ve chosen “Use the default certificate” at this point.

migrate_vrops_04

Enter the name of the master node and a valid NTP server. In my case, the NTP server is my Active Directory Domain Controller.

migrate_vrops_05

Complete the initial setup by clicking “Finish”.

migrate_vrops_06

Now the vROps services must be started. Click “Start vRealize Operations Manager”. The pop up can be answered with “Yes”.

migrate_vrops_07

Log in with “admin” and the password you’ve chosen at the beginning of the initial setup.

migrate_vrops_08

Because I already had a vCOps running in my lab, I’ve chosen “Import Environment”. This selection allows you to import the data from your current vCOps vApp. Click “Next”.

migrate_vrops_09

Carefully read the EULA, enable the checkbox and click “Next”.

migrate_vrops_10

Now you have to add a license key. I used my vExpert vCloud Suite key.

migrate_vrops_11

Enter the IP address or the FQDN of your current vCOps UI VM. A pop-up appears that informs you, that an agent is pushed to the VM. This step takes about 5 minutes to finish. Click “Next”.

migrate_vrops_12

You can add additional solutions at this point. This wasn’t necessary in my case. Click “Next”.

migrate_vrops_13

Everything’s fine. Simply click “Finish” and relax.

migrate_vrops_14

Now the data is imported. Depending on your resources and the amount of data, this step can take some time. In my case this step took about 10 minutes to finish.

migrate_vrops_15

Congratulations! That’s it. Easy, isn’t it? Now you can add additional nodes, e.g. a replica node or additional data nodes.

migrate_vrops_16

The migration process is really easy and straightforward. The database is the most valuable. Because of this, I strongly recommend to migrate the data from your current vCOps environment.

But what to do with the old vCOps vApp? It’s still active and consumes resources from your vSphere cluster. I covered this in part 3 of this series.

How to migrate from VMware vCOps to vROps – Part 1

VMware presented the vRealize Operations Manager 6.0 at the VMworld 2014 in Barcelona. In early december, vROps was available for download.

vROps 6.0 is the successor of VMwares IT Operations Management suite vCenter Operations Manager, or vCOps. VMware has aligned the naming scheme with other products, so this release is the first release under the new brand vRealize.

VMware has made some major improvements to this release. One of the biggest advantages is the ability to scale-out. In prior releases you had to deploy multiple vApps to scale. Now you can add additional vROps instances to a cluster. These appliances provide computing resources, as well as redundancy. This allows you to scale beyond the limits of vCOps 5. Redundancy is provided by a concept which is based on master, replica and data nodes. The fist node in a vROps deployment is the master node. By adding a replica node, you can add redundancy for the case that the master node fails. Master and replica node work in a active/ standby relationship. The data nodes are the secret behind the scalability of vROps. A data node has only one task to perform: Collect data based on the assigned adapter.

Abstract

An adapter is used to collect data from 3rd party systems and it’s provided by different available management packs. Check VMwares Whitepaper “VMware vRealize Operations Guide to Third-Party Solutions” for more information about availble management packs. From my point of view the management packs (which provide the adapters) for HP 3PAR (HP StoreFront), HP OneView, Brocade SAN Analytics, Microsoft SCOM or SAP CCMS are really cool. Just to make this clear: vRealize Operations Manager 6.0 is NOT focused on VMware! You can use vROps with available management packs to get a much better oversight over your IT infrastructure!

VMware has also improved the user interface (UI). There is no need anymore to switch between different UIs. The management, administration and customer UI has been consolidated into a single UI. The first UI access after the initial deployment is redirected to a first-time wizard. This wizard helps you to deploy vROps, or to migrate your vCOps environment. In addition to the UI changes, VMware has added RBAC to vROps to simplify user access management.

When talking about the UI, it’s only a small step to the data visualization and reporting. The visualization of collected data is one of the greatest features of vCOps/ vROps. Only data visualization makes it possible to get a quick oversight over the current health of the IT infrastructure. Prior version of vROps included reports and some fixed dashboards. With the current release, vROps provides fully configurable reports and dashboards with the ability to include any data source, object or metric. Just think about the opportunities: You can build reports or dashboards across the whole stack, from the infrastructure to the application.

Smart Alerts provide a way to trigger an alarm, when multiple symptoms are observed. This is much more flexible than the old alarms you know from vCOps. But even a dynamic threshold may be too inflexible in certain cases. Smart Alerts add more intelligence to the alarming system. When using smart alerts, an alert is triggert when multiple symptoms are observed. Now you know that something is wrong in your datacenter. But how can you solve this? vROps is able to execute basics operations as a reaction to a smart alert. If you need more advanced reactions, vROps can utilize vRealize Orchestrator to accomplish this. Your datacenter will heal itself.

The capacity management was improved. The well known “Demand & Allocation” based capacity model hasn’t changed, but it’s now not limited to vSphere objects. The capacity management can now include all monitored objects. VMware also improved the the “what-if”-analysis. This feature was now extended and is now aware of projects. With this feature, you can add future hardware purchases into a “what-if”-analysis.

VMware dramatically improved the analysis and monitoring of the storage subsystem. Unified Storage Visibility can show you the correlation between applications and your underlying storage infrastructure in an End-2-End manner.

The licensing was simplified. Customers can now install multiple editions in the same vROps deployment. This enables customers to deploy a-la-carte and suite licenses together, e.g. vCloud Suite Standard, vSOM Standard and vROps Standard together in one deployment. You can’t deploy Standard with either an Advanced or Enterprise, e.g. vCloud Site Standard and vSOM Advanced or vROps Advanced/ Enterprise.

The deployment process

You can deploy vROps using a virtual appliance or on top of RHEL or Windows using suitable installation packages. I would always prefer the appliance deployment. A big advantage is, that the new vROps appliance is a single appliance, not a vApp. Especially in environments without DRS deploying the vCOps vApp was pain. You can deploy vROps on every ESX/ESXi host running version 4.0 or later that is managed by a VMware vCenter Server 4.0 Update 2 or later.

I used the good old C# client to deploy the appliance. ;) Start with selecting “Deploy a OVF Template” from the vSphere client.

deploy_vrops_01

Simply click “Next” to get to the EULA.

deploy_vrops_02

Carefully read the EULA and click “Accept”. Then click “Next”.

deploy_vrops_03

Enter a new name for the vROps appliance or accept the default name. Select a location for the VM.

deploy_vrops_04

Select the configuration. If your environment consists of equal or less 2000 VMs, select “small”. In this case the appliance will configured with 4 vCPUs and 16 GB memory. This is equal to the resources of the old vCOps vApp construct.

deploy_vrops_05

Select the host or the cluster into the vROps appliance should be deployed. I selected my management host.

deploy_vrops_06

Select a resource pool if you have RPs configured.

deploy_vrops_07

I deploy my VMs always thin-provisioned in my lab. Select an appropriate disk format for your deployment.

deploy_vrops_08

Choose the port-group to which the appliance should be connected. In my case it’s my infrastructure port-group.

deploy_vrops_09

Now we need to add IP, subnet, gateway, DNS and time zone. Make sure that this applies to the port-group you have chosen one step earlier.

deploy_vrops_10

Double check your settings and click “finish”.

deploy_vrops_11

Depending on your equipment, wait a couple of seconds, get yourself a coffee or go out for a walk. Congratulations, it’s a vROps!

deploy_vrops_12

At this point is the initial deployment finished and we can proceed further. As you maybe noticed, there was already a vCOps deployed in my lab. At this point a vCOps 5.8.x and a vROps 6.0 were running in my lab. Time to migrate vCOps to vROps. This process is covered in part 2 of this series.

Active Directory property ‎’homeMDB‎’ is not writeable on recipient

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

How to enable permissions inheritance?

Start the Active Directory Users and Computers MMC. Click “View” and enable “Advanced Features”. Otherwise the needed tabs will not shown.

permissions_inheritance_1

Open the user that can’t be migrated. Switch to the “Security” tab. Click “Advanced” and enable the “Include inheritable permissions from this object’s parent” check box. Click “OK” and confirm popups regarding added ACL entries.

permissions_inheritance_2

Check if the Exchange Trusted Subsystem group is now part of the ACL. If yes, try the migration of the mailbox again.

Why is  permissions inheritance disabled?

Good question, the customer asked me the same question. But the answer is easy. As above mentioned, is a ACL with disabled permission inheritance called a protected ACL. When should a ACL be protected? If the ACL of the associated object shouldn’t be changed, when a permission from a parent object is inherited. A good example for such an object is the Administrator account. Active Directory protects members of the following groups

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

and the following users

  • Administrator
  • krbtgt

with three simple steps:

  1. Set the value of the AdminCount attribute to 1.
  2. Disable permission inheritance
  3. Apply permissions from the AdminSDHolder template

This process runs once an hour on a Active Directory Domain Controller and recovers the ACL of the two user accounts and the member accounts of the mentioned groups to a protected state and well-known permissions. What happens if you add an user account to the Domain Admins group? AdminCount is set to 1, the permission inheritance is disabled and the permissions specified in the AdminSDHolder template are applied. If you remove the user account from the Domain Admins group, the changes are not rolled back and so  permission inheritance stays disabled! Changes to parent ACLs are not applied to the user account and, in the case of my customer, the Exchange Trusted Subsystem group wasn’t added to the user objects ACL.

You can search for accounts with AdminCount=1 easily with the PowerShell:

Why is this a problem? With AdminCount=1 and disabled permission inheritance you will face a lot of problems:

  • Changes to an object ACLs are rolled back after an hour
  • Send-As doesn’t work
  • ActiveSync doesn’t work
  • Language settings in Outlook Web Access isn’t saved
  • ..and some other nasty effects

I have seen this error so often. Mostly at customers, that use normal work accounts for administrative tasks, or that add normal user accounts to admin groups during the setup of new profiles or clients. The solution is easy: Don’t do it!

Exchange 2013: Event ID 2937 MSExchange ADAccess after public folder migration

I got a couple of warnings (source MSExchange ADAccess, Event ID 2937) after removing a Exchange 2007 server at the end of a Exchange 2007 > 2013 migration. The details of the warning told me, that there was a faulty value set to a attribute of the mailbox database object. Because the public folder migration was part of the migration, the error message seemed plausible.

A quick check with ADSI Edit confirmed the message. To be honest: I made a mistake and searched for the attribute PublicFolderDatabase in the database object, but in the end I found the wrong entry as a value of the msExchHomePublicMDB attribute in the database object. It must be set to the distinguished name of the mailbox database that houses the public folder mailboxes. If you don’t have any public folders in your Exchange 2013 org, then you have to clear the value!

The solution

Start ADSI Edit (Start > Run > adsiedit.msc) and right click the “ADSI Edit”. Select “Connect to…”.

event_2937_3

Open the configuration context.

event_2937_4

Navigate to the database object, which is mentioned in the warning.

event_2937_2

Right click the database object and select “Properties” from the context menu. You can limit the number of attributes if you hit the “Filter” button and select “Show only attributed that have values”. Scroll down to the msExchHomePublicMDB attribute. Double click it and enter the complete CN of the mailbox database, that houses the public folder mailboxes (this is new with Exchange 2013!). If you have not public folders in your Exchange org, than you can clear the value!

event_2937_1

Click “OK” and restart the Exchange Information Store service. After the restart, no new 2937 events should be logged.

EDIT

If you get the message “Your Administrator has made a change and requires you to restart Outlook” (or in german “Der Microsoft Exchange Administrator hat eine Änderung durchgeführt, die einen Neustart von Outlook erfordert”), check if event id 2937 is logged on the Exchange servers. I had several migrations where a removed public folder DB caused Outlook to throw the error message.