Author Archives: Patrick Terlisten

About Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.Feel free to follow him on Twitter and/ or leave a comment.

Supported Active Directory environments for Microsoft Exchange

It is time for some words of wisdom, in regard to Exchange and the supported Active Directory environments. It is the same as with the supported. NET Framework releases: Latest release does not automatically mean “supported”.

To be honest: I nearly nuked a customer environment with ~ 300 users yesterday by preparing the domain for the first Windows Server 2019 Domain Controller.

First things first: Everything is fine! I did not prepared to forest schema for Windows Server 2019.

The support for Windows Server 2008 R2 comes to an end and some customers are still running it. Like my customer yesterday. Some application servers are still on 2008 R2… and the Domain Controllers. The customer is also running Exchange 2013 on Windows Server 2012 R2.

The customer has decided to go to Windows Server 2019 wherever possible. This includes file servers, application servers, and the Domain Controllers. On of the first steps was the deployment of Active Directory-Based Activation. The AD schema needs to be prepared for this and I decided to prepare the schema for Windows Server 2019. I already copied the adprep folder from the Server 2019 ISO and openened a PowerShell. And then I paused. Something felt odd. I wanted to take a look at the Exchange Server supportability matrix.

Exchange 2013 does NOT supported Windows Server 2019 Domain Controllers! Uhh… that was unexpected.

Lessons learned

Always check the Exchange Server supportability matrix. Always! Regardless if it’s because of .NET Framework, Active Directory, Outlook Clients etc. Just check it every time you plan to change something in your environment.

Especially in regard to Microsoft Exchange “newer” does not automatically mean “supported”. Most times the opposite is true.

Microsoft Exchange 2013/ 2016/ 2019 shows blank ECP & OWA after changes to SSL certificates

EDIT
This issue is described in KB2971270 and is fixed in Exchange 2013 CU6.

I published this blog post in July 2015 and it is still relevant. The feedback for this blog post was incredible, and I’m not joking when I say: I saved many admins weekends. ;) It has shown, that this error still occurs with Exchange 2016 and even 2019. Maybe not because of the same, with Exchange 2013 CU6 fixed bug, but maybe for other reasons. And the solution below still applies to it. Because of this I have decided to re-publish this blog post with a modified title and this little preamble.

Feel free to leave a comment if this blog post worked for you. :)

I ran a couple of times in this error. After applying changes to SSL certificates (add, replace or delete a SSL certificate) and rebooting the server, the event log is flooded with events from source “HttpEvent” and event id 15021. The message says:

If you try to access the Exchange Control Panel (ECP) or Outlook Web Access (OWA), you will get a blank website. To solve this issue, open up an elevated command prompt on your Exchange 2013 server.

Check the certificate hash and appliaction ID for 0.0.0.0:443, 0.0.0.0:444 and 127.0.0.1:443. You will notice, that the application ID for this three entries is the same, but the certificate hash for 0.0.0.0:444 differs from the other two entries. And that’s the point. Remove the certificate for 0.0.0.0:444.

Now add it again with the correct certificate hash and application ID.

That’s it. Reboot the Exchange server and everything should be up and running again.

What’s new in Vembu BDR Suite v4.0.1

Vembu Technologies was founded in 2002, and with 60.000 customers and more than 4000 partners, Vembu is a leading provider with a comprehensive portfolio of software products and cloud services to small and medium businesses.

In December 2018, Vembu announced the fourth major release of their BDR Suite. Vembu BDR Suite 4.0.1 is now out for production setups with enhanced performance and bug fixes. Vembu BDR Suite v4.0.1 is an intermediate patch update that addresses the customers reported issues and other support issues on the previous build of v4.0. Vembu BDR Suite v4.0.1 also features a large number of enhancements and significant of those are listed below.

Vembu Technologies/ Vembu BDR Essentials/ Copyright by Vembu Technologies

What’s new?

Beside of bug fixes, BDR Suite v4.0.1 also includes some new enhancements. In my opinion, the most significant enhancements are:

  • Significant performance improvement in Quick VM Recovery on VMware environments
  • Rescan option is introduced in Hyper-V Manager Servers page, which allows you to install Vembu Integration Service on the newly added node of the Hyper-V cluster (or if it’s not available on the existing node)
  • Backups configured through BDR Server console will run in parallel (Default parallel backup count is set to 5 and it is configurable)
  • Ability to add new Hyper-V hosts or choose existing hosts while performing Live Recovery to Hyper-V host

Interested in trying Vembu BDR suite? Try the 30-day free trial now! For any questions, simply send an e-mail to vembu-support@vembu.com or follow them on Twitter.

If you are a small or mid-sized businesses, check out the Vembu BDR Essentials package!

VCAP6.5-DCV Design – Objective 1.3 Determine risks, requirements, constraints, and assumptions

This blog post covers objective 1.3 (Determine risks, requirements, constraints, and assumptions) of the VCAP6.5-DCV Design exam. It is based on the VMware Certified Advanced Professional 6.5 in Data Center Virtualization Design (3V0-624) Exam Preparation Guide (last update August 2017).

The first objective of the exam prep guide has covered the business requirements. Now we have to do similar for the affected applications.

The necessary skills and abilities are documented in the exam prep guide for the older VCAP6-DCV Design exam (3V0-622). I think they also apply to the current version of the exam:

  • Differentiate between the concepts of risks, requirements, constraints, and assumptions
  • Given a statement, determine whether it is a risk, requirement, constraint, or an assumption
  • Analyze impact of VMware best practices to identified risks, constraints, and assumptions

Differentiate between the concepts of risks, requirements, constraints, and assumptions

I wrote a couple of times about risks, requirements, constraints and assumptions, but I missed to explain the meaning of each of these terms. I will use the following order:

  • requirements
  • risks
  • constraints
  • assumptions

So let us start with “What is a requirement“? A requirement is something that a has to be achieved. This can be applied to business or technical things. Without defined requirements, you would have no clue what your design should cover. If you define a requirement, you should test it with the following question: Is the defined requirement

  • specific
  • feasable
  • verifiable
  • traceable
  • unambiguous

When we talk about requirements, we have to differ between functional (WHAT) and non-functional (HOW) requirements. Some examples:

  • Solution must comply with ISO standards
  • The uptime must be at a minumum of 99,9%
  • Users must be able deploy new virtual machine within 15 minutes after approval

A Risks is a potential event, that might prevent us from achieving the defined project goals, or which can cause that the project completely fails. They are often common points in every projekct. The best we can do is to identify and list every risk that might prevent us from successfully finish the project. Some examples:

  • Missing the delivery date
  • Vendor discontinued parts of the solution
  • Hidden incompatibility with currently used frameworks

Constraints can be a limiting factor when we design our solution. They can be understood as cornerstones that set the borders of our solution. Contraints are always very specific. Examples:

  • The costs per user must not exceed 5 €
  • The project has to be finished withing 9 months
  • The solution must include servers from HPE

In opposite to constraints, which are very specific, assumptions are considered to be true without proof in the planning phase. This is pretty important! We are talking about the time, when we put our design together. Examples:

  • Rackspace will be available when the HW needs to be deployed
  • A MS SQL database server will be available at the installation date
  • A specific decision is made when needed

Summary

As I wrote at the beginning of this article: It is important to understand these terms. In simple words:

  • requirements: Things that have to met to successfully finish the project
  • risks: Things that might happen and that put our project at risk
  • constraints: Limiting factors to our project design
  • assumptions: Things that are considered to be true, but that are not proofed during the planning phase

That is a pretty simple summary, but it should be good enough to be memorized. :)

Links

NetScaler Gateway – Cannot complete your request

A customer reported a weird problem with his NetScaler Gateway. Upon the first load of the website, they got an error “Cannot complete your request”. After clicking OK the error disappeared and does not occured again after reloading the website. Only after closing and re-opening the browser. I got this message in Firefox and Internet Explorer, but not from a remote machine, e.g. my PC at the office.

I found no configuration error or something, that would have explained this message. Finally, I found something that caught my attention:

I found this using the Firefox Web Development Tools (I only had a Firefox and IE on my remote machine). With this message I found CTX244520 which also explained this error. The issue is caused by a hidden feature for caching web site data of the Gateway vServer. If you don’t have Integrated Cache feature licensed or enable, this feature failes. It is called Static Page Caching.

My customer is currently running NS12.0 60.10, and this issue is fixed in 12.0 61.8. And the customer is using a custom theme, which is based on one of the included themes.

If possible you can enable Integrated Caching. If you can’t enable Integrated Caching, you can simple disable this feature:

VCAP6.5-DCV Design – Objective 1.2 Gather and analyze application requirements

This blog post covers objective 1.2 (Gather and analyze application requirements) of the VCAP6.5-DCV Design exam. It is based on the VMware Certified Advanced Professional 6.5 in Data Center Virtualization Design (3V0-624) Exam Preparation Guide (last update August 2017).

The first objective of the exam prep guide has covered the business requirements. Now we have to do similar for the affected applications.

The necessary skills and abilities are documented in the exam prep guide for the older VCAP6-DCV Design exam (3V0-622). I think they also apply to the current version of the exam:

  • Gather and analyze application requirements for a given scenario
  • Determine the requirements for a set of applications that will be included in the design
  • Collect information needed in order to identify application dependencies
  • Given one or more application requirements, determine the impact of the requirements on the design

Gather and analyze application requirements for a given scenario

As a result of our already done work, we should know with what applications we have to deal in our project. Now ee have to gather the requirements of those applications. The necessary techniques are already known to us:

  • interviews with the relevant stakeholders and/ or developers or engineers
  • existing documentation about the deployment
  • our documented baseline from objective 1.1
  • vendor documentation/ support/ knowledge base articles

It is pretty important to understand what requirements these applications have. It depends on the workload and the applications itself. Tools like perfmon or capacity planning tools can help us to get a solid knowledge about the current and planned capacity/ performance requirements.

But we should not only focus on performance. There is much more to take into account, to be more specific: AMPRS

It stands for

  • Availability
  • Manageability
  • Performance
  • Recoverability, and
  • Security

You can read an detailed explanation here.

Determine the requirements for a set of applications that will be included in the design

This is similar to the written above. When we talk about a set of applications, we have to take the dependencies between these applications into account.

Collect information needed in order to identify application dependencies

To gain the necessary information, we have to talk to the right people, which means that we have to talk to developers, engineers and/ or end-users. We have to deep dive into existing customer and/ or vendor documentation. And we need to use the right tools to map the found dependencies. This can be done with Microsoft Visio, OmniGraffle or similar.

Given one or more application requirements, determine the impact of the requirements on the design

With the knowledge about the applications and the dependencies between them, it is time to make some design decisions. These decisions must support the documented requirements, especially when we think about the requirements in regard of availability, manageability, performance, recoverability, and security.

The key is to understand the impact of the made decisions for the rest of the design.

Summary

I will try to summarize this objective. The last blog post has covered the business requirements and the process from gathering the required information, over the documentation, until the point at which we can start creating a design. This blog post covers the same, but not for the business requirements, but for the applications and the requirements of these applications.

We can gather the necessary information by talking to the relevant stakeholders, engineers, developers etc. Customer and/ or vendor documentation and other sources can be used to get a better understanding of the different application requirements. We also need to understand the dependencies between the different applications, especially if only a subset of applications is virtualized. Our work is supported by different tools, especially for performance analysis, capacity planning and documentation.

With the gathered information we will able to make design decisions that fulfill the requirements (Think about AMPRS).

Links

VCAP6.5-DCV Design – Objective 1.1 Gather and analyze business requirements

This blog post covers objective 1.1 (Gather and analyze business requirements) of the VCAP6.5-DCV Design exam. It is based on the VMware Certified Advanced Professional 6.5 in Data Center Virtualization Design (3V0-624) Exam Preparation Guide (last update August 2017).

When you get the task to design something , you will instinctively start gathering information about the requirements that have to be fulfilled. Everything IT is doing should support the business in some way.

The necessary skills and abilities are documented in the exam prep guide for the older VCAP6-DCV Design exam (3V0-622). I think they also apply to the current version of the exam:

  • Associate a stakeholder with the information that needs to be collected
  • Utilize inventory and assessment data from a current environment to define a baseline state
  • Analyze customer interview data to explicitly define customer objectives for a conceptual design
  • Determine customer priorities for defined objectives
  • Ensure that Availability, Manageability, Performance, Recoverability and Security (AMPRS) considerations are applied during the requirements gathering process
  • Given results of the requirements gathering process, identify requirements for a conceptual design
  • Categorize requirements by infrastructure qualities to prepare for logical design requirements

Associate a stakeholder with the information that needs to be collected

Let’s start with the stakeholders and why they are important for us. But what is a stakeholder? A stakeholder is a person with an interest or concern in something, especially a business (Oxford). Stakeholders can be internal or external parties. An internal stakeholder is someone with a direct relationship to the company. An external stakeholder has no direct connection to the company, but it is affected in some way. This can be suppliers, the government, or other groups. A stakeholder can be anyone, but in our context stakeholders are typically

  • C-Level Executives (CEO, CFO, CIO etc.)
  • Vice Presidents
  • Managers, but also
  • Engineers and end users

As always: It depends. :)

Utilize inventory and assessment data from a current environment to define a baseline state

We also need to understand the current environment and what is currently deployed at the company. Interviews with the stakeholders are important, but in most cases they will not answer all questions. Depending on what is currently deployed, different tools can be used to gain the necessary data. Some examples:

  • RVTools, PowerCLI, vSphere Web Client, vROps etc
  • Custom scripts
  • Windows Server Manager
  • Network Monitoring Tools, like HPE Intelligent Management Center
  • Asset Management

It is important to document the results of the assessment. This is the baseline state of the current environment.

Analyze customer interview data to explicitly define customer objectives for a conceptual design

Now we need to get back to the results of the interviews that we did with the stakeholders to define the goals and the scope of the design. We also need to understant the the

  • Constraints
  • Assumptions,
  • Requirements, and
  • Risks

When we talk about requirements, we have to differ between functional (WHAT) and non-functional (HOW) requirements.

These information will allow us to create a conceptual design, which is written down in a workbook document.

Determine customer priorities for defined objectives

The next step is to define the priorities over the defined objectives. It is important to weight e.g. requirements and risks. Milestones have to be defined. They will help us to measure the success of the project and keep it on track.

Ensure that AMPRS considerations are applied during the requirements gathering process

AMPRS stands for

  • Availability
  • Manageability
  • Performance
  • Recoverability, and
  • Security

It is important to understand the meaning of each of these terms.

Availability considerations address the availability requirements of our design. These are typically expressed by percent uptime of a specific system. For example: 99,5% availability for file services.

Manageability considerations address the management and operational requirements of our design. This can be alerting, reports, access concepts etc.

Performance considerations express the required performance characteristics of the design. For example: Mails per second by a given size.

Recoverability considerations cover the ability to recover from an unexpected incident or disaster. This topic typically addresses backup and recovery of our design.

Security considerations cover the requirements around data control, access management, governance, risk management etc.

Given results of the requirements gathering process, identify requirements for a conceptual design

Now we have collected information from the relevant stakeholders, including the goals, scope, and CARR (constraints, assumptions, requirements, risks), and we have collected details about the current environment. Now it is time to put these information together and create a conceptual design.

The conceptual design must be approved by the stakeholders. This assures that everything is covered. Creating a conceptual design is an iterative process. The conceptual design is finished when the relevant stakeholders have approved it.

Categorize requirements by infrastructure qualities to prepare for logical design requirements

Sounds simple, but it can be challenging: The documented requirements have to be grouped by infrastructure categories, eg.

  • Networking
  • Storage
  • Recovery
  • Compute
  • VM
  • Security

Based on the CARR and the AMPRS considerations, we made design decisions. These decisions affect each of the infrastructure categories. At this point, we can review each of our decisions and mapping the requirements to the infrastructure will ease the creation of a high-level logical design.

Summary

Let me try to simplify this complex process a bit.

We were asked to solve a problem for a company. To solve this problem, we have to design a solution. To create this design, we have to identify the relevant stakeholders. These stakeholders will help us to gather information about the goals, the scope, about constraints, assumptions, requirements and risks. Especially when it comes to the requirements, we have to take availability, manageability, performance, recoverability and security considerations into account.

We can use different tools to collect information about the current environment.

At this point we know WHAT the company want, and we know WHAT they are currently running.

Now we can start with the creation of a conceptual design, which has to be approved by the relevant stakeholders.

To prepare the logical design, we need to map the documented requirements to the different categories of the infrastructure.

Links

VMware Certified Advanced Professional — Data Center Virtualization Design 2019 Study Guide

Last year in September I’ve passed the VCAP6-DCV Deployment exam. After a busy first half of 2019 it’s time to start preparing the VMware Certified Advanced Professional — Data Center Virtualization Design 2019 exam.

There are many great study guides out there, but in most cases I need “my own study guide” to feel well prepared. I hope the step to publish my notes helps me to stay focused and motivated.

Image by Pexels from Pixabay

In opposite to the Deploy exam, the Design exam is a MC exam. 135 Minutes for 60 questions. Sounds easy, but it’s told that it’s one of the hardest exams available by VMware.

The exam is split into three sections:

  • Section 1 – Create a vSphere 6.5 Conceptual Design
  • Section 2 – Create a vSphere 6.x Logical Design from an Existing Conceptual Design
  • Section 3 – Create a vSphere 6.x Physical Design from an Existing Logical Design

Each section contains several objects.

  • Objective 2.1 – Map business requirements to a vSphere 6.x logical design
  • Objective 2.2 – Map service dependencies
  • Objective 2.3 – Build availability requirements into a vSphere 6.x logical design
  • Objective 2.4 – Build manageability requirements into a vSphere 6.x logical design
  • Objective 2.5 – Build performance requirements into a vSphere 6.x logical design
  • Objective 2.6 – Build recoverability requirements into a vSphere 6.x logical design
  • Objective 2.7 – Build security requirements into a vSphere 6.x logical design
  • Objective 3.1 – Transition from a logical design to a vSphere 6.x physical design
  • Objective 3.2 – Create a vSphere 6.x physical network design from an existing logical design
  • Objective 3.3 – Create a vSphere 6.x physical storage design from an existing logical design
  • Objective 3.4 – Determine appropriate computer resources for a vSphere 6.x physical design
  • Objective 3.5 – Determine virtual machine configuration for a vSphere 6.x physical design
  • Objective 3.6 – Determine data center management options for a vSphere 6.x physical design

I will try to cover each objective in a blog post and add a link here. Feel free to add comments, corrections and questions. :)

Out of space – first steps when a datastore runs out of space

This is a situation that never should happen, and I had to deal with it only a couple of times in more than 10y working with VMware vSphere/ ESXi. In most cases, the reason for this was the usage of thin-provisioned disks together with small datastores. Yes, that’s a bad design. Yes, this should never happen.

There is a nearly 100% chance that this setup will fail one day. Either because someone dumps much data into the VMs, or because of VM snapshots. But such a setip WILL FAIL one day.

Yesterday was one of these days and five VMs have stopped working on a small ESXi in a site of one of my customers. A quick look into the vCenter confirmed my first assumption. The datastore was full. My second thought: Why are there so many VMs on that small ESXi host, and why they are thin-provisioned?

The vCenter showed me the following message on each VM:

There is no more space for virtual disk $VMNAME.vmdk. You might be able to continue this session by freeing disk space on the relevant volume, and clicking Retry. Click Cancel to terminate this session.

Okay, what to do? First things first:

  1. Is there any unallocated space left on the RAID group? If yes, expand the VMFS.
  2. Are there any VM snapshots left? If yes, remove them
  3. Configure 100% memory reservation for the VMs. This removes the VM memory swap files and releases a decent amout of disk space
  4. Remove ISO files from the datastore
  5. Remove VMs (if you have a backup and they are not necessary for the business)

This should allow you to continue the operation of the VMs. To solve the problem permanently:

  1. Add disks to the server and expand the VMFS, or create a new datastore
  2. Add a NFS datastore
  3. Remove unnecessary VMs
  4. Setup a working monitoring , setup alarms, do not overprovision datastores, or switch to eager-zeroed disks

Such an issues should not happen. It is not rude to say here: This is simply due to bad design and lack of operational processes.

User vdcs does not have the expected uid 1006

Sorry for the long delay since my last blog post – busy times, but with lots of vSphere. :) Today, I did an upgrade of a standalone vCenter Server Appliance at one of my healthcare customers. The vCenter was on 6.0 U3 and I had to upgrade it to 6.7 U2. It was only a small deployment with three hosts, so nothing fancy. And as with in many other vSphere upgrades, I came across this warning message:

Warning User vdcs does not have the expected uid 1006
Resolution Please refer to the corresponding KB article.

I saw this message multiple times, but in the past, there was no KB article about this, only a VMTN thread. And this thread mentioned, that you can safely ignore this message, if you don’t use a Content Library. Confirmation enough to proceed with the upgrade. :)

Meanwhile, there is a KB article:

Uploading content to the library fails with error: Content Library Service does not have write permission on this storage backing (52559)

This is a statement from the KB article:

Note: You can safely ignore this message if you are not using Content Library Service before the upgrade, or using it only for libraries not backed by NFS storage.

Currently, I don’t have cusomters with NFS backed Content Libraries, but if you do, you might want to take a look at it. Especially if you have done an upgrade from 6.0 to 6.5 or 6.7 and you want to start using Content Libraries now.