Tag Archives: server

Logon problems after demoting a branch office Domain Controller

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer of mine is currently refreshing his branch office server infrastructure. A part of this project is to demote the Active Directory Domain Controllers, that are currently running in each branch office. The customer has multiple branch offices and each branch office has an Active Directory Domain Controller which is acting as file-/ print- and DHCP server. Each branch office has its own Active Directory site. The Domain Controller and the used IP subnets are assigned to the corresponding AD site. Due to this configuration the clients at the branch office choose the site-local Domain Controller as logon server. This works totally flawless since a couple of years. Over the year bandwidth of site connection has increased and even small branch offices have a redundant MPLS connection to the HQ. And no one likes single domain AD forests with 20 or more Domain Controllers…

After demoting the Domain Controller in the first branch office I visited, my colleague and I discovered an interesting behaviour: The removal of the Domain Controller was flawless. Everything went fine. But when we tried to logon at a client, we got no GPOs and no network drives mapped. The name resolution was fine, so this was not our problem. I checked the content of the % LOGONSERVER% variable and discovered that it contained the name of the (now) demoted Domain Controller. After another logout and login, everything was fine. The client had chosen a new domain controller, now from the HQ AD site. This was correct and an expected behaviour. The branch office IP subnets were changed at the same time and the new IP subnets were assigned to the HQ AD site days before we demoted the DC.

Assumption: The client (Windows 7 Enterprise) used cached credentials to logon. These credentials included the old Domain Controller. During the second logon, a new Domain Controller is discovered based in the AD site.

To the lab!

Because I had to do the same in other branch offices, I searched for a solution. I used a couple of VMs to create a similar situation.

2 sites
Each site has its own port group
1 IP subnet per site
Router VM routes traffic between IP subnets
Subnets were assigned to AD sites
1 DC per AD site
Client gets IP address from a DHCP in the site
Client is moved from one site to another site by switching port group
Client uses “last” logonserver (from the old site)
After the second logon, the site-local DC is chosen
nltest shows the correct DC for the site
If usage of cached credentials is disabled, client uses the site-local DC at every logon

I took some screenshots to clarify this. Logon as [email protected] on client2 in site 2.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I powered off the VM, switched the port group and powered on the VM. I logged on as [email protected] on client2 in site 1. You can see, that the client still uses DC2 as logon server. The client got an IP from site 1.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Logout and logon as [email protected] on client2 in site 1. Now the client uses DC als logon server.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I tried this several times. The behaviour was always the same. Then I disabled cached credentials using a GPO (“Interactive logon: Number of previous logons to cache” set to 0). Now the client always chose the site-local DC on the first attempt.

Solution?

I don’t know if this is willed behavior. It’s reproducible and I don’t think that this is the result of a misconfiguration or a bug. If you demote a Domain Controller in a branch office, and it’s the only Domain Controller, the clients will try to reach it on the next logon. If the Domain Controller is still available, maybe because you moved it to another site, everything’s fine. But if it’s gone, you will get in trouble due to cached credentials.

In my case, the customer and I decided to assign all used IP subnets from the branch offices to the HQ AD site. Even if the branch offices still have a Domain Controller, the clients now chose the Domain Controller from the HQ. The Domain Controller in the branch offices now acts as only as DHCP and DNS until they are demoted.

Load Balancing inbound SMTP connection with HAProxy

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In my last blog post I have highlighted how HAProxy can be used to distribute client connections to two or more servers with Exchange 2013 CAS role. But there is another common use case for load balancers in a Exchange environment: SMTP. Let’s take a look at this drawing:

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The inbound SMTP connections are distributed to two Mail Transfer Agents (often a cluster of appliances, like Cisco IronPort or Symantec Messaging Gateway) and the MTAs forward the e-mails to the Exchange servers. Sometimes the e-mails are not directly forwarded to the Exchange servers, but to mail security appliances instead (like Zertificon Z1 SecureMail Gateway). After the e-mails have been processed by the mail security appliances, they are forwarded to the Exchange backend. Such setups are quite common. If a load balancer isn’t used, the MX records often point to the public IP address of a specific MTA. In this case, two or more MX records have to be set to ensure that e-mails can be received, even if a MTA fails.

A setup with a load balancers allows you to have a single MX record in your DNS, but two or more servers that can handle inbound SMTP connections. This makes maintenance easier und allows you to scale without having to fumble on the DNS. It’s without saying that your Load Balancer should be highly available, if you decide to realize such a setup.

It’s not hard to persuade HAProxy to distribute inbound SMTP connections. All you have to do is to add this to your haproxy.conf. To get the full config, check my last blog post about HAProxy.

 mode tcp
    no option http-server-close
    balance roundrobin
    option smtpchk HELO mail.terlisten-consulting.de
    server mail1 192.168.200.107:25 send-proxy check
    server mail2 192.168.200.108:25 send-proxy check

The “send-proxy” parameter ensures, that the incoming IP address is forwarded to the servers behind the load balancer. This is important if you use Greylisting or real-time blacklists on your MTA or mail server. When running Postfix 2.10 or later, please make sure that you add this line to your main.cf:

smtpd_upstream_proxy_protocol = haproxy

This option add support for the PROXY protocol. Incoming requests are distributed alternating to the servers behind the load balancer. The “balance roundrobin” parameter ensures this. Please make sure that the MTA, that is running on your Linux host, doesn’t listen on the external IP. In my case, Postfix listens only on 127.0.0.1.

[[email protected] haproxy]# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 192.168.200.103:25      0.0.0.0:*               LISTEN      0          228433     22876/haproxy
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          15431      1309/master

The statistics page can be used to verify the success of the configuration (click the picture to enlarge).

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Alternatively you can use Telnet to connect to the load balancer on port 25/tcp. As you can see in the screenshot, using the FQDN mailin.vcloudlab.local resulted in an alternating connection to the backend servers.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

STOP c00002e2 after changing SCSI Controller to PVSCSI

This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I changed the SCSI controller type for my Windows VMs in my lab from LSI SAS to PVSCSI. Because the VMs were installed with LSI SAS, I used the procedure described in VMware KB1010398 (Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters) to change the SCSI controller type. The main problem is, that Windows doesn’t have a driver for the PVSCSI installed. You can force the installation of the driver using this procedure (taken from KB1010398):

Power off the virtual machine.
Create a new temporary 1GB disk(SCSI 1:0) and assign a new SCSI controller (default is LSI LOGIC SAS).
Change the new SCSI controller to PVSCSI for the new SCSI controller.
Click Change Type.
Click VMware Paravirtual and click OK.
Click OK to exit the Virtual Machine Properties dialog.
Power on the virtual machine.
Verify the new disk was found and is visible in Disk Management. This confirms the PVSCSI driver is now installed.
Power off the virtual machine.
Delete the temporary 1GB vmdk disk and associated controller (SCSI 1:0).
Change the original SCSI controller(SCSI 0:X) to PVSCSI as detailed in Steps 3 to 5.
Power on the virtual machine.

Please note, that this change is not a supported method to change the controller type. Usually you should install a server with disks already attached to a PVSCSI controller.

The problem

I changed the controller type for a couple of VMs using the above described method. This worked really fine until I changed the controller type of my Domain Controller. The DC failed to boot with the new controller. I changed the controller type back to LSI SAS and the VM started without problems. A change in the type of controller led to another BSoD (Blue Screen of Death). But it was not the usual STOP 0x0000007B.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

STOP: c00002e2 Directory Services could not start because of the following error:
A device attached to the system is not functioning.
Error Status: 0xc000000f.
Please shutdown this system and reboot into Directory Services Restore Mode, check the event log for more detailed information.

Something new… I booted into Directory Services Restore Mode (DSRM) and logged in with my directory services restore mode password (I hope you remember your password…).

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I checked the disks and found my SYSVOL volume offline.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I usually let DCPROMO install the NTDS and SYSVOL onto a separate drive instead using the system volume. In my case this drive was offline! This causes that Windows failed to start.

The solution

The solution was simple: Bring the volume online and reboot the VM.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Windows started without problems and the AD was also okay.

Final words

I think it’s clear, that you can’t replace the controller type without additional steps. And it should also be clear, that changing the controller type isn’t a good idea until you’re really know what you’re doing. I also think that it’s unsupported to change the controller type (but I haven’t found a statement if it’s supported or not). I did this a couple of times and I never had problem with the above described procedure. In this case, the problem is related to the controller type change. The cause for the BSoD was the offline NTDS/ SYSVOL volume. Bringing the volume online in DSRM solved the problem.

Power on HP ProLiant servers with iLO, SSH & Plink

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some weeks ago, Frank Denneman wrote a short blog post about accessing his Supermicro IPMI with SSH. He used this access to power on his lab servers.I don’t use Supermicro boards in my lab, but I have four HP ProLiants with iLO and iLO has a also a SSH interface. This way to power on my servers seemed very practical, especially because the iLO web interface isn’t the fastest. But I wanted it a bit more automated, so I decided to use Plink to send commands via SSH.

Create a new user account

I created a new user account in the iLO user database. This user has only the rights to change the power state of the server. Login into the iLO web interface. Click on “Administration”, then “User Administration” and “New”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Fill in the required fields. You have to enter a password, even if you later login with SSH public key authorization. Only allow “Virtual Power and Reset”. All other rights should be disallowed. Click “Save User Information”.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Create SSH key pair

I used the PuTTY Key Generator to create the necessary SSH key pair. Click “Generate” and move the mouse in the blank field.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Enter the username of the new created user in the “Key comment” field. Copy the public key into a textfile. You need this file for the key import into iLO. Then save the public and private key.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Key import

To import the key, login into the iLO web interface again. Click “Administration”, then “Security” in the “Settings” area on the left. Click “Browse…” and select the text file with the SSH public key. The key that is shown in the “Key” area of the PuTTY Key Generator differs from the saved public key. Both are public keys, but they have a different format. You have to import the key, that is shown in the “Key” area.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

If you have imported the right key, the key is automatically assigned to the new user.

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The test

Open a CMD and change to the directory with the Plink executable and the SSH private key. The following command turns the server on.

C:\Program Files (x86)\PuTTY>plink -i sshlogin.ppk sshlogin@ilo-esx1.lab.local "power on"

To turn off, simply use this command:

C:\Program Files (x86)\PuTTY>plink -i sshlogin.ppk sshlogin@ilo-esx1.lab.local "power off"

A warm reset can be requested by using this command:

C:\Program Files (x86)\PuTTY>plink -i sshlogin.ppk sshlogin@ilo-esx1.lab.local "power warm"

A cold reset can be requested by using this command:

C:\Program Files (x86)\PuTTY>plink -i sshlogin.ppk sshlogin@ilo-esx1.lab.local "power reset"

You can put these commands into a batchfile to power on/ off a couple of servers with a single click.

Homelabs: It’s evolution, baby

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A discourse is going on in the community. I can’t say who has started the discourse, but the number of blog postings to this discourse is an indication for the interest at this topic. But what’s the topic?

Homelabs

A homelab is the datacenter of the poor man. Some people have the luck to use a fully populated data center for test and study purposes. Our job requires to work with the latest technology and products, so we need an environment for test- and study purposes. Back in the days it was sufficient to have some VMs on you computer or laptop. But as virtualization moved into the data center, it was necessary to have this in the lab. At this point homelabs began to explode.

Why homelabs began to explode?

Lets assume that you wish to play with VMware vSphere. Playing with a single host is lame. So you need at least two hosts to build a cluster. If you want to use the cool features like HA, vMotion, DRS etc., you need shared Storage. Virtulization without VMs is also lame. So you need some CPU power and memory. Wow. At least two hosts and a shared storage. That escalated quickly… Okay, lets look at Microsoft Hyper-V. Mhh… at least two hosts and a shared storage if you seriously want to work with it. Now you have two options:

Physical Equipment
- real server HW
- Whitebox HW
Nested Enviroment

Physical HW has some benefits, because nothing is shared or virtualized. If it’s server HW, the chance is high that it’s on the HCL and you will not face issuses due to unsupported HW. But there are disadvantages: Think about space and power consumption, heat or the WAF (Wife Acceptance Factor – higher is better). Real server HW will violate requirements to space and power consumption, heat and WAF. You can go for whitebox HW, which means that you build your own server out of different components, that are not necessarily supported. But it’s cheap (if I look at Franks and Eriks homelabs this is not necessarily true…), you can focus on power consumption, noise and WAF. But what if you get in trouble because the HW is unsupported? What if HW currently works, but with the next release of VMware ESXi or Microsoft Hyper-V not? You can skip dedicated HW and go for nested environments. In this case you virtualize virtualization environments. Sounds spooky? Yes, sometimes it is. And it has some disadvantages, especially in case of performance or things that simply didn’t work (VMware FT with 64 bit guests). But it’s easy, and that is a big advantage. All you need is VMware Workstation, Fusion or ESXi and AutoLab. An awesome source for nested environments is virtuallyGhetto, William Lams blog.

The “scientific” discourse

There are some really nice blog posts came up in the last days. Take a look into the comments section!

Frank Denneman – vSphere 5.5 Home lab
Erik Bussink – The homelab shift…
Erik Bussink – Homelab 2014 upgrade
Vladan SEGET – vSphere Homelabs in 2014 – scale up or scale out?

It’s evolution, baby…

… and sometimes there are different, at the same time extending developments. Time will show which architecture will make the race. I chose server equipment, because due to some circumstances I came to four HP ProLiants. But I will not run them at home. ;)