Tag Archives: windows

Important foot note: Windows 10 Enterprise LTSB 2016 requires a new KMS host key

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today, I have stumbled upon a fact that is worth being documented.

TL;DR: Use the “Windows Srv 2016 DataCtr/Std KMS” host key (CSVLK), if you want to activate Windows 10 Enterprise LTSB 2016 using KMS. Or use AD-based activation. For more information read the blog post of the Ask the Core Team: Windows Server 2016 Volume Activation Tips.

A customer wants to deploy Windows 10 Enterprise LTSB 2016. A Windows Server 2012 R2 is acting as KMS host, and successfully activates Windows Server 2012 R2 and Microsoft Office 2013 Professional Plus. The “Windows Srv 2012R2 DataCtr/Std KMS for Windows 10” CSVLK was successfully installed. Nevertheless, the “current count” value does not increase. The client logged the event 12288:

On the server-side, I found the event 12290:

Error 0xC004F042 means:

The Software Protection Service determined that the specified Key Management Service (KMS) cannot be used.

It took a time to find the root cause, because I knew that Windows 10 Enterprise LTSB 2015 can successfully activated with this key. At the end it’s a question of the right search terms… Finally, I found the Ask the Core Team blog post. With the rightkey, the “current count” value increased, and finally the deployed Windows 10 Enterprise VMs were successfully activated.

Receive Connector role not selectable in Exchange 2016 CU2

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Another bug in Exchange 2016 CU2. The Role of a new receive connector is greyed out. You can select “Front-End-Transport”. This is a screenshot from a german Exchange 2016 CU2.

receive_connect_role_not_selectable

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Solution

Use the Exchange Management Shell to create a new receive connector. Afterwards, you can modify it with the Exchange Control Panel (ECP).

Microsoft has confirmed, that this is a bug in Exchange 2016 CU2.

Setting up split DNS using Windows DNS server

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes it’s necessary to have two DNS servers that are authoritative for the same DNS namespace. This is the case if you use the same namespace for your web site and your internal Active Directory domain, e.g. terlisten-consulting.de. Or that you have created the zone terlisten-consulting.de in your Windows DNS to point specific hosts to internal IP addresses. The DNS servers at your ISP would be authoritative, and the domain controllers of your Active Directory would also be authoritative for the same domain. The response to a query depends on which DNS server you ask. So what would happen if you try to resolve www.terlisten-consulting.de, and the internal DNS has no record for it?

In this case, the domain controller in my lab is authoritative for terlisten-consulting.de. But he doesn’t has a A record for www.terlisten-consulting.de. If I remove the zone from my domain controller, or if I use an external DNS server, I get a non-authoritative answer.

This, the same DNS namespace on different DNS server, is called “split DNS” (sometimes also called split-horizon DNS, split-view DNS or split-brain DNS).

Do it right

Split DNS is pretty handy, and sometimes it’s necessary. When it comes to Microsoft Exchange, it a common practice to use the same external DNS namespace for the internal and external URLs. This requires, that I create a zone for the externally used DNS namespace on my internal DNS (in most cases: Microsoft Windows Activice Directory domain controllers). The downside: I must create all DNS entries on my internal DNS, and I must point them to their external IP addresses, except the ones that should point to an internal IP.

FQDNInternal/ External IP address
www.terlisten-consulting.deexternal IP address
exchange.terlisten-consulting.deinternal IP address
shop.terlisten-consulting.deexternal IP address

Otherwise, users that use the domain controllers as DNS server, wouldn’t be able to resolve www or shop. This is challenging. But there’s a solution.

Create split DNS for single hosts

The Domain Name System is hierarchy organized. Because of this, I can tell my DNS server to be authoritative only for a sub-tree of a domain, e.g. exchange.terlisten-consulting.de. If I try to resolve www.terlisten-consulting.de, the DNS server would go down the hierarchy starting at the DNS root servers (or it would ask a forwarder). Instead of creating a zone for the whole namespace, create a zone for the host. Simply add

  • a new primary zone
  • don’t allow dynamic updates to the zone, and
  • create a new A or AAAA record for the host

Make sure

  • to leave the name field empty
  • don’t create a PTR record
  • point it to the internal IP of the host
single_host_zone

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

A simple nslookup will show if split DNS works as expected.

Works as expected. Make sure to clear the DNS server cache after you have added the zones.

Windows DNS Server Policies

Windows Server 2016 will introduce Windows DNS Server Policies. DNS Policies will allow you to control how a DNS Server handles answers to queries based on parameters like source IP address, IP address of the network interface that has received the query etc. In future, DNS Server Policies can be used to configure split DNS.

Microsoft Windows: Avoiding COM port proliferation

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This is not a specific problem of Alcatel-Lucent Enterprise (ALE) OmniSwitches, but I’m affected by this behaviour and it’s really, really annoying. It’s not a problem with the switch, but with the device handling of Windows.

ALE delivers a micro USB-to-USB cable with each OmniSwtich 6860E. This cable is used to connect to the console port of the switch. Each time you connect the cable, Windows will discover a new USB-to-UART bridge and creates a new COM port. This happens each time you connect to a new switch or if you choose another USB port. Over time, you will see the number of COM ports increasing (COM 2, COM 3, COM 4, COM 5…).

usb_com_ports_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Furthermore, you have to reconfigure your client software (PuTTY etc.) each time. This is annoying! But there’s a workaround.

Workaround

All you need is the product ID and the vendor ID. You can find these values in the properties of the device.

usb_com_ports_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

The VID for this device is 10C4, the PID EA60. Now create a reg file, which is used to import registry values.

You can see that the VID and PID was added to the string “IgnoreHWSerNum”. Import the reg file (double click or import it using RegEdit) and remove all unnecessary COM ports from the device manager.

Next time, Windows will not create a new COM port, as long as you use the same USB port. If you change the USB port, Windows will create an additional COM port.

WSUS on Windows 2012 (R2) and KB3159706 – WSUS console fails to connect

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As any other environment, my lab needs some maintenance from time to time. I use a Windows 2012 R2 VM with the Windows Server Update Service (WSUS) role to keep my Windows VMs up to date. Like many others, I was surprised by KB3148812 (Update enables ESD decryption provision in WSUS in Windows Server 2012 and Windows Server 2012 R2), which broke my WSUS. But the fix was easy: Uninstall KB3148812 and reboot the server. The WSUS product team published an artice about this known issue in their blog: Known Issues with KB3148812. In the meantime, Microsoft has published a new update, which supersedes KB3148812: KB3159706.

WSUS dead again?

Today I wanted to check the update status of my VMs. Unfortunately, the WSUS console was unable to connect to the WSUS server.

wsus_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I checked the status of the service and found the WSUS service stopped. But even after I had started the service, the WSUS console was unable to connect to the server. I found an error in the event logs (ID 507, source Windows Server Update Services), but the message “Update Services failed its initialization and stopped” wasn’t helpful. More helpful was a log entry:

After some searching and examination of the recently installed updates, I came across KB3159706.

Manual steps required to complete the installation of KB3159706

Open an elevated CMD and run this command:

The output should look similar to this:

Then you have to install the “HTTP Activation” feature under “.NET Framework 4.5” features.

wsus_3

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

After a restart of the WSUS service, the WSUS should work again.

Summary

The installation of KB3148812 on a WSUS server will break the WSUS installation. Because of this, Microsoft has published KB3159706. If you install this update (in my case it was installed automatically over WSUS…), you have to execute some manual steps to ensure that WSUS works as expected. The WSUS product team is aware of this and they pointed this out in their blog article “The long-term fix for KB3148812 issues” (you will find a hint directly at top of the blog article).

Windows recieves wrong DNS server from DHCP after DHCPINFORM

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week, I was surprisingly booked by a customer who observed a problem in his network. Unfortunately, colleagues worked on this network some day before (moving servers, routers etc. to a new pair of HP 7509 new core switches).

It was quickly clear, that some of the clients have received the wrong DNS servers from the DHCP server. The environment is a bit unusual. The customer is running two Active Directory domains (root and sub domain) in a single layer 2 broadcast domain. This nothing unusual, but he is also running two DHCP servers in the same layer 2 broadcast domain. To get this working, the customer uses exclusion ranges and reservations. This guarantees, that the client receives the correct DHCP information.

Observations

It was quicky clear, that some of the clients have received the wrong DNS servers from the DHCP server. That is the (defaced) output of a SUBDOM client with the correct IP settings.

And this is the output of the same client, after a reboot:

As you can see, the client got its DHCP information from the same DHCP server, but with the wrong DNS settings. The DNS servers are the servers from the ROOTDOM.

  • only clients from SUBDOM were affected
  • only Windows XP and Windows 7 clients were affected
  • Windows 8.1 and Windows 10 clients were not affected
  • Igel Thin Clients were not affected
  • after an “ipconfig /renew”, the correct DNS servers were registered
  • after an “ipconfig /release” and “ipconfig /renew”, the wrong DNS servers were registered
  • the same happened after a reboot
  • Wireshard packet trace showed, that the correct DNS information were included in the DHCPOFFER

The sum of observation told me, that this has nothing (or less) to do with the network changes. Interestingly, the correct DNS information were included in the DHCPOFFER and the behaviour was only observed on Windows XP and Windows 7. In addition, only clients of SUBDOM were affected.

The smoking gun

The packet trace with Wireshark showed, that the correct DNS information was included in the DHCPOFFER. But I also saw, that the client has sent a DHCPINFORM, which was answered by all available DHCP servers.

dhcp_inform

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

DHCPINFORM is used by a client to discover more information, e.g. router, proxy, static routes… or DNS. The DHCPINFORM request was only sent after a reboot, or after a DHCPRELEASE and a subsequent DHCPDISCOVER. I saw that all available DHCP servers answered the DHCPINFORM request with a DHCPACK. This DHCPACK included the requested information, including DNS. I quickly developed the hypothesis, that the DHCPACK from the ROOTDOM DHCP servers was used by the client, to add the wrong DNS information to the configuration.

With this information, I quickly found a references (MerakiLaurent Gaffié) to a registry key, that can be used to disable DHCPINFORM. This registry key is valid for Windows 2000, 2003, Windows XP, Windows Vista and Windows 7. Especially the blog post from Laurent Gaffié gave is interesting:

A vulnerability in Windows DHCP (http://www.ietf.org/rfc/rfc2131.txt) was found on Windows OS versions ranging from Windows 2000 through to Windows server 2003.  This vulnerability allows an attacker to remotely overwrite DNS, Gateway, IP Addresses, routing, WINS server, WPAD, and server configuration with no user interaction.

It’s useful to disable DHCPINFORM, even if you don’t have a problem!

Disable DHCPINFORM

To disable DHCPINFORM, you must add a registry key for the network interface, that shouldn’t sent DHCPINFORM messages.

Unfortunately the GUID of the interface differs between clients. I build this Visual Basic script (like Dr. Frankenstein: Different sources, plugged together, but it works) to add the registry key. You can run this script as part of a startup script with a Group Policy.

You should test this script very carefully! I provide this script “AS IS” with no warranties.

I don’t know why this has happened. I assume that the customer had this problem for some time. But due to some strange effects, he never noticed it. One hypothesis is, that the sequence of the DHCPACK messages after a DHCPINFORM has an influence. The DHCP server of ROOTDOM was moved to the new core switches, and maybe this changed the sequence of the answer packets. But it’s only a hypothesis, not a theory.

Using Microsoft certreq.exe to generate a certificate signing request (CSR)

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Generating a certificate signing request (CSR) is the first step towards a signed certificate. The requests is generated with the applicants private key and consists of the public key, a name and optional attributes.

To generate a CSR, you can use tools like OpenSSL on a Linux box, or sometimes the application itself can generate a CSR. But if you have a Windows box, you don’t have OpenSSL by default. And it’s unhandy to install something just for a single CSR. You can use certreq.exe to create a CSR. This tool is mostly unknown, but it’s included since Server 2000. The syntax slightly differs between the version, so I focus on the version that is shipped with Server 2008/ Windows Vista and newer.

To generate a CSR, you have to create a configuration file. This file specifies the key length, the common name, if the private key is exportable etc. This is a configuration file which includes additional names (subject alternative names, SAN).

This CSR includes three subject alternative names, which are listed below the [Extension] section. The syntax of this file is very important!

To create a CSR, open a CMD and change to the directory where the CSR is stored:

The csr-server1.req file can be used to create a CA signed certificate. The result is a signed certificate, based on the issued CSR. Very handy, especially in VMware Horizon View deployments in which you do not have access to a Windows-based Enterprise CA.

Exchange Management Shell (EMS) and new PowerShell releases

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some day ago, I installed a new Exchange 2013 CU11 for some test ins my lab. Nothing fancy, just a single server deployment on a Windows Server 2012 R2 VM. I deployed this Windows Server from a template, which was updated with the latest Windows Patches and WMF some days ago. The Exchange setup went smooth. I updated the SSL certificates and the internal and external URLs for the virtual directories. Then I started the Exchange Management Shell (EMS), to update the Autodiscover URL in the service connection point (SCP) of the Active Directory.

Well… that doesn’t look successful. I quickly switched to a PowerShell windows and imported the Exchange snap-in manually.

Looks better, isn’t it?

I compared my lab setup to a running Exchange 2013 single server deployment and I stumbled over the PowerShell version. In addition, I found the Windows Management Framework 5 Production Preview (KB3066437) on my freshly deployed Windows Server 2012 R2 VM.

After checking the Exchange Server Supportability Matrix, it was clear what had happened: WMF 5 is not supported (Source). Not supported with Exchange 2013, and also not supported with Exchange 2016.

exchange_supported_wmf

After I had removed KB3066437 from my Exchange server, the EMS loaded successfully.

You should ALWAYS check if installed applications are supported with newer version of PowerShell/ WMF! Currentyl, no Exchange version is supported with PowerShell 5/ WMF 5.

An attempt to restore the reputation of IPv6

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

IPv6 is not really new. According to Google, 10% of all users that access Google, do this over an IPv6 connection (Source). My blog is also accessible over IPv6 since its start in January 2014 (and since January 2016 only over HTTPS – thanks to Let’s Encrypt!).

When I talk with customers about IPv6, I often hear things like “Oh, we had to disable it. Too much problems!” or “We had to disable it. With IPv6 enabled, we had connectivity problems.”. Sometimes it went wrong. Especially in this cases, where IPv6 was only unbind from the network adapter. That’s the wrong way to “disable” IPv6.

Troublemaker IPv6

I have often heard that IPv6 causes connectivity problems. One of the most common examples is slow internet access. Often in conjuction with Windows Vista, Server 2008 or later, or with Linux (I don’t have a Mac and I don’t have many customers that use Macs a significant number). But every time, disabling IPv6 was the “solution”.

Another example: Domain-joined Windows Clients lose connection to the Active Directory, or can’t join an Active Directory domain. Disabling IPv6 was the “solution”.

This third example is particularly nice: Some of you will know this. It’s the default setting on Active Directory Domain Controllers since Server 2008:

domain_controller_ipv6

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

This setting results in an ugly “unknown” in nslookup.

Many admins change the setting to “Obtain DNS server address automatically”. Then they will start wondering, why the Active Directory domain has stopped working. Solution? Disabling IPv6.

IPv6 seems to be quite a waste, right? Sometimes it helps to understand why something has happened.

IPv6 Autoconfiguration

With IPv4, you need a DHCP server or static IP addresses. Okay, RFC3927 describes a third way (Dynamic Configuration of IPv4 Link-Local Addresses), but this was added long after the release of IPv4. IPv6 includes an autoconfiguration process. This process is specified in RFC4862 (IPv6 Stateless Address Autoconfiguration). This process is called SLAAC, StateLess Address AutoConfiguration. RFC4862 describes:

The autoconfiguration process includes generating a link-local address, generating global addresses via stateless address autoconfiguration, and the Duplicate Address Detection procedure to verify the uniqueness of the addresses on a link.

These IPv6 link-local addresses are assigned, or changed, whenever an interfaces is

  • initialized (for example on system startup),
  • an interface is enabled for IPv6,
  • an interfaces is re-initialized or,
  • an interface is attached to a new network

These IPv6 link-local addresses share a common IPv6 prefix (FE80::/10) and are only valid for the local network (layer 2 domain). They will not be routed. An IPv6 link-local address is dynamically generated and consists of the link-local prefix (FE80::/10) and an interface identifier. Latter is generated from the world-wide unique MAC address. This should ensure that a IPv6 link-local address is unique. Neverless, there is a Duplicate Address Detection (DAD). The relationship between link-local and MAC address is handy, but to protect your privacy, the interface identifier will include a random part (RFC4941 – Privacy Extensions for Stateless Address Autoconfiguration in IPv6).

Open a terminal/ command prompt and check your IP addresses. You will find a IPv6 IP starting with FE80 for each interface (if not, continue reading and re-enable IPv6…). This output is taken from my laptop running Windows 8.1 (german localization).

Notice the IPv6 link-local address in the output: fe80:0000:0000:0000:89d4:4eb0:3ff4:9172, or shorter fe80::89d4:4eb0:3ff4:9172. Check your clients, your servers, your switches and routers. Most of them will have an IPv6 link-local address.

Explore the neighborhood

With an IPv6 link-local address, your host has everything to explore the neighborhood. RFC4861 specifies the “Neighbor Discovery for IP version 6 (IPv6)”.

IPv6 nodes on the same link use Neighbor Discovery to discover each other’s presence, to determine each other’s link-layer addresses, to find routers, and to maintain reachability information about the paths to active neighbors.

Now multicast comes into play. The host will send a request to the multicast address ff02::2 (all routers in the local network). This is called a Router Solicitation. A router will respond to this request with a Router Advertisement. This advertisement include information about the MTU, lifetime and the prefix. The host can use this prefix to create its own local (fc00::/7) or global (2000::/3) unicast IPv6 address. This address also sets together from prefix (included in the Router Advertisement) and an interface identifier (randomly generated due to privacy extension or based on the MAC address of the physical interface). Router Advertisements are sent periodically (default 200 seconds) or as response to a Router Solicitation. Router Advertisements are only sent by routers. Because of this, the sender address of a Router Advertisement is used as default gateway by the host. If multiple routers are available, a in RFC4861 described algorithm will make the selection. If there are no routers available on the local network, there will be no response on a Router Solicitation and the default gateway will not be changed. A Router Advertisement doesn’t include information about DNS servers, domain name etc. The packet offers two flags, that can be used by the host to determine if there is a DHCPv6 server: The M-bit (managed address config flag) is set to 1 if a stateful DHCPv6 is available. The O-bit (other config flag) is set to 1 if the host should ask a stateless DHCPv6 server for DNS servers, domain name etc. If the M-bit is set to 1, the O-bit will be ignored. If the M-bit is set, the IPv6 unicast address is assigned by the stateful DHCPv6 server.

A stateful DHCPv6 server can assign IPv6 IP addresses. A statefless DHCPv6 can’t! Latter can only provide information about optional IP parameters like DNS server, domain name, SNTP server etc.

Neighbor Discovery can do so much more. There are five ICMPv6 packet types, and Router Solicitation and Advertisement are only two of them. The Neighbor Solicitation is used by hosts to determine the link-local address of a neighbor host. It’s also used to verify that a neighbor host is still reachable via a cached link-local address. Neighbor Advertisements are used in the same way as Router Advertisements: They are used to respond a Neighbor Solicitation. Redirect is used to inform hosts that there is a better first hop router for a specific destination. Some admins have the strong wish to block ICMP. Don’t even think about blocking ICMPv6. You will get into trouble.

Putting the pieces together

Imagine you have a network with multiple clients, servers, routers etc. Now you add a new router to your network. It’s a router for home users, but it was cheap and it’s only for a cable connection, that you want to use for downloads. You add the router to your network and the fun starts. Clients can’t connect to the domain network. Your domain controllers are going mad. Branch offices aren’t reachable any more. You disable IPv6 on clients and servers and everything is fine. You enable IPv6, you remove the new router, and everything is fine. But with your new router active and IPv6 enabled, everything is fucked up.

Let me explain what has happened. The new router has started to send Router Advertisements. Your clients and servers now have a new, additional default gateway. Since Microsoft Server 2008 and Windows Vista, IPv6 has precedence over IPv4. Maybe the new router acts as stateless DHCPv6 server. Your clients and servers are now using your ISPs DNS servers. It is logical that services, that relies on your corporate DNS, doesn’t work anymore. If internet is slow, this is mainly a DNS problem. Your ISP or the router treats AAAA DNS requests wrong.

Lessons learned

You should not disable IPv6. Microsoft doesn’t recommend this. It will break things.

We do not recommend that you disable IPv6 or its components, or some Windows components may not function.

If you have trouble with IPv6, check what is wrong and fix it. If a device sends RA or acting as a stateless DHCPv6, configure the device the right way. But stop blaming IPv6. It’s that easy. ;)

Using HP StoreOnce as target for Windows Server Backup (WSB)

This posting is ~3 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, I blogged about the new HP StoreOnce software release 3.13.0. This release included several fixes. One fix wasn’t mentioned by me, although it’s interesting.

  • Fixed issue where Windows 2012 R2 built-in native backup was not supported with 3.12.x software (BZ 61232)

Windows Server Backup (WSB) is part of Windows Server since Windows Server 2008. WSB can create bare metal backups and recover those backups. The same applies to system state backups, file level backups, Hyper-V VMs, Exchange etc. Very handy for small environmens. Backup can be stored on disk or on a file share. With Server 2012, the file share must be SMB3 capable. So if it’s not a Windows file server, the NAS that offers the file share has to be SMB3 capable. This doesn’t apply to Windows Server 2008 (R2).

With StoreOnce 3.13.0, HP has fixed this. Starting with 3.13.0, you can use a CIFS share on a StoreOnce appliance as a target for Windows Server Backup. This allows you to take advantage of the benefits of StoreOnce, like industry-leading deduplication and replication technology.

I was able to test this new feature with StoreOnce VSA appliances in my lab, as well as with a customers StoreOnce 4700 appliance.

Download you free copy of the HP StoreOnce Free 1 TB VSA today and give it a try!