Tag Archives: active directory

Why you should change your KRBTGT password prior disabling RC4

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While chilling on my couch, I stumbled over this pretty interesting Reddit thread: Story Time – How I blew up my company’s AD for 24 hours and fixed it : sysadmin (reddit.com)

Long story short: A poor guy applied some STIG hardening and his Active Directory blew up. Root cause was disabling RC4, which caused Kerberos failures, primarily documented by errors like “The encryption type requested is not supported by the KDC.” The guy fixed it by shutdown all domain controllers, changing the KRBTGT account password on one domain controller, and finally, everything came back

So why blew everything up after disabling RC4? Let’s travel back in time. Microsoft released Active Directory with the release of Windows 2000. At this time, Active Directory supported DES and RC4 to encrypt Kerberos tickets. With RFC 6649 (Deprecate DES, RC4-HMAC-EXP, and Other Weak Cryptographic Algorithms in Kerberos), DES was retired in July 2012, but Microsoft disabled DES with the release of Windows Server 2008 R2 and Windows 7. Disabling DES was no big deal, because Active Directory was designed to select the highest supported cipher for encrypting the Kerberos tickets. In addition to this, support for AES was added with Windows Server 2008.

If you deploy a new domain controllers, with a higher Windows OS version, and you remove all older domain controllers, you will be able to raise the Domain, and the Forest Functional Level (DFL/ FFL). The functional levels determine the available domain or forest capabilities. With the DFL of Windows 2008, Microsoft added AES support. But only if you raise the DFL vom Windows 2003 to 2008, or any higher DFL, the KRBTGT password will be changed to get it stored AES encrypted.

Let me make this clear: This only happens when raising the DFL vom 2003 to 2008 or any higher version. Not if you go from 2008 to 2012 R2, from 2012 R2 to 2016 etc. Only from 2003 to 2008 or higher. This should be done automatically and you can verify this by checking the PasswordLastSet of the user account:

PS C:\windows\system32> Get-ADUser "krbtgt" -Property Created, PasswordLastSet


Created           : 6/22/2005 2:48:12 PM
DistinguishedName : CN=krbtgt,CN=Users,DC=mlnetwork,DC=local
Enabled           : False
GivenName         :
Name              : krbtgt
ObjectClass       : user
ObjectGUID        : dfc8490d-374f-4570-944e-d5fa41d601ab
PasswordLastSet   : 3/3/2015 8:29:08 AM
SamAccountName    : krbtgt
SID               : S-1-5-21-3103332001-754687911-2831376874-502
Surname           :
UserPrincipalName :

As you can see, the user account was created on the 22. June 2005, which is creation date of this Active Directory domain. The PasswordLastSet dates back to the 3rd March 2015, possibly the date where the DFL was raised to 2008 or above. If you don’t see a suitable date in the PasswordLastSet attribute, you should change the KRBTGT password! It looks like that there are some domains out there where the KRBTGT password wasn’t changed during the DFL raise.

Domain Controller with Windows 2008 and later will always use AES for the Ticket Granting Ticket (TGT). Once your domain functional level (DFL) is 2008 or higher, the KRBTGT account will always default to AES. For any other accounts (user and computer) the selected encryption type is determined by the msDS-supportedEncryptionTypes attribute of the account.

And let me get this pretty clear: As long as you are running Windows Server 2000, 2003, or Windows XP, you can’t disable RC4, because these operating systems simply doesn’t support AES (Source)!

So prior disable RC4, or do any hardening regarding Kerberos, make sure that you change the KRBTGT password. And change it twice. Or use the KRBTGT Reset script. You should also confirm, that your TGTs are encrypted with AES. You can check this with klist tgt.  If the TGTs are still being issued with RC4, you should check the pwdLastSet attribute on the KRBTGT account.

What should I do?

To be honest: You should change the KRBTGT password regularly, e.g. every 180 days. This is possible sind Server 2008 (and DFL/ FFL Windows 2008). This blog post of Quest gives you a pretty good summary of why and how. You might also want to take a look at this Microsoft website, if you want to know more about the KRBTGT account. You don’t need to document the password. Simply change it. If you want a more controlled way, you can use this script: Microsoft KRBTGT Reset script.

Mail notification for specific Active Directory security events

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer used PRTG Network Monitor to notify him in case of account lockouts. This worked quite well until we implemented Admin Tiering. In order to get a mail notification in case of an account lockout, or other security-relevant events in Active Directory, I customized some scripts from my PowerShell dump.

The solution is pretty simple: I used the Task Planner to run a PowerShell script if a specific event id occurs. The events are generated in case of a various number of Active Directory events. You have to enable audit policy to get the needed events in the security event log. Take a look at Microsoft audit policy recommendations and enable what you need. I recommend to enable the stronger settings.

Image by Vitor Dutra Kaosnoff from Pixabay

I implemented five scripts:

  • Account lockout
  • New account in Active Directory
  • New member in domain-local group
  • New member in domain-global group
  • New member in universal group

The implementation is pretty easy. Create a basic task and execute this task if a specific event occurs.

The action is “Start a program”, like in the following screenshot.

Save the task and then change the user, which is used to run this task, to SYSTEM.

Please note, that you run these scripts with SYSTEM privileges. So make sure that NO ONE can easily edit these scripts! Best way is to restrict it to specific domain admins, restrict access to your domain controllers etc!

Repeat these steps for each script and implement them on each domain controller.

Please leave a comment with feedback :)

Veeam Backup & Replication: Backup of Microsoft Active Directory Domain Controller VMs

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

To backup a virtual machine, Veeam Backup & Replication needs two permissions:

  • permission to access and backup the VM, as well as the
  • permission to do specific tasks inside the VM

to guarantee a consistent backup. The former persmission is granted by the user account that is used to access the VMware vCenter server (sorry for the VMW focust at this point). Usually, this account has the Administrator role granted at the vCenter Server level. The latter permission is granted by a user account that has permissions inside the guest operating system.

geralt / pixabay.com/ Creative Commons CC0

Something I often see in customer environments is the usage of the Domain Administrator account. But why? Because everything works when this account is used!

There are two reasons for this:

  • This account is part of the local Administrator group on every server and client
  • customers tend to grant the Administrator role to the Domain Admins group on vCenter Server level

In simple words: Many customers use the same account to connect to the vCenter, and for the application-aware processing of Veeam Backup & Replication. At least for Windows servers backups.

Houston, we have a problem!

Everything is fine until customers have to secure their environments. One of the very first things customers do, is to protect the Administrator account. And at this point, things might go wrong.

Using a service account to connect to the vCenter server is easy. This can be any account from the Active Directory, or from the embedded VMware SSO domain. I tend to create a dedicated AD-based service account. For the necessary permissions in the vCenter, you can grant this account Administrator permissions, or you can create a new user role in the vCenter. Veeam offers a PDF document which documents the necessary permissions for the different Veeam tasks.

The next challenge is the application-aware processing. For Microsoft SQL Server, the user account must have the sysadmin privileges on the Microsoft SQL Server. For Microsoft Exchange, the user must be member of the local Administrator group. But in case of a Active Directory Domain Contoller things get complicated.

A Domain Controller does not have a local user database (SAM). So what user account or group membership is needed to backup a domain controller using application-aware processing?

This statement is from a great Veeam blog post:

Permissions: Administrative rights for target Active Directory. Account of an enterprise administrator or domain administrator.

So the service account used to backup a domain controller is one of the most powerful accounts in the active directory.

There is no other way. You need a Domain or Enterprise Administrator account. I tend to create a dedicated account for this task.

I recommend to create a service account to connect the vCenter, and which is added to the local Administrator group on the servers to backup, and I create a dedicated Domain/ Enterprise Administrator account to backup the virtual Domain Controllers.

The advantage is that I can change apply different fine-grained password policies to this accounts. Sure, you can add more security by creating more accounts for different servers, and applications, add a dedicated role to the vCenter for Veeam etc. But this apporach is easy enough to implement, and adds a significant amount of user account security to every environment that is still using DOMAIN\Administrator to backup their VMs.

NetScaler native OTP does not work for users with many group memberships

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, I have implemented one-time passwords (OTP) for NetScaler Gateway for one of my customers. This feature was added with NetScaler 12, and it’s a great way to secure NetScaler Gateway with a native NetScaler feature. Native OTP does not need any third party servers. But you need a NetScaler Enterprise license, because nFactor Authentication is a requirement.

To setup NetScaler native OTP, I followed the availbe guides on the internet.

The setup is pretty straightforward. But I used the AD extensionAttribute15  instead of userParameters, because my customer already used userParameters  for something else. Because of this, I had to change the search filter from userParameters>=#@  to extensionAttribute15>=#@ .

Everything worked as expected… except for some users, that could not register their devices properly. They were able to register their device, but a test of the OTP failed. After logoff and logon, the registered device were not available anymore. But the device was added to the extensionAttribute. While I was watching the nsvpn.log with tail -f , I discovered that the built group string for $USERNAME  seemed to be cut off (receive_ldap_user_search_event). My first guess was, that the user has too many group memberships, and indeed, the users was member for > 50 groups. So I copied the user, and the copied user had the same problem. I removed the copied user from some groups, and at some point the test of the OTP worked (on the /manageotp website).

With this information, I quickly stumbled over this thread: netscaler OTP not woring for certain users. This was EXACTLY what I discovered. The advised solution was to change the “Group Attribute” from memberOf  to userParameter , or in my case, extensionAttribute15. Problem solved!

Logon problems after demoting a branch office Domain Controller

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer of mine is currently refreshing his branch office server infrastructure. A part of this project is to demote the Active Directory Domain Controllers, that are currently running in each branch office. The customer has multiple branch offices and each branch office has an Active Directory Domain Controller which is acting as file-/ print- and DHCP server. Each branch office has its own Active Directory site. The Domain Controller and the used IP subnets are assigned to the corresponding AD site. Due to this configuration the clients at the branch office choose the site-local Domain Controller as logon server. This works totally flawless since a couple of years. Over the year bandwidth of site connection has increased and even small branch offices have a redundant MPLS connection to the HQ. And no one likes single domain AD forests with 20 or more Domain Controllers…

j4mib

After demoting the Domain Controller in the first branch office I visited, my colleague and I discovered an interesting behaviour: The removal of the Domain Controller was flawless. Everything went fine. But when we tried to logon at a client, we got no GPOs and no network drives mapped. The name resolution was fine, so this was not our problem. I checked the content of the % LOGONSERVER% variable and discovered that it contained the name of the (now) demoted Domain Controller. After another logout and login, everything was fine. The client had chosen a new domain controller, now from the HQ AD site. This was correct and an expected behaviour. The branch office IP subnets were changed at the same time and the new IP subnets were assigned to the HQ AD site days before we demoted the DC.

Assumption: The client (Windows 7 Enterprise) used cached credentials to logon. These credentials included the old Domain Controller. During the second logon, a new Domain Controller is discovered based in the AD site.

To the lab!

Because I had to do the same in other branch offices, I searched for a solution. I used a couple of VMs to create a similar situation.

  • 2 sites
  • Each site has its own port group
  • 1 IP subnet per site
  • Router VM routes traffic between IP subnets
  • Subnets were assigned to AD sites
  • 1 DC per AD site
  • Client gets IP address from a DHCP in the site
  • Client is moved from one site to another site by switching port group
  • Client uses “last” logonserver (from the old site)
  • After the second logon, the site-local DC is chosen
  • nltest shows the correct DC for the site
  • If usage of cached credentials is disabled, client uses the site-local DC at every logon

I took some screenshots to clarify this. Logon as Administrator@vcloudlab on client2 in site 2.

client_ad_site_logon_server_01

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I powered off the VM, switched the port group and powered on the VM. I logged on as Administrator@vcloudlab on client2 in site 1. You can see, that the client still uses DC2 as logon server. The client got an IP from site 1.

client_ad_site_logon_server_02

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Logout and logon as Administrator@vcloudlab on client2 in site 1. Now the client uses DC als logon server.

client_ad_site_logon_server_03

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I tried this several times. The behaviour was always the same. Then I disabled cached credentials using a GPO (“Interactive logon: Number of previous logons to cache” set to 0). Now the client always chose the site-local DC on the first attempt.

Solution?

I don’t know if this is willed behavior. It’s reproducible and I don’t think that this is the result of a misconfiguration or a bug. If you demote a Domain Controller in a branch office, and it’s the only Domain Controller, the clients will try to reach it on the next logon. If the Domain Controller is still available, maybe because you moved it to another site, everything’s fine. But if it’s gone, you will get in trouble due to cached credentials.

In my case, the customer and I decided to assign all used IP subnets from the branch offices to the HQ AD site. Even if the branch offices still have a Domain Controller, the clients now chose the Domain Controller from the HQ. The Domain Controller in the branch offices now acts as only as DHCP and DNS until they are demoted.

STOP c00002e2 after changing SCSI Controller to PVSCSI

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I changed the SCSI controller type for my Windows VMs in my lab from LSI SAS to PVSCSI. Because the VMs were installed with LSI SAS, I used the procedure described in VMware KB1010398 (Configuring disks to use VMware Paravirtual SCSI (PVSCSI) adapters) to change the SCSI controller type. The main problem is, that Windows doesn’t have a driver for the PVSCSI installed. You can force the installation of the driver using this procedure (taken from KB1010398):

  1. Power off the virtual machine.
  2. Create a new temporary 1GB disk(SCSI 1:0) and assign a new SCSI controller (default is LSI LOGIC SAS).
  3. Change the new SCSI controller to PVSCSI for the new SCSI controller.
  4. Click Change Type.
  5. Click VMware Paravirtual and click OK.
  6. Click OK to exit the Virtual Machine Properties dialog.
  7. Power on the virtual machine.
  8. Verify the new disk was found and is visible in Disk Management. This confirms the PVSCSI driver is now installed.
  9. Power off the virtual machine.
  10. Delete the temporary 1GB vmdk disk and associated controller (SCSI 1:0).
  11. Change the original SCSI controller(SCSI 0:X) to PVSCSI as detailed in Steps 3 to 5.
  12. Power on the virtual machine.

Please note, that this change is not a supported method to change the controller type. Usually you should install a server with disks already attached to a PVSCSI controller.

The problem

I changed the controller type for a couple of VMs using the above described method. This worked really fine until I changed the controller type of my Domain Controller. The DC failed to boot with the new controller. I changed the controller type back to LSI SAS and the VM started without problems. A change in the type of controller led to another BSoD (Blue Screen of Death). But it was not the usual STOP 0x0000007B.

bsod_after_pvscsi_change

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

STOP: c00002e2 Directory Services could not start because of the following error:

A device attached to the system is not functioning.

Error Status: 0xc000000f.
Please shutdown this system and reboot into Directory Services Restore Mode, check the event log for more detailed information.

Something new… I booted into Directory Services Restore Mode (DSRM) and logged in with my directory services restore mode password (I hope you remember your password…).

boot_into_dsrm

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I checked the disks and found my SYSVOL volume offline.

offline_sysvol_disk

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

I usually let DCPROMO install the NTDS and SYSVOL onto a separate drive instead using the system volume. In my case this drive was offline! This causes that Windows failed to start.

The solution

The solution was simple: Bring the volume online and reboot the VM.

online_sysvol

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Windows started without problems and the AD was also okay.

Final words

I think it’s clear, that you can’t replace the controller type without additional steps. And it should also be clear, that changing the controller type isn’t a good idea until you’re really know what you’re doing. I also think that it’s unsupported to change the controller type (but I haven’t found a statement if it’s supported or not). I did this a couple of times and I never had problem with the above described procedure. In this case, the problem is related to the controller type change. The cause for the BSoD was the offline NTDS/ SYSVOL volume. Bringing the volume online in DSRM solved the problem.

Active Directory property ‎’homeMDB‎’ is not writeable on recipient

This posting is ~9 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

During an Exchange 2013 migration project the  first attempt to migrate a mailbox failed with the following error:

Error: MigrationPermanentException: Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'. --> Active Directory property 'homeMDB' is not writeable on recipient 'testing.local/Users/Dummy'.

The error message clearly stated, that this was a permission issue. A quick search pointed me to the right direction. I found a thread in the TechNet forums, in which the same error message were discussed. This error occurs, if the Exchange Trusted Subsystem group is missing in the ACL of the user object. This group contains the exchange server and it’s usually inherited from the domain object to all child containers and objects. I checked the ACL of the user and the Exchange Trusted Subsystem group was missing in the ACL. This was caused by the disabled permissions inheritance. An object ACL with disabled permissions inheritance is sometimes called a protected ACL. Bill Long wrote a nice Power Shell script to search for object which have permissions inheritance disabled.

How to enable permissions inheritance?

Start the Active Directory Users and Computers MMC. Click “View” and enable “Advanced Features”. Otherwise the needed tabs will not shown.

permissions_inheritance_1

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Open the user that can’t be migrated. Switch to the “Security” tab. Click “Advanced” and enable the “Include inheritable permissions from this object’s parent” check box. Click “OK” and confirm popups regarding added ACL entries.

permissions_inheritance_2

Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0

Check if the Exchange Trusted Subsystem group is now part of the ACL. If yes, try the migration of the mailbox again.

Why is  permissions inheritance disabled?

Good question, the customer asked me the same question. But the answer is easy. As above mentioned, is a ACL with disabled permission inheritance called a protected ACL. When should a ACL be protected? If the ACL of the associated object shouldn’t be changed, when a permission from a parent object is inherited. A good example for such an object is the Administrator account. Active Directory protects members of the following groups

  • Enterprise Admins
  • Schema Admins
  • Domain Admins
  • Administrators
  • Account Operators
  • Server Operators
  • Print Operators
  • Backup Operators
  • Cert Publishers

and the following users

  • Administrator
  • krbtgt

with three simple steps:

  1. Set the value of the AdminCount attribute to 1.
  2. Disable permission inheritance
  3. Apply permissions from the AdminSDHolder template

This process runs once an hour on a Active Directory Domain Controller and recovers the ACL of the two user accounts and the member accounts of the mentioned groups to a protected state and well-known permissions. What happens if you add an user account to the Domain Admins group? AdminCount is set to 1, the permission inheritance is disabled and the permissions specified in the AdminSDHolder template are applied. If you remove the user account from the Domain Admins group, the changes are not rolled back and so  permission inheritance stays disabled! Changes to parent ACLs are not applied to the user account and, in the case of my customer, the Exchange Trusted Subsystem group wasn’t added to the user objects ACL.

You can search for accounts with AdminCount=1 easily with the PowerShell:

PS C:\Users\Administrator> ([adsisearcher]"(AdminCount=1)").findall()

Path                                                        Properties
----                                                        ----------
LDAP://CN=Administrator,CN=Users,DC=testing,DC=local        {msexchrecipientdisplaytype, logoncount, codepage, objec...
LDAP://CN=Administrators,CN=Builtin,DC=testing,DC=local     {objectcategory, usnchanged, distinguishedname, grouptyp...
LDAP://CN=Print Operators,CN=Builtin,DC=testing,DC=local    {iscriticalsystemobject, usnchanged, distinguishedname, ...
LDAP://CN=Backup Operators,CN=Builtin,DC=testing,DC=local   {iscriticalsystemobject, usnchanged, distinguishedname, ...
LDAP://CN=Replicator,CN=Builtin,DC=testing,DC=local         {iscriticalsystemobject, usnchanged, distinguishedname, ...
LDAP://CN=krbtgt,CN=Users,DC=testing,DC=local               {logoncount, codepage, objectcategory, description...}
LDAP://CN=Domain Controllers,CN=Users,DC=testing,DC=local   {usnchanged, distinguishedname, grouptype, whencreated...}
LDAP://CN=Schema Admins,CN=Users,DC=testing,DC=local        {objectcategory, usnchanged, distinguishedname, grouptyp...
LDAP://CN=Enterprise Admins,CN=Users,DC=testing,DC=local    {objectcategory, usnchanged, distinguishedname, grouptyp...
LDAP://CN=Domain Admins,CN=Users,DC=testing,DC=local        {objectcategory, usnchanged, distinguishedname, grouptyp...
LDAP://CN=Server Operators,CN=Builtin,DC=testing,DC=local   {iscriticalsystemobject, usnchanged, distinguishedname, ...
LDAP://CN=Account Operators,CN=Builtin,DC=testing,DC=local  {iscriticalsystemobject, usnchanged, distinguishedname, ...
LDAP://CN=Read-only Domain Controllers,CN=Users,DC=testi... {usnchanged, distinguishedname, grouptype, whencreated...}

Why is this a problem? With AdminCount=1 and disabled permission inheritance you will face a lot of problems:

  • Changes to an object ACLs are rolled back after an hour
  • Send-As doesn’t work
  • ActiveSync doesn’t work
  • Language settings in Outlook Web Access isn’t saved
  • ..and some other nasty effects

I have seen this error so often. Mostly at customers, that use normal work accounts for administrative tasks, or that add normal user accounts to admin groups during the setup of new profiles or clients. The solution is easy: Don’t do it!