Author Archives: Patrick Terlisten

About Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible. Feel free to follow him on Twitter and/ or leave a comment.

Why you should change your KRBTGT password prior disabling RC4

While chilling on my couch, I stumbled over this pretty interesting Reddit thread: Story Time – How I blew up my company’s AD for 24 hours and fixed it : sysadmin (reddit.com)

Long story short: A poor guy applied some STIG hardening and his Active Directory blew up. Root cause was disabling RC4, which caused Kerberos failures, primarily documented by errors like “The encryption type requested is not supported by the KDC.” The guy fixed it by shutdown all domain controllers, changing the KRBTGT account password on one domain controller, and finally, everything came back

So why blew everything up after disabling RC4? Let’s travel back in time. Microsoft released Active Directory with the release of Windows 2000. At this time, Active Directory supported DES and RC4 to encrypt Kerberos tickets. With RFC 6649 (Deprecate DES, RC4-HMAC-EXP, and Other Weak Cryptographic Algorithms in Kerberos), DES was retired in July 2012, but Microsoft disabled DES with the release of Windows Server 2008 R2 and Windows 7. Disabling DES was no big deal, because Active Directory was designed to select the highest supported cipher for encrypting the Kerberos tickets. In addition to this, support for AES was added with Windows Server 2008.

If you deploy a new domain controllers, with a higher Windows OS version, and you remove all older domain controllers, you will be able to raise the Domain, and the Forest Functional Level (DFL/ FFL). The functional levels determine the available domain or forest capabilities. With the DFL of Windows 2008, Microsoft added AES support. But only if you raise the DFL vom Windows 2003 to 2008, or any higher DFL, the KRBTGT password will be changed to get it stored AES encrypted.

Let me make this clear: This only happens when raising the DFL vom 2003 to 2008 or any higher version. Not if you go from 2008 to 2012 R2, from 2012 R2 to 2016 etc. Only from 2003 to 2008 or higher. This should be done automatically and you can verify this by checking the PasswordLastSet of the user account:

PS C:\windows\system32> Get-ADUser "krbtgt" -Property Created, PasswordLastSet


Created           : 6/22/2005 2:48:12 PM
DistinguishedName : CN=krbtgt,CN=Users,DC=mlnetwork,DC=local
Enabled           : False
GivenName         :
Name              : krbtgt
ObjectClass       : user
ObjectGUID        : dfc8490d-374f-4570-944e-d5fa41d601ab
PasswordLastSet   : 3/3/2015 8:29:08 AM
SamAccountName    : krbtgt
SID               : S-1-5-21-3103332001-754687911-2831376874-502
Surname           :
UserPrincipalName :

As you can see, the user account was created on the 22. June 2005, which is creation date of this Active Directory domain. The PasswordLastSet dates back to the 3rd March 2015, possibly the date where the DFL was raised to 2008 or above. If you don’t see a suitable date in the PasswordLastSet attribute, you should change the KRBTGT password! It looks like that there are some domains out there where the KRBTGT password wasn’t changed during the DFL raise.

Domain Controller with Windows 2008 and later will always use AES for the Ticket Granting Ticket (TGT). Once your domain functional level (DFL) is 2008 or higher, the KRBTGT account will always default to AES. For any other accounts (user and computer) the selected encryption type is determined by the msDS-supportedEncryptionTypes attribute of the account.

And let me get this pretty clear: As long as you are running Windows Server 2000, 2003, or Windows XP, you can’t disable RC4, because these operating systems simply doesn’t support AES (Source)!

So prior disable RC4, or do any hardening regarding Kerberos, make sure that you change the KRBTGT password. And change it twice. Or use the KRBTGT Reset script. You should also confirm, that your TGTs are encrypted with AES. You can check this with klist tgt. If the TGTs are still being issued with RC4, you should check the pwdLastSet attribute on the KRBTGT account.

What should I do?

To be honest: You should change the KRBTGT password regularly, e.g. every 180 days. This is possible sind Server 2008 (and DFL/ FFL Windows 2008). This blog post of Quest gives you a pretty good summary of why and how. You might also want to take a look at this Microsoft website, if you want to know more about the KRBTGT account. You don’t need to document the password. Simply change it. If you want a more controlled way, you can use this script: Microsoft KRBTGT Reset script.

Use app-only authentication with the Microsoft Graph PowerShell SDK

In the previous blog post I have showed you how to interactively log in into the Microsoft Graph API. You had to enter a username, a password, and you had to enter a second factor. This is typically not want you want if you want to automate things. But there is another way to get access to the Microsoft Graph API.

Create an app registration

To get access, you have to register an app in your AzureAD. Go to your Azure portal and select “App registration” from the “Manage” section. Add a new registration by clicking to “New registration”.

Give your registration a meaningful name. Usually, only accounts in your AzureAD should be able to use this app.

The next step is to add permissions. This is equivalent to defining permission scopes during an interactive login. Make sure that you only follow the least-privilege method. In contrast to delegate access, this login type is truly limited to the permissions you grant in this step.

Select “Microsoft Graph” from the list.

Choose “Application permissions”.

Then select the necessary permissions.

Grant the permissions and select “Grant admin consent”. This step is pretty important. You, the admin, consent to the selected permissions. There is no further question to consent to the enduser.

Login with a client secret

But before we can use, we have to add something to use in the authentication process. There are two different methods to authenticate:

Certificate, or
client secret

A client secret is okay for test or dev environments. But I would not recommend the usage in a prod environment. You have to add this secret to a script or something, which is hard to protect. Create a new client secret. Please note, that a client secret has a lifetime. And make sure that you copy it. Tge client secret will be hidden later!

Make sure that you give your client secret a descriptive name.

The usage of a client secret is a two-step process. We need to get an access token, using the client secret, and use the token to connect to the Graph API. To get an access token, you need to install the Microsoft Authentication Libraries (MSAL) PowerShell module.

Install-Module MSAL.PS -Scope CurrentUser

Then we can aquire the token.

$AppId = '525b0e65-xxxx-xxxx-xxxx-7f8c32536247'
$TenantId = 'ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe'
$ClientSecret = 'NmO8Q~PPzVqZnxxxxxxxxi0vfRBhj8_xxxxxxx'
 
$Token = Get-MsalToken -TenantId $TenantId -ClientId $AppId -ClientSecret ($ClientSecret | ConvertTo-SecureString -AsPlainText -Force)
 
Connect-Graph -AccessToken $MsalToken.AccessToken

As you can see, this service principal login was made by the client secret.

Login with a certificate

Something more appropriate for a prod environment, is to use a certificate for the login. You can create a self-signed certificate, or use any other kind of X.509 certificate to authenticate. I used a S/MIME certificate in this case.

Make sure that you only upload the public key!! The certificate with the private key must be stored in the computer or user certificate store on the machine from which you want to access.

Upload the public key of a certificate.

Next step is to use the certificate hash during the login process:

Connect-MgGraph -ClientId 525b0e65-xxxx-xxxx-xxxx-7f8c32536247 -TenantId ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe -CertificateThumbprint DC427652498895A6F453671275BC69B352F3510A

Same result, a successful login, but different authentication method.

As already mentioned: I would prefer certificate over client secret. :)

Getting started with the Microsoft Graph PowerShell SDK

There is a new API in town… naa, not really new, but the Microsoft Graph API will replace most, if not all, other Azure AD/ Microsoft 365 APIs. Actually, Microsoft has planned to retire Azure AD Graph API and ADAL in Juni 2022. Now they have postponed this date to somewhere after December 2022. This will give you some extra time to refactor your PowerShell scrips and move them to use the PowerShell SDK for Graph.

What is Microsoft Graph? Microsoft Graph is the spider web that connects everything in Microsoft 365. One part of Graph is a single and unified API endpoint, which allows you to access and manage Azure AD and Microsoft 365 services, such as Teams and Exchange Online, Intune etc.

Instead of different endpoints, the Graph API uses a single endpoint (https://graph.microsoft.com/v1.0), which is the pretty nice thing if it comes down to Firewall rules and traffic management. This single endpoint, together with a single access token, allows you to manage all M365 services using REST API calls. So you don’t have to use the PowerShell SDK, you can also use Python or curl. Furthermore, the Microsoft Graph PowerShell SDK is Open Source, it offers cross-platform support (Linux, MacOS, Windows), and its available on PowerShell 5.1 nd above. This is a pretty important thing for me. Now I can use a single PowerShell Module to manage all my M365 services. This was PITA in the past. Depending on the service, I had to use different PowerShell modules, and I had to switch between PowerShell and PoSh Core.

Install the Microsoft Graph PowerShell SDK

The installation is pretty simple. All you need is

at least PowerShell 5.1 or later,
.NET 4.7.2 or later, and
the PowerShell script execution policy must be set to remote signed or a less restrictive execution policy

The installation is done by calling Install-Module.

Install-Module Microsoft.Graph -Scope CurrentUser

Sure, you can also use -Scope AllUsers, but I prefer to install PowerShell modules in my CurrentUser context. That’s it! Now let’s authenticate to the Graph API.

Authentication & Authorization

First of all: The Graph API knows two types of authentication:

delegated access, and
app-only access

Delegates access allows an application to act as a specific user, where as application access allows an application to act as its own entity. If you want that an API call is executed with the users permissions, then you should use delegated access. If you want to run scripts, or a service, then you want to use app-only access. Some API call are only possible with delegated access, some calls can only be made with app-only access, and some calls can be made with either of the two methods.

As part of the login process, you must define the requested permission scope. This is to ensure that the least possible privileges are used. For example:

Connect-MgGraph -Scopes "User.Read.All","Group.ReadWrite.All"

As you can see, this command will open a connection to the Graph API and you must consent the requested permissions after entering valid user credentials. Side note: Graph offers all the modern authentication stuff that we want to use today.

The tenant is specified by the singed-in user. If you want to connect to a specific tenant (to avoid usage of a cached access token), then you can add –TenantId <TenantId> to the command above.

As you can see, I now have a working connection using the Graph API using delegated access and with the requested permission scopes.

This map of Azure AD and MSOnline cmdlets was pretty helpful when playing around with Graph.

If you want to run a script, you might want to register an app in your tenant. This allows you to define the required permission scopes as properties of the app itself. I will show this in a separate blog post. This blog post will also cover the certificate-based authentication, as well as the authentication using a client secret.

Microsoft rolls back decision to block Office macros by default

Scrolling through my Twitter timeline is a common task to start my day. This morning, a tweet from @BleepinComputer has caught my attention.

Microsoft rolls back decision to block Office macros by default – @serghei https://t.co/9BK0slNuEw
— BleepingComputer (@BleepinComputer) July 7, 2022

My first reaction: WHAT. THE. FUCK?! Microsoft added this as feature 88883 in februrary 2022 to the Microsoft 365 roadmap, and I was pretty happy about this feature. Let’s take a look at this change.

The intention for this change was to add an extra layer of security in case that a user tried to open a downloaded file or an email attachment. The primary reason for this was, that Office documents with VBA were the main entry point for malware deployments. A highly customized email is sent to an employee with the goal, that the user opens the document. The embedded VBA code then is used to download and execute the malware.

Now, this highly wanted change is rolled back. IMHO there is only one reason for this: Customer complains about broken business processes. It’s that simple..

I’m not an InfoSec guy, but I know how to keep Ransomware gangs out of my network. There are three very efficient ways to do this:

Implement admin tiering, and avoid giving users admin permissions
Quarantine emails with attachments, and last, but not least
avoid to run unsighed office macros

Sure, you can allow users the use of Office documents with macros, but please make sure that these macros are signed with a proper code singing certicate, and stop allowing users to open documents with unsigned macros.

I really don’t get it why Microsoft is unable to push such an important change to the public. Of course, Microsoft isn’t responsible if Conti tears your store apart, but at least they could make it a little harder for you to let that happen…

No responsible person can tell me today “Sorry, we couldn’t do anything about this cyber attack”. But I also think that vendors should continue to turn off unsafe features. And if it breaks your business processes… yes, then you don’t deserve it any other way.

Wartungsfenster Podcast

Ausnahmsweise ein Blogpost in deutscher Sprache. Grund dafür ist, dass Claudia Kühn und ich seit Januar 2022 einen gemeinsamen Podcast rund um den Themenkomplex Datacenter, Cloud und IT ein. Eine lockere Kaminzimmerrunde in der wir entspannt über unseren Job, und alles was damit zu tun hat, plaudern.

Der Podcast erscheint alle zwei Wochen auf den üblichen Kanälen, oder ihr schaut auf der Homepage des Podcasts vorbei. Lasst gernen einen Kommentar/ Feedback da, und gebt uns eine Bewertung auf iTunes.

Upgrade to ESXi 7.0: Missing dependencies VIBs Error

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

This error gets me from time to time, regardless which server vendor, mostly on hosts that were upgraded a couple of times. In this case it was a ESXi host currently running a pretty old build of ESXi 6.7 U3 and my job was the upgrade to 7.0 Update 3c.

If you add a upgrade baseline to the cluster or host, and you try to remediate the host, the task fails with a dependency error. When taking a closer look into the taks details, you were getting told that the task has failed because of a failed dependency, but not which VIB it caused.

You can find the name if the causing VIB on the update manager tab of the host which you tried to update. The status of the baseline is “incompatible”, and not “non-compliant”.

To resolve this issue you have to remove the causing VIB. This is no big deal and can be done with esxcli. Enable SSH and open a SSH connection to the host. Then remove the VIB.

[[email protected]:~] esxcli software vib list | grep -i ssacli
ssacli                         4.17.6.0-6.7.0.7535516.hpe          HPE        PartnerSupported  2020-06-18
[[email protected]:~] esxcli software vib remove -n ssacli
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: HPE_bootbank_ssacli_4.17.6.0-6.7.0.7535516.hpe
   VIBs Skipped:
[[email protected]:~]

You need to reboot the host after the removal of the VIB. Then you can proceed with the update. The status of the upgrade baseline should be now “not-compliant”.

Mail notification for specific Active Directory security events

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer used PRTG Network Monitor to notify him in case of account lockouts. This worked quite well until we implemented Admin Tiering. In order to get a mail notification in case of an account lockout, or other security-relevant events in Active Directory, I customized some scripts from my PowerShell dump.

The solution is pretty simple: I used the Task Planner to run a PowerShell script if a specific event id occurs. The events are generated in case of a various number of Active Directory events. You have to enable audit policy to get the needed events in the security event log. Take a look at Microsoft audit policy recommendations and enable what you need. I recommend to enable the stronger settings.

Image by Vitor Dutra Kaosnoff from Pixabay

I implemented five scripts:

Account lockout
New account in Active Directory
New member in domain-local group
New member in domain-global group
New member in universal group

The implementation is pretty easy. Create a basic task and execute this task if a specific event occurs.

The action is “Start a program”, like in the following screenshot.

Save the task and then change the user, which is used to run this task, to SYSTEM.

Please note, that you run these scripts with SYSTEM privileges. So make sure that NO ONE can easily edit these scripts! Best way is to restrict it to specific domain admins, restrict access to your domain controllers etc!

Repeat these steps for each script and implement them on each domain controller.

Please leave a comment with feedback :)

Outlook Web Access fails with “440 Login Timeout”

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I faced an interesting problem. A customer told me that their Exchange 2010, which is currently part of a Exchange cross-forest migration project, has an issue with Outlook Web Access and the Exchange Control Panel. Both web sites fail with a white screen and a single message:

440 Login Timeout

I checked some basics, like certificate, configuration of the virtual directories and I found nothing suspicious. Most hints on the internet pointed towards problems with the IUSR_servername user, which is not used with IIS 7 and later. But authentication configuration and filesystem permissions were okay. Also the IIS end event logs were pretty unhelpful.

More interesting was the change date of the web.config! This file is part of the OWA web app and it’s typically stored under C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa.

Long story short: I found this entry in the file and removed it.

<add name=”kerbauth” />

Looks like someone wanted to setup Kerberos auth for OWA, or did not reverse a change.

Modify ProxyAddresses of Office 365 users without Exchange Online

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a Office 365 tenant rebuild, I had to move a custom domain to the new Office 365 tenant. The old tenant was not needed anymore, and the customer had to move to a Non-Profit tenant for compliance reasons. So the migration itself was no big deal:

disable AzureAD sync
change UPN of all users
remove the domain
connect the domain to the new tenant
setup a new AzureAD sync
assign licenses
time for a beer

That was my, honestly, naive plan for this migration.

Disabling the AzureAD sync was easy. Even the change from ADFS to Password Hash Sync was easy. Changing the UPN for all users was a bit challenging, but the PowerSHell code in this article was quite helpful.

$users = Get-MsolUser -All | Where {$_.UserPrincipalName -like "*customdomain.tld"} | select UserPrincipalName 

foreach ($user in $users) {
 
   #Create New User Principal Name
   $newUser = $user.UserPrincipalName -replace "customdomain.tld", "customdomain.onmicrosoft.com"
 
   #Set New User Principal Name
   Set-MsolUserPrincipalName -UserPrincipalName $user.UserPrincipalName -NewUserPrincipalName $newUser
 
   #Display New User Principal Name
   $newUser
 }

But after this, I still was unable to remove the custom domain from the tenant. The domain was still referenced in the ProxyAddresses attribute, which was synced by the AzureAD sync…

Removing the domain from the users in the on-prem Active Directory was not solution. The users were already cloud-only because the sync was switched off. With this in mind my plan was to modify the cloud-only users in the tenant. To be honest: This solution worked in this specific case!

The customer was using Microsoft Teams Commercial Cloud trial licenses, so I had no Exchange Online to edit the proxy addresses. But luckily, the Exchange Online Management PowerShell Module was quite helpful.

Get-MailUser | Select -ExpandProperty emailaddresses | ? {$_ -like "*customdomain.tld"}

This line of code gave me an idea how many users were affected… quite a lot… With my colleague Claudia I quickly developed some dirty PowerShell code to remove all proxy addresses that included the custom domain.

$users = Get-MailUser -ResultSize Unlimited

foreach ($u in $users) {

    Get-MailUser -Identity $u.Alias |select -ExpandProperty emailaddresses | 
    ? {$_ -like "*customdomain.tld"} |
    % {Set-MailUser -Identity $u.Alias -EmailAddresses @{remove="$_"}}
     
}

It tool about 45 minutes to modify ~ 2000 users. After this, I was able to remove the domain and connect it to the new tenant.

This solution worked in my case. Another way might be using the AzureAD sync itself, masking out the custom domain and wait until the domain is removed from all proxy addresses. But I didn’t tested this.

Escaping special characters in proxy auth passwords in vCenter

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

EDIT: It seems that his was fixed in vCenter 7.0 U3.

While debugging a vCener Lifecycle Manager, which was unable to download updates, I’ve stumbled over a weird behaviour, which is (IMHO) by design.

Some of you might use a proxy server. And some of you might use a proxy server which requires credentials. In my case, my customer uses a Sophos SG appliance as a web proxy server with authentication. The customer creaded a user with a complex password. But I was unable to get a working internet connection.

I played a bit with curl on the bash of the vCenter. The proxy settings are stored under /etc/sysconfig/proxy. These settings are used to populate the http_proxy and https_proxy environment variable. It’s important to know, that the credentials stored in the /etc/sysconfig/proxy are encoded with the percent-encoding, also known as URL encoding. So someone with root access can grab credentials from these file.

But then I noticed something weird. I set the http_proxy variable manually with

http://username:password@proxy.domain.tld:8080

and I got this error:

-bash: !": event not found

Okay… there was a ! in the password and the BASH tried to execute the part behind the !. But it was part of the password, so I had to tell the BASH that it has to take this literally.

I escaped the ! in the password with a \. And to my surprise: The vCenter was able to download updates. I decoded the percent-encoded string in the /etc/sysconfig/poxy and found the escaped ! (\!). For example. Instead of Passw0rd! I had to enter Passw0rd\! in the password field.

Long story short: Use a password without special characters, otherwise escape them, because the password is stored in BASH variables.