Tag Archives: microsoft

Why you should change your KRBTGT password prior disabling RC4

While chilling on my couch, I stumbled over this pretty interesting Reddit thread: Story Time – How I blew up my company’s AD for 24 hours and fixed it : sysadmin (reddit.com)

Long story short: A poor guy applied some STIG hardening and his Active Directory blew up. Root cause was disabling RC4, which caused Kerberos failures, primarily documented by errors like “The encryption type requested is not supported by the KDC.” The guy fixed it by shutdown all domain controllers, changing the KRBTGT account password on one domain controller, and finally, everything came back

So why blew everything up after disabling RC4? Let’s travel back in time. Microsoft released Active Directory with the release of Windows 2000. At this time, Active Directory supported DES and RC4 to encrypt Kerberos tickets. With RFC 6649 (Deprecate DES, RC4-HMAC-EXP, and Other Weak Cryptographic Algorithms in Kerberos), DES was retired in July 2012, but Microsoft disabled DES with the release of Windows Server 2008 R2 and Windows 7. Disabling DES was no big deal, because Active Directory was designed to select the highest supported cipher for encrypting the Kerberos tickets. In addition to this, support for AES was added with Windows Server 2008.

If you deploy a new domain controllers, with a higher Windows OS version, and you remove all older domain controllers, you will be able to raise the Domain, and the Forest Functional Level (DFL/ FFL). The functional levels determine the available domain or forest capabilities. With the DFL of Windows 2008, Microsoft added AES support. But only if you raise the DFL vom Windows 2003 to 2008, or any higher DFL, the KRBTGT password will be changed to get it stored AES encrypted.

Let me make this clear: This only happens when raising the DFL vom 2003 to 2008 or any higher version. Not if you go from 2008 to 2012 R2, from 2012 R2 to 2016 etc. Only from 2003 to 2008 or higher. This should be done automatically and you can verify this by checking the PasswordLastSet of the user account:

PS C:\windows\system32> Get-ADUser "krbtgt" -Property Created, PasswordLastSet


Created           : 6/22/2005 2:48:12 PM
DistinguishedName : CN=krbtgt,CN=Users,DC=mlnetwork,DC=local
Enabled           : False
GivenName         :
Name              : krbtgt
ObjectClass       : user
ObjectGUID        : dfc8490d-374f-4570-944e-d5fa41d601ab
PasswordLastSet   : 3/3/2015 8:29:08 AM
SamAccountName    : krbtgt
SID               : S-1-5-21-3103332001-754687911-2831376874-502
Surname           :
UserPrincipalName :

As you can see, the user account was created on the 22. June 2005, which is creation date of this Active Directory domain. The PasswordLastSet dates back to the 3rd March 2015, possibly the date where the DFL was raised to 2008 or above. If you don’t see a suitable date in the PasswordLastSet attribute, you should change the KRBTGT password! It looks like that there are some domains out there where the KRBTGT password wasn’t changed during the DFL raise.

Domain Controller with Windows 2008 and later will always use AES for the Ticket Granting Ticket (TGT). Once your domain functional level (DFL) is 2008 or higher, the KRBTGT account will always default to AES. For any other accounts (user and computer) the selected encryption type is determined by the msDS-supportedEncryptionTypes attribute of the account.

And let me get this pretty clear: As long as you are running Windows Server 2000, 2003, or Windows XP, you can’t disable RC4, because these operating systems simply doesn’t support AES (Source)!

So prior disable RC4, or do any hardening regarding Kerberos, make sure that you change the KRBTGT password. And change it twice. Or use the KRBTGT Reset script. You should also confirm, that your TGTs are encrypted with AES. You can check this with klist tgt.  If the TGTs are still being issued with RC4, you should check the pwdLastSet attribute on the KRBTGT account.

What should I do?

To be honest: You should change the KRBTGT password regularly, e.g. every 180 days. This is possible sind Server 2008 (and DFL/ FFL Windows 2008). This blog post of Quest gives you a pretty good summary of why and how. You might also want to take a look at this Microsoft website, if you want to know more about the KRBTGT account. You don’t need to document the password. Simply change it. If you want a more controlled way, you can use this script: Microsoft KRBTGT Reset script.

Use app-only authentication with the Microsoft Graph PowerShell SDK

In the previous blog post I have showed you how to interactively log in into the Microsoft Graph API. You had to enter a username, a password, and you had to enter a second factor. This is typically not want you want if you want to automate things. But there is another way to get access to the Microsoft Graph API.

Create an app registration

To get access, you have to register an app in your AzureAD. Go to your Azure portal and select “App registration” from the “Manage” section. Add a new registration by clicking to “New registration”.

Give your registration a meaningful name. Usually, only accounts in your AzureAD should be able to use this app.

The next step is to add permissions. This is equivalent to defining permission scopes during an interactive login. Make sure that you only follow the least-privilege method. In contrast to delegate access, this login type is truly limited to the permissions you grant in this step.

Select “Microsoft Graph” from the list.

Choose “Application permissions”.

Then select the necessary permissions.

Grant the permissions and select “Grant admin consent”. This step is pretty important. You, the admin, consent to the selected permissions. There is no further question to consent to the enduser.

Login with a client secret

But before we can use, we have to add something to use in the authentication process. There are two different methods to authenticate:

  • Certificate, or
  • client secret

A client secret is okay for test or dev environments. But I would not recommend the usage in a prod environment. You have to add this secret to a script or something, which is hard to protect. Create a new client secret. Please note, that a client secret has a lifetime. And make sure that you copy it. Tge client secret will be hidden later!

Make sure that you give your client secret a descriptive name.

The usage of a client secret is a two-step process. We need to get an access token, using the client secret, and use the token to connect to the Graph API. To get an access token, you need to install the Microsoft Authentication Libraries (MSAL) PowerShell module.

Install-Module MSAL.PS -Scope CurrentUser

Then we can aquire the token.

$AppId = '525b0e65-xxxx-xxxx-xxxx-7f8c32536247'
$TenantId = 'ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe'
$ClientSecret = 'NmO8Q~PPzVqZnxxxxxxxxi0vfRBhj8_xxxxxxx'
 
$Token = Get-MsalToken -TenantId $TenantId -ClientId $AppId -ClientSecret ($ClientSecret | ConvertTo-SecureString -AsPlainText -Force)
 
Connect-Graph -AccessToken $MsalToken.AccessToken

As you can see, this service principal login was made by the client secret.

Login with a certificate

Something more appropriate for a prod environment, is to use a certificate for the login. You can create a self-signed certificate, or use any other kind of X.509 certificate to authenticate. I used a S/MIME certificate in this case.

Make sure that you only upload the public key!! The certificate with the private key must be stored in the computer or user certificate store on the machine from which you want to access.

Upload the public key of a certificate.

Next step is to use the certificate hash during the login process:

Connect-MgGraph -ClientId 525b0e65-xxxx-xxxx-xxxx-7f8c32536247 -TenantId ffbc872a-xxxx-xxxx-xxxx-d81b43c67ffe -CertificateThumbprint DC427652498895A6F453671275BC69B352F3510A

Same result, a successful login, but different authentication method.

As already mentioned: I would prefer certificate over client secret. :)

Microsoft rolls back decision to block Office macros by default

Scrolling through my Twitter timeline is a common task to start my day. This morning, a tweet from @BleepinComputer has caught my attention.

My first reaction: WHAT. THE. FUCK?! Microsoft added this as feature 88883 in februrary 2022 to the Microsoft 365 roadmap, and I was pretty happy about this feature. Let’s take a look at this change.

The intention for this change was to add an extra layer of security in case that a user tried to open a downloaded file or an email attachment. The primary reason for this was, that Office documents with VBA were the main entry point for malware deployments. A highly customized email is sent to an employee with the goal, that the user opens the document. The embedded VBA code then is used to download and execute the malware.

Now, this highly wanted change is rolled back. IMHO there is only one reason for this: Customer complains about broken business processes. It’s that simple..

I’m not an InfoSec guy, but I know how to keep Ransomware gangs out of my network. There are three very efficient ways to do this:

  • Implement admin tiering, and avoid giving users admin permissions
  • Quarantine emails with attachments, and last, but not least
  • avoid to run unsighed office macros

Sure, you can allow users the use of Office documents with macros, but please make sure that these macros are signed with a proper code singing certicate, and stop allowing users to open documents with unsigned macros.

I really don’t get it why Microsoft is unable to push such an important change to the public. Of course, Microsoft isn’t responsible if Conti tears your store apart, but at least they could make it a little harder for you to let that happen…

No responsible person can tell me today “Sorry, we couldn’t do anything about this cyber attack”. But I also think that vendors should continue to turn off unsafe features. And if it breaks your business processes… yes, then you don’t deserve it any other way.

Wartungsfenster Podcast

Ausnahmsweise ein Blogpost in deutscher Sprache. Grund dafür ist, dass Claudia Kühn und ich seit Januar 2022 einen gemeinsamen Podcast rund um den Themenkomplex Datacenter, Cloud und IT ein. Eine lockere Kaminzimmerrunde in der wir entspannt über unseren Job, und alles was damit zu tun hat, plaudern.

Der Podcast erscheint alle zwei Wochen auf den üblichen Kanälen, oder ihr schaut auf der Homepage des Podcasts vorbei. Lasst gernen einen Kommentar/ Feedback da, und gebt uns eine Bewertung auf iTunes.

Mail notification for specific Active Directory security events

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer used PRTG Network Monitor to notify him in case of account lockouts. This worked quite well until we implemented Admin Tiering. In order to get a mail notification in case of an account lockout, or other security-relevant events in Active Directory, I customized some scripts from my PowerShell dump.

The solution is pretty simple: I used the Task Planner to run a PowerShell script if a specific event id occurs. The events are generated in case of a various number of Active Directory events. You have to enable audit policy to get the needed events in the security event log. Take a look at Microsoft audit policy recommendations and enable what you need. I recommend to enable the stronger settings.

Image by Vitor Dutra Kaosnoff from Pixabay

I implemented five scripts:

  • Account lockout
  • New account in Active Directory
  • New member in domain-local group
  • New member in domain-global group
  • New member in universal group

The implementation is pretty easy. Create a basic task and execute this task if a specific event occurs.

The action is “Start a program”, like in the following screenshot.

Save the task and then change the user, which is used to run this task, to SYSTEM.

Please note, that you run these scripts with SYSTEM privileges. So make sure that NO ONE can easily edit these scripts! Best way is to restrict it to specific domain admins, restrict access to your domain controllers etc!

Repeat these steps for each script and implement them on each domain controller.

Please leave a comment with feedback :)

Outlook Web Access fails with “440 Login Timeout”

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I faced an interesting problem. A customer told me that their Exchange 2010, which is currently part of a Exchange cross-forest migration project, has an issue with Outlook Web Access and the Exchange Control Panel. Both web sites fail with a white screen and a single message:

440 Login Timeout

I checked some basics, like certificate, configuration of the virtual directories and I found nothing suspicious. Most hints on the internet pointed towards problems with the IUSR_servername user, which is not used with IIS 7 and later. But authentication configuration and filesystem permissions were okay. Also the IIS end event logs were pretty unhelpful.

More interesting was the change date of the web.config! This file is part of the OWA web app and it’s typically stored under C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa.

Long story short: I found this entry in the file and removed it.

<add name=”kerbauth” />

Looks like someone wanted to setup Kerberos auth for OWA, or did not reverse a change.

Modify ProxyAddresses of Office 365 users without Exchange Online

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

As part of a Office 365 tenant rebuild, I had to move a custom domain to the new Office 365 tenant. The old tenant was not needed anymore, and the customer had to move to a Non-Profit tenant for compliance reasons. So the migration itself was no big deal:

  • disable AzureAD sync
  • change UPN of all users
  • remove the domain
  • connect the domain to the new tenant
  • setup a new AzureAD sync
  • assign licenses
  • time for a beer

That was my, honestly, naive plan for this migration.

Image by Gerd Altmann from Pixabay 

Disabling the AzureAD sync was easy. Even the change from ADFS to Password Hash Sync was easy. Changing the UPN for all users was a bit challenging, but the PowerSHell code in this article was quite helpful.

$users = Get-MsolUser -All | Where {$_.UserPrincipalName -like "*customdomain.tld"} | select UserPrincipalName 

foreach ($user in $users) {
 
   #Create New User Principal Name
   $newUser = $user.UserPrincipalName -replace "customdomain.tld", "customdomain.onmicrosoft.com"
 
   #Set New User Principal Name
   Set-MsolUserPrincipalName -UserPrincipalName $user.UserPrincipalName -NewUserPrincipalName $newUser
 
   #Display New User Principal Name
   $newUser
 }

But after this, I still was unable to remove the custom domain from the tenant. The domain was still referenced in the ProxyAddresses attribute, which was synced by the AzureAD sync…

Removing the domain from the users in the on-prem Active Directory was not solution. The users were already cloud-only because the sync was switched off. With this in mind my plan was to modify the cloud-only users in the tenant. To be honest: This solution worked in this specific case!

The customer was using Microsoft Teams Commercial Cloud trial licenses, so I had no Exchange Online to edit the proxy addresses. But luckily, the Exchange Online Management PowerShell Module was quite helpful.

Get-MailUser | Select -ExpandProperty emailaddresses | ? {$_ -like "*customdomain.tld"}

This line of code gave me an idea how many users were affected… quite a lot… With my colleague Claudia I quickly developed some dirty PowerShell code to remove all proxy addresses that included the custom domain.

$users = Get-MailUser -ResultSize Unlimited

foreach ($u in $users) {

    Get-MailUser -Identity $u.Alias |select -ExpandProperty emailaddresses | 
    ? {$_ -like "*customdomain.tld"} |
    % {Set-MailUser -Identity $u.Alias -EmailAddresses @{remove="$_"}}
     
}

It tool about 45 minutes to modify ~ 2000 users. After this, I was able to remove the domain and connect it to the new tenant.

This solution worked in my case. Another way might be using the AzureAD sync itself, masking out the custom domain and wait until the domain is removed from all proxy addresses. But I didn’t tested this.

Veeam B&R backup failes with “No scale-out repository extents are available”

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

One of my customers replaced the old Veeam environment with new gear. The HW was pretty simple designed:

  • two HPE ProLiant
  • per server two HPE D3610 enclosures with 6 TB disks
  • ~ 5km between backup server and backup copy destination

One server was designed to act as the Veeam backup server and repository, and the second server was designed to act as the backup copy destination. Both servers were running Windows Server 2019 Standard. We planned to use Windows Deduplication and ReFS, but it turned out that we have to adjust the filesystem size to get Windows Dedup working. Windows Dedup supports filesystems up to 64 TB. Due to the 24x 6 TB disks, we had to create to logical volumes to stay under 64 TB usable capacity.

I created one Scale-Out Backup Repository per server and configured my backup jobs. At this point things got worse…

The backup ran fine, but as soon as the copy kicked in, the copy job failed. Error “No scale-out repository extents are available”.

Huh? Everything was fine. If no backup were running, the copy ran fine. Setting limits (throughput or concurrent tasks) doesn’t fixed it. So I opened a case at Veeam.

We had to take debug logs to come to a solution.

Solution

The support advised us to set a registry key:

Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
Value Name: SobrForceExtentSpaceUpdate
Value Type: DWORD
Value Data: 1

After a restart of the Veeam services, the backup and copy job ran fine. No further issues.

This key is described in Veeam KB2282. The option was introduced with Backup & Replication 9.5 U2. The customer is running the v10.0.1.4854. The key forces Veeam to update free space information with the real values, and it subtracts the estimated sizes of all the tasks currently going to the selected extent.

Two registry changes to improve physical Horizon View Agent experience

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Using physical clients as Horizon View agents is pretty common for me. My office pc, as well as my Lenovo X250 are often used by using the Horizon View Client and the Blast protocol. But as good as the performance is, there were a couple of things that bugged me.

Bild von Mediamodifier auf Pixabay 

On my office pc, I encountered pretty often a black screen, either on first connect, or on reconnect. The typical issue caused by misconfigured firewall policies, but this was completly out of scope in this case, because my collegues never had issues with black screens. The problem occured with different versions of View Agent.

I finally fixed it after I tried to connected per HTML5 client. I got an error, that the connection server was unable to connect to 172.28.208.1. Huh? I don’t know this address… I checked my office pc and found out that this IP was assigned to the Hyper-V virtual switch.

I fixed this by adding this registry change:

HKLM\Software\VMware, Inc.\VMware VDM\IpPrefix = n.n.n.n/m (REG_SZ)

n.n.n.n/m is the subnet on which your View Agent should be connected by the Connection Server.

The second issue was that my client constantly failed to reboot. I clicked reboot, and the machine was gone. No TeamViewer, no RDP, no PING. Only a hard power-off helped to get it back to work.

I fixed this by adding another registry change:

HKCU\Control Panel\Desktop\AutoEndTasks = 1 (REG_SZ)
HKLM\SYSTEM\CurrentControlSet\Control\WaitToKillAppTimeout = 2000 (REG_SZ)
HKU\.DEFAULT\Control Panel\Desktop\AutoEndTasks = 1 (REG_SZ)

This fixed both issues, black screen and hung on reboot, for me.

Details on Windows 10 E3/ E5 Subscription Activation

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

One of my customers purchased a bunch of Microsoft 365 subscriptions in order to use them with Office 365 and Windows 10 Enterprise. The customer called me because he had trouble to activate the Windows 10 Enterprise license.

Source: Microsoft

I would like so summarize some of the requirements in order to successfuly active Windows 10 Enterprise subscriptions.

License

First of all, there is a licensing requirement. You need at least a Windows 10 Pro or Windows 10 Pro Education. You need one of these licenses! There is no way to use the Windows 10 Enterprise subscription without a base license, because it’s an upgrade!

Source: Microsoft

In case of my customer, the Pro license was missing. After adding and activating a Pro key, the key and edition was automatically updated to Windows 10 Enterprise.

AzureAD

In ordner to activate the license, the devices must be Azure AD-joined or Hybrid Azure AD joined. Workgroup-joined or Azure AD registered devices are not supported!

The Windows 10 Enterprise license must assigned to the user. The license can’t assigned to a device. Without an assigned license, the device can’t upgrade from a Pro to an Enterprise license.