vcloudnine.de https://www.vcloudnine.de ... virtualization on cloud 9 Thu, 10 Mar 2022 08:47:58 +0000 en-US hourly 1 https://wordpress.org/?v=5.9.3 Upgrade to ESXi 7.0: Missing dependencies VIBs Error https://www.vcloudnine.de/upgrade-to-esxi-7-0-missing-dependencies-vibs-error/ https://www.vcloudnine.de/upgrade-to-esxi-7-0-missing-dependencies-vibs-error/#respond Thu, 10 Mar 2022 08:47:54 +0000 https://www.vcloudnine.de/?p=5002 This error gets me from time to time, regardless which server vendor, mostly on hosts that were upgraded a couple of times. In this case it was a ESXi host currently running a pretty old build of ESXi 6.7 U3 and my job was the upgrade to 7.0 Update 3c.

If you add a upgrade baseline to the cluster or host, and you try to remediate the host, the task fails with a dependency error. When taking a closer look into the taks details, you were getting told that the task has failed because of a failed dependency, but not which VIB it caused.

You can find the name if the causing VIB on the update manager tab of the host which you tried to update. The status of the baseline is “incompatible”, and not “non-compliant”.

To resolve this issue you have to remove the causing VIB. This is no big deal and can be done with esxcli. Enable SSH and open a SSH connection to the host. Then remove the VIB.

[root@esx-01:~] esxcli software vib list | grep -i ssacli
ssacli                         4.17.6.0-6.7.0.7535516.hpe          HPE        PartnerSupported  2020-06-18
[root@esx-01:~] esxcli software vib remove -n ssacli
Removal Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed:
   VIBs Removed: HPE_bootbank_ssacli_4.17.6.0-6.7.0.7535516.hpe
   VIBs Skipped:
[root@esx-01:~]

You need to reboot the host after the removal of the VIB. Then you can proceed with the update. The status of the upgrade baseline should be now “not-compliant”.

5/5 - (1 vote)
]]>
https://www.vcloudnine.de/upgrade-to-esxi-7-0-missing-dependencies-vibs-error/feed/ 0
Mail notification for specific Active Directory security events https://www.vcloudnine.de/mail-notification-for-specific-active-directory-security-events/ https://www.vcloudnine.de/mail-notification-for-specific-active-directory-security-events/#respond Sun, 06 Feb 2022 11:16:09 +0000 https://www.vcloudnine.de/?p=4993 A customer used PRTG Network Monitor to notify him in case of account lockouts. This worked quite well until we implemented Admin Tiering. In order to get a mail notification in case of an account lockout, or other security-relevant events in Active Directory, I customized some scripts from my PowerShell dump.

The solution is pretty simple: I used the Task Planner to run a PowerShell script if a specific event id occurs. The events are generated in case of a various number of Active Directory events. You have to enable audit policy to get the needed events in the security event log. Take a look at Microsoft audit policy recommendations and enable what you need. I recommend to enable the stronger settings.

Image by Vitor Dutra Kaosnoff from Pixabay

I implemented five scripts:

  • Account lockout
  • New account in Active Directory
  • New member in domain-local group
  • New member in domain-global group
  • New member in universal group

The implementation is pretty easy. Create a basic task and execute this task if a specific event occurs.

The action is “Start a program”, like in the following screenshot.

Save the task and then change the user, which is used to run this task, to SYSTEM.

Please note, that you run these scripts with SYSTEM privileges. So make sure that NO ONE can easily edit these scripts! Best way is to restrict it to specific domain admins, restrict access to your domain controllers etc!

Repeat these steps for each script and implement them on each domain controller.

Please leave a comment with feedback :)

Rate this post
]]>
https://www.vcloudnine.de/mail-notification-for-specific-active-directory-security-events/feed/ 0
Outlook Web Access fails with “440 Login Timeout” https://www.vcloudnine.de/outlook-web-access-fails-with-440-login-timeout/ https://www.vcloudnine.de/outlook-web-access-fails-with-440-login-timeout/#respond Wed, 03 Nov 2021 17:17:30 +0000 https://www.vcloudnine.de/?p=4989 Today I faced an interesting problem. A customer told me that their Exchange 2010, which is currently part of a Exchange cross-forest migration project, has an issue with Outlook Web Access and the Exchange Control Panel. Both web sites fail with a white screen and a single message:

440 Login Timeout

I checked some basics, like certificate, configuration of the virtual directories and I found nothing suspicious. Most hints on the internet pointed towards problems with the IUSR_servername user, which is not used with IIS 7 and later. But authentication configuration and filesystem permissions were okay. Also the IIS end event logs were pretty unhelpful.

More interesting was the change date of the web.config! This file is part of the OWA web app and it’s typically stored under C:\Program Files\Microsoft\Exchange Server\V14\ClientAccess\Owa.

Long story short: I found this entry in the file and removed it.

<add name=”kerbauth” />

Looks like someone wanted to setup Kerberos auth for OWA, or did not reverse a change.

5/5 - (1 vote)
]]>
https://www.vcloudnine.de/outlook-web-access-fails-with-440-login-timeout/feed/ 0
Modify ProxyAddresses of Office 365 users without Exchange Online https://www.vcloudnine.de/modify-proxyaddresses-of-office-365-users-without-exchange-online/ https://www.vcloudnine.de/modify-proxyaddresses-of-office-365-users-without-exchange-online/#respond Sun, 31 Oct 2021 13:20:14 +0000 https://www.vcloudnine.de/?p=4986 As part of a Office 365 tenant rebuild, I had to move a custom domain to the new Office 365 tenant. The old tenant was not needed anymore, and the customer had to move to a Non-Profit tenant for compliance reasons. So the migration itself was no big deal:

  • disable AzureAD sync
  • change UPN of all users
  • remove the domain
  • connect the domain to the new tenant
  • setup a new AzureAD sync
  • assign licenses
  • time for a beer

That was my, honestly, naive plan for this migration.

Image by Gerd Altmann from Pixabay 

Disabling the AzureAD sync was easy. Even the change from ADFS to Password Hash Sync was easy. Changing the UPN for all users was a bit challenging, but the PowerSHell code in this article was quite helpful.

$users = Get-MsolUser -All | Where {$_.UserPrincipalName -like "*customdomain.tld"} | select UserPrincipalName 

foreach ($user in $users) {
 
   #Create New User Principal Name
   $newUser = $user.UserPrincipalName -replace "customdomain.tld", "customdomain.onmicrosoft.com"
 
   #Set New User Principal Name
   Set-MsolUserPrincipalName -UserPrincipalName $user.UserPrincipalName -NewUserPrincipalName $newUser
 
   #Display New User Principal Name
   $newUser
 }

But after this, I still was unable to remove the custom domain from the tenant. The domain was still referenced in the ProxyAddresses attribute, which was synced by the AzureAD sync…

Removing the domain from the users in the on-prem Active Directory was not solution. The users were already cloud-only because the sync was switched off. With this in mind my plan was to modify the cloud-only users in the tenant. To be honest: This solution worked in this specific case!

The customer was using Microsoft Teams Commercial Cloud trial licenses, so I had no Exchange Online to edit the proxy addresses. But luckily, the Exchange Online Management PowerShell Module was quite helpful.

Get-MailUser | Select -ExpandProperty emailaddresses | ? {$_ -like "*customdomain.tld"}

This line of code gave me an idea how many users were affected… quite a lot… With my colleague Claudia I quickly developed some dirty PowerShell code to remove all proxy addresses that included the custom domain.

$users = Get-MailUser -ResultSize Unlimited

foreach ($u in $users) {

    Get-MailUser -Identity $u.Alias |select -ExpandProperty emailaddresses | 
    ? {$_ -like "*customdomain.tld"} |
    % {Set-MailUser -Identity $u.Alias -EmailAddresses @{remove="$_"}}
     
}

It tool about 45 minutes to modify ~ 2000 users. After this, I was able to remove the domain and connect it to the new tenant.

This solution worked in my case. Another way might be using the AzureAD sync itself, masking out the custom domain and wait until the domain is removed from all proxy addresses. But I didn’t tested this.

Rate this post
]]>
https://www.vcloudnine.de/modify-proxyaddresses-of-office-365-users-without-exchange-online/feed/ 0
Escaping special characters in proxy auth passwords in vCenter https://www.vcloudnine.de/escaping-special-characters-in-proxy-auth-passwords-in-vcenter/ https://www.vcloudnine.de/escaping-special-characters-in-proxy-auth-passwords-in-vcenter/#respond Fri, 08 Oct 2021 07:38:24 +0000 https://www.vcloudnine.de/?p=4980 EDIT: It seems that his was fixed in vCenter 7.0 U3.

While debugging a vCener Lifecycle Manager, which was unable to download updates, I’ve stumbled over a weird behaviour, which is (IMHO) by design.

Some of you might use a proxy server. And some of you might use a proxy server which requires credentials. In my case, my customer uses a Sophos SG appliance as a web proxy server with authentication. The customer creaded a user with a complex password. But I was unable to get a working internet connection.

Image by Ed Webster from Pixabay 

I played a bit with curl on the bash of the vCenter. The proxy settings are stored under /etc/sysconfig/proxy. These settings are used to populate the http_proxy and https_proxy environment variable. It’s important to know, that the credentials stored in the /etc/sysconfig/proxy are encoded with the percent-encoding, also known as URL encoding. So someone with root access can grab credentials from these file.

But then I noticed something weird. I set the http_proxy variable manually with

http://username:password@proxy.domain.tld:8080

and I got this error:

-bash: !": event not found

Okay… there was a ! in the password and the BASH tried to execute the part behind the !. But it was part of the password, so I had to tell the BASH that it has to take this literally.

I escaped the ! in the password with a \. And to my surprise: The vCenter was able to download updates. I decoded the percent-encoded string in the /etc/sysconfig/poxy and found the escaped ! (\!). For example. Instead of Passw0rd! I had to enter Passw0rd\! in the password field.

Long story short: Use a password without special characters, otherwise escape them, because the password is stored in BASH variables.

Rate this post
]]>
https://www.vcloudnine.de/escaping-special-characters-in-proxy-auth-passwords-in-vcenter/feed/ 0
On the road to… nowhere? https://www.vcloudnine.de/on-the-road-to-nowhere/ https://www.vcloudnine.de/on-the-road-to-nowhere/#respond Mon, 26 Jul 2021 16:08:00 +0000 https://www.vcloudnine.de/?p=4963 Its been four month since my last blog post, and the blog frequency was quite low before that. This blog is, to be honest, a giant pile of stuff that has not worked as expected. Okay, some random thoughts or howto’s, but most blog posts are about stuff that failed in some way. That’s a bit “depressing”. I should write more about the fun things in my life

Picture by Gerd Altmann on Pixabay 

For a pretty long time my focus was on infrastructure. And my focus _is_ on infrastrucutre – Networks, lots of storage, virtualization with VMware. And always full stack: Networking, Storage, Servers, Operating System, always with a little focus here and there. Sure, products shifted over time, but in the bigger picture, my focus was always on infrastrucutre and datacenter stuff. No client devices, no end user support, no managed services/ admin tasks, no leadership. Technical stuff and projects. But my focus continued to shift. Microsoft Exchange for example. A product I really hate. Not really infrastructure. But I’m good at it and so I got projects and stuff to do. Or Office 365. Or Microsoft Azure. And since 2013 more leadership tasks. And since January 2020 I held some kind of a higher management position.

I’m doing much less VMware for the past 24 months than I like to do. Therefore much more Office 365 and Azure. And consulting for Microsoft stuff, transition to cloud, transition of IT services into managed services, or deployment of managed services. I lost my VCP/ VCAP through, IMHO, unnecessarily complicated recertification requirements. That was very frustrating for me. Of course, I learned other things in return.

Companions from the last 20 years are now mostly in management positions. Head of … whatever. Most of them are not doing technical stuff anymore. And they are happy with it. It looks like a typical career path, but it’s one that I don’t necessarily like right now. I’m still doing technical stuff, even if I’m in a management position. Actually quite good, but it also feels kind of weird.

I’m turning 40 this year. 23 years in IT behind me, 25 years to go until retirement. Not even half-time. :/ A wife, three nice kids, we just moved to our new house. Actually everything should be really great, but currently I can’t see a career path for me that makes me happy. And this sucks pretty hard.

So, to make a long story short, come back from time to time. Add this blog to your RSS reader. I hope to post nice content here again soon.

Rate this post
]]>
https://www.vcloudnine.de/on-the-road-to-nowhere/feed/ 0
Configure VMware Horizon View client device certificate authentication https://www.vcloudnine.de/configure-vmware-horizon-view-client-device-certificate-authentication/ https://www.vcloudnine.de/configure-vmware-horizon-view-client-device-certificate-authentication/#respond Thu, 11 Mar 2021 07:28:47 +0000 https://www.vcloudnine.de/?p=4944 This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Adding a second factor to your authentication is always a good idea. Typically the second factor is a One-Time Password (OTP) or a push notification. But what if you want to allow the login into your Horizon View environment only from specific devices? This implies that you need some kind of second factore that also identifies the device. At this point the arch enemy of many of us comes into play: Certificates!

To be honest: It is not so hard to get client device certificate authentication to work. All you need is:

  • Unified Access Gateway 2.6 or later
  • Horizon 7 version 7.5 or later
  • A certificate installed on the client device that Unified Access Gateway accepts

Configure X.509 authentication settings

The first step is to configure the UAG to accept a device certificate. To do so, log into the UAG admin interface, expand the authentication settings and open the X.509 settings.

You need to upload the Root CA certificate, which is used to sign the device certificates, as a Base64 coded file. I always recommend to enable “Cert Revocation”. You can enable “Use CRL from Certificates”, if the certificates include the URL to the CRL. Otherwise you can add the CRL location. This location must be accessable for the UAG! Click “Save” and you are ready to configure the Horizon settings.

Configure Horizon settings to use X.509 authentication

After you have configured the X.509 authentication, you have to enable the device certificate authentication for Horizon View. Expand “Horizon Settings” and enter the configuration settings.

It is important to select “Device X.509 Certificate AND Passthrough”.

Save the settings and you are ready to go. At this point a user must use a device with a valid device certificate.

Device Certificate

It is important to know that you have to create a new certificate template. The computer certificate template, which is included in a standard Microsoft PKI, cannot be used! It is mandatory to use the “Microsoft Enhanced RSA and AES Cryptographic Provider” in the template. It only works with this Cryptographic Service Provider (CSP)!

The easiest way is to duplicate the “Computer” template and change the necessary settings. First of all: The CSP must changed to “Microsoft Enhanced RSA and AES Cryptographic Provider” and it must be the only provider.

The subject name of the certificate should automatically be populated with information from the Active Directory, in this case the computer name.

Because the certificate is only for authentication purposes, you should remove “Server Authentication” from the Application Polices. Otherwise this certificate could be used to run a webserver.

Depending on your policies, you should mark the private key as “not exportable”!

The last step is important. After you enrolled the certificate to your computer, you need to add permissions to the user that should be able to use the certificate for authentication! This is necessary because it is a device certificate, and only SYSTEM and the local administrators group has permissions to access the private key of the certificate.

That’s it. If you open the View Client and try to connect to your View environment, then you should get a certificate selection dialog. After chosing the correct certificate, you need to enter user credentials.

Only with a valid certificate and valid credentials a connection to your View environment is possible.

Rate this post
]]>
https://www.vcloudnine.de/configure-vmware-horizon-view-client-device-certificate-authentication/feed/ 0
VMware vCenter 7.0 U2 deployment fails at stage 2 https://www.vcloudnine.de/vmware-vcenter-7-0-u2-deployment-fails-at-stage-2/ https://www.vcloudnine.de/vmware-vcenter-7-0-u2-deployment-fails-at-stage-2/#respond Wed, 10 Mar 2021 19:44:34 +0000 https://www.vcloudnine.de/?p=4941 This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Today I had to deploy a new vCenter appliance. Nothing fancy, new deployment. Stage 1 was easy, but stage 2 failed several times. I re-deployed the vCenter appliance two times, but as the deployment failed for the third time, I took a look into the logs.

The deployment failed without any error, but it didn’t finished. It stopped during the start of different services without any error.

First of all: Log into the appliance using SSH or the console. Use the root account and the root password you have entered during the setup.

A good point to start are the logs under /var/log/firstboot. I used ls -lt to get the last written logs. Most services will write two logs: One log ends with _stdout.log, and the second one will end with _stderr.log. The _stdout.log contails the log messages of the service. The _stderr.log contains the errors. I searched for a service that has written to a _stderr.log – and I found it: scafirstboot.py_10507_stderr.log.

And this log gave me a hint what the root cause was. One of the last log entries was:

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate is not yet valid

What what? A certificate not only has an end date, but also a date before which it is not valid – a start date. And this is often indicates a problem with – NTP. And it was NTP. I have configured NTP for the vCenter, but not for the ESXi on which I deployed the vCenter. -.- If it is not DNS, it’s NTP. Or a invalid certificate. Or both.

Rate this post
]]>
https://www.vcloudnine.de/vmware-vcenter-7-0-u2-deployment-fails-at-stage-2/feed/ 0
Veeam B&R backup failes with “No scale-out repository extents are available” https://www.vcloudnine.de/veeam-br-backup-failes-with-no-scale-out-repository-extents-are-available/ https://www.vcloudnine.de/veeam-br-backup-failes-with-no-scale-out-repository-extents-are-available/#respond Wed, 24 Feb 2021 09:50:07 +0000 https://www.vcloudnine.de/?p=4935 This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

One of my customers replaced the old Veeam environment with new gear. The HW was pretty simple designed:

  • two HPE ProLiant
  • per server two HPE D3610 enclosures with 6 TB disks
  • ~ 5km between backup server and backup copy destination

One server was designed to act as the Veeam backup server and repository, and the second server was designed to act as the backup copy destination. Both servers were running Windows Server 2019 Standard. We planned to use Windows Deduplication and ReFS, but it turned out that we have to adjust the filesystem size to get Windows Dedup working. Windows Dedup supports filesystems up to 64 TB. Due to the 24x 6 TB disks, we had to create to logical volumes to stay under 64 TB usable capacity.

I created one Scale-Out Backup Repository per server and configured my backup jobs. At this point things got worse…

The backup ran fine, but as soon as the copy kicked in, the copy job failed. Error “No scale-out repository extents are available”.

Huh? Everything was fine. If no backup were running, the copy ran fine. Setting limits (throughput or concurrent tasks) doesn’t fixed it. So I opened a case at Veeam.

We had to take debug logs to come to a solution.

Solution

The support advised us to set a registry key:

Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\
Value Name: SobrForceExtentSpaceUpdate
Value Type: DWORD
Value Data: 1

After a restart of the Veeam services, the backup and copy job ran fine. No further issues.

This key is described in Veeam KB2282. The option was introduced with Backup & Replication 9.5 U2. The customer is running the v10.0.1.4854. The key forces Veeam to update free space information with the real values, and it subtracts the estimated sizes of all the tasks currently going to the selected extent.

Rate this post
]]>
https://www.vcloudnine.de/veeam-br-backup-failes-with-no-scale-out-repository-extents-are-available/feed/ 0
WatchGuard Network Security Essentials Exam https://www.vcloudnine.de/watchguard-network-security-essentials-exam/ https://www.vcloudnine.de/watchguard-network-security-essentials-exam/#comments Tue, 19 Jan 2021 16:05:47 +0000 https://www.vcloudnine.de/?p=4931 This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Yesterday, I passed the first exam of the year. In this case the WatchGuards Network Security Essentials exam. The exam covers basic networking and firewalling skills, as well as the necessary knowledge to configure, manage, and monitor a WatchGuard Firebox. If you were familier with networking and firewalls in general, this exam is a “low hanging fruit”. I had to take it due to partner conditions.

WatchGuards offers a pretty good study guide for this exam which you can get for free.. The exam is delivered by  Kryterion and can be taken in a test center or as online proctored exam.

The closed book exam consists of 70 questions. You have 2h and you need at least 70% to pass the exam. The exam covers six different topics:

  • Network and network security basics
  • Administration and setup
  • Monitoring, logging, and reporting
  • Networking and NAT
  • Policies, proxies, and security services
  • Authentication and VPN

I passed the exam with a some preparation (I’ve only used the study guide). As long as you have experience with WatchGuard firewalls, which is mandartory IMHO, it is sufficient to read the study guide a couple of times.

Rate this post
]]>
https://www.vcloudnine.de/watchguard-network-security-essentials-exam/feed/ 1