One of my customers replaced the old Veeam environment with new gear. The HW was pretty simple designed:
two HPE ProLiant
per server two HPE D3610 enclosures with 6 TB disks
~ 5km between backup server and backup copy destination
One server was designed to act as the Veeam backup server and repository, and the second server was designed to act as the backup copy destination. Both servers were running Windows Server 2019 Standard. We planned to use Windows Deduplication and ReFS, but it turned out that we have to adjust the filesystem size to get Windows Dedup working. Windows Dedup supports filesystems up to 64 TB. Due to the 24x 6 TB disks, we had to create to logical volumes to stay under 64 TB usable capacity.
I created one Scale-Out Backup Repository per server and configured my backup jobs. At this point things got worse…
The backup ran fine, but as soon as the copy kicked in, the copy job failed. Error “No scale-out repository extents are available”.
Huh? Everything was fine. If no backup were running, the copy ran fine. Setting limits (throughput or concurrent tasks) doesn’t fixed it. So I opened a case at Veeam.
We had to take debug logs to come to a solution.
The support advised us to set a registry key:
Key: HKLM\SOFTWARE\Veeam\Veeam Backup and Replication\ Value Name: SobrForceExtentSpaceUpdate Value Type: DWORD Value Data: 1
After a restart of the Veeam services, the backup and copy job ran fine. No further issues.
This key is described in Veeam KB2282. The option was introduced with Backup & Replication 9.5 U2. The customer is running the v10.0.1.4854. The key forces Veeam to update free space information with the real values, and it subtracts the estimated sizes of all the tasks currently going to the selected extent.
Using physical clients as Horizon View agents is pretty common for me. My office pc, as well as my Lenovo X250 are often used by using the Horizon View Client and the Blast protocol. But as good as the performance is, there were a couple of things that bugged me.
On my office pc, I encountered pretty often a black screen, either on first connect, or on reconnect. The typical issue caused by misconfigured firewall policies, but this was completly out of scope in this case, because my collegues never had issues with black screens. The problem occured with different versions of View Agent.
I finally fixed it after I tried to connected per HTML5 client. I got an error, that the connection server was unable to connect to 172.28.208.1. Huh? I don’t know this address… I checked my office pc and found out that this IP was assigned to the Hyper-V virtual switch.
One of my customers purchased a bunch of Microsoft 365 subscriptions in order to use them with Office 365 and Windows 10 Enterprise. The customer called me because he had trouble to activate the Windows 10 Enterprise license.
I would like so summarize some of the requirements in order to successfuly active Windows 10 Enterprise subscriptions.
First of all, there is a licensing requirement. You need at least a Windows 10 Pro or Windows 10 Pro Education. You need one of these licenses! There is no way to use the Windows 10 Enterprise subscription without a base license, because it’s an upgrade!
In case of my customer, the Pro license was missing. After adding and activating a Pro key, the key and edition was automatically updated to Windows 10 Enterprise.
In ordner to activate the license, the devices must be Azure AD-joined or Hybrid Azure AD joined. Workgroup-joined or Azure AD registered devices are not supported!
The Windows 10 Enterprise license must assigned to the user. The license can’t assigned to a device. Without an assigned license, the device can’t upgrade from a Pro to an Enterprise license.
While migrating a customer from Exchange 2010 to Exchange 2016, I had to create an Exchange Hybrid Deployment, because the customer wants to use Microsoft Teams. Nothing fancy and I’ve did this a couple of times.
Unfortunantely the Hybrid Connection Wizard failed to create the migration endpoint. A quick check of the logs showed this error:
Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'mail.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure
The customer had not plans to move mailboxes to Exchange Online, so we didn’t care about this error. But the Calendar tab in Teams was not visible, and Teams logs stated that Teams was unable to discover the mailbox. A typical sign of a not working EWS connection.
It’s always TLS… or DNS… or NTP
The customer used a certificate from its own PKI, so it was not trusted by Microsoft. In addition, the Exchange was located behind a Sophos XG which was running Webserver Protection (Reverse Proxy). But this was not the main cause for the problems.
The root cause was the certificate from the customers PKI.
And therefore you should make sure to use a proper certificate from a 3rd CA for Exchange Hybrid Deployments. I really please every customer to stop using self-signed certificates, or certificates from their own PKI for external connections.
The customer has switched to a Let’s Encrypt certificate for testing purposes and the problems went away, without running the HCW again. He will now purchase a certificate from a 3rd party CA.
A couple of days ago, I wrote about our first steps to move our on-prem stuff to Azure. This post will cover how we adopted Office 365 and how we have started with our Azure deployment.
Our first step into Office 365 was Microsoft Teams. We needed a solution for calls (audio/ video) and chat. We skipped Skype 4 Business and started with Microsoft Teams.
Our Microsoft Teams deployment was pretty simple: We used our Microsoft IUR Office 365 E3 plans. Microsoft Azure AD Connect was quickly deployed and the Microsoft Exchange Hybrid Connection Wizard did the rest. Some weeks later we deployed ADFS/ ADFS Proxy. We used this setup over several months and it was pretty slick and was working flawless. At this point, we only used Teams, Planner and OneDrive 4 Business (SharePoint).
Some months went by until we decided to move to Azure.
Resource groups in Azure
You can imagine a resource group (RG) as a container that contains one or more resources, like VMs, NICs, SQL instances etc. The resource group can contain all the resources for the solution, or only those resources that you want to manage as a group.
First question: What do we need to deploy?
The answer was easy:
in sum 9 VMs
Recovery Services Vault
Log Analytics Workspace
Second question: One or multiple resource groups?
An easy rule of thumb is, that a resource group should contain only resources that share the same life cycle and sponsor.
Third question: Who needs delegated priviledges to manage this stuff?
In our case there was no need to fine-graded RBAC. All of our technical staff has a personalized admin account and should be able to do whatever is necessary.
To connect our on-prem network to Azure, we had to setup a Site-2-Site VPN. This was the first thing after creating our first resource group. We used a Gen 1 Basic VPN Gateway, which was sufficient for our needs (max 100 Mbit, no OpenVPN, no BGP).
Keep in mind to choose your networks and subnets wisely. If you need to deploy 9 VMs, don’t use 10.0.0.0/8. ;) In our case we added two network ranges with a single subnet in each network range. One for our server VMs, and a second subnet as gateway subnet.
We deployed our VMs as B-Series VMs. A common mistake is to use the wrong VM size. Start small and right-size a VM if necessary. Most of our VMs are B2s (2 CPUs, 4 GB RAM). Only the Exchange (B4m), the management (B2ms) and the RDS server (B2ms) differ from this. This looks pretty small for Server 2019, but it is working pretty nice.
After deploying the VMs, we assigned static IP addresses to them. To our suprise most things in Azure are lacking proper IPv6 support. :( That hurt a lot.
For most VMs we used Standard HDDs instead of SSDs. Even for your file server, because the bottleneck is not the disk, it is the connection between clients and server. Beside this, we used managed disks for all VMs, and we deployed a second disk for data if necessary (Exchange, domain Controller, file server etc.).
If a server had a DNAT in our on-prem network, we deployed a public IP, and secured the access to it.
All VMs are connected to the same Network Security Group (NSG), which we use to get control over what a VM can reach, and who can access a VM.
Over a couple of days we moved more and more services to Azure, starting with our Domain Controllers, PKI and file services. These were low hanging fruits. The file server was easy because we already had a DFS namespace in place, so all we had to do were to change the DFS Links and point them to the new file server. The data was copied by using DFS replication.
DHCP was moved to our on-prem firewall. A printserver was not necessary any more. Windows Updates were switched back to download from Microsoft and Delivery Optimization.
The applications that were running on our Linux and Windows application server were also easy to migrate. After a couple of days we had our server workload running on Azure.
To get our ERP running, we deployed a single RDS host (quick deployment), and deployed our ERP as a remote app. It was too slow to use it over the VPN. Unfortunately the application lacks a proper database backend. :/ But as a remote app, it is working pretty good.
A bigger challenge was Exchange, but not because of the mailbox migrations.
The migration to Exchange Online was pretty simple. Since our first HCW run, we used the central mail transport, so that all mails are received and sent by our on-prem mail gateway.
The mailbox migration was pretty easy and we had zero issues. Then we tried to switch the mail transport from central of Exchange Online. This was flawless too… except the fact, that our ticket system was unable to send e-mails.
Our ticket system relays its mail over our Exchange server. After switching the mail server in our ticket system to the new Azure based VM, the mails stuck in the outbound queue, even if the server tried to send the mail to our on-prem mail gateway. This quote from Microsoft explains the whole problem:
Starting on November 15, 2017, outbound email messages that are sent directly to external domains (such as outlook.com and gmail.com) from a virtual machine (VM) are made available only to certain subscription types in Microsoft Azure. Outbound SMTP connections that use TCP port 25 were blocked. (Port 25 is primarily used for unauthenticated email delivery.)
This change in behavior applies only to new subscriptions and new deployments since November 15, 2017.
This is the case for MSDN, Azure Pass, Azure in Open, Education, BizSpark, and Free Trial subscriptions!
If you created an MSDN, Azure Pass, Azure in Open, Education, BizSpark, Azure Sponsorship, Azure Student, Free Trial, or any Visual Studio subscription after November 15, 2017, you’ll have technical restrictions that block email that’s sent from VMs within these subscriptions directly to email providers. The restrictions are done to prevent abuse. No requests to remove this restriction will be granted.
If you’re using these subscription types, you’re encouraged to use SMTP relay services, as outlined earlier in this article or change your subscription type.
We accelerated our migration and disabled the central mail transport earlier than planned. Then we configured our Linux application server to authenticate against Exchange Online using SMTP Auth and SMTP Submission (587/tcp). For incoming mails, the mails are routed to the application server using a Exchange Online connector and a transport rule which matches to specific mail addresses.
The Azure based Exchange VM is only needed because we still have an Azure AD Connect running. Microsoft has planned to replace this by a new solution. And until this, we will run this Exchange 2016 in Azure. But it is not part of our mail flow.
Moving Azure AD Connect & decommissioning ADFS
Because we had to get rid of the ADFS server and ADFS Proxy, we deployed Pass-Through Authentication and Seamless SSO. Then we decommissioned the ADFS setup.
Moving Azure AD Connect was a bit quirky. We had conditional access already in place and the Azure AD Connect setup was unable to handle this. The synchronisation account was unable to sync, because it ran into a MFA request. We optimized our policies and got this sorted out.
Decommissioning old stuff
Whenever we moved a service successful to Azure, we switched off the on-prem server, and modified our documentation to reflect the made changes. At the end, we were able to switch off three of our four ESXi hosts. A last ESXi Host is still running for our Horizon View deployment and our firewall.
The next post will cover how we automated this, how we do backups and whatever you’re interested in. Leave a comment! :)
As part of an ongoing Exchange 2010 to 2016 migration, I had to replace the self-signed certificate with a certificate from the customers PKI. Everything went fine, the customer had a suitable template, we’ve added the necessary hostnames and bound IIS and SMTP to the certificate. The mess started with an iisreset /noforce…
The iisreset took longer than expected. After that, I tried to login into the ECP, entered username and password and got an error.
<Provider Name="MSExchange Front End HTTP Proxy" />
<TimeCreated SystemTime="2020-10-22T12:16:38.934123400Z" />
<Data>System.NullReferenceException: Object reference not set to an instance of an object. at
Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication) at
Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication) at
Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
Pretty strange. We switched back to the self-singned certificate, did an iisreset and everyting was fine again.So it was pretty obvious that the error was related to the certificate, or to be more clear, to the certificate template.
A short research confirmed this. The template was a modified v3 web server template from an Enterprise CA running Windows Server 2008 R2.
With Windows Server 2008, Microsoft introduced a new cryptographic API called Cryptography Next Generation (CNG), which separates cryptographic providers (algorithm implementation) from key storage providers (create, delete, export, import, open and store keys). The older CryptoAPI does not differ between this and implements cryptographic algorithms and key storage.
The modified template used CNG instead of CryptoAPI. We noticed this when we checked the certificate with certutil -store my <thumbprint>.
If the listed provider for the certificate is Microsoft Software Key Storage Provider, then you will have to re-import the certificate. If Microsoft RSA SChannel Cryptographic Provider is used, everything is fine.
You have to remove the certificate, then re-import it using
It was a bit quiet here due to the current COVID 19 pandemic. But now I’m back with a pretty interesting story on how my colleagues and I moved most of our on-prem server stuff to Microsoft Azure and Office 365.
It all started with the COVID19 lockdown in Germany in March 2020. We moved into our home offices after setting up a small VMware Horizon View deployment to access our PCs using physical View Agents and manual desktop pools. Most projects were stopped, and we did most of our work remote. No lay-offs or short-time work.
We were running a small VMware vSphere cluster for a couple of years. Nothing fancy: Two HPE ProLiants, vCenter, two DCs, File-/ Printserver, WSUS, Exchange, Linux maschines for web services, Sophos UTM, a pfSense, View Connection Server, UAG, ADFS/ ADFS Proxy, PKI etc. In sum 18 VMs on two hosts, some VLANs with firewalls in between etc. We were running Exchange 2016, AzureAD Sync, Exchange Hybrid, but we only used Microsoft Teams from our Office 365 deployment. Veeam Backup & Replication was used for backups, a backup copy to a NAS and some Robocopy jobs that moved Veeam Backups to USB drives for DR. Everything was pretty simple and designed to work without much operations. Our focus is on our customers, not on our internal IT. It was stable, secure and pretty slick.
In March 2020 we asked “What if?”. What if we lose our offices due to a fire (we are located in a bigger office building and we had a couple of fire alarms this year due to remodeling work). How can we work if your DSL line is cut? How can we get our backups offsite? How can we modernize our IT withtout big invests? Money, that we don’t spend on our internal IT can given to our employees. ;) (By the way, that’s the same reason why we try to drive smaller and more efficient cars…).
We developed a couple of ideas, including new servers, storage etc. and put that stuff into a datacenter. But in the end, we decided to move most of our stuff to Microsoft Azure and Office 365.
I want to share some of the things we have learned on the road to Azure.
We used the Azure Migrate Server Assessment tool to assess our vSphere environment. We wanted to get a ball park on how we had to size the VMs. We knew, that we wont need to migrate all VMs. For example our virtual pfSense firewall, the vCenter, the Sophos UTM or our ADFS setup were not planned to migrate.
After the first assessment, we started to play around the the Azure pricing calculator. Just to get an idea on how different VM sizes affect the costs.
As a Microsoft partner, we were able to use our internal user rights (IUR) for Microsoft Office 365 and Azure. Microsoft offers us 25 Office 365 E3 plans and a 6000 US-$ budget for Azure (Azure Sponsorship Subscription). Our plan was to stretch the Azure budget over 12 months, so that we don’t have additional costs until we re-apply for our Microsoft partnership. Starting with 6000 US-$ Azure budget, it makes ~16 US-$ per day for our complete Azure deployment.
Now, as we knew that we have 16 US-$ per day, we planned our Azure deployment. First of all, we planned the number of VMs. We had 18 VMs on-prem, and we managed to get down to 9 VMs.
two Domain Controller
Remote Desktop Host (all-in-one Deployment)
SQL/ App server
Exchange 2016 for Hybrid Deployment
The View Connection Servers and UAG are still running on-prem. Our virtual pfSense will be moved to a WatchGuard Firebox soon. Sophos UTM and ADFS are gone. A dedicated WSUS server is not necessary any more, we moved back to simple Windows Update and Delivery Optimization.
Instead of D-Series VMs, we decided to go for B-Series VMs. The main reason for this were costs, but today I can say: The performance is quite good. I can’t see any reason for us to move to D-Series.
To connect to our Azure deployment, we had to setup Site-2-Site VPN. We deployed a simple Gen1 Basic SKU VPN Gateway. We had no need for more than 100 Mbit (we’re using a 50/10 VDSL at our office location), BGP or zone redundancy.
Backups are kept in a Recovery Services Vault with pretty simple polices. Either a VM needs to be current, in this case we keep 7 restore points, or we might need to keep more restore points. In this case we keep 7 daily, 5 weekly, 12 monthly and 3 yearly restore points. And this is only the case for our fileserver.
Additional cost savings
But with this setup we would not get under 16 US-$ a day. :( So we took another approach to break the mark: We shut down VMs at night and at the weekends! It took a bit until my colleagues and I get used to this. Nobody wants to shut down servers without a good reason.
But: We are currently at 18 US-$ per workday, and 10 US-$ for saturday and sunday. Everything, except domain controllers and ticket system, is shutdown at night and on the weekends.
We are using an Automation Account with some simple scripts and schedules to shutdown VMs and start them again.
The next blog post will be around how we planned the usage of Office 365, and how we started with Azure.
Public Folders are still a thing. And while companies are moving their stuff into the cloud, Public Folders still need to be accessed by cloud-located mailboxes.
Allowing the access from Exchange Online mailboxes to on-premise hosted Public Folders is well documented by Microsoft, but there are also some fuzz. I had to deal with this during a Office 365 transition project at one of my customers.
The customer is running a single Exchange 2016 server in a Windows Server 2012 R2 forest. AzureAD Sync is running and its syncing on-premise identities to AzureAD. The customer uses Office 365 E5 plans and he wants to move to Exchange Online, aside other O365 services like SharePoint Online, Teams etc.
Something was missing
After setting up the Exchange Hybrid, the customer and I where able to migrate the first mailboxes to Exchange Online.
To our surprise the on-premise Public Folders were not visible from the migrated Exchange Online mailboxes. We had still things to do…
In order to get the access to the Public Folders working, the Public Folder mailbox object needs to be synced to AzureAD. This is not complicated, because all you need to make sure is, that the user object is synced. If you are using an OU filter for the AzureAD sync, make sure that the OU with the Public Folder mailbox user object is included into the sync.
Please note that some of these steps need some time to get active! It will take some time for the background tasks to get some things sorted.
Controlled Connections to Public Folders in Outlook
It is worth mentioning that after enabling the access to Public Folders all Exchange online users can see the on-premise hosted Public Folders. If you need to enable the access only for some Exchange Online users, Microsoft has a solution for you: Controlled Connections to Public Folders.
First, you need to enable the Public Folder access for the users you have selected.
You might got this news some days ago: Starting with September 1, 2020, browsers and devices from Apple, Google, and Mozilla will show errors for new TLS certificates that have a lifespan greater than 398 days. Due to this move from Apple, Google and Mozilla, you have to deal with the replacement of certificates much more often. And we all know: Replacing certificates can be a real PITA!
Replacing TLS certificates used for ADFS and Office 365 can be a challenging task, and this blog post will cover the neccessary steps.
The first service, for which we will replace the certificate, is the ADFS server, or the ADFS server farm. At this point it is important to understand that we are dealing with two different points to which the certificate is bound:
the ADFS service communications certificate, and
the ADFS SSL certificate
The first step is to replace the service communication certificate. After importing the certificate with private key, you need to assign “read” permission to the ADFS service account. Right click on the certificate, then “All Tasks” > “Manage Private Keys”.
Make sure to import the certificate on all farm servers! Next step: Start the ADFS management console on the primary node. Select “Certificates” and then “Select service communication certificate” on the right window pane.
Now we have successfully replaced the service communication certificate. But we are no finished yet! Now we have to set the ADFS SSL certificate. Depending on your OS, you have to run the PowerShell command on the primary node. If your are running Windows Server 2012 R2 or older, you have to run the PowerShell command on EVERY ADFS farm server!
You can get the certificate thumbprint using the Get-AdfsSslCertificate command. Set the ADFS SSL certificate with
The last step is to update thefederated trust with Office 365.
Update the federated trust with Office 365
To update the federated trust with Office 365, you will need the Windows Azure Active Direcotry Module for Windows PowerShell and an elevated PowerShell. Connect to Office 365 and update the federated trust: