Author Archives: Patrick Terlisten

About Patrick Terlisten

vcloudnine.de is the personal blog of Patrick Terlisten. Patrick has a strong focus on virtualization & cloud solutions, but also storage, networking, and IT infrastructure in general. He is a fan of Lean Management and agile methods, and practices continuous improvement whereever it is possible.Feel free to follow him on Twitter and/ or leave a comment.

Exchange HCW8078 – Migration Endpoint could not be created

While migrating a customer from Exchange 2010 to Exchange 2016, I had to create an Exchange Hybrid Deployment, because the customer wants to use Microsoft Teams. Nothing fancy and I’ve did this a couple of times.

Unfortunantely the Hybrid Connection Wizard failed to create the migration endpoint. A quick check of the logs showed this error:

Microsoft.Exchange.MailboxReplicationService.MRSRemotePermanentException: The Mailbox Replication Service could not connect to the remote server because the certificate is invalid. The call to 'https://mail.contoso.com/EWS/mrsproxy.svc' failed. Error details: Could not establish trust relationship for the SSL/TLS secure channel with authority 'mail.contoso.com'. -->The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel. --> The remote certificate is invalid according to the validation procedure

The customer had not plans to move mailboxes to Exchange Online, so we didn’t care about this error. But the Calendar tab in Teams was not visible, and Teams logs stated that Teams was unable to discover the mailbox. A typical sign of a not working EWS connection.

It’s always TLS… or DNS… or NTP

The customer used a certificate from its own PKI, so it was not trusted by Microsoft. In addition, the Exchange was located behind a Sophos XG which was running Webserver Protection (Reverse Proxy). But this was not the main cause for the problems.

The root cause was the certificate from the customers PKI.

And therefore you should make sure to use a proper certificate from a 3rd CA for Exchange Hybrid Deployments. I really please every customer to stop using self-signed certificates, or certificates from their own PKI for external connections.

The customer has switched to a Let’s Encrypt certificate for testing purposes and the problems went away, without running the HCW again. He will now purchase a certificate from a 3rd party CA.

Moving a small on-prem environment to Azure/ O365 – Part 2

A couple of days ago, I wrote about our first steps to move our on-prem stuff to Azure. This post will cover how we adopted Office 365 and how we have started with our Azure deployment.

Our first step into Office 365 was Microsoft Teams. We needed a solution for calls (audio/ video) and chat. We skipped Skype 4 Business and started with Microsoft Teams.

Microsoft 2020 (c)

Our Microsoft Teams deployment was pretty simple: We used our Microsoft IUR Office 365 E3 plans. Microsoft Azure AD Connect was quickly deployed and the Microsoft Exchange Hybrid Connection Wizard did the rest. Some weeks later we deployed ADFS/ ADFS Proxy. We used this setup over several months and it was pretty slick and was working flawless. At this point, we only used Teams, Planner and OneDrive 4 Business (SharePoint).

Some months went by until we decided to move to Azure.

Resource groups in Azure

You can imagine a resource group (RG) as a container that contains one or more resources, like VMs, NICs, SQL instances etc. The resource group can contain all the resources for the solution, or only those resources that you want to manage as a group.

First question: What do we need to deploy?

The answer was easy:

  • in sum 9 VMs
  • VPN gateway
  • Recovery Services Vault
  • Automation Account
  • Log Analytics Workspace

Second question: One or multiple resource groups?

An easy rule of thumb is, that a resource group should contain only resources that share the same life cycle and sponsor.

Third question: Who needs delegated priviledges to manage this stuff?

In our case there was no need to fine-graded RBAC. All of our technical staff has a personalized admin account and should be able to do whatever is necessary.

So we went for a single resource group.

Do yourself a favor and keep the recommended naming conventions in mind!

Site-2-Site VPN

To connect our on-prem network to Azure, we had to setup a Site-2-Site VPN. This was the first thing after creating our first resource group. We used a Gen 1 Basic VPN Gateway, which was sufficient for our needs (max 100 Mbit, no OpenVPN, no BGP).

Keep in mind to choose your networks and subnets wisely. If you need to deploy 9 VMs, don’t use 10.0.0.0/8. ;) In our case we added two network ranges with a single subnet in each network range. One for our server VMs, and a second subnet as gateway subnet.

VM Deployment

We deployed our VMs as B-Series VMs. A common mistake is to use the wrong VM size. Start small and right-size a VM if necessary. Most of our VMs are B2s (2 CPUs, 4 GB RAM). Only the Exchange (B4m), the management (B2ms) and the RDS server (B2ms) differ from this. This looks pretty small for Server 2019, but it is working pretty nice.

After deploying the VMs, we assigned static IP addresses to them. To our suprise most things in Azure are lacking proper IPv6 support. :( That hurt a lot.

For most VMs we used Standard HDDs instead of SSDs. Even for your file server, because the bottleneck is not the disk, it is the connection between clients and server. Beside this, we used managed disks for all VMs, and we deployed a second disk for data if necessary (Exchange, domain Controller, file server etc.).

If a server had a DNAT in our on-prem network, we deployed a public IP, and secured the access to it.

All VMs are connected to the same Network Security Group (NSG), which we use to get control over what a VM can reach, and who can access a VM.

Server Migration

Over a couple of days we moved more and more services to Azure, starting with our Domain Controllers, PKI and file services. These were low hanging fruits. The file server was easy because we already had a DFS namespace in place, so all we had to do were to change the DFS Links and point them to the new file server. The data was copied by using DFS replication.

DHCP was moved to our on-prem firewall. A printserver was not necessary any more. Windows Updates were switched back to download from Microsoft and Delivery Optimization.

The applications that were running on our Linux and Windows application server were also easy to migrate. After a couple of days we had our server workload running on Azure.

To get our ERP running, we deployed a single RDS host (quick deployment), and deployed our ERP as a remote app. It was too slow to use it over the VPN. Unfortunately the application lacks a proper database backend. :/ But as a remote app, it is working pretty good.

A bigger challenge was Exchange, but not because of the mailbox migrations.

Exchange Online

The migration to Exchange Online was pretty simple. Since our first HCW run, we used the central mail transport, so that all mails are received and sent by our on-prem mail gateway.

The mailbox migration was pretty easy and we had zero issues. Then we tried to switch the mail transport from central of Exchange Online. This was flawless too… except the fact, that our ticket system was unable to send e-mails.

Our ticket system relays its mail over our Exchange server. After switching the mail server in our ticket system to the new Azure based VM, the mails stuck in the outbound queue, even if the server tried to send the mail to our on-prem mail gateway. This quote from Microsoft explains the whole problem:

Starting on November 15, 2017, outbound email messages that are sent directly to external domains (such as outlook.com and gmail.com) from a virtual machine (VM) are made available only to certain subscription types in Microsoft Azure. Outbound SMTP connections that use TCP port 25 were blocked. (Port 25 is primarily used for unauthenticated email delivery.)

This change in behavior applies only to new subscriptions and new deployments since November 15, 2017.

Source: Microsoft

This is the case for MSDN, Azure Pass, Azure in Open, Education, BizSpark, and Free Trial subscriptions!

If you created an MSDN, Azure Pass, Azure in Open, Education, BizSpark, Azure Sponsorship, Azure Student, Free Trial, or any Visual Studio subscription after November 15, 2017, you’ll have technical restrictions that block email that’s sent from VMs within these subscriptions directly to email providers. The restrictions are done to prevent abuse. No requests to remove this restriction will be granted.

If you’re using these subscription types, you’re encouraged to use SMTP relay services, as outlined earlier in this article or change your subscription type.

Source: Microsoft

We accelerated our migration and disabled the central mail transport earlier than planned. Then we configured our Linux application server to authenticate against Exchange Online using SMTP Auth and SMTP Submission (587/tcp). For incoming mails, the mails are routed to the application server using a Exchange Online connector and a transport rule which matches to specific mail addresses.

The Azure based Exchange VM is only needed because we still have an Azure AD Connect running. Microsoft has planned to replace this by a new solution. And until this, we will run this Exchange 2016 in Azure. But it is not part of our mail flow.

Moving Azure AD Connect & decommissioning ADFS

Because we had to get rid of the ADFS server and ADFS Proxy, we deployed Pass-Through Authentication and Seamless SSO. Then we decommissioned the ADFS setup.

Moving Azure AD Connect was a bit quirky. We had conditional access already in place and the Azure AD Connect setup was unable to handle this. The synchronisation account was unable to sync, because it ran into a MFA request. We optimized our policies and got this sorted out.

Decommissioning old stuff

Whenever we moved a service successful to Azure, we switched off the on-prem server, and modified our documentation to reflect the made changes. At the end, we were able to switch off three of our four ESXi hosts. A last ESXi Host is still running for our Horizon View deployment and our firewall.

Next steps

The next post will cover how we automated this, how we do backups and whatever you’re interested in. Leave a comment! :)

Exchange Control Panel /ecp broken after certificate replacement

As part of an ongoing Exchange 2010 to 2016 migration, I had to replace the self-signed certificate with a certificate from the customers PKI. Everything went fine, the customer had a suitable template, we’ve added the necessary hostnames and bound IIS and SMTP to the certificate. The mess started with an iisreset /noforce

Bild von Oskars Zvejs auf Pixabay 

The iisreset took longer than expected. After that, I tried to login into the ECP, entered username and password and got an error.

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
  <Provider Name="MSExchange Front End HTTP Proxy" />
  <EventID Qualifiers="49152">1003</EventID>
  <Level>2</Level>
  <Task>1</Task>
  <Keywords>0x80000000000000</Keywords>
  <TimeCreated SystemTime="2020-10-22T12:16:38.934123400Z" />
  <EventRecordID>368718</EventRecordID>
  <Channel>Application</Channel>
  <Computer>server.domain.tld</Computer>
  <Security />
</System>
<EventData>
  <Data>Owa</Data>
  <Data>System.NullReferenceException: Object reference not set to an instance of an object. at 
Microsoft.Exchange.HttpProxy.FbaModule.ParseCadataCookies(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.FbaModule.OnBeginRequestInternal(HttpApplication httpApplication) at 
Microsoft.Exchange.HttpProxy.ProxyModule.<>c__DisplayClass16_0.<OnBeginRequest>b__0() at 
Microsoft.Exchange.Common.IL.ILUtil.DoTryFilterCatch(Action tryDelegate, Func`2 filterDelegate, Action`1 catchDelegate)
</Data>
</EventData>
</Event>

Pretty strange. We switched back to the self-singned certificate, did an iisreset and everyting was fine again.So it was pretty obvious that the error was related to the certificate, or to be more clear, to the certificate template.

A short research confirmed this. The template was a modified v3 web server template from an Enterprise CA running Windows Server 2008 R2.

With Windows Server 2008, Microsoft introduced a new cryptographic API called Cryptography Next Generation (CNG), which separates cryptographic providers (algorithm implementation) from key storage providers (create, delete, export, import, open and store keys). The older CryptoAPI does not differ between this and implements cryptographic algorithms and key storage.

The modified template used CNG instead of CryptoAPI. We noticed this when we checked the certificate with certutil -store my <thumbprint>.

If the listed provider for the certificate is Microsoft Software Key Storage Provider, then you will have to re-import the certificate. If Microsoft RSA SChannel Cryptographic Provider is used, everything is fine.

You have to remove the certificate, then re-import it using

certutil -csp "Microsoft RSA SChannel Cryptographic Provider" -importpfx <CertificateFilename>

You need a PKCS#12 file (PFX) and the password. Re-import it and then you can use the certificate for Exchange. Bind services to it and restart the IIS.

Moving a small on-prem environment to Azure/ O365 – Part 1

It was a bit quiet here due to the current COVID 19 pandemic. But now I’m back with a pretty interesting story on how my colleagues and I moved most of our on-prem server stuff to Microsoft Azure and Office 365.

Microsoft 2020 (c)

It all started with the COVID19 lockdown in Germany in March 2020. We moved into our home offices after setting up a small VMware Horizon View deployment to access our PCs using physical View Agents and manual desktop pools. Most projects were stopped, and we did most of our work remote. No lay-offs or short-time work.

We were running a small VMware vSphere cluster for a couple of years. Nothing fancy: Two HPE ProLiants, vCenter, two DCs, File-/ Printserver, WSUS, Exchange, Linux maschines for web services, Sophos UTM, a pfSense, View Connection Server, UAG, ADFS/ ADFS Proxy, PKI etc. In sum 18 VMs on two hosts, some VLANs with firewalls in between etc. We were running Exchange 2016, AzureAD Sync, Exchange Hybrid, but we only used Microsoft Teams from our Office 365 deployment. Veeam Backup & Replication was used for backups, a backup copy to a NAS and some Robocopy jobs that moved Veeam Backups to USB drives for DR. Everything was pretty simple and designed to work without much operations. Our focus is on our customers, not on our internal IT. It was stable, secure and pretty slick.

In March 2020 we asked “What if?”. What if we lose our offices due to a fire (we are located in a bigger office building and we had a couple of fire alarms this year due to remodeling work). How can we work if your DSL line is cut? How can we get our backups offsite? How can we modernize our IT withtout big invests? Money, that we don’t spend on our internal IT can given to our employees. ;) (By the way, that’s the same reason why we try to drive smaller and more efficient cars…).

We developed a couple of ideas, including new servers, storage etc. and put that stuff into a datacenter. But in the end, we decided to move most of our stuff to Microsoft Azure and Office 365.

I want to share some of the things we have learned on the road to Azure.

Initial assessment

We used the Azure Migrate Server Assessment tool to assess our vSphere environment. We wanted to get a ball park on how we had to size the VMs. We knew, that we wont need to migrate all VMs. For example our virtual pfSense firewall, the vCenter, the Sophos UTM or our ADFS setup were not planned to migrate.

After the first assessment, we started to play around the the Azure pricing calculator. Just to get an idea on how different VM sizes affect the costs.

Subscription

As a Microsoft partner, we were able to use our internal user rights (IUR) for Microsoft Office 365 and Azure. Microsoft offers us 25 Office 365 E3 plans and a 6000 US-$ budget for Azure (Azure Sponsorship Subscription). Our plan was to stretch the Azure budget over 12 months, so that we don’t have additional costs until we re-apply for our Microsoft partnership. Starting with 6000 US-$ Azure budget, it makes ~16 US-$ per day for our complete Azure deployment.

Sizing

Now, as we knew that we have 16 US-$ per day, we planned our Azure deployment. First of all, we planned the number of VMs. We had 18 VMs on-prem, and we managed to get down to 9 VMs.

  • two Domain Controller
  • PKI
  • Fileserver
  • Management Server
  • Remote Desktop Host (all-in-one Deployment)
  • Ticket System
  • SQL/ App server
  • Exchange 2016 for Hybrid Deployment

The View Connection Servers and UAG are still running on-prem. Our virtual pfSense will be moved to a WatchGuard Firebox soon. Sophos UTM and ADFS are gone. A dedicated WSUS server is not necessary any more, we moved back to simple Windows Update and Delivery Optimization.

Instead of D-Series VMs, we decided to go for B-Series VMs. The main reason for this were costs, but today I can say: The performance is quite good. I can’t see any reason for us to move to D-Series.

To connect to our Azure deployment, we had to setup Site-2-Site VPN. We deployed a simple Gen1 Basic SKU VPN Gateway. We had no need for more than 100 Mbit (we’re using a 50/10 VDSL at our office location), BGP or zone redundancy.

Instead of Standard SSD or Premium SSD storage, most of our VMs are using simple Standard HDD storage.

Backups are kept in a Recovery Services Vault with pretty simple polices. Either a VM needs to be current, in this case we keep 7 restore points, or we might need to keep more restore points. In this case we keep 7 daily, 5 weekly, 12 monthly and 3 yearly restore points. And this is only the case for our fileserver.

Additional cost savings

But with this setup we would not get under 16 US-$ a day. :( So we took another approach to break the mark: We shut down VMs at night and at the weekends! It took a bit until my colleagues and I get used to this. Nobody wants to shut down servers without a good reason.

But: We are currently at 18 US-$ per workday, and 10 US-$ for saturday and sunday. Everything, except domain controllers and ticket system, is shutdown at night and on the weekends.

We are using an Automation Account with some simple scripts and schedules to shutdown VMs and start them again.

What’s next?

The next blog post will be around how we planned the usage of Office 365, and how we started with Azure.

Virtually reseated: Reset blade in a HPE C7000 enclosure

After a reboot, a VMware ESXi 6.7 U3 told me that he has no compatible NICs. Fun fact: Right before the reboot everything was fine.

The ILO also showed no NICs. Unfortunately, I wasn’t onsite to pull the blade server and put it back in. But there is a way to do this “virtually”.

You have to connect to the IP address of the Onboard Administrator via SSH. Then issue the reset server command with the bay of the server you want to reset and an argument.

OA1-C7000> reset server 13

WARNING: Resetting the server trips its E-Fuse. This causes all power to be momentarily removed from the server. This command should only be used when physical access to the server is unavailable, and the server must be removed and
reinserted.

Any disk operations on direct attached storage devices will be affected. I/O
will be interrupted on any direct attached I/O devices.

Entering anything other than 'YES' will result in the command not executing.

Do you want to continue ? yes

Successfully reset the E-Fuse for device bay 13.

The server will power up automatically. Please note, that the OBA is unable to display certain information right after this operation. It will take a couple of minutes until all information, like serial number or device bay name are visible again.

Update Manager fails with unknown error during host remediation

During an vSphere 6.5 > 6.7 update a was host failing continously at the remediation with an “unknown error”. The host was updated from ESXI 6.5 to 6.7 using an upgrade baseline. Other hosts were updated to 6.7 and with the latest patches without any issues. Something strange was going on…

The esxupdate.log and the vua.log on the host itself showed nothing special. So I checked the vmware-vum-server-log4cpp.log which was much more informative!

[2020-07-19 13:03:25:217 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [singleHostScanTask, 693] caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?, ?, ?, ?, ?, ?, ?)"
[2020-07-19 13:03:25:219 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [singleHostScanTask, 404] SingleHostScan caught exception: caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?, ?, ?, ?, ?, ?, ?)" with code: -1
[2020-07-19 13:03:25:223 'SingleHostScanTask.SingleHostScanTask{262}' 139762329831168 ERROR] [vciTaskBase, 568] Task execution has failed: caught an odbc error: "ODBC error: (23505) - ERROR: duplicate key value violates unique constraint "pk_vci_scanresults"; Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_SCANRESULTS(endtime, id, scan_status, scan_type, starttime, target_component, target_uid) VALUES (?

Well… ERROR: duplicate key value violates unique constraint “pk_vci_scanresults” is not what I expected, but it is an error, and it occured everytime I tried to remediate the host.

Google found nothing about this error, so I decided to reset the VUM database. Please don’t try this at your customer! Log a call at VMware.

To reset the VUM database:

  1. Connect to vCenter Server Appliance via SSH
  2. Switch to the BASH 
  3. Stop the VMware Update Manager Service with this command

    service-control –stop vmware-updatemgr
     
  4. To reset the VMware Update Manager Database (applies only to VCSA 6.7 and 7.0!)

    /usr/lib/vmware-updatemgr/bin/updatemgr-utility.py reset-db
  1. Delete the contents of the VMware Update Manager Patch Store

    rm -rf /storage/updatemgr/patch-store/*
     
  2. Start the VMware Update Manager Service again

    service-control –start vmware-updatemgr

You will lose all your baselines, so you have to configure them again. And you need to download all patches again.

For vSAN environments this procedure will also remove the vSAN default baselines, but they will recreated automatically when there is a configuration change to vSAN or an update to the HCL DB. Again: Don’t do this at home!

Access to on-premise hosted Public Folders using Exchange Online mailboxes

Public Folders are still a thing. And while companies are moving their stuff into the cloud, Public Folders still need to be accessed by cloud-located mailboxes.

Allowing the access from Exchange Online mailboxes to on-premise hosted Public Folders is well documented by Microsoft, but there are also some fuzz. I had to deal with this during a Office 365 transition project at one of my customers.

The background

The customer is running a single Exchange 2016 server in a Windows Server 2012 R2 forest. AzureAD Sync is running and its syncing on-premise identities to AzureAD. The customer uses Office 365 E5 plans and he wants to move to Exchange Online, aside other O365 services like SharePoint Online, Teams etc.

Something was missing

After setting up the Exchange Hybrid, the customer and I where able to migrate the first mailboxes to Exchange Online.

To our surprise the on-premise Public Folders were not visible from the migrated Exchange Online mailboxes. We had still things to do…

In order to get the access to the Public Folders working, the Public Folder mailbox object needs to be synced to AzureAD. This is not complicated, because all you need to make sure is, that the user object is synced. If you are using an OU filter for the AzureAD sync, make sure that the OU with the Public Folder mailbox user object is included into the sync.

But there is also a second requirement: You also need to sync the Microsoft Exchange System Objects container! This is pretty important and it is not mentioned in the docs (Public folders in Microsoft 365, Office 365, and Exchange Online).

Check if the user is synced by using the Exchange Online PowerShell. This is how the Public Folder mailbox user looks like from the Exchange Online perspective:

PS C:\Users\p.terlisten> Get-MailUser Mailbox1

Name                                     RecipientType
----                                     -------------
Mailbox1                                 MailUser

This is the Public Folder mailbox that is hosted on-premise.

PS C:\Users\p.terlisten> Get-Mailbox -PublicFolder

Name                      Alias                ServerName     ProhibitSendQuota
----                      -----                ----------     -----------------
Mailbox1                  Mailbox1             EX             Unlimited

As long as you can’t see the MailUser in Exchange Online, you have no chance to configure the Public Folder access.

The next step is to synchronize the mail-enabled Public Folder objects to Exchange Online. For this, you have to download two scripts from Microsoft.

  • Sync-ModernMailPublicFolders.ps1
  • Sync-ModernMailPublicFolders.psd1

Run the Sync-ModernMailPublicFolders.ps1 script on your on-premise Exchange server. You will need your Office 365 admin credentials for this task.

PS C:\Users\p.terlisten\Download> .\Sync-ModernMailPublicFolders.ps1 -Credential (Get-Credential) -CsvSummaryFile:sync_summary.csv

The last step is to enable the Public Folder access using the Set-OrganizationConfig cmdlet in the Exchange Online PowerShell session.

PS C:\Users\p.terlisten> Set-OrganizationConfig -PublicFoldersEnabled Remote -RemotePublicFolderMailboxes Mailbox1

Please note that some of these steps need some time to get active! It will take some time for the background tasks to get some things sorted.

Controlled Connections to Public Folders in Outlook

It is worth mentioning that after enabling the access to Public Folders all Exchange online users can see the on-premise hosted Public Folders. If you need to enable the access only for some Exchange Online users, Microsoft has a solution for you: Controlled Connections to Public Folders.

First, you need to enable the Public Folder access for the users you have selected.

Set-CASMailbox "John Doe" -PublicFolderClientAccess $true

Then you enable the feature globally using the Exchange Online PowerShell.

Set-OrganizationConfig -PublicFolderShowClientControl $true

This setting applies only to Exchange Online hosted mailboxes!

Office365/ Exchange OAuth errors after replacing TLS certificate

A customer of mine asked for help to analyse a weird OAuth error. They are using a Microsoft Dynamics 365 Outlook plugin, which came up with an error:

“Can’t connect to Exchange”

In addition to this, they also faced an issueaccessing shared calendars of Exchange Online mailboxes.

Clearly an OAuth error. So we ran the Hybrid Connection Wizard again, which finished without any errors. But the errors persisted. Next stop: OAuth configuration.

We logged into one of the Exchange servers, started an Exchange Management Shell and checked the current OAuth configuration:

[PS] C:\Windows\system32>Get-AuthConfig
RunspaceId : e7c560cd-8316-4d9c-b97d-f4358d665e7d
CurrentCertificateThumbprint : CA420022ACC542D3AF7598456615715E98FFE986
PreviousCertificateThumbprint : E7B56BEAF4704BCBBF8C297F1D8D000215679C6F
NextCertificateThumbprint :
NextCertificateEffectiveDate :
ServiceName : 00000002-0000-0ff1-ce00-000000000000
Realm :
DeploymentId :
IssuerIdentifier :
Name : Auth Configuration
AdminDisplayName :
ExchangeVersion : 0.20 (15.0.0.0)
DistinguishedName : CN=Auth Configuration,CN=EXORG,CN=Microsoft
Exchange,CN=Services,CN=Configuration,DC=domain,DC=tld
Identity : Auth Configuration
Guid : 911b6246-6248-4278-91be-1b7b4943d2f1
ObjectCategory : mlnetwork.local/Configuration/Schema/ms-Exch-Auth-Auth-Config
ObjectClass : {top, container, msExchContainer, msExchAuthAuthConfig}
WhenChanged : 11.04.2020 12:33:12
WhenCreated : 27.02.2015 14:24:37
WhenChangedUTC : 11.04.2020 10:33:12
WhenCreatedUTC : 27.02.2015 13:24:37
OrganizationId :
Id : Auth Configuration
OriginatingServer : DC01.domain.tld
IsValid : True
ObjectState : Unchanged

The CurrentCertificateThumbprint was the thumbprint of the old certificate. So we had to update it.

Set-AuthConfig -NewCertificateThumbprint 45CCEEDFAAB00F3A4312FF56774FA3B3412B9C (get-date)
Set-AuthConfig -PublishCertificate
iisreset /noforce
Set-AuthConfig -ClearPreviousCertificate

After the iisreset, the Dynamics plugin and the Exchange Online calendars started to work again.

Once in a year: How to update TLS certificates on ADFS server and proxies

You might got this news some days ago: Starting with September 1, 2020, browsers and devices from Apple, Google, and Mozilla will show errors for new TLS certificates that have a lifespan greater than 398 days. Due to this move from Apple, Google and Mozilla, you have to deal with the replacement of certificates much more often. And we all know: Replacing certificates can be a real PITA!

Bild von skylarvision auf Pixabay

Replacing TLS certificates used for ADFS and Office 365 can be a challenging task, and this blog post will cover the neccessary steps.

ADFS Server

The first service, for which we will replace the certificate, is the ADFS server, or the ADFS server farm. At this point it is important to understand that we are dealing with two different points to which the certificate is bound:

  • the ADFS service communications certificate, and
  • the ADFS SSL certificate

The first step is to replace the service communication certificate. After importing the certificate with private key, you need to assign “read” permission to the ADFS service account. Right click on the certificate, then “All Tasks” > “Manage Private Keys”.

Make sure to import the certificate on all farm servers! Next step: Start the ADFS management console on the primary node. Select “Certificates” and then “Select service communication certificate” on the right window pane.

Now we have successfully replaced the service communication certificate. But we are no finished yet! Now we have to set the ADFS SSL certificate. Depending on your OS, you have to run the PowerShell command on the primary node. If your are running Windows Server 2012 R2 or older, you have to run the PowerShell command on EVERY ADFS farm server!

You can get the certificate thumbprint using the Get-AdfsSslCertificate command. Set the ADFS SSL certificate with

Set-AdfsSslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then restart the ADFS service.

Restart-Service adfssrv

ADFS Proxies

In most cases you will have one or more ADFS proxies in your DMZ. The ADFS proxy is nothing more than a Web Application Proxy (WAP) and therefore the PowerShell commands for WAP will be used.

First of all: Import the new certificate with the private key on all ADFS proxies, and then get the certificate hash of the new certificate. Then open an elevated PowerShell on each proxy.

Set-WebApplicationProxySslCertificate -Thumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125

Then we have to re-establish the trust between the proxies and the primary ADFS farm server. You will need the local (!) administrator account of the primary farm server.

$cred = Get-Credential
Install-WebApplicationProxy -FederationServiceTrustCredential $cred -CertificateThumbprint 07B3DFE31E020BFC3F3BDF1A0F61807223DBF125 -FederationServiceName adfs.domain.tld

The last step is to update thefederated trust with Office 365.

Update the federated trust with Office 365

To update the federated trust with Office 365, you will need the Windows Azure Active Direcotry Module for Windows PowerShell and an elevated PowerShell. Connect to Office 365 and update the federated trust:

$cred = Get-Credential
Connect-MsolService -Credential $cred
Update-MSOLFederatedDomain –DomainName domain.tld

That’s it! Bookmark this page and set a calendar entry on today +12 months. :)

Passed Microsoft exam AZ-103 – Azure Administrator Associate

Six weeks ago, I passed the Microsoft AZ-103 exam and earned the Azure Administrator Associate. A last minute pass, because AZ-104 was already launched. But better late than never. I had to re-schedule the exam a couple of times because the test center was closed due to COVID19.

The Azure Administrator Associate is a Administrator-role certification and it is all about implementing, managing and monitoring the Azure identity, governance, storage, compute, and virtual network solutions.

The exam covers a couple of topics and you should have knowledge and hands-on experience in administering Azure services using the Azure Portal, PowerShell, Azure CLI, and Azure Resource Manager templates.

Your knowledge is tested over a broad band of topics. These topics are:

  • Manage Azure identities and governance
  • Implement and manage storage
  • Deploy and manage Azure compute resources
  • Configure and manage virtual networking
  • Monitor and back up Azure resources

How to prepare for the exam

Fortunately I have a monthly Azure credit which I can use to gain new skills. I used these Azure credit together with the Microsoft Learning Path for AZ-103 (now 104).

It is pretty important no only to focus on VMs, storage or networking. Web Apps was one of my blind spots, and I had to get my head around it. Azure identities and governance is not so hard, if you are already familiar with Office 365.

I learned a lot from the Microsoft Documentation for Azure, and I was really impressed how much I was able to find, read and learn from there.

Next stop: Microsoft Certified: Azure Solutions Architect Expert

Microsoft has announced to retire all remaining exams associated with Microsoft Certified Solutions Associate (MCSA), Microsoft Certified Solutions Developer (MCSD), Microsoft Certified Solutions Expert (MCSE) on January 31, 2021, so the role-based certifications introduced in September 2018 are the way to go.

I’m currently holding a MCSE for Core Infrastructure and one for Productivity. Based on this, the Azure Solutions Architect Expert is the next step for me.