Category Archives: Software

Missing Microsoft Teams calendar tab with on-premise Exchange

Microsoft Teams got a big push due to the current COVID19 crisis and many of my customers deployed it in the past weeks. At ML Network, we are using Microsoft Teams for more than a year, and we don’t want to miss it anymore.

Source: Microsoft

We are running Exchange 2016 on-premises, currently CU16. We were missing the calendar tab in Teams since we started with Microsoft Teams. when you do some research about this issue, you will find many threads and blog posts, but these are the two key facts:

  • it is supported with on-premises hybrid Exchange deployments
  • it works flawless with Exchange Online

Our Exchange is configured as full-hybrid mode deployment. I did this as we deployed Office 365 at our organization.

Let’s summarize:

  • Exchange 2016 CU16
  • Hybrid Deployment
  • Office 365 with Teams enabled
  • no calendar tab when the Exchange mailbox is hosted on-premises

OAuth FTW!

While doing an Exchange Hybrid deployment for one of my customers some weeks ago, I’ve stumbled over an OAuth error message at the end of the Hybric Connection Wizard. The message was HCW8064

“HCW has completed, but was not able to perform the OAuth portion of your Hybrid configuration”

We were not able to fix this. Microsoft offers two solutions:

Yesterday I did the upgrade from CU15 to CU16 on our Exchange server and while watching the progress bar I did some research on this issue again. I found strong evidence that Microsoft Teams needs working OAuth to display the calendar tab and access the on-premises hosted mailbox. So I gave it a try and used the latest version of the HCW wizard.

What should I say? No OAuth configuration error and after a restart of Microsoft Teams, the calendar tab appeared.

Lessons Learned:

  • always use the latest CU für Exchange
  • always use the latest HCW Wizard

What’s new in Vembu BDR Suite v4.0.1

This posting is ~1 year years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Vembu Technologies was founded in 2002, and with 60.000 customers and more than 4000 partners, Vembu is a leading provider with a comprehensive portfolio of software products and cloud services to small and medium businesses.

In December 2018, Vembu announced the fourth major release of their BDR Suite. Vembu BDR Suite 4.0.1 is now out for production setups with enhanced performance and bug fixes. Vembu BDR Suite v4.0.1 is an intermediate patch update that addresses the customers reported issues and other support issues on the previous build of v4.0. Vembu BDR Suite v4.0.1 also features a large number of enhancements and significant of those are listed below.

Vembu Technologies/ Vembu BDR Essentials/ Copyright by Vembu Technologies

What’s new?

Beside of bug fixes, BDR Suite v4.0.1 also includes some new enhancements. In my opinion, the most significant enhancements are:

  • Significant performance improvement in Quick VM Recovery on VMware environments
  • Rescan option is introduced in Hyper-V Manager Servers page, which allows you to install Vembu Integration Service on the newly added node of the Hyper-V cluster (or if it’s not available on the existing node)
  • Backups configured through BDR Server console will run in parallel (Default parallel backup count is set to 5 and it is configurable)
  • Ability to add new Hyper-V hosts or choose existing hosts while performing Live Recovery to Hyper-V host

Interested in trying Vembu BDR suite? Try the 30-day free trial now! For any questions, simply send an e-mail to vembu-support@vembu.com or follow them on Twitter.

If you are a small or mid-sized businesses, check out the Vembu BDR Essentials package!

Make your life easier – KeeAgent for KeePass

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Using a password safe, or password management system, is not a best practice – it’s a common practice. I’m using KeePass for years, because it’s available for different platforms, it can be used offline, it is Open Source, and it is not bound to any cloud services. Keepass allows me securely store usernames, passwords, recovery codes etc. for different services and websites, and together with features like autotype, Keepass offers a plus security and convenience.

I use 2FA or MFA wherever I can. That’s the reason why I’m a big fan of SSH public key authentication. But SSH key handling is sometimes inconvenient. You simple don’t want to store your SSH private keys on a cloud drive, and you don’t want to store them on a USB stick, or distribute them over different devices. In the past, I stored my SSH private keys on a cloud-drive in an encrypted container. When I needed a key, I decrypted the container and was able to use them. But this solution was inconvenient.

So what to do?

AbsolutVision/ pixabay.com/ Pixybay License

While searching for a solution I stumbled over KeeAgent, which is a plugin for KeePass. Keeagent allows you to store SSH keys in a KeePass database. KeeAgent then acts as SSH agent. I’m using this with PuTTY and MobaXterm and it works like a charm.

Setup KeeAgent

All you need is KeePass 2.x and the KeeAgent plugin. After installing the plugin (simply put the plgx file into C:\Program Files (x86)\KeePass Password Safe 2\Plugins), you can create a new entry in your KeePass database.

The password is the SSH private key passphrase. Then add the public and private key file to the newly created keepass database entry.

The KeeAgent.settings entry will be added automatically. Jump to the “KeeAgent” tab.

If required, keys can be loaded automatically if the database is locked, or you can add them later using the menu “Extras > KeeAgent”. Not every database entry can be used with KeeAgent, you have to enable the first checkbox to allow KeeAgent to use a specific database entry.

I create a database entry for each key pair I want to use with KeeAgent. And I only add frequently used keys automatically to KeeAgent. I have tons of keys and 99% of them are only added if I need them.

With KeeAgent in place, I can start new SSH sessions and KeeAgent delivers the matching key. You can see this in this screenshot “…from agent”.

I really don’t want to miss KeePass and KeeAgent. It makes my life easier and more secure.

Vembu CloudDR – Disaster Recovery as a Cloud Service

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

When it comes to disaster recovery (DR), dedicated offsite infrastructure is a must. If you follow the 3-2-1 backup rule, then you should have at least three copies of your data, on two different media, and one copy should be offsite.

But an offsite copy of your data can be expensive… You have to setup storage and networking in a suitable colocation. And even if you have an offsite copy of your data, you must be able to recover the data. This could be fun in case of terabytes of data and an offsite copy on tape.

A offsite copy in a cloud is much more interesting. No need to provide hardware, software, licenses. Just provide internet-connectivity, book a suitable plan, and you are ready to go.

Replication to Cloud using Vembu CloudDR

Vembu offers a cloud-based disaster recovery plan through its own cloud services, which is hosted in Amazon Web Services (AWS). This product is designed for businesses, who can’t afford, or who are not willing, to setup a dedicated offsite infrastructure for disaster recovery.

The data, which is backuped by the Vembu BDR server, is replicated to the Vembu Cloud. In case of any disaster, the backup data can be directly restored from the cloud at anytime and anywhere. The replication is managed and monitored using the CloudDR portal.

Before you can enable the offsite replication, you have to register your Vembu BDR server with your Vembu Portal account. You can either go to onlinebackup.vembu.com, or you can go to portal.vembu.com and sign up.

Vembu Technologies/ Vembu CloudDR/ Copyright by Vembu Technologies

After configuring schedule, retention and bandwidth usage, Vembu CloudDR is ready to go.

The end is near – time for recovery

CloudDR offers two types of recovery:

  • Image Based Recovery
  • Application Based Recovery

In case of an image based recovery, you can either download a VMDK or VHD(X) image, or you can do a file level recovery. In this case you can restore single files from inside of a chosen image.

You can even download a VHD(X) image of a VMware backup, which allows you some kind of V2V or P2V restores.

In case of a application based recovery, you can recover single application items from

  • Microsoft Exchange
  • Microsoft SharePoint
  • Microsoft SQL Server, or
  • MySQL

Depending on the type of restore, you will get an encrypted and password protected ZIP file with documents, or even MDF/ LDF files. These files can than be used to restore the lost data.

Summary

Vembu CloudDR is a pretty interesting add-on for Vembu customers. It’s easy to setup, has an attractive price tag and therefore consequently addresses the SMB customers.

Feel free to request a demo or try Vembu CloudDR.

Vembu BDR Essentials – Now up to 10 CPU Sockets

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

It is pretty common that vendors offer their products in special editions for SMB customers. VMware offers VMware vSphere Essentials and Essentials Plus, Veeam offers Veeam Backup Essentials, and Vembu has Vembu BDR Essentials.

Now Vembu has extended their Vembu BDR Essentials package significantly to address the needs of mid-sized businesses.

Vembu Technologies/ Vembu BDR Essentials/ Copyright by Vembu Technologies

Affordable backup for SMB customers

Most SMB virtualization deployments consists of two or three hosts, which makes 4 or 6 used CPU sockets. Because of this, Vembu BDR Essentials supportes up to 6 sockets or 50 VMs. Yes, 6 sockets OR 50 VMs. Vembu has no rised this limit to 10 Sockets OR 100 VMs! This allows customers to use up to five 2-socket hosts or 100 VMs with less than 10 sockets.

Feature Highlights

Vembu BDR Essentials support all important features:

  • Agentless VMBackup to backup VMs
  • Continuous Data Protection with support for RPOs of less than 15 minutes
  • Quick VM Recovery to get failed VMs up and running in minutes
  • Vembu Universal Explorer to restore individual items from Microsoft applications like Exchange, SharePoint, SQL and Active Directory
  • Replication of VMs Vembu OffsiteDR and Vembu CloudDR

Needless to say that Vembu BDR Essentials support VMware vSphere and Microsoft Hyper-V. If necessary, customer can upgrade to the Standard or Enterprise edition.

Veeam B&R: “Rescan of Manually Added” failed

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

I got this error in a new deployment of Veeam Backup & Replication 9.5 Update 4. The error occured every day at 9 pm.

24.02.2019 21:00:11 :: Error: Remote deployment and management is available for licensed agents only. Please change your backup server settings to allow managed agents to consume the license, then perform a protection group rescan.

The solution to this issue is pretty simple. Make sure that you allow the consumption of licenses for free agents. You will find this option under General > License.

Another workaround is to disable the protection group. Right click “Manually Added” under “Physical & Cloud Infrastructure” and click “Disable”.

Let me know if one of these workarounds worked for you. :)

Help Vembu and win a gift card!

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Vembu Technologies was founded in 2002, and with 60.000 customers and more than 4000 partners, Vembu is a leading provider with a comprehensive portfolio of software products and cloud services to small and medium businesses.

Backup is important. There is no reason to have no backup. According to an infographic published by Clutch Research at the World Backup Day 2017, 60% of all SMBs that lost all their data will shutdown within 6 months after the data loss. Pretty bad, isn’t it?

When I talk to SMB customers, most of them complain about the costs of backups. You need software, you need the hardware, and depending on the type of used hardware, you need media. And you should have a second copy of your data. In my opinion, tape is dead for SMB customers. HPE for example, offers pretty smart disk-based backup solutions, like the HPE StoreOnce.

Vembu is giving away an Amazon gift cards through a lucky draw for those readers, that take part of a short Survey

Vembu Technologies/ Vembu BDR/ Copyright by Vembu Technologies

Vembu BDR Suite provides a 30-day free trial with no restriction. This gives you the chance to intensively test Vembu BDR Suite prior purchase.

The free edition let you choose between unlimited VMs, that are covered with limited functionality, or unlimited functionality for up to 3 VMs. Check out this comparison of free, standard and enterprise edition. Check out this comparison of free, standard and enterprise edition.

Veeam Backup & Replication: Backup of Microsoft Active Directory Domain Controller VMs

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

To backup a virtual machine, Veeam Backup & Replication needs two permissions:

  • permission to access and backup the VM, as well as the
  • permission to do specific tasks inside the VM

to guarantee a consistent backup. The former persmission is granted by the user account that is used to access the VMware vCenter server (sorry for the VMW focust at this point). Usually, this account has the Administrator role granted at the vCenter Server level. The latter permission is granted by a user account that has permissions inside the guest operating system.

geralt / pixabay.com/ Creative Commons CC0

Something I often see in customer environments is the usage of the Domain Administrator account. But why? Because everything works when this account is used!

There are two reasons for this:

  • This account is part of the local Administrator group on every server and client
  • customers tend to grant the Administrator role to the Domain Admins group on vCenter Server level

In simple words: Many customers use the same account to connect to the vCenter, and for the application-aware processing of Veeam Backup & Replication. At least for Windows servers backups.

Houston, we have a problem!

Everything is fine until customers have to secure their environments. One of the very first things customers do, is to protect the Administrator account. And at this point, things might go wrong.

Using a service account to connect to the vCenter server is easy. This can be any account from the Active Directory, or from the embedded VMware SSO domain. I tend to create a dedicated AD-based service account. For the necessary permissions in the vCenter, you can grant this account Administrator permissions, or you can create a new user role in the vCenter. Veeam offers a PDF document which documents the necessary permissions for the different Veeam tasks.

The next challenge is the application-aware processing. For Microsoft SQL Server, the user account must have the sysadmin privileges on the Microsoft SQL Server. For Microsoft Exchange, the user must be member of the local Administrator group. But in case of a Active Directory Domain Contoller things get complicated.

A Domain Controller does not have a local user database (SAM). So what user account or group membership is needed to backup a domain controller using application-aware processing?

This statement is from a great Veeam blog post:

Permissions: Administrative rights for target Active Directory. Account of an enterprise administrator or domain administrator.

So the service account used to backup a domain controller is one of the most powerful accounts in the active directory.

There is no other way. You need a Domain or Enterprise Administrator account. I tend to create a dedicated account for this task.

I recommend to create a service account to connect the vCenter, and which is added to the local Administrator group on the servers to backup, and I create a dedicated Domain/ Enterprise Administrator account to backup the virtual Domain Controllers.

The advantage is that I can change apply different fine-grained password policies to this accounts. Sure, you can add more security by creating more accounts for different servers, and applications, add a dedicated role to the vCenter for Veeam etc. But this apporach is easy enough to implement, and adds a significant amount of user account security to every environment that is still using DOMAIN\Administrator to backup their VMs.

Veeam and StoreOnce: Wrong FC-HBA driver/ firmware causes Windows BSoD

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

One of my customers bought a very nice new backup solution, which consists of a

  • HPE StoreOnce 5100 with ~ 144 TB usable capacity,
  • and a new HPE ProLiant DL380 Gen10 with Windows Server 2016

as new backup server. StoreOnce and backup server will be connected with 8 Gb Fibre-Channel and 10 GbE to the existing network and SAN. Veeam Backup & Replication 9.5 U3a is already in use, as well as VMware vSphere 6.5 Enterprise Plus. The backend storage is a HPE 3PAR 8200.

This setup allows the usage of Catalyst over Fibre-Channel together with Veeam Storage Snapshots, and this was intended to use.

I wrote about a similar setup some month ago: Backup from a secondary HPE 3PAR StoreServ array with Veeam Backup & Replication.

The OS on the StoreOnce was up-to-date (3.16.7), Windows Server 2016 was installed using HPE Intelligent Provisioning. Afterwards, a drivers and firmware were updated using the latest SPP 2018.11 was installed. So all drivers and firmware were also up-to-date.

After doing zoning and some other configuration tasks, I installed Veeam Backup and Replication 9.5 U3, configured my Catalyst over Fibre-Channel repository. I configured a test backup… and the server failed with a Blue Screen of Death… which is pretty rare since Server 2008 R2.

geralt / pixabay.com/ Creative Commons CC0

I did some tests:

  • backup from 3PAR Storage Snapshots to Catalyst over FC repository – BSoD
  • backup without 3PAR Storage Snapshots to Catalyst over FC repository – BSoD
  • backup from 3PAR Storage Snapshots to Catalyst over LAN repository – works fine
  • backup without 3PAR Storage Snapshots to Catalyst over LAN repository – works fine
  • backup from 3PAR Storage Snapshots to default repository – works fine
  • backup without 3PAR Storage Snapshots to default repository – works fine

So the error must be caused by the usage of Catalyst over Fibre-Channel. I filed a case at HPE, uploaded gigabytes of memory dumps and heard pretty less during the next week.

HPE StoreOnce Support Matrix FTW!

After a week, I got an email from the HPE support with a question about the installed HBA driver and firmware. I told them the version number and a day later I was requested to downgrade (!) drivers and firmware.

The customer has got a SN1100Q (P9D93A & P9D94A) HBA in his backup server, and I was requested to downgrade the firmware to version 8.05.61, as well as the driver to 9.2.5.20. And with this firmware and driver version, the backup was running fine (~ 750 MB/s hroughput).

I found the HPE StoreOnce Support Matrix on the SPOCK website from HPE. The matrix confirmed the firmware and driver version requirement (click to enlarge).

Fun fact: None of the listed HBAs (except the Synergy HBAs) is supported with the latest StoreOnce G2 products.

Lessons learned

You should take a look at those support matrices – always! HPE confirmed that the first level recommendation “Have you trieed to update to the latest firmware” can cause similar problems. The fact, that the factory ships the server with the latest firmware does not make this easier.

Vembu BDR Suite v4.0 is now generally available

This posting is ~2 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Vembu Technologies was founded in 2002, and with 60.000 customers and more than 4000 partners, Vembu is a leading provider with a comprehensive portfolio of software products and cloud services to small and medium businesses.

Last week, Vembu has announced the availability of Vembu BDR Suite v4.0! Vembu’s new release is all about maintaining business continuity and ensuring high availability. Apart from new features, this release features significant enhancements and bug fixes that are geared towards performance improvement.

Vembu Technologies/ Vembu BDR Essentials/ Copyright by Vembu Technologies

The Vembu BDR Suite

The Vembu BDR Suite is an one stop solution to all your backup and disaster recovery needs. That is what Vembu says about their own product. The BDR Suite covers

  • Backup and replication of VMs running on VMware vSphere and Microsoft Hyper-V
  • Backup and bare-metal recovery for physical servers and workstations (Windows Server and Desktop)
  • File and application backups of Microsoft Exchange, Microsoft SQL Server, Microsoft SharePoint, Microsoft Active Directory, Microsoft Outlook, and MySQL
  • Creating of backup copies and transfer of them to a DR site

More blog posts about Vembu:

Vembu BDR Essentials – affordable backup for SMB customers
The one stop solution for backup and DR: Vembu BDR Suite

What’s new in 4.0?

Vembu BDR Suite v4.0 has got some pretty nice new features. IMHO, there are four highlights:

  • Hyper-V Failover Cluster Support for Backup & Recovery
  • Shared VHDX Backup
  • Hyper-V Checksum Based Incremental, and the
  • Credential Manager

There is a significat chance that you use a Hyper-V Failover Cluster if you have more than one Hyper-V host. With v4.0 Vembu added support for backup and recovery for the VMs residing in a Hyper-V Failover Cluster. Even if the VMs running on Hyper-V cluster move from one host to another, the backups will continue to run without any interruption.

A feature, that I’m really missing in VMware and Veeam, is the support for the backup shared VHDX files. v4.0 added support for this.

Vembu BDR Suite v4.0 also added support bot performing incremental backups with Hyper-V. They call it Checksum based incremental method, but it is in fact Change Block Tracking. An important feature for Hyper-V customers!

The Vembu Credential Manager allows you to store the necessary credentials at one place, use it everywhere inside the Vembu BDR Suite v4.0.

But there are also other, very nice enhancements.

  • Handling new disk addition for VMware ESXi and Hyper-V, which allows the backup of newly added disks at the next backup. In prioir releases, newly added disks were only backuped during the next full backup.
  • Reconnection for VMware ESXi and Hyper-V jobs in case of a dropped network connection
  • Application-wware processing for Hyper-V VMs can now enabled on a per-VM basis
  • API for VM list with Storage utilization report which allows you to generate detailed reports whenever you need one

Interested in trying Vembu BDR suite?, Try a 30-day free trial now! For any questions, simply send an e-mail to vembu-support@vembu.com or follow them on Twitter.