Category Archives: Security

Make your life easier – KeeAgent for KeePass

Using a password safe, or password management system, is not a best practice – it’s a common practice. I’m using KeePass for years, because it’s available for different platforms, it can be used offline, it is Open Source, and it is not bound to any cloud services. Keepass allows me securely store usernames, passwords, recovery codes etc. for different services and websites, and together with features like autotype, Keepass offers a plus security and convenience.

I use 2FA or MFA wherever I can. That’s the reason why I’m a big fan of SSH public key authentication. But SSH key handling is sometimes inconvenient. You simple don’t want to store your SSH private keys on a cloud drive, and you don’t want to store them on a USB stick, or distribute them over different devices. In the past, I stored my SSH private keys on a cloud-drive in an encrypted container. When I needed a key, I encrypted the container and was able to use them. But this solution was inconvenient.

So what to do?

AbsolutVision/ Pixybay License

While searching for a solution I stumbled over KeeAgent, which is a plugin for KeePass. Keeagent allows you to store SSH keys in a KeePass database. KeeAgent then acts as SSH agent. I’m using this with PuTTY and MobaXterm and it works like a charm.

Setup KeeAgent

All you need is KeePass 2.x and the KeeAgent plugin. After installing the plugin (simply put the plgx file into C:\Program Files (x86)\KeePass Password Safe 2\Plugins), you can create a new entry in your KeePass database.

The password is the SSH private key passphrase. Then add the public and private key file to the newly created keepass database entry.

The KeeAgent.settings entry will be added automatically. Jump to the “KeeAgent” tab.

If required, keys can be loaded automatically if the database is locked, or you can add them later using the menu “Extras > KeeAgent”. Not every database entry can be used with KeeAgent, you have to enable the first checkbox to allow KeeAgent to use a specific database entry.

I create a database entry for each key pair I want to use with KeeAgent. And I only add frequently used keys automatically to KeeAgent. I have tons of keys and 99% of them are only added if I need them.

With KeeAgent in place, I can start new SSH sessions and KeeAgent delivers the matching key. You can see this in this screenshot “…from agent”.

I really don’t want to miss KeePass and KeeAgent. It makes my life easier and more secure.

Securing VMs – vTPM, VBS, KMS and why you should not simply add a vTPM

Yesterday, I got one of these mails from a customer that make you think “Ehm, no”.

Can you please enable the TPM on all VMs.

The customer

The short answer is “Ehm, no!”. But I’m a kind guy, so I added some explanation to my answer.

Let’s add some context around this topic. The Trusted Platform Module (TPM) is a cryptoprocessor that offers various functions. For example, BitLocker uses the TPM to protect encryption keys. But there are another pretty interesting Windows features that require a TPM: “Virtualization-based Security“, or VBS. In contrast to BitLocker, VBS might be a feature that you want to use inside a VM.

VBS, uses virtualization features to create an isolated and secure region of memory, that is separated from the normal operating system. VBS is required if you want to use Windows Defender Credential Guard, which protects secrets like NTLM password hashes or Kerberos ticket-granting tickets against block pass-the-hash or pass-the-ticket (PtH) attacks. VBS is also required when you want to use Windows Defender Exploit Guard, or Windows Defender Application Control.

Credential Guard, Exploit Guard, and Application Control require a TPM 2.0 (and some other stuff, like UEFI, and some CPU extensions).

So, just add the vTPM module to a VM and you are ready to go? Ehm… no.

Prerequisites – or pitfalls

There are some prerequisites that must be met to use a vTPM:

  • the guest OS you use must be either Windows Server 2016, 2019 or Windows 10
  • the ESXi hosts must be at least ESXi 6.7, and
  • the virtual machine must use UEFI firmware

Okay, no big deal. But there is a fourth prerequisite that must be met:

  • your vSphere environment is configured for virtual machine encryption

And now things might get complicated… or expensive… or both.

Why do you need VM encryption when you want to add a vTPM?

The TPM can be used to securly store encryption keys. So the vTPM must offer a similar feature. In case of the vTPM, the data is written to the “Non-Volatile Secure Storage” of the VM. This is the .nvram file in the VM directory. To protect this data, the .nvram file is encrypted using the vSphere VM Encryption feature. In addition to the .nvram file, parts of the VMX file, the swap file, the vmware.log, and some other files are also encrypted. But not the VMDKs, except you decide to encrypt them.

Before you can start using VM encryption, you have to add a Key Management Server (KMS) to your vCenter. And you better you add a KMS cluster to your vCenter, because you don’t want that the KMS is a single point of failure. The vCenter Server requests keys from the KMS. The KMS generates and stores the keys, and passes them to third party systems, like the vCenter, using the Key Management Interoperability Protocol (KMIP) 1.1

The KMS is not a part of the vCenter or of the PSC. It is a seperate solution you have to buy. The KMS must support KMIP 1.1. Take a look into the Key Management Server (KMS) compatibility documentation offered by VMware for supported KMS products.

Make sure that you think about administrator permissions, role-based access control (RBAC), or disaster recovery. When you have to deal with security, you don’t want to have users use a general, high priviedge administrator account. And think about disaster recovery! You won’t be able to start encrypted VMs, until you have re-established trust between your vCenter and your KMS (cluster). So be prepared, and do not implement a single KMS.


And this is why vTPM is nothing you simply enable on all VMs. Because it’s security. And security has to be done right.

Mike Foley has written two awesome blog posts about this topic. Make sure that you read them.

vSphere 6.7 – Virtual Trusted Platform Modules
Introducing support for Virtualization Based Security and Credential Guard in vSphere 6.7

Notes for a 2-Tier Microsoft Windows PKI

Implementing a public key infrastructure (PKI) is a recurring task for me. More and more customers tend to implement a PKI in their environment. Mostly not to increase security, rather then to get rid of browser warnings because of self-signed certificates, to secure intra-org email communication with S/MIME, or to sign Microsoft Office macros.

tumbledore / Pixybay License

What is a 2-tier PKI?

Why is a multi-tier PKI hierarchy a good idea? Such a hierarchy typically consits of a root Certificate Authority (CA), and an issuing CA. Sometimes you see a 3-tier hierarchy, in which a root CA, a sub CA and an issuing CA are tied together in a chain of trust.

A root CA issues, stores and signs the digital certificates for sub CA. A sub CA issues, stores and signs the digital certificates for issuing CA. Only an issuing CA issues, stores and signs the digital certificates for users and devices.

In a 2-tier hierarchy, a root CA issues the certificate for an issuing CA.

In case of security breach, in which the issuing CA might become compromised, only the CA certificate for the issuing CA needs to be revoked. But what of the root CA becomes compromised? Because of this, a root CA is typically installed on a secured, and powered-off (offline) VM or computer. It will only be powered-on to publish new Certificate Revocation Lists (CRL), or to sign/ renew a new sub or issuing CA certificate.

Lessons learned

Think about the processes! Creating a PKI is more than provisioning a couple of VMs. You need to think about processes to

  • request
  • sign, and
  • revoke

Be aware of what a digital certificate is. You, or your CA, confirms the identity of a party by handing out a digital certificate. Make sure that no one can issue certificates without a proof of his identity.

Think about lifetimes of certificates! Customers tend to create root CA certificates with lifetimes of 10, 20 or even 40 years. Think about the typical lifetime of a VM or server, which is necessary to run an offline root CA. Typically the server OS has a lifetime of 10 to 12 years. This should determine the lifetime of a root CA certificate. IMHO 10 years is a good compromise.

For a sub or issuing CA, a lifespan of 5 years is a good compromise. Using the same lifetime as for a root CA is not a good idea, because an issued certificate can’t be longer valid than the lifetime of the CA certificate of the issuing CA.

A lifespan of 1 to 3 years for thinks like computer or web server certificates is okay. If a certificate is used for S/MIME or code signing, you should go for a lifetime of 1 year.

But to be honest: At the end of the day, YOU decide how long your certificates will be valid.

Publish CRLs and make them accessable! You can’t know if a certificate is revoked by a CA. But you can use a CRL to check if a certificate is revoked. Because of this, the CA must publish CRLs regulary. Use split DNS to use the same URL for internal and external requests. Make sure that the CRL is available for external users.

This applies not only to certificates for users or computers, but also for sub and issuing CAs. So there must be a CRL from each of your CAs!

I recommend to publish CRLs to a webserver and make this webserver reachable over HTTP. An issued certificate includes the URL or path to the CRL of the CA, that has issued the certificate.

Make sure that the CRL has a meaningful validity period. Of an offline root CA, which issues only a few certificates of its lifetime, this can be 1 year or more. For an issuing CA, the validity period should only a few days.

Publish AIA (Authority Information Access) information and make them accessable! AIA is an certificate extension that is used to offer two types of information :

  • How to get the certificate of the issuing or upper CAs, and
  • who is the OCSP responder from where revocation of this certificate can be checked

I tend to use the same place for the AIA as for the CDP. Make sure that you configure the AIA extension before you issue the first certificates, especially configure the AIA and CDP extension before you issue intermediate and issuing CA certificates.

Use a secure hash algorithm and key length! Please stop using SHA1! I recommend at least SHA256 and 4096 bit key length. Depending on the used CPUs, SHA512 can be faster than SHA256.

Create a CApolicy.inf! The CApolicy.inf is located uder C:\Windows and will will be used during the creation of the CA certificate. I often use this CApolicy.inf files.

For the root CA:

For the issuing CA:

Final words

I do not claim that this is blog post covers all necessary aspects of such an complex thing like an PKI. But I hope that I have mentioned some of the important parts. And at least: I have a reference from which I can copy and paste the CApolicy.inf files. :D

Windows NPS – Authentication failed with error code 16

Today, a customer called me and reported, on the first sight, a pretty weired error: Only Windows clients were unable to login into a WPA2-Enterprise wireless network. The setup itself was pretty simple: Cisco Meraki WiFi access points, a Windows Network Protection Server (NPS) on a Windows Server 2016 Domain Controller, and a Sophos SG 125 was acting as DHCP for different WiFi networks.

Pixybay / Pixabay License

Windows clients failed to authenticate, but Apple iOS, Android, and even Windows 10 Tablets had no problem.

The following error was logged into the Windows Security event log.

The credentials were definitely correct, the customer and I tried different user and password combinations.

I also checked the NPS network policy. When choosing PEAP as authentication type, the NPS needs a valid server certificate. This is necessary, because the EAP session is protected by a TLS tunnel. A valid certificate was given, in this case a wildcard certificate. A second certificate was also in place, this was a certificate for the domain controller from the internal enterprise CA.

It was an educated guess, but I disabled the server certificate check for the WPA2-Enterprise conntection, and the client was able to login into the WiFi. This clearly showed, that the certificate was the problem. But it was valid, all necessary CA certificates were in place and there was no reason, why the certificate was the cause.

The customer told me, that they installed updates on friday (today is monday), and a reboot of the domain controller was issued. This also restarted the NPS service, and with this restart, the Wildcard certificate was used for client connections.

I switched to the domain controller certificate, restarted the NPS, and all Windows clients were again able to connect to the WiFi.

Lessons learned

Try to avoid Wildcard certificates, or at least check the certificate that is used by the NPS, if you get authentication error with reason code 16.

Exam prep & experience: Citrix NetScaler Advanced Topics: Security, Management, and Optimization (1Y0-340)

In May 2018, Citrix released their new Citrix Certified Expert – Networking certification, which completet the networking certification path at the upper end (blog post on The track starts with the Associate (CCA-N), the lower-level certification is a requirement for achieving the higher-level certification, continues with the Professional (CCP-N), and ends with the Expert (CCE-N) certification. This is pretty cool, and I’m very happy that Citrix now offers the CCE-N, because the expert-level certification was missing all the time.

kmicican/ Creative Commons CC0

Everything is cool… except you have passed exam 1Y0-351 to gain your CCP-N. In this case, you have to pass 1Y0-340 until Dec 31 2018. Otherwise you have to start with the CCA-N, after the validity period of your CCP-N is over (3y after passing the exam).

Bad move, Citrix, bad move. I’m really disappointed. I passed 1Y0-351 in Nov 2017, and now, 12 months later, I have to book, pay, and pass 1Y0-340 if I not want to start with a CCA-N in Nov 2020. Bad move, Citrix, bad move!

Exam 1Y0-340 is titled as “Citrix NetScaler Advanced Topics: Security, Management, and Optimization”, where as 1Y0-351 was titeld as “Citrix NetScaler 10.5 Essentials and Networking”. You can assume that more in-depth knowledge is needed to pass the exam, as it was necessary for 1Y0-351. Note the “Advanced Topics” in the exam title.

But what are these “advanced topics”?  According to the exam prep guide, the perfect candidate for the 1Y0-340 exam can deploy and/or manage

  • Citrix NetScaler Application Firewall (AppFirewall) to secure application access in a Citrix NetScaler 12 environment, as well as
  • NetScaler Management and Analytics System (NMAS) to administer a Citrix NetScaler environment, or
  • Optimize NetScaler-managed application delivery traffic

Citrix NetScaler Application Firewall (AppFirewall)

You should take an in-depth look at these topics:

  • Application Firewall Overview
  • Application Firewall Profiles and Policies
  • Regular Expression
  • Attacks and Protections
  • Monitoring and Troubleshooting
  • Security and Filtering

NetScaler Management and Analytics System (NMAS)

  • NetScaler MAS: Introduction and Configuration
  • Managing and Monitoring NetScaler Instances
  • Managing NetScaler Configurations
  • NetScaler Web Logging

Optimize NetScaler-managed application delivery traffic

  • Integrated Caching
  • Front-End Optimization
  • Tuning and Optimizations

How to prep?

The exam prep guide referres to the NetScaler documentation, as also to training material. Unfortunately I don’t have access to the newer training material, only to the training material from my CNS-220 course. But hey: At least we have tons of publically available NetScaler 12.0 documentation available!

The exam prep guide has a section in which Citrix outlines sections, objectives and references. You will find links to the NetScaler 12.0 documentation, as well as knowledge base articles, or blog posts. Go through it. Read it carefully!

The exam prep guide also outlines the section titles and weights. Two areas stand out:

  • Section 4: Attacks and Protections, and
  • Section 8: Managing and Monitoring NetScaler Instances

The section weights are directly map to the number of questions in the exam. If the exam has 60 questions, and section 4 has a weight of 21%, at least 12 questions will relate to “Attacks and Protections”.

How did it go?

First things first: I passed with a good score. The exam had 62 questions and I needed at least 62% to pass the exam. I passed with 82%. As a non-native English speaker that took the exam in a country where english is a foreign language, I got 30 minutes extra, resulting in 120 minutes for 62 questions. Plenty of time…

What should I say? It was a multiple choice test. Read the questions carefully. The exam guide did not lie to me. It came pretty close to the topics that were described in the guide. For most questions, my first “educated guess” was right. Sometimes, the least dumb answer seemed to be correct. ;)

It was a bit frustrating that Citrix has changed product names. NetScaler is no “Application Delivery Controller”, MAS is now known as “Citrix Application Delivery Management”. There was a button which showed a mapping table “old name – new name”.

If you are experienced with Citrix ADC deployments and configuration, I think the exam prep guide is enough to pass the exam.

Good luck!

Using Let’s Encrypt DNS-01 challenge validation with local BIND instance

I’m using Let’s Encrypt certificates for a while now. In the past, I used the standalone plugin (TLS-SNI-01) to get or renew my certificates. But now I switched to the DNS plugin. I run my own name servers with BIND, so it was a very low hanging fruit to get this plugin to work.

Clker-Free-Vector-Images/ Creative Commons CC0

To get or renew a certificate, you need to provide some kind of proof that you are requesting the certificate for a domain that is under your control. No certificate authority (CA) wants to be the CA, that hands you out a certificate for or…

The DNS-01 challenge uses TXT records in order to validate your ownership over a certain domain. During the challenge, the Automatic Certificate Management Environment (ACME) server of Let’s Encrypt will give you a value that uniquely identifies the challenge. This value has to be added with a TXT record to the zone of the domain for which you are requesting a certificate. The record will look like this:

This record is for a wildcard certificate. If you want to get a certificate for a host, you can add one or more TXT records like this:

There is a IETF draft about the ACME protocol. Pretty interesting read!

Configure BIND for DNS-01 challenges

I run my own name servers with BIND on FreeBSD. The plugin for certbot automates the whole DNS-01 challenge process by creating, and subsequently removing, the necessary TXT records from the zone file using RFC 2136 dynamic updates.

First of all, we need a new TSIG (Transaction SIGnature) key. This key is used to authorize the updates.

This key has to be added to the named.conf. The key is in the .key file.

The key is used to authroize the update of certain records. To allow the update of TXT records, which are needed for the challenge, add this to the zone part of you named.con.

The records start always with _acme-challenge.domainname.

Now you need to create a config file for the RFC2136 plugin. This file also includes the key, but also the IP of the name server. If the name server is running on the same server as the DNS-01 challenge, you can use as name server address.

Now we have everything in place. This is a --dry-run  from on of my FreeBSD machines.

This is a snippet from the name server log file at the time of the challenge.

You might need to modify the permissons for the directory which contains the zone files. Usually the name server is not running as root. In my case, I had to grant write permissions for the “bind” group. Otherwise you might get “permission denied”.


CloudFlare API v4 and Fail2ban: Fixing the unban action

In January 2017, I wrote an article about how to protect your WordPress blog using the WP Fail2Ban plugin, fail2ban on your Linux/ FreeBSD host, and CloudFlare. Back then, the fail2ban was using the CloudFlare API V1, which was already deprecated since November 2016.

Free-Photos/ Creative Commons CC0

Although the actions were updated later to use CloudFlare API V4, I still had problems with the unbaning of IP addresses. IP addresses were banned, but the unban action failed. 

This is the unban action, which is included in fail2ban (taken from fail2ban- which is shipped with FreeBSD 11.1-RELEASE-p10):

And this is the unban action, which finally solved this issue:

I found the solution at The only difference is an additional tr -d '\n'  in the last line of the statement. Kudos to Jake for fixing this!

To prevent the action file to being overwritten, you should copy the original cloudflare.conf  located in the  action.d  directory, e.g. to mycloudflare.conf , and use the copied action file in your fail definition.

Windows Network Policy Server (NPS) server won’t log failed login attempts

This is just a short, but interesting blog post. When you have to troubleshoot authentication failures in a network that uses Windows Network Policy Server (NPS), the Windows event log is absolutely indispensable. The event log offers everything you need. The success and failure event log entries include all necessary information to get you back on track. If failure events would be logged…

geralt/ Creative Commons CC0

Today, I was playing with Alcatel-Lucent Enterprise OmniSwitches and Access Guardian in my lab. Access Guardian refers to the some OmniSwitch security functions that work together to provide a dynamic, proactive network security solution:

  • Universal Network Profile (UNP)
  • Authentication, Authorization, and Accounting (AAA)
  • Bring Your Own Device (BYOD)
  • Captive Portal
  • Quarantine Manager and Remediation (QMR)

I have planned to publish some blog posts about Access Guardian in the future, because it is a pretty interesting topic. So stay tuned. :)

802.1x was no big deal, mac-based authentication failed. Okay, let’s take a look into the event log of the NPS… okay, there are the success events for my 802.1x authentication… but where are the failed login attempts? Not a single one was logged. A short Google search showed me the right direction.

Failed logon/ logoff events were not logged

In this case, the NPS role was installed on a Windows Server 2016 domain controller. And it was a german installation, so the output of the commands is also in german. If you have an OS installed in english, you must replace “Netzwerkrichtlinienserver” with “Network Policy Server”.

Right-click the PowerShell Icon and open it as Administrator. Check the current settings:

As you can see, only successful logon and logoff events were logged.

The option /success:enable /failure:enable activeates the logging of successful and failed logon and logoff attempts.

Replace SSL certificates on Citrix NetScaler using the CLI

Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.

skylarvision/ Creative Commons CC0

When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.

Update or replace?

There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate  includes the unbinding of the old, and binding the new certificate.

Replacing a certificate

The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.

Do yourself a favor and add the expiration date to the name of the certificate/ key pair.

Now you can unbind the old, and bind the new certificate. Please note, that this causes a short outage of your service!

SSL Cert Unbind Causing NetScaler Crash

You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.

Bypass stateful firewall on a Sophos XG

Usually, bypassing a firewall is not the best idea. But sometimes you have to. One case, where you want to bypass a firewall, is asymmetric routing.

MichaelGaida/ Creative Commons CC0

What is asymmetric routing? Imagine a scenario with two routers on the same network. One router offeres access to the internet, the other router provides access to other sites with site-2-site VPN tunnels.

Asymmetric Routing

Patrick Terlisten/ Creative Commons CC0

Host 1 uses R1 as default gateway. R1 has static routes configured to the networks reachable over the VPN, or it has learned them dynamically using a routing protocol from R2. A packet from host 1 arrives at R1, is routed to R2, and is sent over the VPN tunnel. The answer to this packet arrives at R2, and is sent directly to host 1, because host 1 is the destination. This works because R2 and host 1 are on the same network. This is asymmetric routing, because request and answer go different ways.

In case of routing, this is not a problem. But if R1 is a firewall, this firewall might be stubborn, because it does not see the whole traffic.

Bypass the stateful firewall

I recently had such a setup due to some technical debts. The firewall dropped that “Invalid Traffic”. Fortunately, there is a way to bypass the statefull firewall. You can create advanced firewall rules using the CLI. There is no way to create these rules using the GUI. And this only applies to the Sophos XG (former Cyberoam products).

Login to the device console and select option 4. Then enter on the console the following commands, one per destination:

Make sure that you have a static or dynamically learned route to the networks. This is not a routing entry, it only tells the firewall what traffic should bypass the stateful firewall.