Tag Archives: security

CloudFlare API v4 and Fail2ban: Fixing the unban action

In January 2017, I wrote an article about how to protect your WordPress blog using the WP Fail2Ban plugin, fail2ban on your Linux/ FreeBSD host, and CloudFlare. Back then, the fail2ban was using the CloudFlare API V1, which was already deprecated since November 2016.

Free-Photos/ pixabay.com/ Creative Commons CC0

Although the actions were updated later to use CloudFlare API V4, I still had problems with the unbaning of IP addresses. IP addresses were banned, but the unban action failed. 

This is the unban action, which is included in fail2ban (taken from fail2ban-0.10.3.1 which is shipped with FreeBSD 11.1-RELEASE-p10):

And this is the unban action, which finally solved this issue:

I found the solution at serverfault.com. The only difference is an additional tr -d '\n'  in the last line of the statement. Kudos to Jake for fixing this!

To prevent the action file to being overwritten, you should copy the original cloudflare.conf  located in the  action.d  directory, e.g. to mycloudflare.conf , and use the copied action file in your fail definition.

Windows Network Policy Server (NPS) server won’t log failed login attempts

This is just a short, but interesting blog post. When you have to troubleshoot authentication failures in a network that uses Windows Network Policy Server (NPS), the Windows event log is absolutely indispensable. The event log offers everything you need. The success and failure event log entries include all necessary information to get you back on track. If failure events would be logged…

geralt/ pixabay.com/ Creative Commons CC0

Today, I was playing with Alcatel-Lucent Enterprise OmniSwitches and Access Guardian in my lab. Access Guardian refers to the some OmniSwitch security functions that work together to provide a dynamic, proactive network security solution:

  • Universal Network Profile (UNP)
  • Authentication, Authorization, and Accounting (AAA)
  • Bring Your Own Device (BYOD)
  • Captive Portal
  • Quarantine Manager and Remediation (QMR)

I have planned to publish some blog posts about Access Guardian in the future, because it is a pretty interesting topic. So stay tuned. :)

802.1x was no big deal, mac-based authentication failed. Okay, let’s take a look into the event log of the NPS… okay, there are the success events for my 802.1x authentication… but where are the failed login attempts? Not a single one was logged. A short Google search showed me the right direction.

Failed logon/ logoff events were not logged

In this case, the NPS role was installed on a Windows Server 2016 domain controller. And it was a german installation, so the output of the commands is also in german. If you have an OS installed in english, you must replace “Netzwerkrichtlinienserver” with “Network Policy Server”.

Right-click the PowerShell Icon and open it as Administrator. Check the current settings:

As you can see, only successful logon and logoff events were logged.

The option /success:enable /failure:enable activeates the logging of successful and failed logon and logoff attempts.

Replace SSL certificates on Citrix NetScaler using the CLI

Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.

skylarvision/ pixabay.com/ Creative Commons CC0

When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.

Update or replace?

There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate  includes the unbinding of the old, and binding the new certificate.

Replacing a certificate

The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.

Do yourself a favor and add the expiration date to the name of the certificate/ key pair.

Now you can unbind the old, and bind the new certificate. Please note, that this causes a short outage of your service!

SSL Cert Unbind Causing NetScaler Crash

You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.

Bypass stateful firewall on a Sophos XG

Usually, bypassing a firewall is not the best idea. But sometimes you have to. One case, where you want to bypass a firewall, is asymmetric routing.

MichaelGaida/ pixabay.com/ Creative Commons CC0

What is asymmetric routing? Imagine a scenario with two routers on the same network. One router offeres access to the internet, the other router provides access to other sites with site-2-site VPN tunnels.

Asymmetric Routing

Host 1 uses R1 as default gateway. R1 has static routes configured to the networks reachable over the VPN, or it has learned them dynamically using a routing protocol from R2. A packet from host 1 arrives at R1, is routed to R2, and is sent over the VPN tunnel. The answer to this packet arrives at R2, and is sent directly to host 1, because host 1 is the destination. This works because R2 and host 1 are on the same network. This is asymmetric routing, because request and answer go different ways.

In case of routing, this is not a problem. But if R1 is a firewall, this firewall might be stubborn, because it does not see the whole traffic.

Bypass the stateful firewall

I recently had such a setup due to some technical debts. The firewall dropped that “Invalid Traffic”. Fortunately, there is a way to bypass the statefull firewall. You can create advanced firewall rules using the CLI. There is no way to create these rules using the GUI. And this only applies to the Sophos XG (former Cyberoam products).

Login to the device console and select option 4. Then enter on the console the following commands, one per destination:

Make sure that you have a static or dynamically learned route to the networks. This is not a routing entry, it only tells the firewall what traffic should bypass the stateful firewall.

DOT1X authentication failed on HPE OfficeConnect 1920 switches

The last two days, I have supported a customer during the implementation of 802.1x. His network consisted of HPE/ Aruba and some HPE Comware switches. Two RADIUS server with appropriate policies was already in place. The configuration and test with the ProVision based switches was pretty simple. The Comware based switches, in this case OfficeConnect 1920, made me more headache.

blickpixel/ pixabay.com/ Creative Commons CC0

The customer had already mac authentication running, so all I had to do, was to enable 802.1x on the desired ports of the OfficeConnect 1920. The laptop, which I used to test the connection, was already configured and worked flawless if I plugged it into a 802.1x enabled port on a ProVision based switch. The OfficeConnect 1920 simply wrote a failure to its log and the authentication failed. The RADIUS server does not logged any failure, so I was quite sure, that the switch caused the problem.

After double-checking all settings using the web interface of the switch, I used the CLI to check some more settings. Unfortunately, the OfficeConnect 1920 is a smart-managed switch and provides only a very, very limited CLI. Fortunately, there is a developer access, enabling the full Comware CLI. You can enable the full CLI by entering

after logging into the limited CLI. You can find the password using your favorite internet search engine. ;)

Solution

While poking around in the CLI, I stumbled over this option, which is entered in the interface context:

RADIUS is the authentication domain, which was used on this switch. The command specifies, that the authentication domain RADIUS has to be for 802.1x authentication requests. Otherwise the switch would use the default authentication domain SYSTEM, which causes, that the switch tries to authenticate the user against the local user database.

I have not found any way to specify this setting using the web GUI! If you know how, of if you can provide additional information about this “issue”, please leave a comment.

Security: If it doesn’t hurt, you’re doing it wrong!

The Informationsverbund Berlin-Bonn (IVBB), the secure network of the german government , was breached by an unknown hacker group. Okay, a secure government network might be a worthy target for an attack, but your network not, right? Do you use the same password for multiple accounts? There were multiple massive data breaches in the past. Have you ever checked if your data were also compromised? I can recommend haveibeenpwned.com. If you want to have some fun, scan GitHub for -----BEGIN RSA PRIVATE KEY-----. Do you use a full disk encryption on your laptop or PC? Do you sign and/ or encrypt emails using S/MIME or PGP? Do you use different passwords for different services? Do you use 2FA/ MFA to secury importan services? Do you never work with admin privileges when doing normal office tasks? No? Why? Because it’s uncomfortable to do it right, isn’t it?

My focus is on infrastructure, and I’m trying to educate my customers that hey have to take care about security. It’s not the missing dedicated management network, or the usage of self signed certificates that makes an infrastructre unsecure. Mostly it’s the missing user management, the same password for different admin users, doing office work with admin privileges, or missing security patches because of “never touch a running system”, or “don’t ruin my uptime”. I don’t khow how often I heard the story of ransomware attacks, that were caused by admins opening email attachments with admin privileges…

My theory

Security must approach infinitely near the point, where it becomes unusable.

Security is nothing you can take care about later. It has to be part of the design. It has to be part of the processes. Most security incidents doesn’t happen because of 0-day exploits. It’s because of default passwords for admin accounts, missing security patches, and because of lazy admins or developers.

Don’t be lazy. Do it right. Even if it’s uncomfortable.

NetScaler native OTP does not work for users with many group memberships

Some days ago, I have implemented one-time passwords (OTP) for NetScaler Gateway for one of my customers. This feature was added with NetScaler 12, and it’s a great way to secure NetScaler Gateway with a native NetScaler feature. Native OTP does not need any third party servers. But you need a NetScaler Enterprise license, because nFactor Authentication is a requirement.

To setup NetScaler native OTP, I followed the availbe guides on the internet.

The setup is pretty straightforward. But I used the AD extensionAttribute15  instead of userParameters, because my customer already used userParameters  for something else. Because of this, I had to change the search filter from  userParameters>=#@  to extensionAttribute15>=#@ .

Everything worked as expected… except for some users, that could not register their devices properly. They were able to register their device, but a test of the OTP failed. After logoff and logon, the registered device were not available anymore. But the device was added to the extensionAttribute. While I was watching the nsvpn.log with tail -f , I discovered that the built group string for $USERNAME  seemed to be cut off (receive_ldap_user_search_event). My first guess was, that the user has too many group memberships, and indeed, the users was member for > 50 groups. So I copied the user, and the copied user had the same problem. I removed the copied user from some groups, and at some point the test of the OTP worked (on the /manageotp website).

With this information, I quickly stumbled over this thread: netscaler OTP not woring for certain users. This was EXACTLY what I discovered. The advised solution was to change the “Group Attribute” from memberOf  to userParameter , or in my case, extensionAttribute15. Problem solved!

Meltdown & Spectre: What about Microsoft Exchange?

On January 18, 2018, Microsoft has published KB4074871 which has the title “Exchange Server guidance to protect against speculative execution side-channel vulnerabilities”. As you might guess, Exchange is affected by Meltdown & Spectre – like any other software. Microsoft explains in KB4074871:

Because these are hardware-level attacks that target x64-based and x86-based processor systems, all supported versions of Microsoft Exchange Server are affected by this issue.

Like Citrix, Microsoft does not offer any updates to address this issue, because there is nothing to fix in Microsoft Exchange. Instead of this, Microsoft recommends to run the lates Exchange Server cumulative update and any required security updates. On top, Microsoft recommends to check software before it is deployed into production. If Exchange is running in a VM, Microsoft recommends to follow the instructions offered by the cloud or hypervisor vendor.

Meltdown & Spectre: What about HPE Storage and Citrix NetScaler?

In addition to my shortcut blog post about Meltdown and Spectre with regard of Microsoft Windows, VMware ESXi and vCenter, and HPE ProLiant, I would like to add some additional information about HPE Storage and Citrix NetScaler.

When we talk about Meltdown and Spectre, we are talking about three different vulnerabilities:

  • CVE-2017-5715 (branch target injection)
  • CVE-2017-5753 (bounds check bypass)
  • CVE-2017-5754 (rogue data cache load)

CVE-2017-5715 and CVE-2017-5753 are known as “Spectre”, CVE-2017-5754 is known as “Meltdown”. If you want to read more about these vulnerabilities, please visit meltdownattack.com.

Due to the fact that different CPU platforms are affected, one might can guess that also  other devices, like storage systems or load balancers, are affected. Because of my focus, this blog post will focus on HPE Storage and Citrix NetScaler.

HPE Storage

HPE has published a searchable and continously updated list with products, that might be affected (Side Channel Analysis Method allows information disclosure in Microprocessors). Interesting is, that a product can be affected, but not vulnerable.

ProductImpactedComment
Nimble StorageYesFix under investigation
StoreOnceYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR StoreServYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR Service ProcessorYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR File ControllerYESVulnerable- further information forthcoming.
MSAYESNot vulnerable – Product doesn’t allow arbitrary code execution.
StoreVirtualYESNot vulnerable – Product doesn’t allow arbitrary code execution.
StoreVirtual File ControllerYESVulnerable- further information forthcoming.

The File Controller are vulnerable, because they are based on Windows Server.

So if you are running 3PAR StoreServ, MSA, StoreOnce or StoreVirtual: Relax! If you are running Nimble Storage, wait for a fix.

Citrix NetScaler

Citrix has also published an article with information about their products (Citrix Security Updates for CVE-2017-5715, CVE-2017-5753, CVE-2017-5754).

The article is a bit spongy in its statements:

Citrix NetScaler (MPX/VPX): Citrix believes that currently supported versions of Citrix NetScaler MPX and VPX are not impacted by the presently known variants of these issues.

Citrix believes… So nothing to do yet, if you are running MPX or VPX appliances. But future updates might come.

The case is a bit different, when it comes to the NetScaler SDX appliances.

Citrix NetScaler SDX: Citrix believes that currently supported versions of Citrix NetScaler SDX are not at risk from malicious network traffic. However, in light of these issues, Citrix strongly recommends that customers only deploy NetScaler instances on Citrix NetScaler SDX where the NetScaler admins are trusted.

No fix so far, only a recommendation to check your processes and admins.

The Meltdown/ Spectre shortcut blogpost for Windows, VMware and HPE

TL;DR

Jump to

I will try to update this blog post regularly!

Change History

01-13-2018: Added information regarding VMSA-2018-0004
01-13-2018: HPE has pulled Gen8 and Gen9 system ROMs
01-13-2018: VMware has updated KB52345 due to issues with Intel microcode updates
01-18-2018: Updated VMware section
01-24-2018: Updated HPE section
01-28-2018: Updated Windows Client and Server section
02-08-2018: Updated VMware and HPE section
02-20-2018: Updated HPE section
04-17-2018: Updated HPE section


Many blog posts have been written about the two biggest security vulnerabilities discovered so far. In fact, we are talking about three different vulnerabilities:

  • CVE-2017-5715 (branch target injection)
  • CVE-2017-5753 (bounds check bypass)
  • CVE-2017-5754 (rogue data cache load)

CVE-2017-5715 and CVE-2017-5753 are known as “Spectre”, CVE-2017-5754 is known as “Meltdown”. If you want to read more about these vulnerabilities, please visit meltdownattack.com.

Multiple steps are necessary to be protected, and all necessary information are often repeated, but were distributed over several websites, vendor websites, articles, blog posts or security announcements.

Two simple steps

Two (simple) steps are necessary to be protected against these vulnerabilities:

  1. Apply operating system updates
  2. Update the microcode (BIOS) of your server/ workstation/ laptop

If you use a hypervisor to virtualize guest operating systems, then you have to update your hypervisor as well. Just treat it like an ordinary operating system.

Sounds pretty simple, but it’s not. I will focus on three vendors in this blog post:

  • Microsoft
  • VMware
  • HPE

Let’s start with Microsoft. Microsoft has published the security advisory ADV180002  on 01/03/2018.

Microsoft Windows (Client)

The necessary security updates are available for Windows 7 (SP1), Windows 8.1, and Windows 10. The January 2018 security updates are ONLY offered in one of theses cases (Source: Microsoft):

  • An supported anti-virus application is installed
  • Windows Defender Antivirus, System Center Endpoint Protection, or Microsoft Security Essentials is installed
  • A registry key was added manually

To add this registry key, please execute this in an elevated CMD. Do not add this registry key, if you are running an unsupported antivirus application!! Please contact your antivirus application vendor! This key has to be added manually, only in case if NO antivirus application is installed, otherwise your antivirus application will add it!

OSUpdate
Windows 10 (1709)KB4056892
Windows 10 (1703)KB4056891
Windows 10 (1607)KB4056890
Windows 10 (1511)KB4056888
Windows 10 (initial)KB4056893
Windows 8.1KB4056898
Windows 7 SP1KB4056897

Please note, that you also need a microcode update! Reach out to your vendor. I was offered automatically to update the microcode on my Lenovo ThinkPad X250.

Update 01-28-2018

Microsoft has published an update to disable mitigation against Spectre (variant 2) (Source: Microsoft). KB4078130 is available for Windows 7 SP1, Windows 8.1 and Windows 10, and it disables the mitigation against Spectre Variant 2 (CVE 2017-5715) independently via registry setting changes. The registry changed are described in KB4073119.

A reboot is required to disable the mitigation.

Windows Server

The necessary security updates are available for Windows Server 2008 R2, Windows Server 2012 R2, Windows Server 2016 and Windows Server Core (1709). The security updates are NOT available for Windows Server 2008 and Server 2012!. The January 2018 security updates are ONLY offered in one of theses cases (Source: Microsoft):

  • An supported anti-virus application is installed
  • Windows Defender Antivirus, System Center Endpoint Protection, or Microsoft Security Essentials is installed
  • A registry key was added manually

To add this registry key, please execute this in an elevated CMD. Do not add this registry key, if you are running an unsupported antivirus application!! Please contact your antivirus application vendor! This key has to be added manually, only in case if NO antivirus application is installed, otherwise your antivirus application will add it!

OSUpdate
Windows Server, version 1709 (Server Core Installation)KB4056892
Windows Server 2016KB4056890
Windows Server 2012 R2KB4056898
Windows Server 2008 R2KB4056897

After applying the security update, you have to enable the protection mechanism. This is different to Windows Windows 7, 8.1 or 10! To enable the protection mechanism, you have to add three registry keys:

The easiest way to distribute these registry keys is a Group Policy. In addition to that, you need a microcode update from your server vendor.

Update 01-28-2018

The published update for Windows 7 SP1, 8.1 and 10 (KB4073119) is not available for Windows Server. But the same registry keys apply to Windows Server, so it is sufficient to change the already set registry keys to disable the mitigation against Spectre Variant 2 (CVE 2017-5715).

A reboot is required to disable the mitigation.

VMware vSphere

VMware has published three important VMware Security Advisories (VMSA):

VMware Workstation Pro, Player, Fusion, Fusion Pro, and ESXi are affected by CVE-2017-5753 and CVE-2017-5715. VMware products seems to be not affected by CVE-2017-5754. On 09/01/2017, VMware has published VMSA-2018-0004, which also addresses CVE-2017-5715. Just to make this clear:

  • Hypervisor-Specific Remediation (documented in VMSA-2018-0002.2)
  • Hypervisor-Assisted Guest Remediation (documented in VMSA-2018-0004)

I will focus von vCenter and ESXi. In case of VMSA-2018-002, security updates are available for ESXi 5.5, 6.0 and 6.5. In case of VMSA-2018-0004, security updates are available for ESXi 5.5, 6.0, 6.5, and vCenter 5.5, 6.0 and 6.5. VMSA-2018-0007 covers VMware Virtual Appliance updates against side-channel analysis due to speculative execution.

Before you apply any security updates, please make sure that you read this:

  • Deploy the updated version of vCenter listed in the table (only if vCenter is used).
  • Deploy the ESXi security updates listed in the table.
  • Ensure that your VMs are using Hardware Version 9 or higher. For best performance, Hardware Version 11 or higher is recommended.

For more information about Hardware versions, read VMware KB article 1010675.

VMSA-2018-0002.2

OSUpdate
ESXi 6.5ESXi650-201712101-SG
ESXi 6.0ESXi600-201711101-SG
ESXi 5.5ESXi550-201709101-SG

In case of ESXi550-201709101-SG it is important to know, that this patch mitigates CVE-2017-5715, but not CVE-2017-5753! Please see KB52345 for important information on ESXi microcode patches.

VMSA-2018-0004

OSUpdate
ESXi 6.5ESXi650-201801401-BG, and
ESXi650-201801402-BG
ESXi 6.0ESXi600-201801401-BG, and
ESXi600-201801402-BG
ESXi 5.5ESXi550-201801401-BG
vCenter 6.56.5 U1e
vCenter 6.06.0 U3d
vCenter 5.55.5 U3g

The patches ESXi650-201801402-BG, ESXi 6.0 ESXi600-201801401-BG, and
ESXi550-201801401-BG will patch the microcode for supported CPUs. And this is pretty interesting! To enable hardware support for branch target mitigation (CVE-2017-5715 aka Spectre) in vSphere, three steps are necessary (Source: VMware):

  • Update to one of the above listed vCenter releases
  • Update the ESXi 5.5, 6.0 or 6.5 with
    • ESXi650-201801401-BG
    • ESXi600-201801401-BG
    • ESXi550-201801401-BG
  • Apply microcode updates from your server vendor, OR apply these patches for ESXi
    • ESXi650-201801402-BG
    • ESXi600-201801402-BG
    • ESXi550-201801401-BG

In case of ESXi 5.5, the hypervisor and microcode updates are delivered in a single update (ESXi550-201801401-BG).

Update 01-13-2018

Please take a look into KB52345 if you are using Intel Haswell and Broadwell CPUs! The KB article includes a table with affected CPUs.

All you have to do is:

  • Update your vCenter to the latest update release, then
  • Update your ESXi hosts with all available security updates
  • Apply the necessary guest OS security updats and enable the protection (Windows Server)

For the required security updates:

Make sure that you also apply microcode updates from your server vendor!

VMSA-2018-0007

This VMSA, published on 08/02/2018, covers several VMware Virtual appliances. Relevant appliances are:

  • vCloud Usage Meter (UM)
  • Identity Manager (vIDM)
  • vSphere Data Protection (VDP)
  • vSphere Integrated Containers (VIC), and
  • vRealize Automation (vRA)
ProductPatch pending?Mitigation/ Workaround
UM 3.xyesKB52467
vIDM 2.x and 3.xyesKB52284
VDP 6.xyesNONE
VIC 1.xUpdate to 1.3.1
vRA 6.xyesKB52497
vRA 7.xyesKB52377

 

HPE ProLiant

HPE has published a customer bulletin (document ID a00039267en_us) with all necessary information:

HPE ProLiant, Moonshot and Synergy Servers – Side Channel Analysis Method Allows Improper Information Disclosure in Microprocessors (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754)

CVE-2017-5715 requires that the System ROM be updated and a vendor supplied operating system update be applied as well. For CVE-2017-5753, CVE-2017-5754 require only updates of a vendor supplied operating system.

Update 01-13-2018

The following System ROMs were previously available but have since been removed from the HPE Support Site due to the issues Intel reported with the microcode updates included in them. Updated revisions of the System ROMs for these platforms will be made available after Intel provides updated microcodes with a resolution for these issues.

Update 01-24-2018

HPE will be releasing updated System ROMs for ProLiant and Synergy Gen10, Gen9, and Gen8 servers including updated microcodes that, along with an OS update, mitigate Variant 2 (Spectre) of this issue. Note that processor vendors have NOT released updated microcodes for numerous processors which gates HPE’s ability to release updated System ROMs.

I will update this blog post as soon as HPE releases new system ROMs.

For most Gen9 and Gen10 models, updated system ROMs are already available. Check the bulletin for the current list of servers, for which updated system ROMs are available. Please note that you don’t need a valid support contract to download this updates!

Under Software Type, select “BIOS-(Entitlement Required”) – (Note that Entitlement is NOT required to download these firmware versions.

Update 02-09-2018

Nothing new. HPE has updates the bulletin on 31-01-2018 with an updated timeline for new system ROMs.

Update 02-25-2018

HPE hast published Gen10 system ROMs. Check the advisory: HPE ProLiant, Moonshot and Synergy Servers – Side Channel Analysis Method Allows Improper Information Disclosure in Microprocessors (CVE-2017-5715, CVE-2017-5753, CVE-2017-5754).

Update 04-17-2018

HPE finally published updated System ROMS for several Gen10, Gen9, Gen8, G7 and even G6 models, which also includes bread-and-butter servers like the ProLiant DL360 G6 to Gen10, and DL380 G6 to Gen10.

If you are running Windows on your ProLiant, you can use the online ROM flash component for Windows x64. If you are running VMware ESXi, you can use the systems ROMPaq firmware upgrade for for USB key media.