Tag Archives: netscaler

Load balancing ADFS and ADFS Proxy using Citrix ADC

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last week I had to setup a small Active Directory Federation Services (ADFS) farm that will be used to allow Single Sign-On (SSO) with Office 365.

Active Directory Federation Services (ADFS) is a solution developed by Microsoft to provide users an authenticated access to applications, that are not capable of using Integrated Windows Authentication (IWA).

Required by the customer was a two node ADFS farm located on the internal network, and a two node ADFS Proxy farm located at the DMZ.

An ADFS Proxyserver acts as a reverse proxy and it is typically located in your organizations perimeter network (DMZ).

This picture shows a typical ADFS/ ADFS Proxy setup:

ADFS/ WAP Design/ Citrix/ citrix.com

My customer has decided to use Citrix ADC (former NetScaler) to load balance the requests for the ADFS farm and the ADFS Proxy farm. In addition to load balancing, this offers high availability in case of a failed ADFS server or ADFS Proxy server. Please note that Citrix ADC can act as a ADFS Proxy, but this requires the Advanced Edition license. My customer “only” had a Standard License, so we had to setup dedicated ADFS Proxy servers on the DMZ network.

Citrix ADC setup

The ADFS service name is typically something like adfs.customer.tld. This farm name has to be the same for internal and external access. For internal access, the ADFS service name must be resolved to the VIP of the Citrix ADC. The same applies to external accesss. So you have to setup split DNS.

ADFS uses HTTP and HTTP, so my first attempt was to use this Citrix ADC Content Switch based setup:

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add cs vserver cs_vsrv_adfs SSL x.x.x.x 443 -cltTimeout 180 -caseSensitive OFF
add lb vserver lb_vsrv_adfs SSL 0.0.0.0 0 -persistenceType SSLSESSION -cltTimeout 180

add cs action cs_action_adfs -targetLBVserver lb_vsrv_adfs
add cs policy cs_pol_adfs -rule "HTTP.REQ.URL.SET_TEXT_MODE(IGNORECASE).CONTAINS(\"adfs.customer.tld\")" -action cs_action_adfs
bind cs vserver cs_vsrv_adfsL -policyName cs_pol_adfs -priority 100

add serviceGroup svcgrp_adfs SSL -maxClient 0 -maxReq 0 -cip ENABLED X-MS-Forwarded-Client-IP -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP YES -appflowLog DISABLED

add lb monitor mon_adfs HTTP-ECV -send "GET /federationmetadata/2007-06/federationmetadata.xml" -recv "adfs.customer.tld/adfs/services/trust" -LRTM ENABLED -secure YES

bind serviceGroup svcgrp_adfs srv_gk-adfs1 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs srv_gk-adfs2 443 -CustomServerID "\"None\""
bind serviceGroup svcgrp_adfs -monitorName mon_adfs

bind lb vserver lb_vsrv_adfs svcgrp_adfs

bind ssl vserver lb_vsrv_adfs -certkeyName cert-key-pair
bind ssl vserver cs_vsrv_adfs -certkeyName cert-key-pair

set ssl vserver lb_vsrv_adfs -ssl3 DISABLED
set ssl vserver cs_vsrv_adfs -ssl3 DISABLED

This is a pretty common setup for HTTP/ HTTPS based services. But it doesn’t work… Mainly because the monitor was not getting the required response. So the monitored service was down for the ADC, and therefore the service group, the load balancing virtual server and the content switch won’t came up.

The reason for this is Server Name Indication (SNI), an extension to Transport Layer Security (TLS). SNI is enabled and required since ADFS 3.0. The monitor tries to access the URL http://x.x.x.x/federationmetadata/2007-06/federationmetadata.xml, but the ADFS service won’t answer to those requests, because it includes the ip address, and not the ADFS service name.

But there is a workaround for everything on the Internet! You can change the binding on the ADFS server nodes using netsh.

netsh http add sslcert ipport=<IPAddress:port> certhash=<certhash> appid=<appid> certstorename=MY

I will not add the necessary options to this command, because: DON’T DO THIS!

Yes, the service group, the load balancing virtual server and the content switch will come up after this change. But you will not be able to enable a trust between your ADFS Proxy servers and the ADFS farm.

Microsofts requirements on Load Balancing ADFS

Microsoft offers a nice overview about the requirements when deploying ADFS. There is a section about the Network requirements. Below this, Microsoft clearly documents the requirements when load balancing ADFS servers and ADFS Proxy servers.

The load balancer MUST NOT terminate SSL. AD FS supports multiple use cases with certificate authentication which will break when terminating SSL. Terminating SSL at the load balancer is not supported for any use case.

Requirements for deploying AD FS/ microsoft.com

Okay, with this in mind, the you can’t use a ADC Content Switch as described above. Because it will terminate SSL. You have to switch to a load balancing virtual server and a service group with SSL bridge . Citrix describes SSL bridge as follows:

A SSL bridge configured on the NetScaler appliance enables the appliance to bridge all secure traffic between the SSL client and the SSL server. The appliance does not offload or accelerate the bridged traffic, nor does it perform encryption or decryption. Only load balancing is done by the appliance. The SSL server must handle all SSL-related processing. Features such as content switching, SureConnect, and cache redirection do not work, because the traffic passing through the appliance is encrypted.

But there is a second, very interesting statement:

It is recommended to use the HTTP (not HTTPS) health probe endpoints to perform load balancer health checks for routing traffic. This avoids any issues relating to SNI. The response to these probe endpoints is an HTTP 200 OK and is served locally with no dependence on back-end services. The HTTP probe can be accessed over HTTP using the path ‘/adfs/probe’http://<Web Application Proxy name>/adfs/probe
http://<ADFS server name>/adfs/probe
http://<Web Application Proxy IP address>/adfs/probe
http://<ADFS IP address>/adfs/probe

Requirements for deploying AD FS/ microsoft.com

This is pretty interesting, because it addresses the above described issue with the monitor. The solution to this is a HTTP-ECV monitor with on port 80, a GET to “/adfs/probe” and the check for a HTTP/200.

A working Citrix ADC setup

This setup is divided into two parts: One for the ADFS farm, and a second one for the ADFS Proxy farm. It uses SSL bridge and HTTP for the service monitor.

Load balancing the ADFS farm

add server srv_adfs1 x.x.x.x
add server srv_adfs2 x.x.x.y

add serviceGroup svcgrp_adfs SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfs SSL_BRIDGE x.x.x.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfs srv_adfs1 443
bind serviceGroup svcgrp_adfs srv_adfs2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_http
bind lb vserver lb_vsrv_adfs svcgrp_adfs
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

Load balancing the ADFS Proxy farm

add server srv_adfsproxy1 y.y.y.y
add server srv_adfsproxy2 y.y.y.x

add serviceGroup svcgrp_adfsproxy SSL_BRIDGE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport YES -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO
add lb vserver lb_vsrv_adfsproxy SSL_BRIDGE y.y.y.z 443 -persistenceType SSLSESSION -cltTimeout 180
add lb monitor mon_adfs_proxy_http HTTP -respCode 200 -httpRequest "GET /adfs/probe" -LRTM ENABLED -destPort 80

bind serviceGroup svcgrp_adfsproxy srv_adfsproxy1 443
bind serviceGroup svcgrp_adfsproxy srv_adfsproxy2 443
bind serviceGroup svcgrp_adfs -monitorName mon_adfs_proxy_http
bind lb vserver lb_vsrv_adfsproxy svcgrp_adfsproxy
set ssl vserver lb_vsrv_adfsproxy -ssl3 DISABLED

I have implemented it on a NetScaler 12.1 with a Standard license. If you have feedback or questions, please leave a comment. :)

NetScaler Gateway – Cannot complete your request

This posting is ~4 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

A customer reported a weird problem with his NetScaler Gateway. Upon the first load of the website, they got an error “Cannot complete your request”. After clicking OK the error disappeared and does not occured again after reloading the website. Only after closing and re-opening the browser. I got this message in Firefox and Internet Explorer, but not from a remote machine, e.g. my PC at the office.

I found no configuration error or something, that would have explained this message. Finally, I found something that caught my attention:

HTTP/1.1 412 Precondition Failed

I found this using the Firefox Web Development Tools (I only had a Firefox and IE on my remote machine). With this message I found CTX244520 which also explained this error. The issue is caused by a hidden feature for caching web site data of the Gateway vServer. If you don’t have Integrated Cache feature licensed or enable, this feature failes. It is called Static Page Caching.

My customer is currently running NS12.0 60.10, and this issue is fixed in 12.0 61.8. And the customer is using a custom theme, which is based on one of the included themes.

If possible you can enable Integrated Caching. If you can’t enable Integrated Caching, you can simple disable this feature:

   show aaa parameter
   Configured AAA parameters
           EnableStaticPageCaching: YES
           EnableEnhancedAuthFeedback: NO
           DefaultAuthType: LOCAL  MaxAAAUsers: 1000
           AAAD nat ip: None
           EnableSessionStickiness : NO
           aaaSessionLoglevel : INFORMATIONAL
           AAAD Log Level : INFORMATIONAL
           Dynamic address: OFF
           GUI mode: ON
           Max Saml Deflate Size: 1024
    Done
   set aaa parameter -enableStaticPageCaching NO
    Done

Exam prep & experience: Citrix NetScaler Advanced Topics: Security, Management, and Optimization (1Y0-340)

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In May 2018, Citrix released their new Citrix Certified Expert – Networking certification, which completet the networking certification path at the upper end (blog post on training.citrix.com). The track starts with the Associate (CCA-N), the lower-level certification is a requirement for achieving the higher-level certification, continues with the Professional (CCP-N), and ends with the Expert (CCE-N) certification. This is pretty cool, and I’m very happy that Citrix now offers the CCE-N, because the expert-level certification was missing all the time.

kmicican/ pixabay.com/ Creative Commons CC0

Everything is cool… except you have passed exam 1Y0-351 to gain your CCP-N. In this case, you have to pass 1Y0-340 until Dec 31 2018. Otherwise you have to start with the CCA-N, after the validity period of your CCP-N is over (3y after passing the exam).

Bad move, Citrix, bad move. I’m really disappointed. I passed 1Y0-351 in Nov 2017, and now, 12 months later, I have to book, pay, and pass 1Y0-340 if I not want to start with a CCA-N in Nov 2020. Bad move, Citrix, bad move!

Exam 1Y0-340 is titled as “Citrix NetScaler Advanced Topics: Security, Management, and Optimization”, where as 1Y0-351 was titeld as “Citrix NetScaler 10.5 Essentials and Networking”. You can assume that more in-depth knowledge is needed to pass the exam, as it was necessary for 1Y0-351. Note the “Advanced Topics” in the exam title.

But what are these “advanced topics”?  According to the exam prep guide, the perfect candidate for the 1Y0-340 exam can deploy and/or manage

  • Citrix NetScaler Application Firewall (AppFirewall) to secure application access in a Citrix NetScaler 12 environment, as well as
  • NetScaler Management and Analytics System (NMAS) to administer a Citrix NetScaler environment, or
  • Optimize NetScaler-managed application delivery traffic

Citrix NetScaler Application Firewall (AppFirewall)

You should take an in-depth look at these topics:

  • Application Firewall Overview
  • Application Firewall Profiles and Policies
  • Regular Expression
  • Attacks and Protections
  • Monitoring and Troubleshooting
  • Security and Filtering

NetScaler Management and Analytics System (NMAS)

  • NetScaler MAS: Introduction and Configuration
  • Managing and Monitoring NetScaler Instances
  • Managing NetScaler Configurations
  • NetScaler Web Logging

Optimize NetScaler-managed application delivery traffic

  • Integrated Caching
  • Front-End Optimization
  • Tuning and Optimizations

How to prep?

The exam prep guide referres to the NetScaler documentation, as also to training material. Unfortunately I don’t have access to the newer training material, only to the training material from my CNS-220 course. But hey: At least we have tons of publically available NetScaler 12.0 documentation available!

The exam prep guide has a section in which Citrix outlines sections, objectives and references. You will find links to the NetScaler 12.0 documentation, as well as knowledge base articles, or blog posts. Go through it. Read it carefully!

The exam prep guide also outlines the section titles and weights. Two areas stand out:

  • Section 4: Attacks and Protections, and
  • Section 8: Managing and Monitoring NetScaler Instances

The section weights are directly map to the number of questions in the exam. If the exam has 60 questions, and section 4 has a weight of 21%, at least 12 questions will relate to “Attacks and Protections”.

How did it go?

First things first: I passed with a good score. The exam had 62 questions and I needed at least 62% to pass the exam. I passed with 82%. As a non-native English speaker that took the exam in a country where english is a foreign language, I got 30 minutes extra, resulting in 120 minutes for 62 questions. Plenty of time…

What should I say? It was a multiple choice test. Read the questions carefully. The exam guide did not lie to me. It came pretty close to the topics that were described in the guide. For most questions, my first “educated guess” was right. Sometimes, the least dumb answer seemed to be correct. ;)

It was a bit frustrating that Citrix has changed product names. NetScaler is no “Application Delivery Controller”, MAS is now known as “Citrix Application Delivery Management”. There was a button which showed a mapping table “old name – new name”.

If you are experienced with Citrix ADC deployments and configuration, I think the exam prep guide is enough to pass the exam.

Good luck!

High CPU usage on Citrix ADC VPX

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

While building a small Citrix NetScaler… ehm… ADC VPX (I really hate this name…) lab environment, I noticed that the fan of my Lenovo T480s was spinning up. I was wondering why, because the VPX VM was just running for a couple of minutes – without any load. But the task manager told me, that the VMware Workstation Process was consuming 25% (I have a Intel i5 Quad Core CPU) CPU. So VMware Workstation was just eating a whole CPU core without doing anything. I would not care, but the fan… And it reminded me, that I’ve seen an similar behaviour in various VPX deployments on VMWare ESXi.

Fifaliana/ pixabay.com/ Creative Commons CC0

A quick search lead me to this Citrix Support Knowledge Center article: High CPU Usage on NetScaler VPX Reported on VMware ESXi Version 6.0. That’s exactly what I’ve observed.

The solution is setting the parameter cpuyield  to yes.

> set ns vpxparam -cpuyield YES
 Done
> show ns runningConfig | grep "cpuyield"
set ns vpxparam -cpuyield YES
>

The VPX does not need a reboot. Short after setting the parameter, the fan stopped spinning. Have I mentioned how I love silence on my desk? I’m pretty happy that my T480s is a really quiet laptop.

But what does this parameter is used for? In pretty simple words: To allocate CPU cycles, that are not used by other VMs. Until ADC VPX 11.1, the VPX was sharing CPU with other VMs. This changed with ADC VPX 12.0. Since this release, the VPX was like a child, that was playing with their favorite toy just to make sure, that no other child can play with it. Not very polite…

This is a quote from the Support Knowledge Center article:

Set ns vpxparam parameters:
-cpuyield: Release or do not release of allocated but unused CPU resources.

YES: Allow allocated but unused CPU resources to be used by another VM.

NO: Reserve all CPU resources for the VM to which they have been allocated. This option shows higher percentage in hypervisor for VPX CPU usage.
DEFAULT: NO

I don’t think that I would change this in production. But for lab environments, especially if you run this on VMware Workstation, I would set -cpuyield  to yes .

Replace SSL certificates on Citrix NetScaler using the CLI

This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Sometimes you have to replace SSL certificates instead of updating them, e.g. if you switch from a web server SSL certificate to a wildcard certificate. The latter was my job today. In my case, the SSL certificate was used in a Microsoft Exchange 2016 deployment, and the NetScaler configuration was using multiple virtual servers. I’m using this little script for my NetScaler/ Exchange deployments.

skylarvision/ pixabay.com/ Creative Commons CC0

When using multiple virtual servers, replacing a SSL certificate using the GUI can be challenging, because you have to navigate multiple sites, click here, click there etc. Using the CLI, the same task is much easier und faster. I like the Lean mindset, so I’m trying to avoid “waste”, in this case, “waste of time”.

Update or replace?

There is a difference between updating or replacing of certificates. When using the same CSR and key as for the expired certificate, you can update the certificate. If you use a new certificate/ key pair, you have to replace it. Replacing a certificate  includes the unbinding of the old, and binding the new certificate.

Replacing a certificate

The new certificate usually comes as a PFX (PKCS#12) file. After importing it, you have to install (create) a new certificate/ key pair.

add ssl certKey wildcard_exp_20190712 -cert wildcard_exp_20190712.pem -key wildcard_exp_20190712.pem -passcrypt random -expiryMonitor ENABLED -notificationPeriod 90

Do yourself a favor and add the expiration date to the name of the certificate/ key pair.

Now you can unbind the old, and bind the new certificate. Please note, that this causes a short outage of your service!

unbind ssl vserver lb_vs_exchange_owa_https_443 -certkeyName exchange2016_exp_20180713
bind ssl vserver lb_vs_exchange_owa_https_443 -certkeyName wildcard_exp_20190712

SSL Cert Unbind Causing NetScaler Crash

You should check what NetScaler software release you are running. There is a bug, which is fixed in 12.0 build 57.X, which causes the NetScaler appliance to crash if a SSL certificate is unbound and a SSL transaction is running. Check CTX230965 for more details.

Demystifying “Interfaces on which heartbeats are not seen”

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

By accident, I found a heartbeat/ VLAN issue on a NetScaler cluster at one of my customers. The NetScaler ADC appliances have three interfaces connected to a switch stack. Two of the three interfaces were configured as a channel (LAG). This is a snippet from the config:

set channel LA/1 -tagall ON -throughput 0 -lrMinThroughput 0 -bandwidthHigh 0 -bandwidthNormal 0
...
bind vlan 10 -ifnum 1/3
bind vlan 10 -ifnum LA/1 -tagged
bind vlan 54 -ifnum LA/1 -tagged
bind vlan 55 -ifnum LA/1 -tagged

On the switch stack, the port to which interface 1/3 is connected, is configured as an access port. The ports, to which the channel is connected, is configured as a trunk port with some permitted VLANs. The customer is using HPE Comware based switches. The terminology is the same for Cisco. If you use HPE ProVision or Alcatel Lucent Enterprise, translate “access” to “untagged” and “trunk” to “tagged”. Because the channel is configured as a trunk port on the switch, the tagall option was set.

Issue

While examining the output of  show ha node I saw this:

Interfaces on which heartbeats are not seen : LA/1

Because interface 1/3 was not affected, this had to be a VLAN issue. During the initial troubleshooting, I was able to discover heartbeat packets in VLAN 1 and in VLAN 10.

Solution

The solution was easy: Remove the tagged option for VLAN 10 on LA/1.

bind vlan 10 -ifnum LA/1

instead of

bind vlan 10 -ifnum LA/1 -tagged

Because of the configured tagall  option, all packets sourced by LA/1 are tagged with the corrosponding VLAN ID. But because it’s now explicitly configured without a tag for VLAN 10, VLAN 10 is now also the native VLAN for LA/1.

> show channel

1)      Interface LA/1 (802.3ad Link Aggregate) #14
        flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, HAMON, HEARTBEAT, 802.1q, tagall>
        MTU=1500, native vlan=10, MAC=02:e0:ed:38:9d:d2, uptime 1362h58m51s

Now the NetScaler was sending heartbeat packets with a tag for VLAN 10, and the issue was solved.

Explanation

Heartbeat packets are always send without a VLAN tag (untagged). There are two exceptions:

  • The NSVLAN is configured with a specific VLAN ID, or
  • an interface used for hearbeats is configured with the tagall

In this case, the heartbeat packets are tagged with the ID of the native VLAN ID of the interface. A show interface of the channel showed, that the channel was using VLAN 1 as the native VLAN.

> show channel

1)      Interface LA/1 (802.3ad Link Aggregate) #14
        flags=0x4100c020 <ENABLED, UP, AGGREGATE, UP, HAMON, HEARTBEAT, 802.1q, tagall>
        MTU=1500, native vlan=1, MAC=02:e0:ed:38:9d:d2, uptime 1362h55m13s

How does the NetScaler determine the native VLAN for an interface? The native VLAN is the VLAN, to which an interface is bound untagged. An interface can only be bound untagged to a single VLAN. But it can be bound tagged to multiple VLANs.

If you take a look at the config snippet at the top of this blog post, you might notice, that interface 1/3 is bound untagged to VLAN 10. So this is the native VLAN for interface 1/3. But this interface is not using the tagall  option. Therefore, heartbeat packets are not tagged. The channel LA/1 is bound tagged to VLAN 10. But it was also bound to VLAN 1, without the tagged  option. This caused, that VLAN 1 was used as the native VLAN for channel LA/1. And because LA/1 is configured with the tagall  option, the heartbeats were tagged with a tag for VLAN 1. That’s why I was able to see the heartbeats, that were send over channel LA/1, in VLAN 1.

In the end, the NetScaler appliances were sending heartbearts from interface 1/3 to VLAN 10, and from channel LA/1 to VLAN 1. This caused the message “Interfaces on which heartbeats are not seen: LA/1”.

NetScaler native OTP does not work for users with many group memberships

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Some days ago, I have implemented one-time passwords (OTP) for NetScaler Gateway for one of my customers. This feature was added with NetScaler 12, and it’s a great way to secure NetScaler Gateway with a native NetScaler feature. Native OTP does not need any third party servers. But you need a NetScaler Enterprise license, because nFactor Authentication is a requirement.

To setup NetScaler native OTP, I followed the availbe guides on the internet.

The setup is pretty straightforward. But I used the AD extensionAttribute15  instead of userParameters, because my customer already used userParameters  for something else. Because of this, I had to change the search filter from userParameters>=#@  to extensionAttribute15>=#@ .

Everything worked as expected… except for some users, that could not register their devices properly. They were able to register their device, but a test of the OTP failed. After logoff and logon, the registered device were not available anymore. But the device was added to the extensionAttribute. While I was watching the nsvpn.log with tail -f , I discovered that the built group string for $USERNAME  seemed to be cut off (receive_ldap_user_search_event). My first guess was, that the user has too many group memberships, and indeed, the users was member for > 50 groups. So I copied the user, and the copied user had the same problem. I removed the copied user from some groups, and at some point the test of the OTP worked (on the /manageotp website).

With this information, I quickly stumbled over this thread: netscaler OTP not woring for certain users. This was EXACTLY what I discovered. The advised solution was to change the “Group Attribute” from memberOf  to userParameter , or in my case, extensionAttribute15. Problem solved!

Meltdown & Spectre: What about HPE Storage and Citrix NetScaler?

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

In addition to my shortcut blog post about Meltdown and Spectre with regard of Microsoft Windows, VMware ESXi and vCenter, and HPE ProLiant, I would like to add some additional information about HPE Storage and Citrix NetScaler.

When we talk about Meltdown and Spectre, we are talking about three different vulnerabilities:

  • CVE-2017-5715 (branch target injection)
  • CVE-2017-5753 (bounds check bypass)
  • CVE-2017-5754 (rogue data cache load)

CVE-2017-5715 and CVE-2017-5753 are known as “Spectre”, CVE-2017-5754 is known as “Meltdown”. If you want to read more about these vulnerabilities, please visit meltdownattack.com.

Due to the fact that different CPU platforms are affected, one might can guess that also  other devices, like storage systems or load balancers, are affected. Because of my focus, this blog post will focus on HPE Storage and Citrix NetScaler.

HPE Storage

HPE has published a searchable and continously updated list with products, that might be affected (Side Channel Analysis Method allows information disclosure in Microprocessors). Interesting is, that a product can be affected, but not vulnerable.

ProductImpactedComment
Nimble StorageYesFix under investigation
StoreOnceYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR StoreServYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR Service ProcessorYESNot vulnerable – Product doesn’t allow arbitrary code execution.
3PAR File ControllerYESVulnerable- further information forthcoming.
MSAYESNot vulnerable – Product doesn’t allow arbitrary code execution.
StoreVirtualYESNot vulnerable – Product doesn’t allow arbitrary code execution.
StoreVirtual File ControllerYESVulnerable- further information forthcoming.

The File Controller are vulnerable, because they are based on Windows Server.

So if you are running 3PAR StoreServ, MSA, StoreOnce or StoreVirtual: Relax! If you are running Nimble Storage, wait for a fix.

Citrix NetScaler

Citrix has also published an article with information about their products (Citrix Security Updates for CVE-2017-5715, CVE-2017-5753, CVE-2017-5754).

The article is a bit spongy in its statements:

Citrix NetScaler (MPX/VPX): Citrix believes that currently supported versions of Citrix NetScaler MPX and VPX are not impacted by the presently known variants of these issues.

Citrix believes… So nothing to do yet, if you are running MPX or VPX appliances. But future updates might come.

The case is a bit different, when it comes to the NetScaler SDX appliances.

Citrix NetScaler SDX: Citrix believes that currently supported versions of Citrix NetScaler SDX are not at risk from malicious network traffic. However, in light of these issues, Citrix strongly recommends that customers only deploy NetScaler instances on Citrix NetScaler SDX where the NetScaler admins are trusted.

No fix so far, only a recommendation to check your processes and admins.

Citrix NetScaler and Exchange: Case-sensitivity of internal and external URLs

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Exchange has known the concept of internal and external URLs for the different services (Outlook Web Access, OAB, EWS, ActiveSync etc) since Exchange 2007. And it’s still confusing people. The internal URL is the URL, that is used to access the desired service from the intranet. The external URL represents the URL that is used to access the service from the internet. Best practice is to use the same URL (the external) for both, use a certificate from a public CA, and use split DNS to access the external domain from the inside of your network.

People tend to imply, that URLs are not case-sensitive. This seems to be true in most cases. The World Wide Web Consortium (W3C) states:

URLs in general are case-sensitive (with the exception of machine names). There may be URLs, or parts of URLs, where case doesn’t matter, but identifying these may not be easy. Users should always consider that URLs are case-sensitive.

Source W3C

Citrix NetScaler and URLs

Citrix NetScaler handles URLs as case-sensitive.

A frequently used concept to load balance Microsoft Exchange with a NetScaler is Content Switching. Policies are used to identify traffic, and actions are used to take action on the traffic that matches the policies. The NetScaler uses the advanced policy engine to create expressions for the Content Switching Policies. When creating a Content Switching policy by creating an expression that uses the CONTAINS operator, you might notice that the results are case-sensitive.

This can be a problem in case of Microsoft Exchange, because /Autodiscover/Autodiscover.xml and /autodiscover/autodiscover.xml, or /ews/exchange.asmx and /EWS/Exchange.asmx are handled different.

Solution

To make sure that different cases are handled, you should add SET_TEXT_MODE(IGNORECASE)  to you policy expression. Citrix describes this in CTX115528.

I’ve changed my NetScaler setup script for Exchange to handle this behavior.

Citrix Certified Professional – Networking (CCP-N) exam experience

This posting is ~6 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.

Last friday I passed the 1Y0-351 (Citrix NetScaler 10.5 Essentails and Networking) exam with a pretty good score. The exam was necessary, not only because I will do much more NetScaler projects in the future, but also because Citrix has made it mandatory to have a CCP-N in your company to to sell Citrix NetScaler.

Preparation

My employer booked me a 5-day course (CNS-220 Citrix NetScaler Essentials and Traffic Management). Very nice, although I already had experience with NetScaler deployments. This training was designed for NetScaler 12.0, not for 10.5.

A training might be recommended to prepare for an exam, but usually it is not sufficient to pass it. But I want to pass the exam in the first try, so I took a closer look into the Citrix NetScaler 10.5 Essentials and Networking Preparation Guide.

In addition to the student and lab material, I deployed three NetScaler VPX (10.5,11.1 and 12.0) in my lab. I really recommend this! Especially to learn the CLI and how to read the log files.

The exam

S. Hofschlaeger / pixelio.de

The exam 1Y0-351 is focused on NetScaler 10.5, and will be not available after January 19, 2018. The sucessor of this exam is 1Y0-340, which is based on NetScaler 12.0. It is available since October 20, 2017. You might have noticed that my course was designed for 12.0, but I took the 10.5 exam. Well, I could not identify a question that would have had to be answered differently for NetScaler 12.0. But I really recommend to take the exam matching your course.

You have to answer 72 questions in 120 minutes. I got 30 minutes extra, because I’m a non-native english speaker. I had to answer two survey before the exam. One of them was a self-assessment about my NetScaler skills.

The questions were pretty fair, no trick questions, or questions were multiple answers seemed to be correct. The exam met the exam objectives from the prep guide. And because I already wrote it: You really should work with the CLI, and you really should know the important logs.

In sum: A challenging, but pretty fair exam. No marketing, no factual knowledge from spec sheets etc. When you are quite familiar with NetScalers, there is a good chance to pass the exam in the first attempt.