This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
The last two days, I have supported a customer during the implementation of 802.1x. His network consisted of HPE/ Aruba and some HPE Comware switches. Two RADIUS server with appropriate policies was already in place. The configuration and test with the ProVision based switches was pretty simple. The Comware based switches, in this case OfficeConnect 1920, made me more headache.
blickpixel/ pixabay.com/ Creative Commons CC0
The customer had already mac authentication running, so all I had to do, was to enable 802.1x on the desired ports of the OfficeConnect 1920. The laptop, which I used to test the connection, was already configured and worked flawless if I plugged it into a 802.1x enabled port on a ProVision based switch. The OfficeConnect 1920 simply wrote a failure to its log and the authentication failed. The RADIUS server does not logged any failure, so I was quite sure, that the switch caused the problem.
After double-checking all settings using the web interface of the switch, I used the CLI to check some more settings. Unfortunately, the OfficeConnect 1920 is a smart-managed switch and provides only a very, very limited CLI. Fortunately, there is a developer access, enabling the full Comware CLI. You can enable the full CLI by entering
_cmdline-mode on
after logging into the limited CLI. You can find the password using your favorite internet search engine. ;)
Solution
While poking around in the CLI, I stumbled over this option, which is entered in the interface context:
RADIUS is the authentication domain, which was used on this switch. The command specifies, that the authentication domain RADIUS has to be for 802.1x authentication requests. Otherwise the switch would use the default authentication domain SYSTEM, which causes, that the switch tries to authenticate the user against the local user database.
I have not found any way to specify this setting using the web GUI! If you know how, of if you can provide additional information about this “issue”, please leave a comment.
This posting is ~5 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
During the replacement of some VMware ESXi hosts at a customer, I discovered a recurrent failure of the vSphere Distributed Switch health checks. A VLAN and MTU mismatch was reported. On the physical side, the ESXi hosts were connected to two HPE 5820 switches, that were configured as an IRF stack. Inside the VMware bubble, the hosts were sharing a vSphere Distributed Switch.
The switch ports of the old ESXi hosts were configured as Hybrid ports. The switch ports of the new hosts were configured as Trunk ports, to streamline the switch and port configuration.
Some words about port types
Comware knows three different port types:
Access
Hybrid
Trunk
If you were familiar with Cisco, you will know Access and Trunk ports. If you were familiar with HPE ProCurve or Alcatel-Lucent Enterprise, these two port types refer to untagged and tagged ports.
So what is a Hybrid port? A Hybrid port can belong to multiple VLANs where they can be untagged and tagged. Yes, multiple untagged VLANs on a port are possible, but the switch will need additional information to bridge the traffic into correct untagged VLANs. This additional information can be MAC addresses, IP addresses, LLDP-MED etc. Typically, hybrid ports are used for in VoIP deployments.
The benefit of a Hybrid port is, that I can put the native VLAN of a specific port, which is often referred as Port VLAN identifier (PVID), as a tagged VLAN on that port. This configuration allows, that all dvPortGroups have a VLAN tag assigned, even if the VLAN tag represents the native VLAN of a switch port.
Failing health checks
A failed health check rises a vCenter alarm. In my case, a VLAN and MTU alarm was reported. In both cases, VLAN 1 was causing the error. According to VMware, the three main causes for failed health checks are:
Mismatched VLAN trunks between a vSphere distributed switch and physical switch
Mismatched MTU settings between physical network adapters, distributed switches, and physical switch ports
Mismatched virtual switch teaming policies for the physical switch port-channel settings.
Let’s take a look at the port configuration on the Comware switch:
#
interface Ten-GigabitEthernet1/0/9
port link-mode bridge
description "ESX-05 NIC1"
port link-type trunk
port trunk permit vlan all
stp edged-port enable
#
As you can see, this is a normal trunk port. All VLANs will be passed to the host. This is an except from the display interface Ten-GigabitEthernet1/0/9 output:
PVID: 1
Mdi type: auto
Port link-type: trunk
VLAN passing : 1(default vlan), 2-3, 5-7, 100-109
VLAN permitted: 1(default vlan), 2-4094
Trunk port encapsulation: IEEE 802.1q
The native VLAN is 1, this is the default configuration. Traffic, that is received and sent from a trunk port, is always tagged with a VLAN id of the originating VLAN – except traffic from the default (native) VLAN! This traffic is sent without a VLAN tag, and if frames were received with a VLAN tag, this frames will be dropped!
If you have a dvPortGroup for the default (native) VLAN, and this dvPortGroup is sending tagged frames, the frames will be dropped if you use a “standard” trunk port. And this is why the health check fails!
Ways to resolve this issue
In my case, the dvPortGroup was configured for VLAN 1, which is the default (native) VLAN on the switch ports.
There are two ways to solve this issue:
Remove the VLAN tag from the dvPortGroup configuration
Change the PVID for the trunk port
To change the PVID for a trunk port, you have to enter the following command in the interface context:
[ToR-Ten-GigabitEthernet1/0/9] port trunk pvid vlan 999
You have to change the PVID on all ESXi facing switch ports. You can use a non-existing VLAN ID for this.
vSphere Distributed Switch health check will switch to green for VLAN and MTU immediately.
Please note, that this is not the solution for all VLAN-related problems. You should make sure that you are not getting any side effects.
You have to route the traffic on the access switches.
Routing on the network access, the edge of the network, is not a question of performance. It is more of a management issue. Depending on the size of your network, and the number of subnets, you have to deal with lots of routes. And think about the effort, if you add, change or remove subnets from your network. This is not what you want to do with static routes. You need a routing protocol.
The experiment of the week
We have a core switch C1, consisting of two independent switches (C1-1 and C1-2) forming a virtual chassis. S1 and S2 are two switches at the network access. This is a core-edge design. There is no distribution layer. Each switch at the network access has two uplinks: One uplink to C1-1 and one uplink to C1-2. The ports on each end of the links are configured as routed ports.
Please ignore the 40 GbE ports (FGE) between C1-1 and C1-2. These ports are used for the Intelligent Resilient Framework (IRF), which is used to create a virtual chassis.
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
These are the interfaces on the core switch, that are working in route mode. GE1/0/1 and GE2/0/1 are the uplinks to S1, and GE1/0/2 and GE2/0/2 are the uplinks to S2.
[C1]display interface brief
Brief information on interface(s) under route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Main IP Description
GE1/0/1 UP UP 10.0.0.1
GE1/0/2 UP UP 10.10.0.1
GE2/0/1 UP UP 10.1.0.1
GE2/0/2 UP UP 10.11.0.1
InLoop0 UP UP(s) --
Loop0 UP UP(s) 1.1.1.1
MGE0/0/0 DOWN DOWN --
NULL0 UP UP(s) --
REG0 UP -- --
These are the interfaces on the access switch S1, that are working in route mode. GE1/0/1 and GE1/0/2 are the uplinks to C1. As you can see, GE1/0/1 on C1 and GE1/0/1 on S1 belong to the same /30 network. The same applies to GE2/0/1 on C1 and GE1/0/2 on S1. There are also two SVIs, one on VLAN 1 (192.168.1.0/24) and another on VLAN 2 (192.168.2.0/24). These VLANs are used for client connectivity.
[S1]display interface brief
Brief information on interface(s) under route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Main IP Description
GE1/0/1 UP UP 10.0.0.2
GE1/0/2 UP UP 10.1.0.2
InLoop0 UP UP(s) --
Loop0 UP UP(s) 1.1.1.2
MGE0/0/0 DOWN DOWN --
NULL0 UP UP(s) --
REG0 UP -- --
Vlan1 UP UP 192.168.1.1
Vlan2 UP UP 192.168.2.1
These are the interfaces on S2, that are working in route mode. GE1/0/1 and GE1/0/2 are the uplinks to C1. The interfaces GE1/0/2 on C1, and GE1/0/1 on S2 belong to the same /30 network. The same applies to GE2/0/2 on C1 and GE1/0/2 on S2. There are also two SVIs, one on VLAN 1 (192.168.10.0/24) and another on VLAN 2 (192.168.20.0/24).
You might wonder, because the same VLAN IDs are used on both access switches. They don’t care, because there is no layer 2 connectivity between these two switches. The only way from S1 to S2 is over the routed links to the core switch.
[S2]display interface brief
Brief information on interface(s) under route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Main IP Description
GE1/0/1 UP UP 10.10.0.2
GE1/0/2 UP UP 10.11.0.2
InLoop0 UP UP(s) --
Loop0 UP UP(s) 1.1.1.3
MGE0/0/0 DOWN DOWN --
NULL0 UP UP(s) --
REG0 UP -- --
Vlan1 UP UP 192.168.10.1
Vlan2 UP UP 192.168.20.1
Now let’s have a look at the Open Shortest Path First (OSPF) routing protocol.
Single Area OSPF
The Open Shortest Path First (OSPF) routing protocol is an interior gateway protocols (IGP), and also a link-state routing protocol. The calculation of the shortest path for each route is based on Dijkstra’s algorithm. I don’t want to annoy you with details. Take a look at the Wikipedia article for OSPF.
The simplest OSPF setup is a “Single Area OSPF”. This is an OSPF configuration, which has only a single area. This is the area 0, or the backbone area.
The configuration on the core switch looks like this:
[C1]ospf 1
[C1-ospf-1]display this
#
ospf 1
area 0.0.0.0
network 1.1.1.1 0.0.0.0
network 10.0.0.0 0.255.255.255
#
The networks, that should be associated with this area, are specified with a wildcard mask. The wildcard mask is the opposite of the subnet mask. The wildcard mask 0.255.255.255 corresponds to the subnet mask 255.0.0.0. Because I have used multiple /30 subnets at the core switch, I can summarize them with a single entry for 10.0.0.0.
The same configuration applies to the access switches S1 and S2.
[S1]ospf 1
[S1-ospf-1]display this
#
ospf 1
area 0.0.0.0
network 1.1.1.2 0.0.0.0
network 10.0.0.0 0.255.255.255
network 192.168.0.0 0.0.255.255
#
[S2]ospf 1
[S2-ospf-1]display this
#
ospf 1
area 0.0.0.0
network 1.1.1.3 0.0.0.0
network 10.0.0.0 0.255.255.255
network 192.168.0.0 0.0.255.255
#
With this simple configuration, the switches will exchange their routing information. They will synchronize their link-state databases, and they will be fully adjacent. If a link-state change occurs, OSPF will handle this.
The core switch has two links to each access switch. The router ID represents the access switches. 1.1.1.2 is a loopback interface IP address on S1, 1.1.1.3 is a loopback interface IP address on S2.
<C1>display ospf peer
OSPF Process 1 with Router ID 1.1.1.1
Neighbor Brief Information
Area: 0.0.0.0
Router ID Address Pri Dead-Time State Interface
1.1.1.2 10.0.0.2 1 37 Full/DR GE1/0/1
1.1.1.3 10.10.0.2 1 36 Full/DR GE1/0/2
1.1.1.2 10.1.0.2 1 38 Full/DR GE2/0/1
1.1.1.3 10.11.0.2 1 35 Full/DR GE2/0/2
The same applies to the access switches, in this case S1. The access switches have also two active links to the core switch.
<S1>dis ospf peer
OSPF Process 1 with Router ID 1.1.1.2
Neighbor Brief Information
Area: 0.0.0.0
Router ID Address Pri Dead-Time State Interface
1.1.1.1 10.0.0.1 1 31 Full/BDR GE1/0/1
1.1.1.1 10.1.0.1 1 32 Full/BDR GE1/0/2
If one of the links fail, the access switch has another working link to the core switch, and OSPF will recalculate the shortest paths, taking the link-state change (link down between core and an access switch) into account.
This is the OSPF routing table of the core switch, based on the example above.
What if I add a new subnet on S1? Let’s create a new VLAN and add a SVI to it (VLAN 3 and 192.168.3.1).
[S1]dis int brief
Brief information on interface(s) under route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Main IP Description
GE1/0/1 UP UP 10.0.0.2
GE1/0/2 UP UP 10.1.0.2
InLoop0 UP UP(s) --
Loop0 UP UP(s) 1.1.1.2
MGE0/0/0 DOWN DOWN --
NULL0 UP UP(s) --
REG0 UP -- --
Vlan1 UP UP 192.168.1.1
Vlan2 UP UP 192.168.2.1
Vlan3 UP UP 192.168.3.1
Without touching the OSPF configuration, the core switch C1, and the other access switch S2, added routes to this new subnet.
This is only an example with a single core switch and two access switches. OSPF can be pretty complex, if the size of the network increases. The Dijkstra’s algorithm can be really CPU intensive, and the size of the link-state databases (LSDB) increase with adding more routers and networks. For this reason, larger networks have to be divided into separate areas. It depends on the network size and the CPU/ memory performance of your switches/ routers, but a common practice is a maximum of up to 50 switches/ routers per area. If you have unstable links, the area should be smaller, because each link-state change is flooded to all neighbors and consumes CPU time.
You need a good subnet design, otherwise you have to touch your OSPF configuration too often. You should be able to summarize subnets.
Conclusion
Routing at the network access is nothing for small networks. There are better designs for small networks. But if your network has a decent size, routing at the edge of the network can offer some benefits. Instead of working with SVIs and small transfer VLANs, a routed port is more simple to implement. Routed links can also have a shorter convergence delay, and you can reduce the usage of Spanning Tree Protocol to a minimum.
This posting is ~7 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Many years ago, networks consisted of repeaters, bridges and router. Switches are the successors of the bridges. A switch is nothing else than a multiport bridge, and a traditional switch doesn’t know how to pass traffic to a different broadcast domains (VLANs). Passing traffic between different broadcast domains, is a job for a router. A router has an IP interface in each broadcast domain, and the IP interface is used by the clients in the broadcast domain as a gateway.
Switch Virtual Interface
A Switch Virtual Interface, or SVI, is exactly this: An virtual IP interface in a broadcast domain (or VLAN). It’s used by the connected clients in the broadcast domain to send traffic to other broadcast domains.
This is how a SVI is created on HPE Comware 7. It’s similar to other vendors.
[C1]interface Vlan-interface 1
[C1-Vlan-interface1]ip address 10.100.100.1 30
[C1-Vlan-interface1]display this
#
interface Vlan-interface1
ip address 10.100.100.1 255.255.255.252
#
At least one port is assigned to this VLAN, and as soon as at least one port of this VLAN is online, the SVI is also reachable.
What happens, if you connect two switches with a cable? The broadcast domain spans both switches. Layer 2 traffic is transmitted between the switches. And what would happen if you connect a second cable between the same two switches? As long as you are running Spanning Tree Protocol (STP), or another loop detection mechanism, nothing would happen. But one of the two connection would be blocked. No traffic would be able to pass over this connection. If you want to use multiple, active connections between switches, you have to use Link Aggregation Groups (LAG), or things like Multiple Spanning Tree Protocol (MSTP) and Per VLAN Spanning Tree (PVST).
Routers don’t know this. Multiple connections between the same two routers can’t form a loop. Loops and STP (an some other crappy layer 2 stuff) are legacies of the bridges, still alive in modern switches. Loops are a typical “bridge problem”.
Routed Ports
Some switches offer a way, to change the operation mode of a switch port. After changing this operation mode, a switch port doesn’t act like a bridge port anymore. It’s acting like the port of a router, that only handles layer 3 traffic.
This is again a HPE Comware 7 example. I know that Cisco and Alcatel Lucent Enterprise also offer routed ports.
This is a normal switch port. Please note the “port link-mode bridge”.
[C1]int XGE1/0/49
[C1-Ten-GigabitEthernet1/0/49]display this
#
interface Ten-GigabitEthernet1/0/49
port link-mode bridge
combo enable fiber
#
To “convert” a switch into a routed port, simply change the link-mode of the port.
[C1-Ten-GigabitEthernet1/0/49]port link-mode route
[C1-Ten-GigabitEthernet1/0/49]ip address 10.10.10.1 30
[C1-Ten-GigabitEthernet1/0/49]display this
#
interface Ten-GigabitEthernet1/0/49
port link-mode route
combo enable fiber
ip address 10.10.10.1 255.255.255.252
#
As you can see, you can now assign an IP address directly to the port.
Example
Let’s try to make this clear with an example. C1-1 and C1-2 are two HPE Comware based switched, configured as an IRF stack (virtual chassis). These two switches form the core switch C1. S1 and S2 are two access switches, also HPE Comware based. Each access switch has two uplinks: One uplink to C1-1 and another uplink to C1-2, the two chassis that form C1. The 40 GbE Ports between C1-1 and C1-2 are used for IRF. Please ignore them.
The uplinks between the switches, all ports are Gigabit Ethernet (GE) ports, are configured as routed ports.
Without routed ports, the uplinks must be configured as a LAG, or STP would block one of the two uplinks between the core switches and the access switch. But because routed ports are used, no loop is formed. Most layer 2 traffic can’t pass the routed ports (broadcasts, multicasts etc.)
Patrick Terlisten/ www.vcloudnine.de/ Creative Commons CC0
THe Link Layer Discovery Protocol (LLDP) traffic can pass the routed port. This is what the core switch (C1) “sees” over LLDP.
[C1]display lldp neighbor-information list
Chassis ID : * -- -- Nearest nontpmr bridge neighbor
# -- -- Nearest customer bridge neighbor
Default -- -- Nearest bridge neighbor
System Name Local Interface Chassis ID Port ID
S1 GE1/0/1 34ce-d3a9-0300 GigabitEthernet1/0/1
S2 GE1/0/2 34cf-0690-0400 GigabitEthernet1/0/1
S1 GE2/0/1 34ce-d3a9-0300 GigabitEthernet1/0/2
S2 GE2/0/2 34cf-0690-0400 GigabitEthernet1/0/2
Each routed port as an IP address assigned. The same applies to the routed ports on the access switches. Each uplink pair (core to access) uses a /30 subnet.
As you can see, the interfaces working in bridge mode start counting at GE1/0/3.
[C1]display interfaces brief
Brief information on interface(s) under route mode:
Link: ADM - administratively down; Stby - standby
Protocol: (s) - spoofing
Interface Link Protocol Main IP Description
GE1/0/1 UP UP 10.0.0.1
GE1/0/2 UP UP 10.10.0.1
GE2/0/1 UP UP 10.1.0.1
GE2/0/2 UP UP 10.11.0.1
InLoop0 UP UP(s) --
Loop0 UP UP(s) 1.1.1.1
MGE0/0/0 DOWN DOWN --
NULL0 UP UP(s) --
REG0 UP -- --
Brief information on interface(s) under bridge mode:
Link: ADM - administratively down; Stby - standby
Speed or Duplex: (a)/A - auto; H - half; F - full
Type: A - access; T - trunk; H - hybrid
Interface Link Speed Duplex Type PVID Description
FGE1/0/53 UP 40G F(a) -- --
FGE1/0/54 UP 40G F(a) -- --
FGE2/0/53 UP 40G F(a) -- --
FGE2/0/54 UP 40G F(a) -- --
GE1/0/3 DOWN auto A A 1
....
....
The same applies to STP. The ports, that were configured as routed ports, are not listed in the output. STP is not active on these ports.
[C1]display stp
-------[CIST Global Info][Mode MSTP]-------
Bridge ID : 0.34a9-6908-0100
Bridge times : Hello 2s MaxAge 20s FwdDelay 15s MaxHops 20
Root ID/ERPC : 0.34a9-6908-0100, 0
RegRoot ID/IRPC : 0.34a9-6908-0100, 0
RootPort ID : 0.0
BPDU-Protection : Disabled
Bridge Config-
Digest-Snooping : Disabled
Root type : Primary root
TC or TCN received : 0
Time since last TC : 0 days 0h:27m:23s
----[Port4(GigabitEthernet1/0/3)][DOWN]----
Port protocol : Enabled
Port role : Disabled Port
Port ID : 128.4
Port cost(Legacy) : Config=auto, Active=200000
Desg.bridge/port : 0.34a9-6908-0100, 128.4
Port edged : Config=disabled, Active=disabled
Point-to-Point : Config=auto, Active=false
Transmit limit : 10 packets/hello-time
TC-Restriction : Disabled
Role-Restriction : Disabled
Protection type : Config=none, Active=none
MST BPDU format : Config=auto, Active=802.1s
Port Config-
Digest-Snooping : Disabled
Rapid transition : False
Num of VLANs mapped : 1
Port times : Hello 2s MaxAge 20s FwdDelay 15s MsgAge 0s RemHops 20
BPDU sent : 0
TCN: 0, Config: 0, RST: 0, MST: 0
BPDU received : 0
TCN: 0, Config: 0, RST: 0, MST: 0
What are the implications?
The example shows redundant links between access and core switches. There are no loops, but there’s also no layer 2 connectivity. VLANs are only located on the access switches. There are no VLANs spanning multiple switches. What does this mean? How can a client on S1 reach a server on S2? The answer is simple: You have to route the traffic on the access switches. But that’s a topic for another blog post.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
In January 2014 I wrote a blog post about network flooding because of Windows NLB clusters in unicast mode. Yesterday, Windows NLB, HP switches and I met again.
After moving a customers core network from HP 5400zl switches to two IRF stacks with HP 7506 switches, multiple Windows NLB clusters stopped working. Because the Windows NLB used multicast operation mode, it was instantly clear that the switches were the problem.
The explanation is easy: By default, a Comware based switch does not learn multicast MAC addresses. And because of this, the switch does not add them to the ARP table. And you can’t add static multicast MAC address entries. You have to disable the ARP entry check.
To do so, you have to login and change to the system-view. Then disable the ARP entry check option, which is enabled by default.
<HP> system-view
[HP] undo arp check enable
A few seconds after issuing the command, the NLB clusters started working again and I saw ARP entries with MAC addresses starting with 03-BF.
This posting is ~8 years years old. You should keep this in mind. IT is a short living business. This information might be outdated.
Last week, my colleague Claudia and I have ported a HP ProVision configuration to HP Comware. Unexpectedly, it wasn’t routing or VLANs or OSPF that caused headaches, it was a Wake-on-LAN (WoL). Depending on the used tool, the magic packet (which wakes up the computer) is a broadcast (255.255.255.255) or a subnet-directed broadcast (e.g. 192.168.200.255). So it was important to know what tool the customer used.
This is how HP ProVision implements subnet-directed broadcasts:
ip directed-broadcast
ip udp-bcast-forward
vlan 99
ip address 192.168.200.254 255.255.255.0
ip forward-protocol udp 10.0.0.255 9
vlan 3
ip address 10.0.0.254 255.255.255.0
The first two commands are issued globally. The “ip forward-protocol” statement has to be entered in the source VLAN (from which the magic packets are sent). The “ip forward-protocol” statement includes the protocol (udp), the destination (the broadcast address of the subnet) and the udp port (WoL uses port 9). Pretty simple, right? But please note that this config works only for sunet-directed broadcasts. And it only works for WoL. If you need another port (e.g. udp port 7), you have to add an additional “ip forward-protocol” statement.
If you use HP Comware, the configuration differs in some points. You can enable the reception of subnet-directed broadcasts by entering “ip forward-broadcast” globally in the system-view. This is mandatory.
<HP> system-view
[HP] ip forward-broadcast
The next step is to tell the switch, to which destination it should forward subnet-directed broadcasts. This can be done by entering “ip forward-protocol” in the vlan-interface context.
[HP-Vlan-Interface99] ip forward-broadcast
The downside: All subnet-directed broadcasts will be forwarded, regardless of source, destination or protocol. To avoid this, you have to create a ACL and add this to the “ip forward-broadcast” statement. To create an ACL enter:
You have to bound the ACL to the source VLAN interface. The subnet-directed broadcast will be forwarded to the VLAN interface which is directly connected to the destination subnet, or if there is no directly connected interface, to a router which knows the way to the destination. If you have applied packet filter ACLs to VLAN interfaces, make sure that your forwarded subnet-directed broadcasts aren’t filtered!
To change your privacy setting, e.g. granting or withdrawing consent, click here:
Settings
