Fibre Channel: ISL, ISL trunking & FSPF

Note

This article is a bit Brocade-minded, especially with regard to the terminology.

Redundant paths in a network are always a curse. Always? Not always. It depends on the protocol and the kind of network. In a Ethernet network you have Spanning Tree which eliminates redundant paths and builds a loop free network. What about redundant paths in a storage network? When iSCSI and NFS is used, then usually TCP/IP is used on top of Ethernet. Fibre Channel manages redundant paths a bit different. There’s nothing like Spanning Tree. Redundant paths are nothing fancy. They are just there and they are used.

The Glossary

FSPF: “Fabric shortest path first” is a routing protocol used in Fibre Channel fabrics. It’s used to establish routes accross the fabric and to re-calculate this routes, if a topology change occurs (e.g. link failures).

ISL: An ISL is an inter-switch link. It’s a connection between two Fibre Channel switches.

ISL Trunking: An ISL Trunk is a logical ISL, that provides up to 128 Gbps of bandwidth (8 ports x 16 Gbps).

Context please..

One cool thing about Fibre Channel is the possibility, to use multiple ISLs between switches. No need to worry about loops, no blocked links. Just more ISLs that provide bandwidth and redundancy. If you add more ISLs between switches, Fabric Shortest Path First (FSPF) will care about this topology change. Fibre Channel fabrics are hierarchical and every fabric has a principal switch. He’s the boss and manages changes of name server entries, zone changes, he ensures uniqueness of domain IDs and synchronizes time throughout the fabric. He also analyzes the topology. Each switch is a hop and each ISL between switches has a cost. The costs depend on the link bandwidth.

Bandwidth Costs
1 Gb 1000
2 Gb 500
4 Gb 250
8 Gb 125
10 Gb 100

A link with two hops and 4 Gb bandwidth has costs of 500. A alternate link with only one hop but 2 Gb bandwidth has also a cost of 500. If two routes have the same speed and latency, FSPF will use the route with the least number of hops. A link to a hierarchical higher switch is a upstream connection. A link to a hierarchical lower switch is a downstream connection. FSPF uses a exchange-based load sharing (hash operation over SRC and DST FCID and OX_ID) to distribute the traffic over multiple ISLs. A Fibre Channel exchange is, simply said, a SCSI command. A exchange is made of several sequences and those of several Fibre Channel frames. This way of distributing traffic can lead into a situation, where congestion occurs, because FSPF doesn’t take the link utilization into account.

To avoid congestion ISL Trunking can be used. With ISL Trunking several ISLs are grouped into a logical ISL. When using Brocade switches, a license has to be installed on all switches, that use ISL Trunking. An ISL Trunk is automatically formed, when two or more (up to 8) adjacent ports are used to connect two switches. The adjacent ports must belong to the same port group. That’s the cause why you can’t add more than 8 ports to an ISL Trunk. Instead of using an exchange-based distribution, a frame-based distribution is used for ISL Trunks. This method is more finer and allows a better distribution. The Ports, that belongs to an ISL Trunk, are known as trunking members. One port of the trunking members is the trunking master. This port has a special function, because it assigns traffic to the other trunking members. Even if an ISL Trunk is an logical ISL, it preserves “in-order delivery” of frames.

Final words

Most times FSPF is doing a great job. It just works. But as with Ethernet, it is not done with adding links between Fibre Channel Switches. Sure, you don’t have to deal with Spanning Tree, but just adding more ISLs between switches can cause more problems than they solve. Choose a adequate design based on your requirements. Simplicity is often the key to success. Be sure, that you use an adequate number of links in an ISL. Take care of congestion and use ISL Trunking whenever it’s possible.