FabricPath :: Part II – Forwarding example

This is part two of a three-part series about FabricPath. The slides are from a presentation I presented to my colleagues. The source of the presented information comes from a lot of Googling, day-to-day practice and this very good piece of documentation: “Nexus 7000 FabricPath White Paper Version 2.0″.

Part I: FabricPath – The basics.
Part II: FabricPath – Forwarding example (this post).
Part III: FabricPath – Advanced topics.


Slide22

This example assumes that host A wants to communicate with host B for the first time, both host A and host B are in VLAN 20.

  1. Host A transmits a ARP request to discover the MAC address of host B. The ARP request is received at CE port e1/12 of edge switch 1001.
  2. The source address of host A is learned by edge switch 1001.
  3. Because it is a broadcast frame it is send out of all CE ports of switch 1001. The frame will also be flooded to all FP switches in the domain. Broadcast, non-IP multicast and unknown unicast frames are always forwarded along MDT 1, MDT 1 is addressed by FTAG 1. edge switch 1001 performs a multi destination lookup for MDT 1. The frame will be forwarded out of interface Po1. MDT 1 is colored black in this example.
  4. Before the frame is send out of edge switch 1001 it is encapsulated in a FP header. The iDA (FF) is copied to the ODA of the FP frame. The FTAG field is set to 1 indicating that the frame must be send along MDT tree 1. The hop count is set to 32. The OSA is encoded with the switch ID, the sub-switch ID and the port ID (1001.0.11). The switch ID is set to the source FP switch 1001, the sub-switch ID is not used and the port ID (or LID) is set to 11 indicating that the frame was sourced from port e1/12.
  5. core switch 101 performs a multi destination lookup for MDT 1 and forwards the frame out of ports Po1, Po2,Po3 and Po4 after decreasing the hop count by 1.
  6. core switch 201 performs a multi destination lookup for MDT 1 and forwards the frame out of ports Po1, Po2, Po11, Po12, Po13 and Po14 after decreasing the hop count by 1.
  7. The frame is decapsulated at edge switch 2001, the frame because of its broadcast nature is send out of all its ports (except the port it was received on). The ARP frame reaches host B. Switch 2001 does not learn the source address of host A, remember: the switch only learns the source MAC address if the destination MAC address matches an already learned local MAC entry (host B did not send a fame and was not learned yet by switch 2001).

Slide23

  1. Host B replies with a unicast ARP frame. The destination is host A and the source is set to B. The ARP reply is received at CE port e1/24 of edge switch 2001.
  2. edge switch 2001 learns the source MAC address of the CE frame, this is reflected in the MAC table of edge switch 2001. The MAC address of host A is still missing in the MAC table of edge switch 2001.
  3. Because it is a unknown unicast frame it is send out of all CE ports of switch 2001. The frame will also be flooded to all FP switches in the domain. Broadcast, non-IP multicast and unknown unicast frames are always forwarded along MDT 1, MDT 1 is addressed by FTAG 1. edge switch 2001 performs a multi destination lookup for MDT 1. The frame will be forwarded out of interface Po1. MDT 1 is colored black in this example.
  4. Before the frame is send out of edge switch 2001 it is encapsulated in a FP header. The iDA (MAC A) is NOT copied to the ODA of the FP frame. In this case the ODA is set to 01:0F:FF:C1:01:C0 indicating that the frame is a unknown unicast frame. The FTAG field is set to 1 indicating that the frame must be send along MDT tree 1. The hop count is set to 32. The OSA is encoded with the switch ID, the sub-switch ID and the port ID (1001.0.23). The switch ID is set to the source FP switch 2001, the sub-switch ID is not used and the port ID (or LID) is set to 23 indicating that the frame was sourced from port e1/24.
  5. core switch 201 performs a multi destination lookup for MDT 1 and forwards the frame out of ports Po1, Po2, Po11, Po12, Po13 and Po14 after decreasing the hop count by 1.
  6. core switch 101 performs a multi destination lookup for MDT 1 and forwards the frame out of ports Po1, Po2,Po3 and Po4 after decreasing the hop count by 1.
  7. The frame is decapsulated at edge switch 1001. Because the destination MAC address of A is known the source address of host B is learned. All other switches in the FP domain will not learn MAC address B. edge switch 1001 sends the frame out of interface e1/12.
  8. Host A receives the ARP reply.

Slide24

  1. Host A starts sending traffic to host B, the unicast frame is received at port e1/12 of edge switch 1001.
  2. Switch 1001 refreshes the MAC entry for host A and does a lookup for host B. This time there is a hit, the destination of host B is known (switch ID 2001 port 23). The unicast frame will be forwarded out of a FP core port. To determine which port there will be a lookup in the switch ID table.
  3. The switch ID table tells the switch that switch 2001 can be either reached via Po1 or Po2. Now the ECMP mechanism kicks in and hashing on determents that the frame will be forwarded out of port Po2 (default hashing is executed on source and destination IP plus layer 4 ports).
  4. The frame is encapsulated in a FabricPath header. The ODA and OSA are set to the appropriate values for the switch ID the sub-switch ID and the port ID. In this case is the ODA 2001.0.23 and the OSA 1001.0.11. The hop count is set to the default value of 32.
  5. The frame is received by core switch 102. Switch 102 does a lookup in its switch table for destination SID 2001, the frame is send out of port Po1 after decreasing the hop count by 1.
  6. The frame is received by core switch 202. Switch 202 does a lookup in its switch table for destination SID 2001, the frame is send out of port Po11 after decreasing the hop count by 1.
  7. The frame is decapsulated at switch 2001. Now the MAC address of switch A will be learned. This is because the destination address B is known by edge switch 2001.
  8. The frame is decapsulated and send to host B. Now both host A and B MAC addresses are learned by both edge switch 1001 and 2001. No other switch in the FabricPath domain learned the MAC addresses of host A and B.

Slide25

A host will send a gratuitous ARP if the server interface is bounced (also if a VM in VMware is vMotiond then a gratuitous ARP is send). The Gratuitous ARP is a broadcast frame with destination address FF. This frame would be send along MDT 1 and would also be delivered at switch 2001. Because the source MAC address of A was already know in switch 2001 it will update the existing MAC entry with new information.

The example figure does not show the forwarding of the gratuitous ARP but shows the order of events if host A simply keeps sending unicast frames destined for host B. Actually the order of events is exactly the same as for the gratuitous ARP, The difference is that the ODA is not set to FF but to 01:0F:FF:C1:01:C0 indicating that a unknown unicast is contained in the frame. The result is the same, the MAC table of switch 2001 will be updated with the new FP source address of host A.

The opinions expressed in this blog are my own views and not those of Cisco.

Leave a Reply

Your email address will not be published. Required fields are marked *