Nexus 1000v layer 3 control mode

A N1Kv can either be installed in layer 2 control or in layer 3 control mode. Layer 3 control is currently the recommended mode by Cisco. The advantage of layer 3 control mode is that the VSM and the VEM don’t have to be within the same subnet (or VLAN as required in layer 2 control mode). This means that the VEMs can be in a remote location. In layer 3 control mode the control traffic between the VSM and the VEM is encapsulated in UDP, port 4785. Because every VEM in layer 3 control mode is associated with a VMkernel port, troubleshooting becomes easier. You can simply use the ping command to validate connectivity between the VSM and VEM. Also Layer 3 control makes integration with future Cisco products easier.This article will describe layer 3 control mode.

The VSM is standard equipped with three network adapters the name and function of each adapter is defined as:

Control VLAN: is connected to VSM “network adapter 1”. The control VLAN is used for downloading configuration from the VSM into the VEM. The control interface is also used to signal port status (attachment or detachment of VEM ports) to the VSM (synchronization). Netflow communication and VSM active/standby communication for HA.

Management VLAN: is connected to VSM “network adapter 2”. This management interface is used for management (SSH), monitoring etc. It is also used for connectivity between the VSM and the vCenter server.

Packet VLAN: is connected to VSM “network adapter 3”. The packet VLAN is used for network control packets such as CDP, LACP, and IGMP (It is used by packets that need to be processed by a central CPU).

differences between the layer 2 and layer 3 modes:

  • In layer 2 mode the control traffic is not routable, in layer 3 mode it is.
  • Management traffic is routable in in both layer 2 and layer 3 modes.
  • In layer 2 mode the packet VLAN is used as described above. In layer 3 mode the packet VLAN functionality is combined with management VLAN. Effectively the packet interface has no function in L3 mode. But the packet interface can’t be removed from the VSM, the VSM expects the packet interface to be present and connected.
VSM to VSM communication is always L2 traffic (in layer 2 mode and in layer 3 mode). This means the VSM hosts must be L2 adjacent. The primary VSM heartbeat interface is the control interface and the management interface is used as backup.

Layer 3 mode can be deployed in several ways. Which type of deployment you choose depends on your requirements and you personal preferences. We start with the most simple setup which I will call “Integrated management and control”. After that we will separate the control traffic from the management traffic in the “Split management and control” setup.

Integrated management and control
The following picture shows the most simple setup. All traffic (management, control and packet data) is flowing over the management interface, adapter 2 of the VSM. The red VLAN represents the combined management and control VLAN. The green VLAN represents the production VLAN.

The mgmt0 interface of the VSM, “adapter 2” is configured with IP address The control0 interface within the VSM, “adapter 1” has no configuration applied. Only the heartbeat (L2) between the VSMs is using “adapter 1” in this scenario.

Nexus 1000v layer 3 control mode - combined

 The red VLAN combines several management functions:

  • Allow the network administrator to SSH to the VSM (
  • Communication between the VSM and vCenter.
  • Communication between vCenter and the ESX(i) VMkernel management interface (VMK).
  • Transfer of opaque data from the vCenter server to the ESX(i) host. Opaque data provides initial information to the VEM, whereafter the VEM is able to contact the VSM and download its configuration from the VSM.

Next to management the red VLAN also provides the control path between the VSM and the VEMs. Control traffic is encapsulated in UDP, port 4785.

Because the N1Kv is a layer 2 switch and cannot provide layer 3 interfaces the VMkernel management port of the ESX server is used for control traffic. To make this work the port-profile of the VMkernel management port is configured with the “capability l3control” command.

The striped VLAN in the VEM represents the usage of a system VLAN. System VLANs are special VLANs that always forward traffic even if the VEM (or vswitch) is not (yet) configured. This mechanism makes it possible to place the VSM on top of a VEM. Or vCenter on top of ESXi. The system VLAN mechanism is independent from any virtual switch.By defining the management VLAN as a system VLAN, no matter what, there will be always connectivity with the VSM.

Split management and control
The integrated setup shown is the most simple setup, but maybe not everyone is comfortable combining management and control functionalities. The split design separates management and control by using different VLANs and L3 interfaces for each functionality. The control interface of the VSM is used for control and packet functionality. The management interface is used for management functionality. The design is show in the next figure.

The mgmt0 interface of the VSM is configured IP with address The control0 interface of the VSM is configured with IP address

To make L3 communication between the VSM and the VEM work a second VMkernel interface for control traffic must be created (The purple VLAN is the control VLAN). The control traffic VMkernel interface must be configured with “capability l3control”. And just like the management VLAN the control VLAN is configured as system VLANs to guarantee a forwarding path.

Nexus 1000v layer 3 control mode - separate

In this post I showed two designs, more designs are possible. For example vSphere management and VSM management could be separated via a firewall. It is also possible to leave the VMK for ESXi management at the standard vswitch (some admins are more confident leaving vSphere functionality at the standard switch).

The opinions expressed in this blog are my own views and not those of Cisco.

Leave a Reply

Your email address will not be published. Required fields are marked *