Cloud Foundation on VxRail offers flexibility with regards to the selection of a physical network architecture to support the planned deployment. The most common network topology for Cloud Foundation on VxRail, and is considered a best practice, is a spine-leaf topology. In this model, the VxRail nodes connect directly to the leaf-layer switches, and multiple VxRail clusters can be supported on a single pair of leaf-layer switches. The spine layer is positioned primarily for aggregating upstream traffic, providing connectivity to external resources  and enabling VTEP tunneling between racks.

Decisions must be made regarding the location of the Layer 2 and Layer 3 boundaries to support Cloud Foundation on VxRail networking. The NSX edge gateways depend on peering with a router upstream in the physical network using External Border Gateway Protocol (eBGP) to update routing tables in the virtual network. The NSX edge gateway collects the routing tables from the adjacent routers in the physical network, and passes the updates down to the NSX controllers.

The VLANs used in Cloud Foundation on VxRail to support the guest virtual machine networks terminate at these upstream routing devices. Therefore, using the route mapping for the applications planned for the VI workload domains drives the decisions for the locations for the NSX edge virtual devices in Cloud Foundation domains, and guides the process for selecting the location of the adjacent routers in the physical network.

In most cases, routing outside of the virtual network is positioned in either the spine layer or leaf layer. If you choose to deploy a spine-leaf network topology, enabling Layer 3 at either the spine layer or the leaf layer is not required. However, this means routing traffic must pass through both the leaf and the spine layers to reach the routers. This option is more suitable for small-scale deployments, and it is easy to deploy and configure. It is appealing for sites that have low routing requirements, or the plan is to deploy only a few edge virtual devices.

Figure 21            Options for Layer 2/Layer 3 boundaries in spine-leaf network topology

Establishing the router layer at the spine layer means the uplinks on the leaf layer are trunked ports, and pass through all of the required VLANs to the routing services on the spine layer. This topology has the advantage of enabling the Layer 2 networks to span across all of the switches at the leaf layer. This topology can simplify VxRail networks that extend beyond one rack. Switches at the leaf layer do not need to support Layer 3 services.

Figure 22            VxRail cluster nodes extended beyond one physical rack

A major drawback to this topology is scalability because Ethernet standards enforce a limitation of addressable VLANs to 4094, which can be a problem with a shared switch layer fabric. Do not select this topology option if your deployment might breach this threshold.

This VLAN limitation can be overcome by establishing the routing at the leaf layer. This will optimize routing traffic, as it requires the least number of hops for the edge virtual devices to peer with an adjacent router. But it requires Layer 3 services to be licensed and configured at the leaf layer. In addition, since Layer 2 networks now terminate at the leaf layer, they cannot span leaf switches without additional configuration. Both NSX-V and NSX-T support spanning Layer 2 networks between switches using Layer 3 services for the virtual machine traffic. To overcome the problem of extending Layer 2 networks across racks and switches, best practice is for the leaf switch to support hardware-based tunneling.

The key points to consider for the decisions regarding the network architecture and topology are:

1. Select Ethernet switches that support the features required for Cloud Foundation on VxRail:

• Border Gateway Protocol: Required for peering with NSX edge gateways
• Multicast & Unicast: Required for VxRail and VXLAN traffic
• Jumbo Frames: Required for VXLAN and GENEVE
• Hardware-based tunneling (VTEP): Required to extend Layer 2 VM traffic over a Layer 3 network at the physical switch layer
• DHCP ‘helper’: Switches that support DHCP ‘helper’ functionality will ease connecting DHCP services in the data center to the Cloud Foundation on VxRail environment     

2. Decide which physical network layer will support Layer 3 routing services.