Traditionally, network infrastructures have always had a clear demarcation zone between Layer 3 and Layer 2 switching.
In a traditional deployment model, any connection from the leaf towards the spine is Layer 3 and Layer 2 from the leaf to the end-host. This means all end-hosts that need to communicate with a different end-host in a different network need to have a Layer 3 gateway as a first hop.
In a combined switching environment, Layer 2 comes with performance and scaling limitations related to spanning-tree and the number of VLANs. To overcome these limitations, Layer 3 end-to-end should be deployed, however, this requires the end-hosts to be able to participate in the entire Layer 3 protocol stack.
Routing on the host was introduced with BGP unnumbered interface to achieve Layer 3 end-to-end and mitigate the performance and scaling limitations created by Layer 2 deployments. Routing on the host has the following key benefits:
- VM mobility
- Layer 3 control plane for VM discovery and reachability
- Eliminates the need for leaf interconnect links
- Layer 3 loop avoidance through Time to Live (TTL) implementation
The following figure shows the typical Layer 3 and Layer 2 (Hybrid) deployment where all end-hosts terminate their traffic at a Layer 3 gateway. This gateway can be configured at the leaf switches through a virtual switch interface with an IP address, or it can be configured at the spine switches. See Layer 2 fabric deployment for additional design methodology information.
The following figure shows routing on the host deployment where Layer 3 end-to-end is deployed including at the end-host. With this deployment insufficient number of VLANs, limited number of MAC addresses, or inefficient use of switch interlinks are removed.
Since Layer 3 is deployed end-to-end, the only limiting factor in this deployment is the route table size of the switch and the end-host's Layer 3 protocol stack support.
With Dell Enterprise SONiC, routing on the host is enabled by configuring BGP unnumbered on all the leaf and spine switch interfaces. The end-hosts running Layer 3 protocol stack advertise their own network dynamically to the directly connected leaf switches. This in turn transforms all end-hosts connections as single IP addresses that can move freely within and across the fabrics, that is, mobility.
Another differentiator with routing on the host is the elimination of interconnect links between leaf switches that provide MC-LAG features. MC-LAG limits redundancy to two leaf switches at any given time. This creates a potential reduction performance by 50% if a leaf switch were to fail.
With routing on the host, the host can advertise and connect to all the leaf switches thereby providing full connectivity regardless of a link failure, and through equal cost multipath (ECMP) equal load sharing is achieved.
Deployment best practices
Routing on the host leverages Layer 3 concepts and has similar Layer 3 deployment guidelines:
- Routing on the host uses BGP. OSPF is not supported.
- Configure /32 subnet on all end-hosts.
- Connect end-host to all leaf switches to achieve full mesh connectivity.
- Configure ECMP across all end hosts links towards the leaf switches.