As an enhancement and combination of the best of both fabrics (Layer 2 and Layer3), EVPN VxLAN is an overlay control plane technology for VxLAN.
It implements an overlay that permits the extension of a Layer 2 domain across a Layer 3 fabric where virtualization and multitenancy deployments leverage Layer 2 connectivity.
The figure below shows a BGP EVPN VxLAN fabric where Tenants A and B are being stretched across a Layer 3 fabric implemented by BGP.
The inter-switch (leaf to spine) links are Layer 3 while the downstream connections to the end-hosts are Layer 2. MC-LAG is implemented at the leaf switch layer to provide link redundancy to the end-hosts.
Virtual tunnel endpoints (VTEPs) are created on each leaf switch pair. These VTEPs establish the tunnels used by each tenant to create a Layer 2 connectivity across the Layer 3 fabric.
The border leaf switch pair connected to an external router or switch may or may not have workloads connected to them.
The links from the border leafs to the external routers are Layer 3 connections with a /31 subnet.
There are two VxLAN deployment models: Layer 2 VxLAN and Layer 3 VxLAN
In a Layer 2 VxLAN, all Layer 2 traffic is directed towards a single Layer 3 point. This Layer 3 point can be the edge or border leaf switches, or an external router or firewall connected to the edge or border leaf switches.
In a Layer 3 VxLAN, tenant VRFs, an anycast gateway, and a transport VNI are configured to provide cross tenant communication while maintaining a Layer 2 domain stretched across a Layer 3 fabric.
In a Layer 3 VxLAN deployment, two types of integrated bridging and routing (IRB) implementations are available:
- In symmetric routing, all VTEPs can perform routing, and routing decisions are made on both ingress and egress VTEPs.
- In asymmetric routing, all VTEPs can perform routing. Routing decisions are made only on ingress VTEPs. Egress VTEPs only perform bridging.
Deployment best practices
Like all scalable high-performance fabrics where virtualization and multitenancy are key requirements, the following best practices are recommended:
- Create a Layer 3/Layer 2 demarcation point. From the leaf layer to the spine layer, Layer 3 should be implemented, in this case, BGP EVPN VxLAN.
- Any connections downstream from the leaf layer should be Layer 2.
- Implement switch redundancy at the leaf layer with MC-LAG.
- Symmetric IRB provides better scalability as each VTEP does not have to store information for each tenant VLAN in the routing table memory.
- Asymmetric IRB is better suited for small and medium-sized data centers and for deployments where there is Layer 2 VxLAN (no tenant data traffic routing is needed) and Layer 3 VxLAN (multiple tenants can communicate with each other).
- Broadcast, unknown unicast, and multicast traffic should be controlled whenever possible to avoid network performance degradation using the storm-control command.
- For simplicity, unnumbered BGP should be configured between leaf and spine links.
- Enable "neighbor suppression" on Layer 3 tenant VLANs to reduce ARP flooding in the overlay network.
- Link state tracking should be enabled on the leaf and spine interlinks. Link state tracking helps to minimize traffic loss during a reset or loss of a switch in the fabric.
- Enable "max-med on-startup" to allow BGP convergence to take place on the affected node (leaf or spine) before data traffic is allowed on the fabric.