Home > Networking Solutions > Enterprise/Data Center Networking Solutions > Enterprise SONiC Networking Solutions > Guides > Dell Enterprise SONiC Deployment for VxRail > Layer 3 BGP EVPN VXLAN fabric multirack deployment
Use the following instructions for L3 BGP EVPN VXLAN deployment. This is a two-rack setup with two leaf-pairs and two spines. L3 uplinks are used for this deployment example. In the two-rack setup the VxRail nodes are deployed across the two racks.
Note the changes in:
For more information, see Figure 4.
Confirm the following configuration to the interface naming to Standard and update the switch hostname. In this case, the hostname is “LeafX,” where X is the leaf number.
Leaf1A | Leaf1B |
sonic-cli | sonic-cli configure terminal |
Note: You must exit the MF-CLI session to the Linux shell to activate the interface naming mode and reenter the MF-CLI. The following message is displayed during the SONiC session.
“Broadcast message: Interface naming mode has changed. Users running 'sonic-cli' are required to restart your session.”
“Broadcast message: Hostname has been changed from sonic to Leaf1x. Users running 'sonic-cli' are suggested to restart your session.”
Leaf1A | Leaf1B |
interface-naming standard hostname Leaf1A end write memory exit
| interface-naming standard hostname Leaf1B end write memory exit |
Leaf2A | Leaf2B |
sonic-cli interface-naming standard exit sonic-cli | sonic-cli interface-naming standard exit sonic-cli |
Leaf router ID assigned to Loopback0.
Leaf1A | Leaf1B |
configure terminal interface Loopback 0 description Router-ID ip address 10.0.2.1/32 exit end write memory | configure terminal interface Loopback 0 description Router-ID ip address 10.0.2.2/32 exit end write memory |
Leaf2A-Leaf2B pair
Use the configuration settings shown in the following table:
Leaf2A | Leaf2B |
configure terminal interface Loopback 0 description Router-ID ip address 10.0.2.3/32 exit end write memory | configure terminal interface Loopback 0 description Router-ID ip address 10.0.2.4/32 exit end write memory |
VXLAN provides a L2 overlay mechanism on an existing L3 network by encapsulating (tunneling) L2 frames in L3 packets. The VXLAN-shared forwarding domain allows hosts, such as virtual and physical machines in tenant L2 segments, to communicate over the shared IP network.
Create the following VxRail VLANs. VLAN 1811 is for External MGMT VLAN and 3939 (Internal discovery MGMT VLAN). VLAN 1812 is for vMotion, VLAN 1813 is for vSAN, and VLAN 1814 is for guest workload VLAN.
In this deployment, VLAN 1 is the default VLAN. The VLAN 2001 is required for VRF VNI mapping and routing. All VxRail node-facing switch ports are trunked for VLANs 1811,1812,1813,1814, and 3939.
Leaf1A | Leaf1B |
configure terminal no shutdown ip anycast-mac-address 00:00:00:11:11:11 ip vrf VrfTenant1 ip vrf mgmt ssh-server vrf mgmt link state track leaf timeout 180 downstream all-mclag end write memory | configure terminal no shutdown ip anycast-mac-address 00:00:00:11:11:11 ip vrf VrfTenant1 ip vrf mgmt ssh-server vrf mgmt link state track leaf timeout 180 downstream all-mclag end write memory |
Leaf1A | Leaf1B |
configure terminal interface Vlan1 no shutdown interface vlan2001 description ForVrfVniMapping ip vrf forwarding VrfTenant1 no shutdown ! end write memory | configure terminal interface Vlan1 no shutdown description ForVrfVniMapping ip vrf forwarding VrfTenant1 no shutdown ! end write memory |
Leaf1A | Leaf1B |
configure terminal interface Vlan1811 mtu 9216 ! mtu 9216 ! mtu 9216 ! mtu 9216 ! ! mtu 9216 no shutdown end write memory | configure terminal interface Vlan1811 mtu 9216 ! mtu 9216 ! mtu 9216 ! mtu 9216 ! ! mtu 9216 no shutdown end write memory |
The following tables show the virtual network configuration:
Leaf1A | Leaf1B |
configure terminal interface vxlan vtep1 source-ip 10.222.222.1 primary-ip 10.0.2.1 map vni 101811 vlan 1811 map vni 101812 vlan 1812 map vni 101813 vlan 1813 map vni 101814 vlan 1814 map vni 102001 vlan 2001 map vni 103001 vlan 1 map vni 103939 vlan 3939 map vni 102001 vrf VrfTenant1 !
| configure terminal interface vxlan vtep1 source-ip 10.222.222.1 primary-ip 10.0.2.2 map vni 101811 vlan 1811 map vni 101812 vlan 1812 map vni 101813 vlan 1813 map vni 101814 vlan 1814 map vni 102001 vlan 2001 map vni 103001 vlan 1 map vni 103939 vlan 3939 map vni 102001 vrf VrfTenant1 ! end |
Leaf2A-Leaf2B pair
Repeat configuration settings shown in the following table:
Leaf2A | Leaf2B |
configure terminal no shutdown ip anycast-address enable ipv6 anycast-address enable ip vrf mgmt ssh-server vrf mgmt link state track leaf timeout 180 downstream all-mclag ! interface Vlan1 description default ! interface vlan2001 description ForVrfVniMapping ip vrf forwarding VrfTenant1 no shutdown ! mtu 9216 ! mtu 9216 ! mtu 9216 no shutdown ! mtu 9216 ! ! interface Vlan3939 ! interface vxlan vtep2 source-ip 10.222.222.2 primary-ip 10.0.2.3 map vni 101811 vlan 1811 map vni 101812 vlan 1812 map vni 101813 vlan 1813 map vni 101814 vlan 1814 map vni 102001 vlan 2001 map vni 103001 vlan 1 map vni 103939 vlan 3939 map vni 102001 vrf VrfTenant1 ! | configure terminal ! ip anycast-address enable ip vrf VrfTenant1 ip vrf mgmt ssh-server vrf mgmt link state track leaf timeout 180 downstream all-mclag ! interface Vlan1 description default ! interface vlan2001 description ForVrfVniMapping ip vrf forwarding VrfTenant1 no shutdown ! mtu 9216 ! mtu 9216 no shutdown ! mtu 9216 no shutdown ! mtu 9216 ! ! ! interface vxlan vtep2 source-ip 10.222.222.2 primary-ip 10.0.2.4 map vni 101811 vlan 1811 map vni 101812 vlan 1812 map vni 101813 vlan 1813 map vni 101814 vlan 1814 map vni 102001 vlan 2001 map vni 103001 vlan 1 map vni 103939 vlan 3939 map vni 102001 vrf VrfTenant1 ! exit |
A multichassis LAG (MC-LAG) allows you to create a logical switch in which multiple interfaces on peer switches are bundled. On each MC-LAG peer, create an MC-LAG domain and enter domain configuration mode. Valid domain numbers are 1 to 4095. Only one MC-LAG domain is supported on a switch. The interfaces Ethernet 1/49 - 52 are the peer link members used to create the MCLAG.
Leaf1A | Leaf1B |
configure terminal interface PortChannel 256 mode on description MCLAG-Leaf1AB-PeerLink switchport trunk allowed Vlan 1,1811-1814,2001,3939 no shutdown mtu 9216 | configure terminal interface PortChannel 256 mode on description MCLAG-Leaf1AB-PeerLink switchport trunk allowed Vlan 1,1811-1814,2001,3939 no shutdown mtu 9216 |
Leaf1A | Leaf1B |
! interface Eth1/49 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/50 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/51 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/52 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown end write memory | ! interface Eth1/49 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/50 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/51 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/52 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown end write memory |
This domain ID should be with a local source IPv4 address, peer IPv4 address, and the port channel created to keep both peers synchronized.
Leaf1A | Leaf1B |
configure terminal mclag domain 1 source-ip 10.0.2.1 peer-ip 10.0.2.2 peer-link PortChannel256 mclag-system-mac 00:00:00:00:00:01 keepalive-interval 1 session-timeout 30 end | configure terminal mclag domain 1 source-ip 10.0.2.2 peer-ip 10.0.2.1 peer-link PortChannel256 mclag-system-mac 00:00:00:00:00:01 keepalive-interval 1 session-timeout 30 end |
Leaf2A-Leaf2B pair
Repeat configuration settings shown in the following table:
Leaf2A | Leaf2B |
! interface PortChannel 256 mode on description MCLAG-Leaf2AB-PeerLink switchport trunk allowed Vlan 1,1811-1814,2001,3939 mtu 9216 no shutdown ! interface Eth1/49 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/50 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/51 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/52 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! ! mclag domain 2 source-ip 10.0.2.3 peer-ip 10.0.2.4 peer-link PortChannel256 mclag-system-mac 00:00:00:00:00:02 keepalive-interval 1 session-timeout 30 end | ! interface PortChannel 256 mode on description MCLAG-Leaf2AB-PeerLink switchport trunk allowed Vlan 1,1811-1814,2001,3939 mtu 9216 no shutdown ! interface Eth1/49 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/50 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/51 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! interface Eth1/52 description mclag mtu 9216 speed 100000 channel-group 256 no shutdown ! ! mclag domain 2 source-ip 10.0.2.4 peer-ip 10.0.2.3 peer-link PortChannel256 mclag-system-mac 00:00:00:00:00:02 keepalive-interval 1 session-timeout 30 end |
The links between the leaf and spine switches, as shown in Figure 8, are referred to as “interlinks.” To configure these links:
Leaf1A to Spines | Leaf1B to Spines |
configure terminal ! interface Eth1/55 description Spine1 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! interface Eth1/56 description Spine2 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! end write memory | configure terminal ! interface Eth1/55 description Spine1 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! interface Eth1/56 description Spine2 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! end write memory |
Leaf2A-Leaf2B pair: Repeat configuration settings are shown in the following table.
Leaf2A to Spines | Leaf2B to Spines |
configure terminal interface Eth1/55 description Spine1 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! interface Eth1/56 description Spine2 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! end write memory | configure terminal interface Eth1/55 description Spine1 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! interface Eth1/56 description Spine2 mtu 9216 speed 100000 no shutdown ipv6 enable link state track leaf upstream ! end write memory |
In this deployment setup, there are three VxRail nodes connected to the leaf pair switches. The leaf switch ports enable connectivity to the VxRail nodes. Each leaf switch port connected to the VxRail node is configured as a trunk tagged port for the VxRail networks. There are two 25 GbE links in active/standby configuration from each VxRail node. One jumphost is connected to the leaf pair switches. The jumphost is used to access the VxRail Deployment Wizard. In this example, the jumphost uses an untagged leaf switch port. During the initial setup of VxRail nodes, the jumphost uses switchport access VLAN1 as default VLAN; post deployment, the jumphost uses VLAN1811.
Initial config uses access VLAN 1 for the initial deployment to access the VxRail external management. See note below for jumphost.
Leaf1A | Leaf1B |
configure terminal interface Eth1/1 description VxRail_01 no shutdown mtu 9216 switchport access vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! interface Eth1/2 description VxRail_02 mtu 9216 no shutdown switchport access vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! ! interface Eth1/15 description Jumphost01 no shutdown switchport access vlan 1 ! end write memory | configure terminal interface Eth1/1 description VxRail_01 no shutdown mtu 9216 switchport access vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! interface Eth1/2 description VxRail_02 mtu 9216 no shutdown switchport access vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! ! interface Eth1/15 description Jumphost01 no shutdown switchport access vlan 1 ! end write memory |
Note: The jumphost is initially configured for VLAN 1. Approximately 30 percent into initial VxRail deployment jumphost port use VLAN 1811 with a jumphost IP address corresponding to 1811 network, for example 172.18.11.201. Both Leaf1A and Leaf1B are shown, but only one switchport is needed. Adjust jumphost switchport speed as required for your jumphost interface. For this deployment, only one VxRail node is attached to rack 2 second leaf pair.
Note: Currently Dell Enterprise SONiC does not support LACP Individual, the examples used in this deployment guide do not use any LACP configuration from VxRail nodes to the Leaf switches. Continue to check for future Dell Enterprise SONiC Distribution updates on Dell.com.
Leaf2A-Leaf2B pair: The configuration settings are shown in the following table:
Leaf2A | Leaf2B |
configure terminal ! interface Eth1/1 description vxRail_03 mtu 9216 no shutdown switchport access Vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! end write memory | configure terminal ! interface Eth1/1 description vxRail_03 mtu 9216 no shutdown switchport access Vlan 1 switchport trunk allowed Vlan 1811-1814,3939 link state track leaf downstream ! end write memory |
Only Leaf1A and Leaf1B switch ports connect to external network for this deployment. Multiplier links are used for redundancy. The Layer 3 uplinks give the VxRail nodes access to the DNS and NTP services.
Note: Some uplinks may require fec RS configuration for connectivity depending on the external switch vendor and optics.
Leaf1A | Leaf1B |
configure terminal interface Eth1/53 description uplink_01 no shutdown ip vrf forwarding VrfTenant1 ip address 192.168.1.1/31 ! interface Eth1/54 description uplink_02 no shutdown ip vrf forwarding VrfTenant1 ! end write memory | configure terminal interface Eth1/53 description uplink_01 no shutdown ip vrf forwarding VrfTenant1 ip address 192.168.1.3/31 ! interface Eth1/54 description uplink_02 no shutdown ip vrf forwarding VrfTenant1 ! end write memory |
Configure the unnumbered BGP configurations on the leaf switches. An unnumbered interface does not have a user-configured IP address. BGP unnumbered interfaces use the extended next-hop encoding (ENHE) feature. Unnumbered BGP uses the link-local address to set up a BGP session with the neighbor. BGP unnumbered interfaces use the extended next-hop to advertise IPv4 routes with an IPv6 next hop.
Leaf1A | Leaf1B |
configure terminal router bgp 65101 vrf VrfTenant1 advertise-pip ip 10.0.2.1 remote-as external timers connect 30 advertisement-interval 0 ! address-family ipv4 unicast activate send-community both ! neighbor 192.168.1.0 peer-group Legacy ! neighbor 192.168.2.0 peer-group Legacy | configure terminal router bgp 65101 vrf VrfTenant1 advertise-pip ip 10.0.2.2 remote-as external timers connect 30 advertisement-interval 0 ! address-family ipv4 unicast activate send-community both ! neighbor 192.168.1.2 peer-group Legacy ! neighbor 192.168.2.2 peer-group Legacy |
Leaf1A | Leaf1B |
router bgp 65101 max-med on-startup 420 advertise-pip ip 10.0.2.1 peer-ip 10.0.2.2 | router bgp 65101 max-med on-startup 420 advertise-pip ip 10.0.2.2 peer-ip 10.0.2.1 |
Leaf1A | Leaf1B |
! allowas-in 2 neighbor interface Eth1/55 peer-group spine ! neighbor interface Eth1/56 peer-group spine ! | ! allowas-in 2 ! neighbor interface Eth1/55 peer-group spine ! neighbor interface Eth1/56 peer-group spine ! end write memory |
Leaf2A-Leaf2B pair: The following table shows repeat configuration settings:
Leaf2A | Leaf2B |
configure terminal router bgp 65102 vrf VrfTenant1 advertise-pip ip 10.0.2.3 max-med on-startup 420 advertise-pip ip 10.0.2.3 peer-ip 10.0.2.4 allowas-in 2 ! neighbor interface Eth1/55 peer-group spine ! neighbor interface Eth1/56 peer-group spine ! end write memory | configure terminal router bgp 65102 vrf VrfTenant1 address-family l2vpn evpn ! max-med on-startup 420 ! activate ! neighbor interface Eth1/55 peer-group spine ! neighbor interface Eth1/56 peer-group spine ! end write memory |