Example 2: Layer 3 with Dell leaf and spine
Example 2: Layer 3 with Dell leaf and spine
-
Example 2: Layer 3 with Dell leaf and spine
Switches using eBGP
Dell Networking S5248F-ON switches are used at the leaf layer and Dell Networking Z9264F-ON switches are used at the spine layer. While the S5248F-ON configuration is shown in this section, Figure 13 provides eBGP configuration examples to build the Layer 3 leaf-spine topology.
Figure 14. Example 2: Layer 3 leaf-spine topology
The following configuration details describe S5248F-Leaf 1 and S5248F-Leaf 2. The configurations for S5248F-Leaf 3 and S5248F-Leaf 4 are similar.
Note: On S5248F-ON switches running OS10, Telnet is disabled and SSH is enabled by default. The S5248F-ON has admin/admin default credentials. The switch is accessible using SSH with those credentials. It is recommended to change the admin password to something more complex after login.
- Set the hostname and configure the OOB management interface and default gateway.
- Enable RSTP as a precaution.
S5248F-Leaf 1 is configured as the primary RSTP root bridge using the spanning-tree priority 0 command. S5248F-Leaf2 is configured as the secondary RSTP root bridge using the spanning-tree priority 4096 command.
S5248F-Leaf 1
S5248F-Leaf 2
enable
configure terminal
hostname S5248F-Leaf1
interface mgmt 1/1/1
no IP address dhcp
IP address 100.67.170.30/24
no shutdown
management route 0.0.0.0/0
100.67.170.254
spanning-tree mode rstp
spanning-tree rstp priority 0
enable
configure terminal
hostname S5248F-Leaf2
interface mgmt 1/1/1
no IP address dhcp
IP address 100.67.170.29/24
no shutdown
management route 0.0.0.0/0
100.67.170.254
spanning-tree mode rstp
spanning-tree rstp priority 4096
- Configure the VLT interconnect between S5248F-Leaf 1 and S5248F-Leaf 2.
In this configuration, remove each interface from Layer 2 mode with the no switchport setting for interfaces eth 1/1/47-1/1/48.
- Enter the VLT domain and add the backup destination.
- Add the interfaces participating in VLTi. Finally enable VLT peer routing.
S5248F-Leaf 1
S5248F-Leaf 2
interface range Ethernet 1/1/47-1/1/48
no switchport
vlt-domain 127
backup destination 100.67.170.29
discovery-interface Ethernet 1/1/47
discovery-interface ethernet 1/1/48
peer-routing
interface range Ethernet 1/1/47-1/1/48
no switchport
vlt-domain 127
backup destination 100.67.170.30
discovery-interface Ethernet 1/1/47
discovery-interface ethernet 1/1/48
peer-routing
- Create a server facing VLAN interface. Use the same VLAN ID on both leaf switches.
- Assign an IP address to the VLAN interface. The address must be unique but on the same network on both leaf switches.
- Configure VRRP to use VRRP version 3. Create a VRRP group and specify the group’s virtual IP address.
Note: In this example, the NIC for Server 1 is configured as an LACP NIC team. It is assigned the IP address 172.16.1.7/24. The VRRP VIP address, 172.16.1.254, is specified as the default gateway for Server 1.
S5248F-Leaf 1
S5248F-Leaf 2
interface Vlan 50
IP address 172.16.1.1/24
no shutdown
exit
vrrp version 3
interface vlan 50
vrrp-group 50
virtual-IP address 172.16.1.254
interface Vlan 50
ip address 172.16.1.2/24
no shutdown
exit
vrrp version 3
interface vlan 50
vrrp-group 50
virtual-address 172.16.1.254
- Configure each downstream server-facing interface with an LACP port channel.
- Configure each port channel for VLT. Port channel 10 connects downstream to Server 1 and is configured as an RSTP edge port.
S5248F-Leaf 1
S5248F-Leaf 2
interface port channel 10
description "Server 1"
switchport access vlan 50
vlt-port-channel 10
spanning-tree port type edge
interface Ethernet 1/1/1
description "Server 1"
channel-group 10 mode active
no shutdown
interface port channel 10
description "Server 1"
switchport access vlan 50
vlt-port-channel 10
spanning-tree port type edge
interface Ethernet 1/1/1
description "Server 1"
channel-group 10 mode active
no shutdown
- The two upstream Layer 3 interfaces connected to the spine switches are configured.
- Assign IP addresses as shown in Table 2.
- Configure a loopback interface to be used as the router ID. This is used with BGP or OSPF.
Note: If multiple loopback interfaces exist on a system, the interface with the highest numbered IP address is used as the router ID. This configuration only uses one loopback interface.
S5248F-Leaf 1
S5248F-Leaf 2
interface Ethernet 1/1/53
description "Z9264F-Spine1 eth 1/1/1"
no switchport
IP address 192.168.1.1/31
no shutdown
interface Ethernet 1/1/54
description "Z9264F-Spine2 eth 1/1/1
no switchport
IP address 192.168.2.1/31
no shutdown
interface loopback 0
description "Router ID"
IP address 10.0.2.1/32
no shutdown
interface Ethernet 1/1/53
description "Z9264F-Spine1 eth 1/1/2"
no switchport
IP address 192.168.1.3/31
no shutdown
interface Ethernet 1/1/54
description "Z9264F-Spine2 eth 1/1/2”
no switchport
IP address 192.168.2.3/31
no shutdown
interface loopback 0
description "Router ID"
IP address 10.0.2.2/32
no shutdown
- Configure a route map and IP prefix-list to redistribute all loopback addresses and leaf networks using BGP. The command seq 10 permit 10.0.0.0/8 ge 24 includes all addresses in the 10.0.0.0/8 address range with a mask greater than or equal to 24. This includes all loopback addresses used as router IDs and the 10.60.1.0/24 network used on leaf switches 3 and 4.
- The command seq 20 permit 172.16.0.0/16 ge 24 includes the 172.16.1.0/24 network used on Leaf switches 1 and 2.
S5248F-Leaf 1
S5248F-Leaf 2
route-map spine-leaf permit 10
match IP address prefix-list spine-leaf
ip prefix-list spine-leaf seq 10 permit 10.0.0.0/8 ge 24
ip prefix-list spine-leaf seq 20 permit 172.16.0.0/16 ge 24
route-map spine-leaf permit 10
match IP address prefix-list spine-leaf
ip prefix-list spine-leaf seq 10 permit 10.0.0.0/8 ge 24
ip prefix-list spine-leaf seq 20 permit 172.16.0.0/16 ge 24
- Configure UFD to shut the downstream interfaces if all uplinks fail. The hosts attached to the switch use the remaining LACP LAG member to continue sending traffic across the fabric.
- Exit configuration mode and save the configuration.
S5248F-Leaf 1
S5248F-Leaf 2
uplink-state-group 1
description "Disable downstream ports in event all uplinks fail"
downstream port channel 10
upstream Ethernet 1/1/53
upstream ethernet 1/1/54
enable
uplink-state-group 1
description "Disable downstream ports in event all uplinks fail"
downstream port channel 10
upstream Ethernet 1/1/53
upstream ethernet 1/1/54
enable
Use the following steps to configure eBGP.
- Enable eBGP with the router bgp ASN command.
The bgp bestpath as-path multipath-relax enables ECMP. The maximum-eBGP 2 command specifies the maximum number of parallel paths to a destination to add to the routing table. In this topology, there are two equal cost best paths from a spine to a host, one to each leaf that the host is connected.
- Configure BGP neighbors and enable neighbor fall-over.
- Restart enables the data plane to continue forwarding traffic for a time if the BGP process fails or quits.
- Set BGP hello and hold down timers to three and nine seconds, respectively. Neighbor fall-over will trigger route withdrawal when IP connectivity between BGP is lost. The more aggressive timers (default is 60 seconds hello and 180 seconds hold down) are for backup.
- Set the advertisement interval to one second. This prevents BGP speakers from advertising updates immediately upon receipt. Instead, they will advertise them in batched intervals of one second. This delay prevents overhead.
- Exit configuration mode and save the configuration.
S5248F-Leaf 1
S5248F-Leaf 2
router bgp 64701
bestpath as-path multipath-relax
maximum-paths ebgp 2
address-family ipv4 unicast
redistribute connected route-map spine-leaf
exit
graceful-restart role receiver-only
template spine-leaf
fall-over
advertisement-interval 1
timers 3 9
exit
neighbor 192.168.1.0
remote-as 64601
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.2.0
remote-as 64602
inherit template spine-leaf
no shutdown
end
write memory
router bgp 64702
bestpath as-path multipath-relax
maximum-paths ebgp 2
address-family ipv4 unicast
redistribute connected route-map spine-leaf
exit
graceful-restart role receiver-only
template spine-leaf
fall-over
advertisement-interval 1
timers 3 9
exit
neighbor 192.168.1.2
remote-as 64601
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.2.2
remote-as 64602
inherit template spine-leaf
no shutdown
end
write memory
- Enable BFD in BGP.
- Establish BGP ASN assignment.
- Establish BFD sessions with all neighbors.
- Exit configuration mode and save the configuration.
S5248F-Leaf 1
S5248F-Leaf 2
bfd enable
router bgp 64701
bfd all-neighbors interval 200 min_rx 200 multiplier 6 role active
bfd enable
router bgp 64702
bfd all-neighbors interval 200 min_rx 200 multiplier 6 role active
Note: Establish BFD sessions with ASN eBGP neighbors on leaf2,3,4, spine1 and spine 2.
Note: By default, BGP uses the following BFD parameters for it's neighbors: min_tx = 200 msec, min_rx = 200 msec, multiplier = 3, role = active. If BFD will be given high priority if it is configured under interface context.
Use the following steps to configure Z9264F-Spine 1 and Z9264F-Spine 2.
- Set the hostname, configure the OOB management interface, and use the default gateway.
Z9264F-Spine 1
Z9264F-Spine 2
enable
configure terminal
hostname Z9264F-Spine1
interface mgmt 1/1/1
no IP address dhcp
IP address 100.67.173.36/24
no shutdown
management route 0.0.0.0/0 100.67.173.254
enable
configure terminal
hostname Z9264F-Spine2
interface mgmt 1/1/1
no IP address dhcp
IP address 100.67.173.35/24
no shutdown
management route 0.0.0.0/0 100.67.173.254
- Configure the four point-to-point interfaces connected to leaf switches.
- Assign IP addresses as shown in Table 2.
- Configure a loopback interface to be used as the router ID. This is used with BGP or OSPF.
Z9264F-Spine 1
Z9264F-Spine 2
interface Ethernet 1/1/1
description "S5248F-Leaf1 eth 1/1/53"
no switchport
IP address 192.168.1.0/31
no shutdown
interface Ethernet 1/1/2
description "S5248F-Leaf2 eth 1/1/53"
no switchport
IP address 192.168.1.2/31
no shutdown
interface Ethernet 1/1/3
description "S5248F-Leaf3 eth 1/1/53"
no switchport
IP address 192.168.1.4/31
no shutdown
interface Ethernet 1/1/4
description "S5248F-Leaf4 eth 1/1/53"
no switchport
IP address 192.168.1.6/31
no shutdown
interface loopback 0
description "Router ID"
IP address 10.0.1.1/32
no shutdown
interface Ethernet 1/1/1
description "S5248F-Leaf1 eth 1/1/54"
no switchport
IP address 192.168.2.0/31
no shutdown
interface Ethernet 1/1/2
description "S5248F-Leaf2 eth 1/1/54"
no switchport
IP address 192.168.2.2/31
no shutdown
interface Ethernet 1/1/3
description "S5248F-Leaf3 eth 1/1/54"
no switchport
IP address 192.168.2.4/31
no shutdown
interface Ethernet 1/1/4
description "S5248F-Leaf4 eth 1/1/54"
no switchport
IP address 192.168.2.6/31
no shutdown
interface loopback 0
description "Router ID"
IP address 10.0.1.2/32
no shutdown
- Configure a route map and IP prefix-list to redistribute all loopback addresses and leaf networks via BGP or OSPF. The command seq 10 permit 10.0.0.0/8 ge 24 includes all addresses in the 10.0.0.0/8 address range with a mask greater than or equal to 24. This includes all addresses used as router IDs and the 10.60.1.0/24 network used on leaf switches 3 and 4
The command seq 20 permit 172.16.0.0/16 ge 24 includes the 172.16.1.0/24 network used on Leaf switches 1 and 2.
Z9264F-Spine 1
Z9264F-Spine 2
route-map spine-leaf permit 10
match IP address prefix-list spine-leaf
ip prefix-list spine-leaf seq 10 permit 10.0.0.0/8 ge 24
ip prefix-list spine-leaf seq 20 permit 172.16.0.0/16 ge 24
route-map spine-leaf permit 10
match IP address prefix-list spine-leaf
ip prefix-list spine-leaf seq 10 permit 10.0.0.0/8 ge 24
ip prefix-list spine-leaf seq 20 permit 172.16.0.0/16 ge 24
Use the following steps to configure eBGP.
- Enable eBGP with the router bgp ASN command.
- The bgp bestpath as-path multipath-relax enables ECMP. The maximum-eBGP 2 command specifies the maximum number of parallel paths to a destination to add to the routing table. In this topology, there are two equal cost best paths from a spine to a host, one to each leaf that the host is connected.
- Configure BGP neighbors and enable neighbor fall-over.
- Restart enables the data plane to continue forwarding traffic for a time if the BGP process fails or quits.
- Set BGP hello and hold down timers to three and nine seconds, respectively. Neighbor fall-over will trigger route withdrawal when IP connectivity between BGP is lost. The more aggressive timers (default is 60 seconds hello and 180 seconds hold down) are for backup.
- Set the advertisement interval to one second. This prevents BGP speakers from advertising updates immediately upon receipt. Instead, they will advertise them in batched intervals of one second. This delay prevents overhead.
- Exit configuration mode and save the configuration.
Z9264F-Spine 1
Z9264F-Spine 2
router bgp 64601
bestpath as-path multipath-relax
maximum-paths ebgp 2
address-family ipv4 unicast
redistribute connected route-map spine-leaf
exit
graceful-restart role receiver-only
template spine-leaf
fall-over
advertisement-interval 1
timers 3 9
exit
neighbor 192.168.1.1
remote-as 64701
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.1.3
remote-as 64702
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.1.5
remote-as 64703
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.1.7
remote-as 64704
inherit template spine-leaf
no shutdown
router bgp 64602
bestpath as-path multipath-relax
maximum-paths ebgp 2
address-family ipv4 unicast
redistribute connected route-map spine-leaf
exit
graceful-restart role receiver-only
template spine-leaf
fall-over
advertisement-interval 1
timers 3 9
exit
neighbor 192.168.2.1
remote-as 64701
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.2.3
remote-as 64702
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.2.5
remote-as 64703
inherit template spine-leaf
no shutdown
exit
neighbor 192.168.2.7
remote-as 64704
inherit template spine-leaf
no shutdown
- Enable BFD in BGP.
- Establish BGP ASN assignment.
- Establish BFD sessions with all neighbors.
- Exit configuration mode and save the configuration.
Note: By default, BGP uses the following BFD parameters for it's neighbors: min_tx = 200 msec, min_rx = 200 msec, multiplier = 3, role = active. If BFD will be given high priority if it is configured under interface context.
S9264F-Spine 1
S9264F-Spine 2
bfd enable
router bgp 64601
bfd all-neighbors interval 200 min_rx 200 multiplier 6 role active
bfd enable
router bgp 64602
bfd all-neighbors interval 200 min_rx 200 multiplier 6 role active