Configure the MX9116n FSE management, production, and uplink modules, which include both the MX9116n-A1 and MX9116n-A2 modules. Differences between the switch configuration for the MX9116n-A1 and MX9116n-A2 are stated. All other configurations are required on both switches.
Note: Some VLAN names and IDs might vary.
- Use SSH to connect to the switch.
- Check the code level and verify that operating system version 10.5.0.3P1 is installed:
MX9116N-A1# show version
Dell EMC Networking OS10 Enterprise
Copyright (c) 1999-2019 by Dell Inc. All Rights Reserved.
OS Version: 10.5.0.3P1
Build Version: 10.5.0.3P1.612
Build Time: 2019-12-18T19:19:25+0000
System Type: MX9116N-ON
Architecture: x86_64
- Verify full switch mode:
MX9116N-A1# show switch-operating-mode
Switch-Operating-Mode: Full Switch Mode
- Configure port groups.
The following table shows the port groups and cable types used for this POC:
Table 6. POC port groups and cable types
100g-2x | QSFP28-DD – 200 Gb DAC/AOC | MX7116n FEM and peer links |
25g-4x | 4-port 25 Gb DAC/AOC breakout cable | PowerFlex Manager connectivity |
40g-1x | 40 g SFP+ | Customer uplink |
- Change the following six ports-groups accordingly:
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 100g-2x
!
port-group 1/1/15
mode Eth 25g-4x
- Create additional required customer VLAN IDs, as shown in the following table:
Table 7. Customer VLAN IDs
interface vlan106
description ESXi_vmotion
no shutdown
!
interface vlan848
description ESXi_mgmt
no shutdown
- Run the following commands to configure bonds by using the required port channels, as shown in the following table:
Table 8. Required port channels
11 | MGMT/vMotion for the first CO node |
12 | MGMT/vMotion for second CO node |
13 | MGMT/vMotion for third CO node |
14 | MGMT/vMotion for fourth CO node |
21 | MGMT for Customer Uplink |
31 | MGMT/vMotion for PFM |
interface port-channel11
description "CO ESXimgmt/vmotion/prod 236"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,848
lacp fallback enable
mtu 9216
vlt-port-channel 11
!
interface port-channel12
description "CO ESXimgmt/vmotion/prod 237"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,848
lacp fallback enable
mtu 9216
vlt-port-channel 12
!
interface port-channel13
description "CO ESXimgmt/vmotion/prod 238"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,848
lacp fallback enable
mtu 9216
vlt-port-channel 13
!
interface port-channel14
description "CO ESXimgmt/vmotion/prod 239"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,848
lacp fallback enable
mtu 9216
vlt-port-channel 14
!
interface port-channel21
description “Customer uplink”
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 848
lacp fallback enable
mtu 9216
vlt-port-channel 21
!
interface port-channel31
description "PxFM ESXI/PROD"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,848
spanning-tree port type edge
spanning-tree bpduguard enable
spanning-tree guard root
speed 25000
lacp fallback enable
mtu 9216
vlt-port-channel 31
- Verify that port channels exist:
MX9116N-B1# show port-channel summary
Flags: D - Down I - member up but inactive P - member up and active
U - Up (port-channel) F - Fallback Activated
--------------------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
--------------------------------------------------------------------------------
12 port-channel11 (U) Eth DYNAMIC 1/1/1(P)
14 port-channel12 (U) Eth DYNAMIC 1/1/3(P)
16 port-channel13 (U) Eth DYNAMIC 1/71/1(P)
18 port-channel14 (U) Eth DYNAMIC 1/71/2(P)
20 port-channel21 (U) Eth DYNAMIC 1/1/33:1(P) 1/1/35:1(P)
31 port-channel31 (U) Eth STATIC 1/1/43:1(P)
1000 port-channel1000 (U) Eth STATIC 1/1/37(P) 1/1/38(P) 1/1/39(P) 1/1/40(P)
Note: At this step, member ports are not displayed as they have not yet been defined.
- Configure each MX9116n module to connect to its attached MX9117n module.
As shown in the following table, F0SHS13 is the fabric extender that is connected to the MX9116n-A1 port. Similarly, F0TJH13 is connected to the MX9116n-A2 port:
Table 9. Connection configuration
Port-group 1/1/1 Port-group 1/1/1 Mode Eth 25g-8x fabric-expander mode Unit-provision 1/71 F0SHS13 | Port-group 1/1/1 Mode Eth 25g-8x fabric-expander mode Unit-provision 1/71 F0TJH13 |
- Validate the MX9116-A1 connection to its attached MX9117n module:
MX9116N-A1# show unit-provision
Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+------------------+--------------------------+-------|
1 | 71 | F0SHS13 | F0SHS13 | up |
1 | 72 | | | |
1 | 73 | | | |
- Validate the MX9116-A2 connection to its attached MX9117n module:
MX9116N-A2# show unit-provision
Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+------------------+--------------------------+-------|
1 | 71 | F0TJH13 | F0TJH13 | up |
1 | 72 | | | |
1 | 73 | | | |
- Configure peer links and Virtual Link Trunking (VLT) configurations using ports 37 through 40 using two 200 Gb DAC/AOC cables.
In the following configuration:
- backup destination points to the peer switch.
- vlt-mac is identical on each switch, as it is a common virtual interface.
- Ports 37 through 38 are one physical QSFP28-DD port.
- Ports 39 through 40 are one physical QSFP28-DD port.
Table 10. Peer link and VLT configuration
vlt-domain 1 backup destination 10.234.90.244 discovery-interface ethernet1/1/37-1/1/40 primary-priority 4196 vlt-mac 00:11:22:33:44:5a ! interface ethernet1/1/37 description "Peer-link 9116-A2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/38 description "Peer-link 9116-A2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/39 description "Peer-link 9116-A2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/40 description "Peer-link 9116-A2" no shutdown no switchport flowcontrol receive off | interface ethernet1/1/37 description "Peer-link 9116-A1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/38 description "Peer-link 9116-A1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/39 description "Peer-link 9116-A1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/40 description "Peer-link 9116-A1" no shutdown no switchport flowcontrol receive off |
- Configure customer uplinks.
Each of the two switches contains two QSFP28 ports that are dedicated to customer uplinks. Ports 1/1/41 through 1/1/42 are used for this purpose. When setting the ports to 40g-1x mode (as configured in step 5), the use of 40 Gb DAC cables as uplinks is allowed. These ports become available in the configuration as 1/1/41:1 and 1/1/42:1.
interface ethernet1/1/41:1
description 9332
shut
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/42:1
description 9332
shut
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
- Enable the customer uplinks
Note: STP+ must be enabled on the customer switches before you enable the uplinks.
interface range ethernet1/1/41:1-1/1/42:1
no shut
- Ensure that the port channel is up:
MX9116N-B1# show vlt 1 vlt-port-detail
…
vlt-port-channel ID : 21
VLT Unit ID Port-Channel Status Configured ports Active ports
-------------------------------------------------------------------------------
1 port-channel21 up 2 2
* 2 port-channel21 up 2 2
Note: The asterisk (*) denotes the active switch. If the first uplink switch configured is active, only one entry is available, and the port value is 2 by default.
- Configure the compute-only node links.
The following example configures the correct internal ports for the compute-only nodes. Configure ports initially as trunked access ports; these ports are converted to Link Aggregation Control Protocol (LACP) later.
Note: The port mapping is different for internal nodes because of the two chassis having different switches per I/O module. (See MX740c compute nodes architecture.)
Table 11. Compute-only node links configuration
interface ethernet1/1/1 description "M740-236 – vmnic0" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-237 – vmnic0" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-238 – vmnic0" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/71/2 description "M740-239 – vmnic0" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off | interface ethernet1/1/1 description "M740-238 - vmnic1" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-239 - vmnic1" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-236 - vmnic1" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off ! interface ethernet1/71/2 description "M740-237 - vmnic1" no shutdown fec off switchport access vlan 848 mtu 9216 flowcontrol receive off |
- Configure the PowerFlex node links.
Note: On all four MX9116n FSE modules, PowerFlex is connected using 4-port 25 Gb breakout cables to port 41.
Table 12. PowerFlex node link configuration
interface ethernet1/1/43:1 description "PFM vmnic4 ESXI/PROD" no shutdown channel-group 31 mode active spanning-tree port type edge trunk spanning-tree bpduguard enable spanning-tree guard root speed 25000 lacp fallback enable no switchport flowcontrol receive off | interface ethernet1/1/43:1 description "PFM vmnic6 ESXI/PROD" no shutdown channel-group 31 no switchport flowcontrol receive off |
Note: If the PowerFlex appliance contains 10 Gb NICs, the port-group mode must be changed to mode Eth 10g-4x, and 4-port 10 Gb breakout cables must be used.
The following table provides examples of the full switch configurations for the POC lab switches:
Table 13. Example A1 and A2 configurations
Note: This information might be different for your deployment.
