When the management/production modules have been configured, configure the storage-only network connections, which are the MX9116n-B1 and MX9116n-B2 FSE modules.
Notes:
- These instructions have been modified from the original switch code to allow PowerFlex to use the correct port.
- Any switch configuration differences between the MX9116n-B2 and MX9116n-A1 modules are noted; otherwise, configure both switches as described in this guide.
- The VLAN IDs in these instructions might differ from the standard.
- Use SSH to connect to the switch.
- Check the code level and verify that OS Version 10.5.0.3P1 is installed:
MX9116N-B1# show version
Dell EMC Networking OS10 Enterprise
Copyright (c) 1999-2019 by Dell Inc. All Rights Reserved.
OS Version: 10.5.0.3P1
Build Version: 10.5.0.3P1.612
Build Time: 2019-12-18T19:19:25+0000
System Type: MX9116N-ON
Architecture: x86_64
- Verify full switch mode:
MX9116N-B1# show switch-operating-mode
Switch-Operating-Mode: Full Switch Mode
- Configure the port groups.
The following table shows the port groups and cable types used for this POC:
Table 14. Port group configuration
100g-2x | QSFP28-DD – 200 Gb DAC/AOC | MX7116n FEM and peer links |
25g-8x | 8-port 25 Gb DAC/AOC breakout cable | Storage-only node connectivity |
25g-4x | 4-port 25 Gb DAC/AOC breakout cable | PowerFlex Manager connectivity |
40g-1x | 40 g SFP+ | Customer uplink |
The following commands set the configuration of the 10 port-groups to the correct cable types:
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/5
mode Eth 25g-8x
!
port-group 1/1/6
mode Eth 25g-8x
!
port-group 1/1/9
mode Eth 25g-8x
!
port-group 1/1/10
mode Eth 25g-8x
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 40g-1x
!
port-group 1/1/15
mode Eth 40g-1x
- Create additional VLAN IDs, as shown in the following table:
Table 15. Additional VLAN IDs
858 | Flex MGMT |
152 | Flex Data1 |
160 | Flex Data2 |
168 | Flex Data3 |
176 | Flex Data4 |
interface vlan152
description data1
no shutdown
mtu 9216
!
interface vlan160
description data2
no shutdown
mtu 9216
!
interface vlan168
description data3
no shutdown
mtu 9216
!
interface vlan176
description data4
no shutdown
mtu 9216
!
interface vlan858
description flex_mgmt
no shutdown
- Run the following commands to configure bonds by using the required port channels shown in the following table:
Table 16. Required port channels
1 | Data1, Data3, and Flex MGMT for the first SO node |
2 | Data2 and Data4 for the first SO node |
3 | Data1, Data3, and Flex MGMT for the second SO node |
4 | Data2 and Data4 for the second SO node |
5 | Data1, Data3, and Flex MGMT for the third SO node |
6 | Data2 and Data4 for the third SO node |
7 | Data1, Data3, and Flex MGMT for the fourth storage-only node |
8 | Data2 and Data4 for the fourth storage-only node |
9 | Data1, Data3, and Flex MGMT for the fifth storage-only node |
10 | Data2 and Data4 for the fifth storage-only node |
15 | Data1, Data2, Data3, and Data4 for the first compute-only node |
16 | Data1, Data2, Data3, and Data4 for the second compute-only node |
17 | Data1, Data2, Data3, and Data4 for the third compute-only node |
18 | Data1, Data2, Data3, and Data4 for the fourth compute-only node |
22 | Customer uplink |
30 | Flex MGMT, Data1, Data2, Data3, and Data4 for PFM |
interface port-channel1
description "231 mgmt d1 d3"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,858
lacp fallback enable
mtu 9216
vlt-port-channel 1
!
interface port-channel2
description "231 d2 d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 168,176
lacp fallback enable
mtu 9216
vlt-port-channel 2
!
interface port-channel3
description "232 mgmt d1 d3"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,858
lacp fallback enable
mtu 9216
vlt-port-channel 3
!
interface port-channel4
description "232 d2 d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 168,176
lacp fallback enable
mtu 9216
vlt-port-channel 4
!
interface port-channel5
description "233 mgmt d1 d3"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,858
lacp fallback enable
mtu 9216
vlt-port-channel 5
!
interface port-channel6
description "233 d2 d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 168,176
lacp fallback enable
mtu 9216
vlt-port-channel 6
!
interface port-channel7
description "234 mgmt d1 d3"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,858
lacp fallback enable
mtu 9216
vlt-port-channel 7
!
interface port-channel8
description "234 d2 d4 "
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 168,176
lacp fallback enable
mtu 9216
vlt-port-channel 8
!
interface port-channel9
description "235 mgmt d1 d3"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,858
lacp fallback enable
mtu 9216
vlt-port-channel 9
!
interface port-channel10
description "235 d2 d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 168,176
lacp fallback enable
mtu 9216
vlt-port-channel 10
!
interface port-channel15
description "CO 236 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 15
!
interface port-channel16
description "CO 237 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 16
spanning-tree port type edge
!
interface port-channel17
description "CO 238 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 17
!
interface port-channel18
description "CO 239 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 18
!
interface port-channel22
description “Customer uplink”
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 858
lacp fallback enable
mtu 9216
vlt-port-channel 22
!
interface port-channel30
description "PFM D1-D4/STIOMGMT"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 152,160,168,176,858
lacp fallback enable
mtu 9216
vlt-port-channel 30
- Verify that the port channels exist:
MX9116N-B1# show port-channel summary
Flags: D - Down I - member up but inactive P - member up and active
U - Up (port-channel) F - Fallback Activated
--------------------------------------------------------------------------------
Group Port-Channel Type Protocol Member Ports
--------------------------------------------------------------------------------
1 port-channel1 (U) Eth DYNAMIC 1/1/25:1(P)
2 port-channel2 (U) Eth DYNAMIC 1/1/27:1(P)
3 port-channel3 (U) Eth DYNAMIC 1/1/25:2(P)
4 port-channel4 (U) Eth DYNAMIC 1/1/27:2(P)
5 port-channel5 (U) Eth DYNAMIC 1/1/25:3(P)
6 port-channel6 (U) Eth DYNAMIC 1/1/27:3(P)
7 port-channel7 (U) Eth DYNAMIC 1/1/33:1(P)
8 port-channel8 (U) Eth DYNAMIC 1/1/35:1(P)
9 port-channel9 (U) Eth DYNAMIC 1/1/33:2(P)
10 port-channel10 (U) Eth DYNAMIC 1/1/35:2(P)
15 port-channel15 (U) Eth DYNAMIC 1/1/1(P)
16 port-channel16 (U) Eth DYNAMIC 1/1/3(P)
17 port-channel17 (U) Eth DYNAMIC 1/71/1(P)
18 port-channel18 (U) Eth DYNAMIC 1/71/2(P)
22 port-channel22 (U) Eth DYNAMIC 1/1/41:1(P) 1/1/42:1(P)
30 port-channel30 (U) Eth DYNAMIC 1/1/43:1(P)
1000 port-channel1000 (U) Eth STATIC 1/1/37(P) 1/1/38(P) 1/1/39(P) 1/1/40(P)
Note: At this step, member ports do not contain links.
- Configure the fabric extenders to connect to the MX7116n FEM modules. As shown in the following table, F0SGS 13 is the fabric extender that is connected to MX9116n-B1 port 1 and similarly F0THS13 is connected to MX9116n-B2 port 1.
Table 17. Fabric extender configuration
|
|
Port-group 1/1/1 Mode Eth 25g-8x fabric-expander mode Unit-provision 1/71 F0SGS13 | Port-group 1/1/1 Mode Eth 25g-8x fabric-expander mode Unit-provision 1/71 F0THS13 |
- Validate that the MX9116-B1 module is connected to the correct MX7116n FEM:
MX9116N-B1# show unit-provision
Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+------------------+--------------------------+-------|
1 | 71 | F0SGS13 | F0SGS13 | up |
1 | 72 | | | |
| 73 | | | |
- Validate that MX9116-B2 module is connected to the correct MX7116n FEM:
MX9116N-B2# show unit-provision
Node ID | Unit ID | Provision Name | Discovered Name | State |
---------+---------+------------------+--------------------------+-------|
1 | 71 | F0THS13 | F0THS13 | up |
1 | 72 | | | |
| 73 | | | |
- Configure peer links and VLT configurations using ports 37 through 40 using two 200 Gb DAC/AOC cables.
Table 18. Peer link and VLT configuration
vlt-domain 1 backup destination 10.234.90.245 discovery-interface ethernet1/1/37-1/1/40 primary-priority 4196 vlt-mac 00:11:22:33:44:47 ! interface ethernet1/1/37 description "Peer-link 9116-B2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/38 description "Peer-link 9116-B2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/39 description "Peer-link 9116-B2" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/40 description "Peer-link 9116-B2" no shutdown no switchport flowcontrol receive off | vlt-domain 1 backup destination 10.234.90.243 discovery-interface ethernet1/1/37-1/1/40 primary-priority 8192 vlt-mac 00:11:22:33:44:47 ! interface ethernet1/1/37 description "Peer-link 9116-B1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/38 description "Peer-link 9116-B1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/39 description "Peer-link 9116-B1" no shutdown no switchport flowcontrol receive off ! interface ethernet1/1/40 description "Peer-link 9116-B1" no shutdown no switchport flowcontrol receive off |
- Configure storage-only node links.
Note: On each storage MX9116n FSE (B1, B2), ports 25 through 28 and ports 33 through 36 are used with 25g-8x breakout cables to connect to the external storage-only nodes. The connections alternate among the nodes to provide resiliency. See the following example and table for clarification.
Example: Switch port 1/1/33 was set to allow a 25g-8x breakout cable so that eight virtual ports in the switch are created in the following manner: 1/1/33:1 through 1/1/33:8.
Note: Not all breakout cable ports are used in this POC.
Table 19. Switch port connections
1/1/33 | 1/1/33:1 | 1 | Node 1, port 03:01 |
1/1/33 | 1/1/33:2 | 2 | Node 2, port 03:01 |
1/1/33 | 1/1/33:3 | 3 | Node 3, port 03:01 |
The ports are configured with LACP. The following example configures the correct external ports for the storage-only nodes (applies to both modules):
interface ethernet1/1/25:1
description "231 - p2p1 - mgmt d1 d3 rep1"
no shutdown
channel-group 1 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/25:2
description "232 - p2p1 - mgmt d1 d3 rep1"
no shutdown
channel-group 3 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/25:3
description "233 - p2p1 - mgmt d1 d3 rep1"
no shutdown
channel-group 5 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/27:1
description "231 - p3p2 - d2 d4 rep2"
no shutdown
channel-group 2 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/27:2
description "232 - p3p2 - d2 d4 rep2"
no shutdown
channel-group 4 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/27:3
description "233 - p3p2 - d2 d4 rep2"
no shutdown
channel-group 6 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/33:1
description "234 - p2p1 - mgmt d1 d3 rep1"
no shutdown
channel-group 7 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/33:2
description "235 - p2p1 - mgmt d1 d3 rep1"
no shutdown
channel-group 9 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/35:1
description "234 - p3p2 - d2 d4 rep2"
no shutdown
channel-group 8 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/35:2
description "235 - p3p2 - d2 d4 rep2"
no shutdown
channel-group 10 mode active
no switchport
mtu 9216
flowcontrol receive off
!
- Configure the compute-only node links.
Note: The following example configures the correct internal ports for the compute-only nodes so that they have access to the storage network. The port mapping is different for internal nodes because the two chassis have different switches per I/O module.
Table 20. Compute-only node links configuration
interface ethernet1/1/1 description "M740-236 – vmnic2" no shutdown channel-group 15 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-237 – vmnic2" no shutdown channel-group 16 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-238 – vmnic2" no shutdown channel-group 17 mode active fec off no switchport mtu 9216 flowcontrol receive off! ! interface ethernet1/71/2 description "M740-239 - vmnic3" no shutdown channel-group 18 mode active fec off no switchport mtu 9216 flowcontrol receive off | interface ethernet1/1/1 description "M740-238 – vmnic4" no shutdown channel-group 17 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/1/3 description "M740-239 – vmnic4" no shutdown channel-group 18 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/1 description "M740-236 – vmnic4" no shutdown channel-group 15 mode active fec off no switchport mtu 9216 flowcontrol receive off ! interface ethernet1/71/2 description "M740-237 - vmnic0" no shutdown channel-group 16 mode active fec off no switchport mtu 9216 flowcontrol receive off |
- Configure PowerFlex node links:
Note: On all four MX9116n FSE modules, PowerFlex is connected using 4-port 25 Gb breakout cables to port 41.
Table 21. PowerFlex node links configuration
interface ethernet1/1/43:1 description "PFM vmnic5 ESXI/PROD" no shutdown channel-group 30 no switchport flowcontrol receive off | interface ethernet1/1/43:1 description "PFM vmnic7 ESXI/PROD" no shutdown channel-group 30 no switchport flowcontrol receive off |
If the PowerFlex appliance contains 10 Gb NICs, the port-group mode must be changed to mode Eth 10g-4x and 4-port 10 Gb breakout cables must be used.
The following table provides examples of the full switch configurations for the POC lab switches:
Table 22. Example B1 and B2 configurations
Note: This information might be different for your deployment.
