Two-layer configuration
Two-layer configuration
-
The MX7000 compute nodes used two-layer architecture explained in Figure 2 to connect to the existing PowerFlex storage-only nodes.
The same VLANs and subnets for the storage network were extended across the entire two-layer network allowing unrouted communication between PowerFlex and MX7000.
The two-layer configuration is based on a standard storage-only build for PowerFlex rack using PowerFlex Manager. Any modifications required are detailed in the sections to follow. The MX7000 was deployed based on Dell EMC practices, and the PowerFlex rack was deployed following the PowerFlex design standards.
The following changes were applied to the MX switches, the PowerFlex aggregation, and access switches:
- VLANs were pushed northbound of both the MX9116n modules and the PowerFlex 93180 aggregation switches.
- The MX7000 Sleds were configured with the same Storage data networks used with the storage-only nodes in the PowerFlex rack.
The following table lists the VLANs that were extended to the customer switches:
Table 4. VLAN details
VLAN ID
Purpose
848
Flex node management
858
Flex storage management
152
Flex data1
160
Flex data2
168
Flex data3
176
Flex data4
Customer switches
The customer switches provided connectivity between the MX7000 chassis and the PowerFlex rack. The storage VLANs that were extended northbound of the MX9116n and the 9332 traversed this pair of switches.
Note: VLANs 848 and 858 were Layer 3, and the gateways existed northbound of these customer switches.
The sample code below is from the Dell EMC laboratory Cisco 9332 switches, which operated as the customer switches.
Note: Some VLAN names may vary.
vlan 152
name data1
vlan 160
name data2
vlan 168
name data3
vlan 176
name data4
vlan 848
name ESXi_mgmt
vlan 858
name SIO_mgmt
interface port-channel20
description "PowerFlex Interconnect"
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
no lacp suspend-individual
vpc 20
interface port-channel51
description 9332 Peer-link
switchport
switchport mode trunk
spanning-tree port type network
vpc peer-link
interface Ethernet1/19
description MX9116 B
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 22 mode active
no shutdown
interface Ethernet1/20
description MX9116 A
switchport
switchport mode trunk
switchport trunk allowed vlan 848,858
mtu 9216
channel-group 21 mode active
no shutdown
interface Ethernet1/21
description MX9116 B
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 22 mode active
no shutdown
interface Ethernet1/22
description MX9116 A
switchport
switchport mode trunk
switchport trunk allowed vlan 848,858
mtu 9216
channel-group 21 mode active
no shutdown
interface Ethernet1/25
description peer-link
switchport
switchport mode trunk
channel-group 51 mode active
no shutdown
interface Ethernet1/26
description peer-link
switchport
switchport mode trunk
channel-group 51 mode active
no shutdown
interface Ethernet1/31
description to PowerFlex Rack
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
interface Ethernet1/32
description to PowerFlex Rack
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
PowerFlex rack
Both the PowerFlex rack access and aggregation switches required changes to allow the two-layer storage VLANs to reach the customer switches. The following table lists the storage VLANs that were extended to the customer switches:
Table 5. PowerFlex rack VLAN details
VLAN ID
Purpose
152
Storage Data 1
160
Storage Data 2
168
Storage Data 3
176
Storage Data 4
Note: The VLAN IDs used in the POC are not as per PowerFlex standard.
Access switches
Cisco 93180 access switches were used in the PowerFlex rack and configured as per PowerFlex standards, except for extending the Data1─ Data4 VLANs north.
The following table shows the required modification for this solution:
Table 6. Access switches VLAN details
Switch 1
Switch 2
interface Ethernet1/49
description wolfpack-n9332-l2agg-a
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
spanning-tree guard loop
mtu 9216
channel-group 41 mode active
no shutdown
interface Ethernet1/50
description wolfpack-n9332-l2agg-b
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
spanning-tree guard loop
mtu 9216
channel-group 41 mode active
no shutdown
interface port-channel41
description "wolfpack uplink"
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
mtu 9216
no lacp suspend-individual
vpc 41
interface Ethernet1/49
description wolfpack-n9332-l2agg-b
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
spanning-tree guard loop
mtu 9216
channel-group 41 mode active
no shutdown
interface Ethernet1/50
description wolfpack-n9332-l2agg-a
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
spanning-tree guard loop
mtu 9216
channel-group 41 mode active
no shutdown
interface port-channel41
description “wolfpack uplink”
switchport
switchport mode trunk
switchport trunk native vlan 10
switchport trunk allowed vlan 152,160,168,176,858,848
spanning-tree port type network
mtu 9216
no lacp suspend-individual
vpc 41
Aggregation switches
Cisco 9332 aggregation switches were used in the PowerFlex rack and were configured per PowerFlex standards, except for extending the storage data VLANs north.
The following table shows the required modification for this solution:
Table 7. Aggregation switch VLAN details
Aggregation Switch A
Aggregation Switch B
interface Ethernet1/16
description to Customer SwitchA
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
interface Ethernet1/20
description To Customer SwitchB
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
interface port-channel20
description To Customer Switches
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
no lacp suspend-individual
vpc 20
interface Ethernet1/16
description to Customer SwitchB
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
interface Ethernet1/20
description To Customer SwitchA
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
channel-group 20 mode active
no shutdown
interface port-channel20
description To Customer Switches
switchport
switchport mode trunk
switchport trunk allowed vlan 152,160,168,176,848,858
mtu 9216
no lacp suspend-individual
vpc 20
MX7000
This section details the PowerEdge MX7000 changes required for allowing Layer 2 connectivity to extend through the customer network and describes the applicable MX7000 modular configuration.
Network I/O modules
The Dell EMC Networking MX9116n Fabric Switching Engines (FSE) provided both northbound access to the customer switches and internal connectivity to the compute nodes. The Dell EMC MX7116n Fabric Expander Modules provided connectivity to multiple chassis.
Figure 4. MX9116n modules
Note: Management and customer VLANs were on the A switches, while storage data VLANs were on the B switches.
Management and Customer traffic - MX9116n FSE
The Dell EMC Networking MX9116n Fabric Switching Engines in Slot A were dedicated to management and customer VLANs.
Table 8. LACP port channel definitions and purposes
Port Channel ID
Purpose
Target
11
ESXi MGMT
CO Node 1
12
ESXi MGMT
CO Node 2
13
ESXi MGMT
CO Node 3
14
ESXi MGMT
CO Node 4
21
Customer Uplink
9332A, 9332B
Table 9. Switch configuration details
MX9116n A1
MX9116n A2
unit-provision 1/71 F0SHS13
!
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 40g-1x
!
interface vlan848
description ESXi_mgmt
no shutdown
!
interface port-channel21
description 9332
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 848
lacp fallback enable
mtu 9216
vlt-port-channel 21
!
interface ethernet1/1/1
description "M740-236 - vmnic0"
no shutdown
channel-group 11 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/3
description "M740-237 - vmnic0"
no shutdown
channel-group 12 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/37
description "Peer-link 9116-A2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/38
description "Peer-link 9116-A2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/39
description "Peer-link 9116-A2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/40
description "Peer-link 9116-A2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/41:1
description customer-9332
no shutdown
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/42:1
description customer-9332
no shutdown
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/1
description "M740-238 - vmnic0"
no shutdown
channel-group 13 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/2
description "M740-239 - vmnic0"
no shutdown
channel-group 14 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
unit-provision 1/71 F0TJH13
!
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 40g-1x
!
interface vlan848
description ESXi_mgmt
no shutdown
!
interface port-channel21
description 9332
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 848
lacp fallback enable
mtu 9216
vlt-port-channel 21
!
interface ethernet1/1/1
description "M740-238 - vmnic1"
no shutdown
channel-group 11 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/3
description "M740-239 - vmnic1"
no shutdown
channel-group 12 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/37
description "Peer-link 9116-A1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/38
description "Peer-link 9116-A1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/39
description "Peer-link 9116-A1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/40
description "Peer-link 9116-A1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/41:1
description customer-9332
no shutdown
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/42:1
description customer-9332
no shutdown
channel-group 21 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/1
description "M740-236- vmnic1"
no shutdown
channel-group 13 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/2
description "M740-237 - vmnic1"
no shutdown
channel-group 14 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
Note:
- VLAN 848, designated for ESXi management, was also used as the customer VLAN.
- The “show unit-provision” command displayed the unit ID and service tag of the Fabric Expanders attached to the Fabric Switching Engine.
- MX9116n port-group settings were determined based on the type of wire used for the physical port.
Storage traffic ─ MX9116n FSE
The Dell EMC Networking MX9116n Fabric Switching Engines in Slot B were dedicated to storage traffic and allowed all VLANs northbound.
Table 10. LACP port channel definitions and purposes
Port Channel ID
Purpose
Target
15
Data1-Data4, vMotion
CO Node 1
16
Data1-Data4, vMotion
CO Node 2
17
Data1-Data4, vMotion
CO Node 3
18
Data1-Data4, vMotion
CO Node 4
22
Customer Uplink
9332A, 9332B
Table 11. VLAN definitions and purposes
VLAN ID
Purpose
152
Storage Data 1
160
Storage Data 2
168
Storage Data 3
176
Storage Data 4
106
vMotion
Table 12. Switch configuration
MX9116n B1
MX9116n A2
unit-provision 1/71 F0SGS13
!
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 40g-1x
!
interface vlan104
description pxe
no shutdown
!
interface vlan106
description vmotion
no shutdown
!
interface vlan152
description data1
no shutdown
mtu 9216
!
interface vlan160
description data2
no shutdown
mtu 9216
!
interface vlan168
description data3
no shutdown
mtu 9216
!
interface vlan176
description data4
no shutdown
mtu 9216
!
interface vlan858
description fos_mgmt
no shutdown
!
interface port-channel15
description "CO 236 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 15
!
interface port-channel16
description "CO 237 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 16
spanning-tree port type edge
!
interface port-channel17
description "CO 238 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,152,160,168,176
lacp fallback enable
mtu 9216
vlt-port-channel 17
!
interface port-channel18
description "CO 239 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,720-723
lacp fallback enable
mtu 9216
vlt-port-channel 18
!
interface port-channel22
description 9332
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 104,106,720-723,858
lacp fallback enable
mtu 9216
vlt-port-channel 22
!
interface ethernet1/1/1
description "M740-236 - vmnic2"
no shutdown
channel-group 15 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/3
description "M740-237 - vmnic2"
no shutdown
channel-group 16 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/37
description "Peer-link 9116-B1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/38
description "Peer-link 9116-B1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/39
description "Peer-link 9116-B1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/40
description "Peer-link 9116-B1"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/41:1
description customer-9332
no shutdown
channel-group 22 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/42:1
description customer-9332
no shutdown
channel-group 22 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/1
description "M740-238 - vmnic2"
no shutdown
channel-group 17 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/2
description "M740-239 - vmnic2"
no shutdown
channel-group 18 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
unit-provision 1/71 F0THS13
!
port-group 1/1/1
mode Eth 25g-8x fabric-expander-mode
!
port-group 1/1/11
mode Eth 100g-2x
!
port-group 1/1/12
mode Eth 100g-2x
!
port-group 1/1/13
mode Eth 40g-1x
!
port-group 1/1/14
mode Eth 40g-1x
!
interface vlan104
description pxe
no shutdown
!
interface vlan106
description vmotion
no shutdown
!
interface vlan152
description data1
no shutdown
mtu 9216
!
interface vlan160
description data2
no shutdown
mtu 9216
!
interface vlan168
description data3
no shutdown
mtu 9216
!
interface vlan176
description data4
no shutdown
mtu 9216
!
interface vlan858
description fos_mgmt
no shutdown
!
interface port-channel15
description "CO 236 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,720-723
lacp fallback enable
mtu 9216
vlt-port-channel 15
!
interface port-channel16
description "CO 237 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,720-723
lacp fallback enable
mtu 9216
vlt-port-channel 16
spanning-tree port type edge
!
interface port-channel17
description "CO 238 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,720-723
lacp fallback enable
mtu 9216
vlt-port-channel 17
!
interface port-channel18
description "CO 239 vmotion d1-d4"
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 106,720-723
lacp fallback enable
mtu 9216
vlt-port-channel 18
!
interface port-channel22
description 9332
no shutdown
switchport mode trunk
switchport access vlan 1
switchport trunk allowed vlan 104,106,720-723,858
lacp fallback enable
mtu 9216
vlt-port-channel 22
!
interface ethernet1/1/1
description "M740-238 - vmnic3"
no shutdown
channel-group 15 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/3
description "M740-239 - vmnic3"
no shutdown
channel-group 16 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/37
description "Peer-link 9116-B2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/38
description "Peer-link 9116-B2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/39
description "Peer-link 9116-B2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/40
description "Peer-link 9116-B2"
no shutdown
no switchport
flowcontrol receive off
!
interface ethernet1/1/41:1
description customer-9332
no shutdown
channel-group 22 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/1/42:1
description customer-9332
no shutdown
channel-group 22 mode active
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/1
description "M740-236 - vmnic4"
no shutdown
channel-group 17 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
!
interface ethernet1/71/2
description "M740-237 - vmnic4"
no shutdown
channel-group 18 mode active
fec off
no switchport
mtu 9216
flowcontrol receive off
MX740c compute sleds
The MX7000 chassis slots 1 and 2 were occupied by two MX740c compute sleds in each chassis and were designed to function as compute-only nodes.
Figure 5. MX740c sleds - slot placement
VMware ESXi
Prerequisites
Ensure that the network configuration on the MX9116n I/O modules is completed.
Procedure
The following steps provide high-level configuration details required to get the compute-only network operational:
- Follow best practices using the version detail provided in Appendix A ─ Validated components section to install VMware ESXi.
- Connect the newly created CO hosts to the PowerFlex management vCenter.
- Check that MX740 sleds are added to PowerFlex Management vCenter.
- Create vSphere Distributed Switches as per PowerFlex standards.
The following example shows CO-DSwitch1 with an MTU of 9000:
- Add the hosts to the VDS following VMware best practices.
The following table provides correct data network assignments for the specific sleds used in this POC:
Vmnic
Network
Vmnic2, Vmnic3
CO Data 1
Vmnic2, Vmnic3
CO Data 2
Vmnic2, Vmnic3
CO Data 3
Vmnic2, Vmnic3
CO Data 4
Vmnic2, Vmnic3
CO vMotion
Vmnic0, Vmnic1
CO Management
- Enable LACP.
The following example shows the final LAG group configuration:
- Create VMkernel adapters as per PowerFlex standards.
The following example shows VMK configuration details:
Results
- VDS Topology Overview for the Management/Data vDS.
- VDS Topology Overview for the Storage vDS.
- Ensure that the Jumbo Frames are enabled and operational using the ESXi console.
Example: ping Data1 gateway, when Data1 = vmk2 and gateway = 10.1.152.1
vmkping -I vmk2 -d -s 8972 10.1.152.1
SDC Install and Configuration
Install the SDC with the appropriate parameters to connect with an existing PowerFlex system. This procedure is relevant both for adding more SDCs to an existing system, and for adding SDCs to a two-layer system during initial deployment activities.
Prerequisites
Ensure that you have:
- The virtual IP address or MDM IP address of the existing system. If an MDM virtual IP address is not in use, obtain the IP addresses of all the MDM managers.
- Login credentials for the intended SDC host.
- Downloaded the required installation software package for the SDC operating system from the Technical Support site.
- A GUID (Global Unique Identifier) string for the SDC. These strings can be generated by tools that are freely available online. The GUID must conform to OSF DCE 1.1. The expected format is xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx where each x can be a digit (0–9) or a letter (a–f).
Procedure
The following procedure explains how to manually install an external SDC on an ESXi server using esxcli command line:
Note: This procedure requires two server reboots.
- Install the SDC on an ESXi 7.0.x server:
- Copy the SDC file to the tmp directory on the local datastore on the host.
- SSH to the host and run the following command:
esxcli software vib install -d /tmp/sdc-3.5.1100.101-esx7.x.zip --no-sig-check
- Reboot the PowerFlex node.
- Set the IP address of the MDM by running the following command:
esxcli system module parameters set -m scini -p "IoctlIniGuidStr=<XXXXXX> IoctlMdmIPStr=<LIST_VIP_MDM_IPS> IoctMdmPasswordStr=<ip>-<password> bBlkDevIsPdlActive=1 blkDevPdlTimeoutMillis=60000"
where IoctlIniGuidStr is the unique value defined earlier.
- Perform another reboot.
- Ensure that the driver is loaded.
vmkload_mod -l | grep scini
Output similar to the following indicates that the driver is loaded:
scini 2 808
Results
The SDC component is installed on the ESXi host.
Verify if the SDC is added using the PowerFlex presentation UI. The following images show a successfully connected SDC.
