Your Browser is Out of Date

ShareDemos uses technology that works best in other browsers.
For a full experience use one of the browsers below
Reference Guide—Network Integration and Host Network Configuration Options
Notes, cautions, and warnings Copyright Introduction to network integration Scalable Architecture Switchless Architecture
Switchless storage networking
Requirements for switchless storage network topologies
Cabling instructions
Deployment instructions for four-node dual-link full mesh switchless interconnect
Additional guidance for all configurations Appendix

Cabling instructions

Cabling instructions

  • Follow these cabling best practices when deploying a full mesh switchless interconnect. The Dell EMC factory populates PCIe network adapters consistently, but the slot enumeration sequence depends on platform and riser configuration. PCIe slot identification is labeled on the chassis, and Port 1 and 2 are physically labeled on the PCIe NIC adapter bracket.

    Note: PCIe riser orientation may change the port orientation on each network adapter. Therefore, some network adapters may have a physically reversed orientation depending on which PCIe slot the network adapter is installed. Be certain to check the orientation of Port 1 and 2 on each network adapter, as shown in the following figure.

    Figure 15. Port orientation
    Image showing PCIe slots

    Use the following procedure when cabling the switchless cluster nodes for single-link or dual-link full mesh storage interconnect:

    1. Starting with Node 1, cable the PCIe network adapter in the lowest slot number, Port 1 to the same adapter slot, Port 1 on Node 2.
    2. Continue cabling a single-link on Node 1 to every other cluster node. Cable the adapter ports on Node 1 in ascending order (for example, Slot 3 Port 1, Slot 3 Port 2, Slot 4 Port 1) to the adapter in the lowest slot, lowest port of each incremental cluster node or nodes.
    3. For Node 2, cable the next available adapter port to the next available adapter port on Node 3. Then incrementally cable an additional port on Node 2 to the next available port on Node 4 (where applicable).
    4. For Node 3, cable the next available adapter port to the next available adapter port on Node 4.
    5. For a dual-link full mesh interconnect, repeat steps 2 to 4 beginning with the next available adapter port on each cluster node.
    6. Apply labels to each end of each cable with RackU, Slot, and Port identification to ensure that the cables are correctly connected in case a server chassis needs to be uncabled.
      The figures in the next sections illustrate the AX-740xd platform to display the networks required for storage network traffic. Different platforms and riser configurations use different PCIe slots and orientations. However, the cabling guidance remains consistent with the best practices that are listed above.

    Three-node single-link full mesh switchless interconnect

    The following figure illustrates an example of a three-node cluster that is configured with a single-link full mesh storage network interconnect. External network traffic uses the LOM/rNDC/OCP integrated network adapter, while storage network traffic uses one RDMA-capable PCIe network adapter per server node. The management network offers some redundancy in the event of a storage network connectivity (cable or port) failure.

    This figure shows the three storage networks that are needed to create a single-link full mesh interconnect between three clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe network adapter in the lowest slot number, Port 1.

    Note: In the following examples, the RDMA-capable network adapter is installed on Slot #6. Depending on the chassis configuration, the ports may be reversed.

    After cabling storage, connect all nodes, LOM/rNDC/OCP, Ports 1 and 2 to a management/VM network.

    Figure 16. Three-node single-link full mesh cabling
    Image showing three-node single-link full mesh cabling
    Table 3. Port link connections
    Node First port link connection Second port link connection
    Node 1 Storage1-2 Storage1-3
    Node 2 Storage1-2 Storage2-3
    Node 3 Storage1-3 Storage2-3

    Three-node dual-link full mesh switchless interconnect

    The following figure illustrates an example of a three-node cluster that is configured with a dual-link full mesh storage network interconnect. External network traffic uses the LOM/rNDC/OCP integrated network adapter, while storage network traffic uses two RDMA-capable PCIe adapters per server node. In this case, each clustered node has redundant storage network connectivity to every other node in the cluster.

    The following figure indicates the six storage networks needed to create a dual-link full mesh interconnect between three clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA adapter in the lowest slot number, Port 1. After cabling storage, connect all nodes, LOM/rNDC/OCP, Ports 1 and 2 to a management/VM network.

    Figure 17. Three-node dual-link full mesh cabling
    Image showing three-node dual-link full mesh cabling
    Table 4. Port link connections
    Node First port link connection Second port link connection Third port link connection Fourth port link connection
    Node 1 Storage1-2 Storage1-3 Storage2-1 Storage3-1
    Node 2 Storage1-2 Storage2-3 Storage2-1 Storage3-2
    Node 3 Storage1-3 Storage2-3 Storage3-1 Storage3-2

    Four-node single-link full mesh switchless interconnect

    The following figure shows the six storage networks needed to create a single-link full mesh interconnect between four clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe network adapter in the lowest slot number, Port 1. After cabling storage, connect all nodes, LOM/rNDC/OCP, Ports 1 and 2 to a management/VM network.

    Figure 18. Four-node single-link full mesh cabling
    Image of four-node single-link full mesh cabling
    Table 5. Port link connections
    Node First port link connection Second port link connection Third port link connection
    Node 1 Storage1-2 Storage1-3 Storage1-4
    Node 2 Storage1-2 Storage2-3 Storage2-4
    Node 3 Storage1-3 Storage2-3 Storage3-4
    Node 4 Storage1-4 Storage2-4 Storage3-4

    Four-node dual-link full mesh switchless interconnect

    The following figure shows the twelve storage networks needed to create a dual-link full mesh interconnect between four clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe adapter in the lowest slot number, Port 1. After cabling storage, connect all nodes, LOM/rNDC/OCP, Ports 1 and 2 to a management/VM network.

    Figure 19. Four-node dual-link full mesh cabling
    Image showing 4-node dual-link full mesh cabling
    Table 6. Port link connections
    Node First port link connection Second port link connection Third port link connection Fourth port link connection Fifth port link connection Sixth port link connection
    Node 1 Storage1-2 Storage1-3 Storage1-4 Storage2-1 Storage3-1 Storage4-1
    Node 2 Storage1-2 Storage2-3 Storage2-4 Storage2-1 Storage3-2 Storage4-2
    Node 3 Storage1-3 Storage2-3 Storage3-4 Storage3-1 Storage3-2 Storage4-3
    Node 4 Storage1-4 Storage2-4 Storage3-4 Storage4-1 Storage4-2 Storage4-3