Your Browser is Out of Date

Nytro.ai uses technology that works best in other browsers.
For a full experience use one of the browsers below
E2E Deployment and Operations Guide with Switchless Networking (Windows)
Deployment Introduction Solution Overview Solution Deployment Network integration overview Switchless Architecture
Switchless Architecture
Switchless storage networking
Requirements for switchless storage network topologies
Cabling instructions
Deployment instructions for four-node, dual-link full mesh switchless interconnect
Additional guidance for all configurations Deploying and Configuring a Host Cluster Persistent Memory for Windows Server HCI Day 1 operations Managing and monitoring clusters with Windows Admin Center Dell OpenManage Integration with Windows Admin Center Updates and maintenance

Cabling instructions

Cabling instructions

  • Follow these cabling best practices when deploying a full mesh switchless interconnect. The Dell factory populates PCIe network adapters consistently, but the slot enumeration sequence depends on platform and riser configuration. PCIe slot identification is labeled on the chassis, and Port 1 and Port 2 are physically labeled on the PCIe NIC adapter bracket.

    Note: PCIe riser orientation may change the port orientation on each network adapter. Therefore, some network adapters may have a physically reversed orientation depending on which PCIe slot the network adapter is installed. Be certain to check the orientation of Port 1 and Port 2 on each network adapter, as shown in the following figure.

    Figure 3. Port orientation
    Image showing PCIe slots

    Use the following procedure when cabling the switchless cluster nodes for single-link or dual-link full mesh storage interconnect:

    1. Starting with Node 1, cable the PCIe network adapter in the lowest slot number (Port 1) to the same adapter slot (Port 1) on Node 2.
    2. Continue cabling a single-link on Node 1 to every other cluster node. Cable the adapter ports on Node 1 in ascending order (for example, Slot 3 Port 1, Slot 3 Port 2, Slot 4 Port 1) to the adapter in the lowest slot that is in the lowest port of each incremental cluster node or nodes.
    3. For Node 2, cable the next available adapter port to the next available adapter port on Node 3. Then incrementally cable an additional port on Node 2 to the next available port on Node 4 (where applicable).
    4. For Node 3, cable the next available adapter port to the next available adapter port on Node 4.
    5. For a dual-link full mesh interconnect, repeat steps 2 to 4 beginning with the next available adapter port on each cluster node.
    6. Apply labels to each end of each cable with rack unit (RackU), slot, and port identification to ensure that the cables are correctly connected in case a server chassis needs to be uncabled.
      The figures in the next sections illustrate the AX-740xd platform to display the networks required for storage network traffic. Different platforms and riser configurations use different PCIe slots and orientations. However, the cabling guidance remains consistent with the best practices that are listed above.

    Three-node, single-link full mesh switchless interconnect

    The following figure shows an example of a three-node cluster that is configured with a single-link full mesh storage network interconnect. External network traffic uses the LOM/rNDC/OCP integrated network adapter, while storage network traffic uses one RDMA-capable PCIe network adapter for each server node. The management network offers some redundancy in the event of a storage network connectivity (cable or port) failure.

    This figure shows the three storage networks that are needed to create a single-link full mesh interconnect between three clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe network adapter in the lowest slot number, Port 1.

    Note: In the following examples, the RDMA-capable network adapter is installed on Slot #6. Depending on the chassis configuration, the ports may be reversed.

    After cabling storage, connect all nodes, LOM/rNDC/OCP, and Ports 1 and 2 to a management/VM network.

    Figure 4. Three-node, single-link full mesh cabling
    Image showing three-node single-link full mesh cabling
    Table 4. Port link connections
    Node First port link connection Second port link connection
    Node 1 Storage1-2 Storage1-3
    Node 2 Storage1-2 Storage2-3
    Node 3 Storage1-3 Storage2-3

    Four-node, single-link full mesh switchless interconnect

    The following figure shows the six storage networks needed to create a single-link full mesh interconnect between four clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe network adapter in the lowest slot number (Port 1). After cabling storage, connect all nodes, LOM/rNDC/OCP, and Ports 1 and 2 to a management/VM network.

    Figure 5. Four-node, single-link full mesh cabling
    Image of four-node single-link full mesh cabling
    Table 5. Port link connections
    Node First port link connection Second port link connection Third port link connection
    Node 1 Storage1-2 Storage1-3 Storage1-4
    Node 2 Storage1-2 Storage2-3 Storage2-4
    Node 3 Storage1-3 Storage2-3 Storage3-4
    Node 4 Storage1-4 Storage2-4 Storage3-4

    Two-node, back-to-back connected architecture

    The following figure shows the two storage networks needed to create a dual-link direct interconnect between two clustered nodes (also known as back-to-back). The numbered blue lines indicate the recommended cabling order, beginning with the network adapter in the lowest slot number (port 1). After cabling storage, connect all nodes, LOM/rNDC/OCP, and ports 1 and 2 to a management/VM network.

    Note: The two-node, back-to-back storage topology supports all available RDMA-based network adapters that are supported in the solution.

    Figure 6. Two-node dual-link direct cabling
    Image of two-node dual-link direct cabling

    Three-node, dual-link full mesh switchless interconnect

    The following figure shows an example of a three-node cluster that is configured with a dual-link full mesh storage network interconnect. External network traffic uses the LOM/rNDC/OCP integrated network adapter, while storage network traffic uses two RDMA-capable PCIe adapters for each server node. In this case, each clustered node has redundant storage network connectivity to every other node in the cluster.

    The following figure shows the six storage networks needed to create a dual-link full mesh interconnect between three clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA adapter in the lowest slot number (Port 1). After cabling storage, connect all nodes, LOM/rNDC/OCP, and Ports 1 and 2 to a management/VM network.

    Figure 7. Three-node, dual-link full mesh cabling
    Image showing three-node dual-link full mesh cabling
    Table 6. Port link connections
    Node First port link connection Second port link connection Third port link connection Fourth port link connection
    Node 1 Storage1-2 Storage1-3 Storage2-1 Storage3-1
    Node 2 Storage1-2 Storage2-3 Storage2-1 Storage3-2
    Node 3 Storage1-3 Storage2-3 Storage3-1 Storage3-2

    Four-node, dual-link full mesh switchless interconnect

    The following figure shows the twelve storage networks needed to create a dual-link full mesh interconnect between four clustered nodes. The numbered blue lines indicate the recommended cabling order, beginning with the RDMA-capable PCIe adapter in the lowest slot number, Port 1. After cabling storage, connect all nodes, LOM/rNDC/OCP, and Ports 1 and 2 to a management/VM network.

    Figure 8. Four-node, dual-link full mesh cabling
    Image showing 4-node dual-link full mesh cabling
    Table 7. Port link connections
    Node First port link connection Second port link connection Third port link connection Fourth port link connection Fifth port link connection Sixth port link connection
    Node 1 Storage1-2 Storage1-3 Storage1-4 Storage2-1 Storage3-1 Storage4-1
    Node 2 Storage1-2 Storage2-3 Storage2-4 Storage2-1 Storage3-2 Storage4-2
    Node 3 Storage1-3 Storage2-3 Storage3-4 Storage3-1 Storage3-2 Storage4-3
    Node 4 Storage1-4 Storage2-4 Storage3-4 Storage4-1 Storage4-2 Storage4-3