Traditional TCP/IP networks offer adequate bandwidth for many workloads, although it comes with some limitations, including CPU overhead and higher than wanted round-trip latency. The RDMA protocol reduces CPU overhead and latency by skipping through layers of the stack, resulting in rapid data transfers.
Two network implementations of RDMA, the Internet Wide Area RDMA protocol (iWARP) and RDMA over Converged Ethernet (RoCE), are embedded in the network adapter hardware, offloading nearly all the work from the CPU to offer faster networking:
- iWARP implements RDMA over IP networks using TCP, making it ideal for organizations that want to use RDMA over their existing IP network infrastructure without any specialized hardware. iWARP requires no additional configuration at the ToR switches for its implementation.
- RoCE v2 uses UDP and requires QoS (Quality of Service) to ensure packet delivery. It relies on an Ethernet network configured to use Layer 2 Priority Flow Control (PFC) or Layer 3 DSCP PFC to minimize congestive packet loss.
Topic iWARP (Internet Wide Area RDMA Protocol) RoCE (RDMA over Convergent Ethernet) Ease of deployment No special network configuration (simple to install/configure) Requires network configuration (more complex to install/configure) QoS Not needed PAUSE frames, or DCB Routable Yes Yes with RoCE V2 Latency Comparable to RoCE except for HPC applications Best Throughput Similar to RoCE Best Vendor support QLogic, Intel Mellanox and Intel Deployment size recommendation Data center or cluster ideal for remote communication (storage replica) May be more appropriate for clusters Interoperable between vendors Not supported Yes - RoCE from two different vendors should not reside in the same server
See Reference Guide: Switch Configurations – RoCE for details about the switch configurations for all RoCE-based deployments for Microsoft Azure Stack HCI Solutions from Dell Technologies.
See Reference Guide: Switch Configurations - iWARP for details about the switch configurations for all iWARP-based deployments for Microsoft Azure Stack HCI Solutions from Dell Technologies.