Home > Storage > PowerScale (Isilon) > Product Documentation > Protocols > PowerScale OneFS NFS Design Considerations and Best Practices > NFS over RDMA overview
Remote Direct Memory Access (RDMA) originated with InfiniBand and evolved gradually on Ethernet network environment. Currently, the following network protocols support RDMA on Ethernet, including Internet Wide Area RDMA Protocol (iWARP), and RDMA Over Converged Ethernet (RoCE). For more details, see RoCE.
NFS over RDMA is defined in RFC8267. Starting with OneFS 9.2.0, OneFS supports NFSv3 over RDMA by leveraging the ROCEv2 (also known as Routable RoCE or RRoCE) network protocol. Starting from OneFS 9.8.0, NFSv4.x over RDMA is also supported. With NFS over RDMA support, direct memory access between OneFS and NFS clients is available with consuming less client CPU resource, improving OneFS cluster network performance with lower latency, lower CPU load and higher throughput.
The Figure 12 shows the architecture of NFS over RDMA in OneFS. NFS is implemented over RDMA for data transferring. Please note that the NFSv3 auxiliary protocols (mount, nlm, nsm, rpc portmapper) still work on TCP/UDP. You must add PowerScale nodes RoCEv2 capable front-end network interfaces into an IP pool before the NFS clients can access OneFS cluster data over RDMA.
Table 4 shows the TCP/UDP port requirement of using NFS over RDMA.
Service | Protocol | Usage description | |
4791 | RoCEv2 | UDP | In RoCEv2, the RDMA payload is encapsulated as UDP payload with the 4791 UDP destination port |
300 | mountd | TCP/UDP | Only used by NFSv3 mount service |
302 | statd | TCP/UDP | Only used by NFSv3 Network Status Monitor (NSM) |
304 | lockd | TCP/UDP | Only used by NFSv3 Network Lock Manager (NLM) |
111 | rpc.bind | TCP/UDP | Only used by NFSv3. ONC RPC portmapper that is used to locate services such as NFS, mountd. |
Note: To optimize performance in case of network packet loss, we recommend enabling flow control on switch ports to optimize performance when losing network packet. When mounting NFS export over RDMA, you may need to specify an NFS over RMDA port 20049 depending on your client Kernel version. The port is used to for RPC binding of RDMA interconnect internally and is not required to be allowed in network firewalls. For more details, see RFC5666 RPC Binding. For example, the command may be: mount -t nfs -o nfsvers=4.1,rdma,port=20049 server_fqdn:/ifs/data /mnt