Accelerating your Network File System (NFS) Workloads with RDMA
Wed, 15 Sep 2021 13:19:26 -0000
|Read Time: 0 minutes
The NFS protocol is widely used in datacenters by NAS storage nowadays. It was originally designed for storing and managing data centrally, then sharing data across networks. As technologies evolved, NFS has been used for critical production workloads by many organizations.
NFS is usually implemented over TCP to transfer data. With the emergence of higher speed Ethernet and heavier application workloads running in datacenters, the speed of transferring the ever-increasing volume of data is critical to organizations. The industry has been pursuing new ways to improve NFS protocol performance and to adapt to emerging application workloads. This has made possible using NFS over Remote Direct Memory Access (RDMA).
RDMA enables accessing memory data on a remote machine without passing the data through the CPUs on the system. RDMA therefore enables data to be transferred between storage and clients with higher throughput and lower CPU usage. NFS over RDMA, as defined in RFC8267, uses the advantages of RDMA. Starting with OneFS 9.2.0, OneFS supports NFSv3 over RDMA based on the ROCEv2 (also known as Routable RoCE or RRoCE) network protocol.
To evaluate the improvements and advantages of NFSv3 over RDMA, as compared to NFSv3 over TCP, we ran some FIO sequential read tests, and observed the throughput and CPU usage under different thread counts. The following figure shows the test environment topology and resource configuration.
| Cluster nodes | Clients |
Quantity | 48-node cluster | 10 |
OS Version | OneFS 9.2.1.0 | CentOS Linux release 8.3.2011 |
Model | F600 | Dell PowerEdge C4140 clients |
Network device | 2 * MT28800 Family [ConnectX-5 Ex] * 100GE | 2 * MT28908 Family [ConnectX-6] * 100GE |
The following chart shows the throughput comparison for RDMA vs. TCP. We found that NFSv3 over RDMA delivers higher throughput than NFSv3 over TCP. (Note: Because 10 test clients cannot overload a 48-node F600 cluster, the throughput number is only used for RDMA and TCP comparison and does not represent the maximum cluster performance.)
The following chart shows the clients’ CPU usage comparison for RDMA vs. TCP. We found that clients consume fewer CPU resources when using NFSv3 over RDMA.
Conclusion
The NFSv3 over RDMA performance improvement does vary as the client thread number increases, as compared to NFSv3 over TCP. Overall, NFSv3 over RDMA delivers higher throughput while providing significant reduction of clients’ CPU overhead. Sequential workloads and CPU-intensive workloads can therefore benefit from using NFSv3 over RDMA on OneFS.
Author: Lieven Lin, LinkedIn