Home > Workload Solutions > High Performance Computing > White Papers > Enhance Availability of Storage Services with an NFS Storage System > IPoIB sequential writes and reads N-N
To evaluate the performance for moving large files, the IOzone version 3.492 benchmark is used to read and write sequentially using 1 MiB transfers, the maximum NFS transfer size for read or write operations. These tests are conducted on multiple thread counts starting at 1 thread and increasing in powers of 2, up to 64 threads. At each thread count, an equal number of files are generated, since this test works on one file per thread or N-N case. An aggregate file size of 512GiB has been selected, as it is twice the amount of RAM in the server in order to avoid cache effects that prevent getting reliable sustained performance results. The 512GiB of data is equally divided among the number of client threads used on any given test. Also, after each step, either sequential writes or reads, client nodes unmounted the NFS file system and dropped all the OS caches before remounting NFS to continue with the next operation.
Figure 2 provides a comparison of the sequential I/O performance of the Dell Technologies Validate Design for NFS Storage and the previous version. From the figure, it is observed that the current and previous systems have similar peak performance, with a small performance gain on the new Design for NFS Storage.
Peak performance for the new solution for sequential reads summits at 7.25 GB/s at 16 threads (one thread per client node), and for sequential writes at almost 5 GB/s with 8 threads. Sequential read performance is higher than the Dell EMC PowerVault ME4084 controllers read specs at 7 GiB/s, but when converted to powers of ten units, 7 GiB/s is a bit over 7.5 GB/s.
The write performance has a boost, mostly at low thread counts, 2, 4, 8 and 32 threads of 11.5%, 29%, 34.9% and 11.6% respectively. The read performance registered a smaller increase at 4, 8 and 64 threads of 17.7%, 15.9% and 11.3%. Since RHEL 8 improved several components in the data path (including the Block IO Layer and XFS file system), more efficient caching, faster server components (3200 MT/s memory, CPUs with more LLC, HDR higher messaging rates), the result was more performance in regions where the Dell EMC PowerVault ME4084 storage controllers are not already at their performance limits. Smaller negative and positive variations (~10%) were observed at other thread counts, but in these areas, the PowerVault ME4084 controllers are already at their limits.
As observed in the last version of the HPC NFS Storage solution, read performance decreases after the peak, and drops faster at 64 threads. As part of the root cause investigation, IOzone is executed locally on the server, without networking and NFS components playing a role. Write performance did not start to fall up to 128 threads. Read performance was at a peak at 16 and 32 threads, and then started to decrease.
That means that NFS or networking is partially responsible for the decrease in performance at thread counts 32 and higher, at least for sequential writes. Performance at higher threads counts could be affected by many factors. For example, concurrent sequential IO patterns from too many concurrent processes cause an increasing number of seek operations that is to move from file to file as different threads get a core to run, and for the storage array, the IO load starts to behave more like a random load as the number of seek operations increases. One tunable investigation is the number of NFS server processes, but performance did not improve using 512 nfsd processes (instead of 256). Therefore, the performance drop at higher thread counts requires further investigation.