Home > Storage > PowerScale (Isilon) > Product Documentation > Protocols > PowerScale OneFS NFS Design Considerations and Best Practices > Linux client
Use NFS over TCP
The advantage using NFS over TCP is that it works far better than UDP over unstable networks. When using TCP, a single dropped packet can be retransmitted, without the retransmission of the entire RPC request, resulting in better performance over unstable networks.
In addition, TCP will handle network speed differences better than UDP, due to the underlying flow control at the network level. When using UDP, if network speeds of the client and server are not identical, dropped packets and retransmissions might cause performance to be extremely slow when the faster entity tries to send data to the slower entity.
The overhead incurred by the TCP protocol will result in somewhat slower performance than UDP under ideal network conditions, but the cost is not severe, and is often not noticeable without careful measurement. If you are using 10 GB Ethernet or above from end to end, you might also investigate the usage of jumbo frames. The high-speed network may allow the larger frame sizes without encountering increased collision rates, particularly if you have set the network to full duplex.
Client support for NFS over TCP is integrated into all Linux kernel 2.4 and later. You can check your client kernel version with command uname –r before you use NFS over TCP. By default, the client will attempt to mount NFS export with TCP if supported, you can also use option –proto=tcp to explicitly use TCP.
Security-Enhanced Linux (SELinux)
Security-Enhanced Linux (SELinux) is an implementation of a mandatory access control mechanism in the Linux kernel, checking for allowed operations after standard discretionary access controls are checked.
By default, NFS mounts on the client side are labeled with a default context defined by policy for NFS volumes. In common policies, this default context uses the nfs_t type. Depending on policy configuration, services such as Apache HTTP Server and MySQL may not be able to read files labeled with the nfs_t type. This may prevent file systems labeled with this type from being mounted and then read or exported by other services.
If you would like to mount an NFS volume and read or export that file system with another service, use the -context option when mounting to override the nfs_t type. For example, use the following context option to mount NFS volumes so that they can be shared through the Apache HTTP Server:
mount -o context="system_u:object_r:httpd_sys_content_t:s0" onefs_server:/export /mnt/mount_point
SELinux can enforce rules on files and processes in a Linux system and on their actions based on defined policies. Because SELinux will introduce a performance overhead, disable the SELinux in the Linux client unless it is explicitly required. To permanently disable SELinux, follow these steps:
NFS read size (rsize) and write size (wsize)
The mount options rsize and wsize specify the size of the data block size in bytes to be transferred at one time between the client and server. Setting a larger data block size would improve the performance. But be careful when changing these values; some older Linux kernels and network cards do not work well with larger block sizes. For most of modern systems, these settings are negotiated during mount and the server will specify the recommended settings. By default in OneFS, a rsize of 128 KB and wsize of 512 KB are advertised for NFSv3 Linux clients, but can be set as high as 1 MB. NFSv4.x defaults to 1 MB for both rsize and wsize.
Setting these values incorrectly will result in slower performance, so we recommended that you experiment before you change rsize and wsize. You can test your options with some simple commands if your network environment is not heavily used. Here is an example using command dd to do a simple test to see the speed while using different rsize and wsize.
Commands use to write file to a OneFS cluster and read file from a OneFS cluster:
time dd if=/dev/zero of=/mnt/test.file bs=1024k count=30720
time dd if=/mnt/test.file of=/dev/null bs=1024k
Result shows in Figure 6 while mounting NFS with options nfsvers=4,rsize=1048576,wsize=1048576, which is the default value advised by OneFS. And leaves the other mount options as default.
Result shows in Figure 7 while mounting NFS with options nfsvers=4,rsize=32768,wsize=32768. And leaves the other mount options as default.
From these results, we can see that the performance will decrease if the inappropriate rsize and wsize values are used. We recommend that you do not specify these options when mounting a OneFS cluster NFS export. If you want to specify these options, we recommend that you verify them, as shown, before you apply these setting in your production environment.
Note: The test result shown may vary widely in different test environments—for example, in a different Linux OS/Kernel or network. If you achieve unexpected test results or if you would like to test different types of workloads (sequential, random, mix), use more complex benchmarks such as IOZone, FIO.
Hard mounts and soft mounts, timeout, and retransmissions
Setting these options incorrectly would cause unnecessary retransmissions in some cases, for example, in a high latency network. So unless you have an explicitly requirement to these options for your application and system, you should leave these options as default.
Sync and async mounts
The options is async by default, the NFS client delays sending application writes to the server until any of these events occur:
In other words, under normal circumstances, data written by an application may not immediately appear on the server that hosts the file. If the sync option is specified on a mount point, any system call that writes data to files on that mount point causes that data to be flushed to the server before the system call returns control to user space.
So the sync write introduces a significant performance overhead while providing better data cache coherence between clients. Applications can use the O_SYNC open flag to force application writes to individual files to go to the server immediately without the use of the sync mount option. It is recommended to use async mounts and the application control the safe write behavior by writing with the O_SYNC open flag or flush data with sync.
Attribute caching (ac/noac)
Use the noac mount option to achieve attribute cache coherence among multiple clients. Almost every file system operation checks file attribute information. The client keeps this information cached for a period to reduce network and server load. When noac is in effect, a client's file attribute cache is disabled, so each operation that needs to check a file's attributes is forced to go back to the server. Besides, the noac option forces application writes to become synchronous so that a client sees changes to a file upon opening, at the cost of many extra network operations. By default, the attribute caching is enabled when mounting the NFS. Enable the attribute caching to improve the attribute checking performance and reduce the NFS operation latency.
nconnect
This mount option exists in all Linux distributions with kernel 5.3 or higher and can be set up to a limit of 16. It allows clients establish multiple TCP connections to a OneFS cluster node for a specific NFS version, as it can be used with NFSv3 and NFSv4.x with each individual version being tracked separately. This option is set during the client's first mount for a particular OneFS node and NFS version combination. If the client performs another NFS mount for the same OneFS node and NFS version, it will reuse the connections established by the first mount. Subsequent mount commands cannot override the nconnect value already established. To set a new nconnect value, all client-mounted NFS file systems which point to a certain OneFS node and NFS version must be unmounted, and you must remount the NFS mount with the desired nconnect value.
Avoid using same hostname for multiple NFSv4 clients
Based on NFSv4.1 RFC, NFS clients should generate a unique co_ownerid string so that multiple clients do not present the same string to the NFS server. Specifically, when multiple clients have same hostname, the co_ownerid string is identical across these clients. Therefore, the OneFS client table records only a single entry for the shared hostname. Consequently, when one client attempts to renew its client ID, the OneFS system responds with an NFS4ERR_STALE_CLIENTID error to the other clients. This forces those clients to issue a new SETCLIENTID operation to obtain a fresh client ID. For more details, see NFSv4 client identifiers.