Home > Storage > PowerScale (Isilon) > Product Documentation > Protocols > Dell PowerScale: Solution Design and Considerations for SMB Environments > Performance
In the next section, we will discuss some performance considerations and useful CLI commands to identify performance issues with the SMB protocol and OneFS. For general troubleshooting guide, see the white paper Dell PowerScale OneFS Cluster Performance Metrics Hints and Tips.
Authentication provider
It is recommended to use Kerberos authentication instead of NT LAN Manager (NTLM) for performance reasons. As user load and authentication requests increase, NTLM can degrade performance because NTLM-based authentication inherently requires multiple round trips. For more detail about NTLM and Kerberos authentication, see the Microsoft article Explained: Windows Authentication in ASP.NET 2.0.
For Kerberos authentication, it is important to verify that the service principal names (SPNs) have been set up correctly, otherwise the cluster may fall back to NLTM authentication. See the Kerberos authentication section for more guideline.
Data Access Pattern
PowerScale can be used for different types of workloads. The Data Access Pattern setting defines the optimization settings for accessing concurrent, streaming, or random data types. Files and directories use a concurrent access pattern by default. To optimize performance, select the pattern dictated by your workflow. Concurrent is a good default setting unless your workflow is very well understood.
Since access patterns can be defined on a per directory basis, it is worthwhile testing and evaluating different access patterns on different workflow data against different tools. Test and validate the access pattern against each data set and the jobs accessing it.
The settings can be configured by from the WebUI by navigating to File System > Storage Pools > File Pools and editing the desired file pool policy. Figure 5 shows the configuration of Data Access Pattern through the WebUI.
Figure 5. Data Access Pattern configuration in WebUI
Table 7 lists the different types of Data Access Pattern options in OneFS and its general description with different workflow examples.
Table 7. OneFS Data Access Pattern Options
OneFS Data Access Pattern | Description | Examples |
Concurrent(default) | Concurrency access is the middle ground with moderate prefetching. Use this for file sets with a mix of both random and sequential access. | General Files and Directories |
Streaming | Streaming access works best for sequentially read medium to large files. This access pattern uses aggressive prefetching to improve overall read throughput, and on disk layout spreads the file across a large number of disks to optimize access. | Large SMB Files and Directories A workflow heavy in video editing Increase sequential-read performance on MapReduce jobs |
Random | The Random access setting performs little to no read-cache prefetching to avoid wasted disk access. This works best for small files (< 128KB) and large files with random small block accesses. | Typical VMware environment for random IO workflow. |
SSDs for performance benefit
Solid-state drives (SSDs) can be used in various ways within OneFS to improve performance. Prior to SmartFlash, or L3 cache, SSDs were used exclusively as file system devices. SmartPools provided the mechanism to use these SSDs primarily as metadata acceleration devices, but also for reducing latency of actual data reads and writes for the appropriate workloads.
Table 6 is a comparison of L3 cache with the other OneFS SmartPools SSD usage strategies. The principle benefits of L3 cache are around metadata read activity, user data read activity, and assistance with job engine performance.
Figure 6. Comparison of L3 cache and other SSD usage strategies.
You may need to select different SSD strategy based on the workload. For most of workflows with write component, it is recommended to select Metadata Read/Write SmartPools SSD option. For repeated random read workloads, the recommendation is to use L3 cache and you will observe latency reduced. Figure 7 shows the decision tree of various (non-L3 cache) SmartPools SSD options, and their requirements and dependencies.
For more details about common practices and considerations of different SSD strategy, see article Dell PowerScale OneFS SmartFlash and SmartPool and SSDs.
Figure 7. SSD strategy decision tree
isi statistics is a utility to access kernel counters that measure OneFS performance. These counters can give us a better understanding of the latency seen in various parts of the filesystem. It is an excellent tool for users to troubleshoot performance issues including SMB performance. Here are some useful commands for you to troubleshooting SMB performance issues:
Checking SMB clients and connections
You can use the following commands to identify how many clients are connecting to PowerScale nodes with SMB protocols.
# isi statistics query current --nodes=all --stats=node.clientstats.connected.smb,node.clientstats.active.smb1,node.clientstats.active.smb2
As Windows client statistics can only show SMB1 and SMB2. SMB3 is considered by Microsoft to be a "dialect" of SMB2, rather than a distinct version. As such, SMB3 client stats are included with the SMB2 statistics. The output displays the current number of SMB sessions per node and how many of those SMB connections are active on each node.
The output in Figure 8 shows there are 36 SMB sessions to node 1 with 8 active SMB2 sessions. The 36 SMB sessions represent clients that were connected to the node, but these clients did not send any requests during the time the command was run. As a result, these sessions are considered idle connections. The 8 active connections represent clients that sent an SMB2 request during the time the counter was collected.
Figure 8. Output of isi statistics query command
The number of active SMB2 connections that a node can process depends on the type of node. The more CPUs and RAM that a node has, the more active connections the node can process. The kernel imposes memory constraints on the OneFS protocol daemons, such as the input-output daemon (lwio), and these constraints limit the number of SMB2 connections.
To ensure that a node does not become overloaded with connections, you should monitor the number of SMB2 connections to each node. For example, for OneFS 8.1.1, it is recommended that SMB active connections per node are under 3,000 and the idle connections are under 27,000. Keep in mind that these are maximums that our fastest nodes with lots of memory could conceivably support. Slower nodes and nodes with lower memory configurations will support numbers lower than these. For general SMB connection limits guideline, see the white paper PowerScale OneFS Technical Specification Guide.
You can also use the following syntax to show the SMB2 client statistics displayed in ‘top’ format where data is continuously overwritten in a single table.
# isi statistics client --protocols=smb2 --format=top
Checking SMB protocol performance
You can use the following command to break out detailed SMB protocol performance. It can break out by detailed protocol operations with following command:
# isi statistics protocol --nodes=all --protocols=smb1,smb2 --interval 5 --repeat 2 --degraded --sort=Class
Figure 9 shows the output of the detailed protocol operations by type, and average latency.
For more detailed information about how to understand isi statistics protocol output, see the article How to Read and understand the output of Isi Statistics Protocol Command.
Figure 9. Output of isi statistics protocol command
Table 8 outlines the common expectations about protocol latency time. The output TimeAvg needed converted to milliseconds (ms) if you are comparing to standard expectations in the table.
Table 8. Common expectations about protocol latency time
Namespace metadata | Read | Write |
<10ms Good | Dependent on I/O Size | Dependent on I/O Size |
10ms – 20ms Normal | ||
>20ms Bad | ||
>50ms Investigate |
You can also use following command to check if the authentication provider is causing an issue. For more detailed information about slow SMB authentication issue, see the article Intermittent slow SMB authentication or share enumeration performance; isi_cbind_d DNS delays.
# isi statistics protocol --nodes=all --protocols=lsass_in,lsass_out --totalby Class --interval 5 --repeat 12 –degraded
Checking disks and systems
Once we narrow down the performance issues could be related specific SMB operations, you can also check disks and system performance to identify the root cause by using following commands:
# isi statistics drive --nodes=all --interval 5 --repeat 12 --degraded
Figure 10 shows the output of disk performance. The most useful statistics are TimeInQ and Queued, along with TimeAvg which is an accurate indication of drive load.
For more detailed information about how to understand isi statistics drive output, see PowerScale OneFS Cluster Performance Metrics Hints and Tips and How to Understand Isi Statistics Drive Output.
Figure 10. Output of isi statistics drive command
A good overview of the cluster can be obtained using the isi statistics system command. This will show CPU, core protocols, network, disk, and totals in a single line.
Performance dataset monitoring
For a workload interacting with OneFS through Protocol Operations, System Jobs, or System Services, performance dataset monitoring is introduced to provide improved workload monitoring starting with OneFS 8.2.0. The performance dataset monitoring allows administrators to define metrics in datasets for performance monitoring as needed.
To view all supported dataset metric, using the following command. Table 9 shows the details of each metric.
# isi performance metrics list
Table 9. Supported dataset metrics
Description | |
username | The user that the current workload belongs to. For example, a user access data using SMB protocol. |
groupname | The user group that the current workload belongs to in OneFS. For any dataset with group metric, only primary groups are reported. However, when the datasets with group metric are pinned or have a filter applied for a group, the supplementary groups will also be scanned and reported. An example is shown in Error! Reference source not found.. |
zone_name | The access zone name that the current workload belongs to. |
share_name | The SMB share name that the current workload belongs to. |
export_id | The NFS export id that the current workload belongs to. Supported in OneFS 8.2.2 and above. |
protocol | Support SMB and NFS protocols in OneFS 8.2.2 and above. Can be set to smb1, smb2, nfs3, or nfs4. Only support SMB protocol for OneFS 8.2.0 and OneFS 8.2.1, can be set to smb1 or smb2. |
system_name | This is only available for the predefined System dataset. The system name of a given workload: For services started by isi_mcp/lwsm/isi_daemon, this is the service name itself. For SMB protocol, this is named smb. For job engine jobs, this is formed with Job: job_id . Anything ran using "isi_run -w" is formed with Run: pid. |
job_type | This is for job engine jobs, it is formed with job_type[job_phase]. For example, AutoBalance[0] for a AutoBalance job phase 1 execution. For more details about supported job types, see the OneFS Job Engine white paper. |
local_address | Local IP address, CIDR subnet, or IP address range of the client causing the workload. |
remote_address | Remote IP address, CIDR subnet or IP address range of the client causing the workload. |
path | For SMB protocol only. The path under /ifs that the current workload belongs to. It is only reported if they are pinned or have a filter applied. There will be double accounting when using this metric as a file can belong to multiple path. Thus, any double accounting will be aggregated into the Overaccounted workload. An example is shown in Figure 15. |
When viewing the workload statistics with defined dataset, there are multiple workload types to tell how resources are consumed by the dataset and system. Table 10 lists the details of each workload type.
Table 10. Workload types
Workload Type | Description |
Dynamic | Top-N tracked workloads based on defined dataset. By default, top 1024 workloads will be listed. Can be modified using command isi performance settings modify. |
Pinned | Make a specific workload always visible regardless of resource usage. |
Overaccounted | The sum of all statistics that have been counted twice within a same dataset. This would be used when path metric or groupname metric are used in a dataset. Examples are shown in Figure 14 and Figure 15. |
Excluded | The amount of resources consumed by workloads that do not match the filter conditions in a dataset. |
Additional | The aggregate of dynamic workloads not in the top-N workloads. The N is 1024 by default. |
System | The amount of resources consumed by the kernel. |
Unknown | The amount of resources that cannot be categorized as the above workload types. |
SMB performance dataset monitoring examples
Use the following command to create a dataset containing metrics (username, protocol, share_name) without filters required. Figure 11 shows the results of performance statistics using the dataset.
# isi performance datasets create --name=smb_ds username protocol share_name
Figure 11. Output using dataset without filters
A dataset also supports adding filters for better visibility. Multiple filters can be applied to the same dataset, and a workload will be included if it matches any one of the filters. Any workload that does not match a filter will be aggregated into an Excluded workload. The following commands create another new dataset with applying a filter to view performance metrics only for username “bob”. The result appears in Figure 12.
# isi performance datasets create --name=smb_ds1 username protocol share_name --filters=username
# isi performance filters apply --dataset=smb_ds1 --name=bob_filter username:bob
Figure 12. Output using dataset with a filter
To make a specific workload always visible regardless of resource usage, you can pin the workload for a specific dataset using the following command. The result appears in Figure 13.
# isi performance workloads pin --dataset=smb_ds1 --name=root_pin username:root protocol:smb2 share_name:smbshare
Figure 13. Output of a pinned workload using dataset with a filter
Shown in Figure 14, a file is written to a cluster with user bob, group02 is the primary group of the user, and group01 is the supplementary group of the user. We pin the group01 so the workload is accounted twice, and the total amount is aggregated into the Overaccounted workload.
Figure 14. Groupname metric overaccounted
Shown in Figure 15, a file is written to directory /ifs/path1/path2, and the workload is monitored with the path metric at the same time. We find the workload is accounted twice and the total amount is aggregated into the Overaccounted workload.
Figure 15. Path metric overaccounted
The following sections will discuss the common practices and consideration of client OS performance management and tuning methodologies. Since the environment of clients may vary, the general guideline is the tunings are not identical and the impact must be evaluated before applying them.
Windows parameter management
There are a few SMB parameters can be used to tune SMB client performance. The following section will list the most significant one. For a complete list of the parameters, see Performance Tuning for File Servers.
Bandwidth throttling
Bandwidth throttling can be controlled with the key - DisableBandwidthThrottling (REG_DWORD) under HKLM\System\CurrentControlSet\Services\LanManWorkstation\Parameters. This key does not exist by default and has an assumed value of 0. The SMB2 client will try to limit its network throughout on links. In some scenarios, especially like Virtual Desktop Infrastructure (VDI) deployments, the drives mapped to PowerScale are connected by high speed frontend networks, so there is no need to try and limit the throughput of the SMB sessions. This value should be set to 1.
macOS parameter management
Many parameters can be tuned in macOS that affect SMB connections. For details on all the configurations and tuning methodologies, see the white paper PowerScale macOS Performance Optimization which is focused on macOS 10.13 “High Sierra” and 10.14 “Mojave”.