The recommended approach for measuring and optimizing performance is as follows:

• Establish reference network performance using common tools such as Secure Copy (SCP) or NFS copy from cluster to cluster. This provides a baseline for a single thread data transfer over the existing network.
• After creating a policy and before running the policy for the first time, use the policy assessment option to see how long it takes to scan the source cluster dataset with default settings.
• Use file rate throttling to roughly control how much CPU and disk I/O SyncIQ consumes while jobs are running through the day.
• Remember that “target aware synchronizations” are much more CPU-intensive than regular baseline replication but they potentially yield much less network traffic if both source and cluster datasets are already seeded with similar data.
• Use IP address pools to control which nodes participate in a replication job and to avoid contention with other workflows accessing the cluster through those nodes.
• Use network throttling to control how much network bandwidth SyncIQ can consume through the day.

Worker and performance scalability

For releases prior to OneFS 8.0, the number of primary and secondary workers is calculated between both clusters based on two factors. First, the lowest number of nodes between the two clusters is considered. The lowest number of nodes is then multiplied by the number of workers per node, which is a configurable value. The default value for workers per node is three. SyncIQ randomly distributes workers across the cluster with each node having at least one worker. If the number of workers is less than the number of nodes, then all nodes will not participate in the replication. The following figure shows a calculation example:

Figure 41.    Calculating primary and secondary workers for release earlier than OneFS 8.0

In OneFS 8.0, the limits have increased to provide additional scalability and capability in line with cluster sizes and higher-performing nodes that are available. The maximum number of workers and workers per policy both scale as the number of nodes in the cluster increase. The defaults should be changed only with the guidance of PowerScale Technical Support.

• A maximum of 1,000 configured policies and 50 concurrent jobs are now available.
• Maximum workers per cluster are determined by the total number of physical cores in the node’s CPUs. The default is 4 * [total physical cores in the cluster]
• Maximum workers per policy is determined by the total number of nodes in the cluster. The default is 8 * [total nodes in the cluster]
• Instead of a static number of workers as in previous releases, workers are dynamically allocated to policies, based on the size of the cluster and the number of running policies. Workers from the pool are assigned to a policy when it starts, and the number of workers on a policy will change over time as individual policies start and stop. The goal is that each running policy always has an equal number (+/- 1) of the available workers assigned.
• Maximum number of target workers remains unchanged at 100 per node

Note: The source and target cluster must have the same number of workers, as each set of source and target workers create a TCP session. Any inconsistency in the number of workers results in failed sessions. As stated above, the maximum number of target workers is 100 per node, implying the total number of source workers is also 100 per node.

Note: The following example is provided to show how a node’s CPU type affects worker count, how workers are distributed across policies, and how SyncIQ works on a higher level. The actual number of workers is calculated dynamically by OneFS based on the node type. The calculations in the example are not a tuning recommendation and are merely for illustration. If the worker counts require adjustment, contact PowerScale Technical Support, as the number of physical cores, nodes, and other factors are considered before making changes.

As an example, consider a 4-node cluster, with 4 cores per node. Therefore, there are 16 total cores in the cluster. Following the previous rules:

• Maximum workers on the cluster = 4 * 16 = 64 workers
• Maximum workers per policy = 8 * 4 = 32

When the first policy starts, it will be assigned 32 workers (out of the maximum 64). A second policy starting will also be assigned 32 workers. The maximum number of workers per policy has been determined previously as 32, and there are now a total of 64 workers—the maximum for this cluster. When a third policy starts, assuming the first two policies are still running, the maximum of 64 workers are redistributed evenly, so that 21 workers are assigned to the third policy, and the first two policies have their number of workers reduced from 32 to 21 and 22 respectively, as 64 does not split into 3 evenly. Therefore, there are 3 policies running, each with 21 or 22 workers, keeping the cluster maximum number of workers at 64. Similarly, a fourth policy starting would result in all four policies having 16 workers. When one of the policies is complete, the reallocation again ensures that the workers are distributed evenly among the remaining running policies.

Note: Any reallocation of workers on a policy occurs gradually to reduce thrashing when policies are starting and stopping frequently.

Specifying a maximum number of concurrent SyncIQ jobs

Administrators may want to specify a limit for the number of concurrent SyncIQ jobs running. Limiting the number is particularly useful during peak cluster usage and client activity. Forcing a limit on cluster resources for SyncIQ ensures that clients do not experience any performance degradation.

Note: Consider all factors prior to limiting the number of concurrent SyncIQ jobs, as policies may take more time to complete, affecting RPO and RTO times. As with any significant cluster update, testing in a lab environment is recommended before a production cluster is updated. Also, a production cluster should be updated gradually to minimize impact and allow measurements of the impacts.

To limit the maximum number of concurrent SyncIQ jobs, perform the following steps from the OneFS CLI:

1. Modify /ifs/.ifsvar/modules/tsm/config/siq-conf.gc using a text editor.

2. Change the following line to represent the maximum number of concurrent jobs for the cluster: scheduler.max_concurrent_jobs
3. Restart SyncIQ services by running the following command: isi sync settings modify --service off;sleep5; isi sync settings modify --service on

Performance tuning for OneFS 8.x releases

OneFS 8.0 introduced an updated SyncIQ algorithm taking advantage of all available cluster resources, improving overall job run times significantly. SyncIQ is exceptionally efficient in network data scaling and uses 2 MB TCP windows, considering WAN latency while delivering maximum performance.

Note: The steps and processes mentioned in this section may significantly affect RPO times and client workflow. Prior to updating a production cluster, test all updates in a lab environment that mimics the production environment. Only after successful lab trials, should the production cluster be considered for an update. As a best practice, gradually implement changes and closely monitor the production cluster after any significant updates.

SyncIQ achieves maximum performance by using all available cluster resources. SyncIQ consumes the following resources if they are available:

• All available CPU bandwidth
• Worker global pool—Default compute is based on node count and total cluster size, as previously described
• All available bandwidth

As SyncIQ consumes cluster resources, this may affect current workflows depending on the environment and available resources. If data replication is affecting other workflows, consider tuning SyncIQ as a baseline as follows:

• Limit CPU to 33% per node
• Limit workers to 33% of global—Factoring in lower performance nodes
• Configure bandwidth rules—For example, limit to 10 GB during business hours and 20 GB during off-hours

For information about updating the variables above, see SyncIQ performance rules. After the baseline is configured, gradually increase each parameter and collect measurements to ensure that workflows are not affected. Also consider modifying the maximum number of SyncIQ jobs, as described in Specifying a maximum number of concurrent SyncIQ jobs.

Note: The baseline variables provided here are only for guidance and are not a one-size-fits-all metric. Every environment is different. Carefully consider cluster resources and workflow when the intersection of workflow affects SyncIQ performance.