Home > Storage > PowerScale (Isilon) > Product Documentation > Storage (general) > PowerScale OneFS Job Engine > SmartThrottling
OneFS 9.8 introduces SmartThrottling. This is an automatic impact control mechanism for the Job Engine that is built upon the OneFS partitioned performance framework. SmartThrottling functions as follows:
There is also a secondary throttling mechanism that uses drive IO statistics. This backup throttling is for situations when no protocol load exists, and attempts to maintain disk IO health within set thresholds in order to prevent the Job Engine from commandeering all the cluster resources.
The SmartThrottling thresholds, and their associated rate of ramping up or down the amount of work jobs can perform, differ based on the impact setting of a specific job. The legacy Job Engine impact configuration remains unchanged, and individual jobs can be set to Low, Medium, or High impact. Similarly, each job still has the same default impact level, which can be further adjusted if needed.
Note: The new SmartThrottling feature is disabled by default in OneFS 9.8 and can be manually enabled if needed. It is currently recommended for customers who have experienced challenges related to the impact that the Job Engine has on their workloads.
To enable SmartThrottling, use the following CLI command:
# isi job settings modify –-smartthrottling true
Similarly, to immediately disable SmartThrottling:
# isi job settings modify –-smartthrottling false
Running this command causes the Job Engine to restart, temporarily pausing and resuming any running jobs, after which they will continue where they left off and run to completion as normal.
To display the current state of throttling, use the following command:
# isi job settings view
Parallel Restriper Mode: All
Smartthrottling: True
For advanced configuration, there are three main threshold options. These are the target read and write latency thresholds for protocol operations, and the disk IO time in queue threshold. These can be viewed with the following:
# isi performance settings view | grep -i in
Time In Queue Threshold (ms): 10.00
Target read latency in microseconds: 12000.0
Target write latency in microseconds: 12000.0
Medium impact job latency threshold modifier in microseconds: 12000.0
High impact job latency threshold modifier in microseconds: 24000.0
The target read and write latency thresholds default to 12 milliseconds (ms) for low impact jobs and are the thresholds at which SmartThrottling begins to throttle the job engine’s work. There are also modifiers for both medium and high impact jobs, which are set to an additional 12 ms and 24 ms respectively by default. For medium impact jobs, throttling will start to activate around 20 ms, and then heavily throttle the job engine at 24 ms. Similarly, for the high impact jobs, throttling starts at 30 ms and ramps up at 36 ms.
Because SmartThrottling is currently configured globally and tuned for an average cluster, the job engine throttling thresholds can be adjusted for specific customer environments, if necessary. For example, powerful all-flash clusters usually respond well to setting thresholds lower than 12 ms. This can be performed using the ‘isi performance settings modify’ CLI command with the following options:
Config Option | Threshold | Default |
--target-protocol-read-latency-usec | Target read latency | 12ms |
--target-protocol-write-latency-usec | Target write latency | 12ms |
--medium-impact-modifier-usec | Medium impact job latency modifier | 12ms |
--high-impact-modifier-usec | High impact job latency modifier | 24ms |
--target-disk-time-in-queue-ms | Time In Drive Queue | 10ms |