Data throughput scaling problems arise from slow metadata performance and a lack of parallelism. If metadata performance is slow, then the number of read/write requests will decline or reach a GB/s performance speed limit strictly correlated with the metadata processing speed. Small file workloads are often slow due to this phenomenon. The underlying storage devices are capable of much faster performance, but an internal speed limit is imposed by the capabilities of the metadata management system. Assuming metadata processing scales well, data throughput can still be architecturally limited by a lack of parallelism or an inability to smoothly schedule and dispatch extremely large volumes of work across a cluster of server nodes.

ScoutAM is a scale out solution capable of utilizing all of the resources within a cluster for highly parallel data transfers and extremely high sustained aggregate data throughput.  The workflow within ScoutAM is packetized and dispatched by a “scheduler” to each node where it is accepted and processed by an “executor”. Each node in the cluster may have as many data “slots” or channels (connections to tape drives or interfaces to object storage or cloud resources) as the server nodes are equipped to provide.

All of the slots on all of the nodes are available for parallel reading and writing of data.  ScoutAM is capable of segmenting a single large file or object using a configurable segment size then sending the segments to all or a subset of available slots in the cluster thus enabling the parallel writing of a very large file across potentially dozens of tape drives or object interfaces simultaneously. Smaller quantities of data may be dispatched across the available slots in a cluster using a round robin algorithm.  The scheduler and executors work together to ensure parallel uninterrupted high bandwidth streams of data are delivered across the slots for maximum aggregate streaming throughput.  Additional throughput can be added by increasing the number of slots per server node (vertical scaling) or increasing the number of nodes (horizontal scaling).

In addition to high streaming throughput performance, ScoutFS includes a powerful indexing capability that enables extremely fast searching of indexed attributes created by ScoutAM, as well as user-defined file and object attributes. The speed of ScoutFS metadata searches is only relative to the number of files matching the search criteria, and not the total population of files. Even searches that require walking the namespace are fast because the resources of the entire cluster are available to execute the search.