Managing traffic flow to ECS

We recommend utilizing an external IP traffic load balancer with ECS to manage the health and distribute traffic to all the ECS nodes in the cluster.

remote.s3.use_delimiter to improve listing performance

ECS Release 3.2.2 is the first release that works with the default value for the SmartStore parameter remote.s3.use_delimiter. 

This means that ECS Release 3.2.2 or greater cannot be used with file system enabled buckets when this parameter is enabled, as file system semantics for ECS native NFS and Hadoop support require a standard “/” delimiter in file paths.

Disabling this parameter for use with legacy versions of ECS or filesystem enabled buckets has neither been tested nor is it a best practice for ECS performance purposes.

Local storage requirements

Splunk recommends to, at a minimum, provision enough storage to keep at least 7-10 days of data in cache, as searches typically occur on data indexed within the last 7 - 10 days.  Review the local storage requirements section in the Splunk 7.3 documentation for guidance.

Cold Buckets

SmartStore index buckets generally roll to frozen directly from warm.

Cold buckets can exist in a SmartStore-enabled index, but only under limited circumstances. Specifically, if you migrate an index from non-SmartStore to SmartStore, any migrated cold buckets use the existing cold path as their cache location, post-migration.

In all respects, cold buckets are functionally equivalent to warm buckets. The cache manager manages the migrated cold buckets in the same way that it manages warm buckets. The only difference is that the cold buckets will be fetched into the cold path location, rather than the home path location.

Data Retention

Data retention policy for SmartStore indexes on indexer clusters is configured using settings similar to those for non-SmartStore indexes. However, with SmartStore indexes, data retention is managed cluster-wide, rather than on a per-indexer basis.

Multi-part upload/download part size

The default setting for multipart download and upload part size is 128MB and should not be modified unless that value has proven to improve throughput.

Versioning

SmartStore supports versioning, used primarily for frozen data, that exceeds the configured data retention time.

When the parameter remote.s3.supports_versioning=false, SmartStore will put a delete marker on any corresponding bucket that gets frozen and this data/bucket is ignored by SmartStore for any subsequent searches. This prevents any accidental deletion. 

Note: The ECS bucket being used to store the index data must have versioning enabled.

Note: It is recommended to set a lifecycle policy on the ECS bucket to eventually remove the SmartStore frozen bucket and free capacity.

If remote.s3.supports_versioning=true, which is the default, then the data is deleted by SmartStore once the data ages out and is frozen.