When deploying L3 cache, the following considerations should be kept in mind:
- All the SSDs within a node pool can either be used for L3 cache, or for SmartPools data strategies (metadata-ro, metadata-rw, data) – but not mixed L3 cache/SmartPools usage.
- L3 cache is not applicable for nodes containing 16 or more SSDs, and all SSD node pools are not eligible for L3 cache enablement.
- Enabling L3 cache on an existing node pool with SSDs takes some time. Data and metadata on the SSDs must be evacuated to other drives before the SSDs can be formatted for caching. Conversely, disabling L3 cache is a fast operation because no data needs to be moved and drive reformatting can begin immediately.
- If you are concerned about metadata being evicted from L3 cache, you can either:
- Deploy more SSDs per node to accommodate a large working set.
- Disable L3 cache and stay with traditional SmartPools metadata acceleration (either metadata read-only or read/write) for the particular node pool.
- You can have GNA and L3 cache in the same cluster (different node pools). It requires some manual setup, including a SmartPools policy to avoid SSD storage on L3 cache node pool, is required.
Note: L3 cache node pool hard drive space does count towards GNA limits.
- All the SSDs in an L3 cache node pool must be the same size.
- If an L3 cache SSD fails, OneFS does not have to run FlexProtect or AutoBalance jobs, like with a regular file system SSD. However, after the failed SSD is replaced, some time is required before the cache is repopulated.
- New clusters with SSD have L3 cache enabled by default, and L3 cache is enabled by default on any new node pool containing SSD. Existing node pools with SSD will not be modified to use L3 cache on upgrade.
- SSDs displace hard drives. More SSDs and fewer hard drive spindles can impact streaming and concurrency performance towards total capacity.
- The L3 cache is intentionally avoided for streaming reads during data prefetch operation. This approach keeps the streaming requests to the spinning disks (hard drives), while using the SSDs for the random I/O.
- L3 cache node pool hard drive space does not count in GNA SSD percentage calculations.
- In L3 cache, metadata is preferentially cached over data blocks.
- When a node reboots, there is no automatic flushing of L2 cache blocks to L3 cache.
- Unlike hard drives and SSDs that are used for storage, when an SSD used for L3 cache fails, the drive state immediately changes to REPLACE without a FlexProtect job running. An SSD drive used for L3 cache contains only cache data that does not require protection by FlexProtect. After the drive state changes to REPLACE, you can pull and replace the failed SSD.
- Although there is no percentage completion reporting shown when converting node pools to use L3 cache, you can estimate percentage completion by tracking SSD space usage throughout the job run. You can also change the job impact policy of the Flexprotect_Plus or SmartPools job responsible for the L3 cache conversion so that the job runs faster or slower.
- InsightIQ reports current and historical L3 cache statistics.
- For L3 cache, the isi_cache_stats prefetch statistics will always read zero because L3 cache is purely an eviction cache and does not use data or metadata prefetch.
- L3 cache has a metadata only mode (as opposed to data and metadata) to support archive-series storage nodes.