Inline data reduction
Inline data reduction
-
OneFS inline data reduction is available on the all-flash F910, F900, F810, F710, F600, F210, F200 nodes, the hybrid H700/7000 and H5600 chassis, and the archive A300/3000 platforms. The architecture consists of the following principal components:
- Data Reduction Platform
- Compression Engine and Chunk Map
- Zero block removal phase
- Deduplication In-memory Index and Shadow Store Infrastructure
- Data Reduction Alerting and Reporting Framework
- Data Reduction Control Path
The inline data reduction write path consists of three main phases:
- Zero Block Removal
- In-line Deduplication
- In-line Compression
If both inline compression and deduplication are enabled on a cluster, zero block removal is performed first, followed by deduplication, and then compression. This order allows each phase to reduce the scope of work each subsequent phase has to perform.
Inline data reduction workflow
The F810 is different from the other inline data reduction supporting nodes in that it includes a hardware compression offload capability, with each node containing a Mellanox Innova-2 Flex Adapter. The Mellanox adapter transparently performs compression and decompression with minimal latency, avoiding the need for consuming a node’s expensive CPU and memory resources.
The OneFS hardware compression engine uses zlib, with a software implementation of igzip as fallback if there is a compression hardware failure. OneFS employs a compression chunk size of 128 KB, with each chunk consisting of sixteen 8 KB data blocks. This is optimal because it is also the same size that OneFS uses for its data protection stripe units, providing simplicity and efficiency by avoiding the overhead of additional chunk packing.
Compression chunks and OneFS transparent overlay
After compression, this chunk is reduced from sixteen to six 8 KB blocks in size. This means that this chunk is now physically 48 KB in size. OneFS provides a transparent logical overlay to the physical attributes. This overlay describes whether the backing data is compressed and which blocks in the chunk are physical or sparse, such that file system consumers are unaffected by compression. As such, the compressed chunk is logically represented as 128 KB in size, regardless of its actual physical size.
Efficiency savings must be at least 8 KB (one block) in order for compression to occur, otherwise that chunk or file will be passed over and remain in its original, uncompressed state. For example, a file of 16 KB that yields 8 KB (one block) of savings would be compressed. Once a file has been compressed, it is then FEC protected.
Compression chunks will never cross node pools. This avoids the need to decompress or recompress data to change protection levels, perform recovered writes, or otherwise shift protection-group boundaries.