With the PowerMaxOS Q3 2019 release, PowerMax introduced end-to-end NVMe over FC fabrics, or FC-NVMe. Internally, PowerMaxOS was already accessing both NAND and SCM flash media using NVMe protocol. New to this release is that now also the database servers can use the NVMe protocol over FC fabric (SAN switches) to access the PowerMax storage system for all read and write I/O operations.

To use this protocol, 32 Gb PowerMax front-end modules are required, together with Gen 6 FC switch and HBAs (32 Gb). This infrastructure allows for either FC or FC-NVMe deployment. The only difference is the protocol that the servers use to access storage devices. When PowerMax front-end adapter ports are set for FC connectivity they are listed as FAs; when set for FC-NVMe they are listed as FNs.

Why use NVMe?

Starting with NAND SSD flash storage, but much more importantly with SCM, storage latency is no longer measured in milliseconds but in microseconds. In addition, each SCM supports hundreds of thousands of IOPS.

NVMe protocol was built from the ground-up to replace legacy SAS and SATA storage access protocols meant for spinning drives. NVMe is NUMA-optimized to take advantage of multi-core CPUs so cores can share control over I/O submission queues. It is built around fast PCIe access between flash media and CPU, and it allows for much higher queue depth, which provides higher concurrency. This is important, because such high-performance drives often service higher I/O density (concurrency of I/Os per GB storage).

Why use FC-NVMe?

While on-board server PCIe NVMe flash drives are not new, there are a few disadvantages as compared to SCM inside a storage system, connected to the server using PCIe FC-NVMe protocol. First, high-performance NVMe flash drives such as SCM cost more than NAND SSD. Placing such drives in the server requires high server utilization to justify the cost. However, servers perform at their peak capacity for only a small percentage of their daily, weekly, or monthly usage. The rest of the time their utilization is low. This creates a potential waste of money and resources.

Placing the SCM drives in the storage system allows many servers to access the high-performance media over NVMe fabrics. Therefore, regardless of which server peaks in activity, the media is shared and can always be highly utilized. This adds value to the investment in SCM and allows for better resource utilization. In addition, PowerMax Service Levels set priorities to storage groups’ consumption of SCMs.

A similar comparison can be made with HCI (Hyper-Converged Infrastructure). HCI is based on a group of servers that provide compute (CPU), memory, networking, and storage resources for the deployment of applications. When on-board server SCM flash drives are used, given the nature of HCI, where applications may end up on any node in the cluster, it is very likely that the SCM drives are now needed on all the servers equally.

If the workload is not balanced, only a few of the servers will be busy at any given time. Either the busy servers could benefit from access to more SCM drives than those available locally (there will be other servers with SCM drives possibly underutilized), or in an opposite example, the busy servers are not busy enough to fully utilize even a single SCM drive installed locally.

By using FC-NVMe, these high-performance resources can be shared by any server that needs them since they are located in the storage system. This saves the cost of placing SCM drives in all the HCI nodes, and increases the utilization of the drives, because all servers connected to the storage system can leverage them, based on their designated Service Levels.

Because FC-NVMe is relatively new, Linux OS and multipathing software support is still limited. Refer to the section FC-NVMe multipathing options for more information.