Home > Storage > PowerMax and VMAX > Storage Admin > Dell PowerMax and VMware vSphere Configuration Guide > Optimizing for interoperability
The Dell PowerMax product line includes the PowerMax 2000 and 8000 and the PowerMax 2500 and 8500. The Dell PowerMax is a fully redundant, high-availability storage processor providing non-disruptive component replacements and code upgrades. The PowerMax system features high levels of performance, data integrity, reliability, and availability. Configuring the PowerMax storage array appropriately for a VMware ESXi environment is critical to ensure scalable, high-performance architecture. This section discusses these best practices.
Dell PowerMax storage arrays offer customers a wide choice of physical drives to meet different workloads including high performance Storage Class Memory (SCM) and Enterprise Flash Drives (EFD). On the PowerMax, various drive sizes can be intermixed on the same storage array to allow customers with the option of providing different applications with the appropriate service level.
The PowerMax storage array is a high-performance NVMe backend delivering increased IOPS and lower latency. NVMe is a high-performance protocol designed for modern media. It was architected to take advantage of the parallelism of modern CPUs and SSDs and SCM to overcome the limitations of storage protocols designed for hard disk drives. PowerMax platforms exhibit significantly improved data efficiency by using deduplication to reduce the number of duplicate copies of identical data.
A Service level is a value or property assigned to a storage group on the PowerMax that defines a target, average response time for IO operations providing storage performance simplification for users. Each service level has a priority relative to the other service levels. A higher priority service level will experience a faster response time than a lower one. Each service level also has a compliance range of response times (since the target is an average) that when violated will generate an alert. Some service levels also have a “floor” response time, meaning IO cannot be serviced any more quickly than that value. The PowerMax will throttle IO at the front-end to meet this floor. This ensures higher priority service levels are not impacted.
Service levels are addressed more fully in Chapter 4 in the section “Service levels”.
The most common configuration of a VMware ESXi cluster presents the storage to the virtual machines as flat files in a VMFS. It is, therefore, tempting to present the storage requirement for the VMware ESXi hosts as one large LUN as vSphere 6 and higher support datastores up to 64 TB in size. Using a singular, large LUN, however, can be detrimental to the scalability and performance characteristics of the environment. The next section will discuss this in detail.
For the majority of environments, Dell recommends using the default queue values for both the HBAs and the VMware specific queues;[1] however, due to performance demands, some customers may require adjustments. VMware may also make such recommendations for very IO intensive workloads or large VMs. While there is no supportability issue using larger queues, Dell recommends that any queue changes are thoroughly tested before enabling in production. The following discusses HBA and VMware-specific queues, but first a short explanation of queues in an NVMeoF environment is required.
The PowerMax implementation of NVMeoF supports multiple queues which is a feature of NVMeoF. The array controls the queue setting. There are 8 IO queues per connection or path, each with a queue depth of 128. This provides excellent scalability. No adjustments are possible.
VMware has a number of queues which can be adjusted if environmental conditions require it. For example, if the desire is to use a very few, very large LUNs in vSphere, this forces the VMkernel to serially queue IOs from all of the virtual machines utilizing the LUN. To help alleviate the queue, a per-device VMware parameter, Disk.SchedNumReqOutstanding (DSNRO), can be adjusted to prevent one virtual machine from monopolizing the Fibre Channel queue for the LUN when more than one virtual machine resides on the device. Note that the HBA queue depth may also require a change to match the new DSNRO value. In most environments, however, the default settings are sufficient and recommended for both the host and VMware queues since the workload is distributed.
Note: Adjusting the DSNRO can have a significant impact on the environment so caution is warranted. If queuing issues arise that impact performance, a better solution is to add more datastores and redistribute the virtual machines.
VMware also has an adaptive queue depth algorithm that is controlled by two parameters: Disk.QFullSampleSize and Disk.QFullThreshold. Disk.QFullSampleSize is the count of queue full conditions needed before ESXi starts throttling. It is off by default (zero value). Disk.QFullThreshold is the number of “good” condition responses needed for ESXi to increase the queue depth limit after hitting the queue full (the value of Disk.QFullSampleSize). It is also off by default. These parameters can be set on a per-device basis. When using PowerMax storage, Dell recommends leaving the QFullSampleSize and the QFullThreshold parameters unset. The PowerMax arrays would only issue a queue full if the array itself was under duress. In such circumstances these parameters are unlikely to provide any relief.
A long queue against the LUN results in unnecessary and unpredictable elongation of response time. This problem can be further exacerbated in configurations that allow multiple VMware ESXi hosts to share a single LUN, particularly across VMware clusters. In this configuration, all VMware ESXi hosts utilizing the LUN share the queues provided by the Fibre Channel port. In a large environment with multiple active virtual machines, despite the PowerMax’s ability to share front-end CPU cores and use of multiple queues on the array, it is possible to generate a queue that could impact performance and make it more likely VMware and HBA queues will require adjusting.
The potential response time elongation and performance degradation can be addressed by presenting more devices to the VMware ESXi cluster and decreasing the VM-density of each datastore. However, this does impose a minor overhead for managing the virtual infrastructure.
The anticipated IO activity influences the maximum size of the LUN that can be presented to the VMware ESXi hosts. The appropriate size for an environment should be determined after a thorough analysis of the performance data collected from existing physical or virtual environments.[2]
Note: If multiple storage arrays are presented to the same ESXi hosts along with PowerMax, be they Dell (e.g., XtremIO) or non-Dell, the VMware queues can be adjusted on a per-device basis so there is no conflict. Note, however, this does not hold true for any HBA queues which will be host-wide or other VMware-specific parameters. In such cases careful consideration must be taken when adjusting the default values as some arrays can handle higher queues than others and while performance may be improved on one, at the same time it could be reduced on another. Dell offers recommended settings when using two different Dell storage arrays with the same ESXi host which can be found in Dell KB 303782.
VMFS supports concatenation of multiple SCSI disks to create a single file system. While in the past, spanned VMFS could be useful when VMware restricted datastores to 2 TB extents, this is no longer a concern as VMware supports 64 TB datastores with a single extent. Dell does not recommend spanning VMFS as it adds unnecessary complexity to environments and negates the benefits of using PowerMax features particularly around replication. If the intent is to increase queue scalability, and thus performance, multiple datastores should be employed instead.
Virtualization enables better utilization of IT assets and fortunately the fundamentals for managing information in the virtualized environment are no different from a physical environment. Dell recommends the following best practices for a virtualized infrastructure:
Dell makes no general recommendation as to the number or size of datastores (VMFS) in a customer’s environment, nor how many VMs should be in each datastore. There are many variables which will determine the best configuration for a customer and the VM density they can support in each datastore. These variables may include response time, VM size, customer SLAs, type of application, backup and restore requirements, etc. To meet these needs, some customers use a few very large datastores, e.g., 8 TB or more, while others prefer a tiered model of small, medium, and large datastores. In the end, if a customer’s performance, availability, and management requirements are met, the configuration is deemed appropriate by Dell.
The PowerMax supports VMware Virtual Volumes, an integration and management framework (referred to as the vVol framework) that moves management from the datastore to the virtual machine. vVols virtualize storage, enabling a more efficient operational model that is optimized for virtualized environments and centered on the application instead of the infrastructure. vVols map virtual disks, configuration files, clones, and snapshots, directly to virtual volume objects on a storage system. This mapping allows VMware to offload more storage operations to the array, such as VM snapshots.
vVols virtualize SAN devices by abstracting physical hardware resources into logical pools of storage called Storage Containers which replace traditional storage volumes. These containers are created on the PowerMax and are assigned storage capabilities which can be all, or a subset, of the SLs the particular PowerMax array supports. These storage containers are presented to the vCenter when a VASA Provider is registered. The VASA Provider runs on the storage side and integrates with vSphere Storage Monitoring Service (SMS) and ESXi vvold service to manage all aspects of the storage in relation to vVols. All communication between VMware and the VASA Provider is out of band.
From the presented storage containers, the VMware Admin creates vVol datastores through the wizard, just like a traditional datastore. A vVol datastores has a one-to-one relationship with a storage container and is its logical representation on the ESXi host. A vVol datastore holds VMs and can be browsed like any datastore. Each vVol datastore then will inherit the capabilities of the container, i.e., SLs - Bronze, Silver, etc. The VMware Admin creates Storage Policies that are mapped to these capabilities so when a VM is deployed, the user selects the desired capability and is presented with compatible vVol datastores in which to deploy the VM.
Unlike traditional devices, ESXi has no direct SCSI visibility to vVols on the PowerMax. Instead, ESXi hosts use a front-end access point called a Protocol Endpoint.[3] A Protocol Endpoint (PE) is a special device created on the PowerMax and mapped and masked to the ESXi hosts. Each ESXi host requires a single, unique PE. The ESXi host uses the PE to communicate with the volumes and the disk files the vVols encapsulate. By utilizing PEs, ESXi establishes data paths from the virtual machines (VM) to the virtual volumes through binding.
vVols simplify the delivery of storage service levels to individual applications by providing finer control of hardware resources and allowing the use of native array-based data services such as SnapVX at the VM level. vVols present the VMware Admin a granularity of control over VMs on shared storage that cannot be achieved with traditional architecture due to the device limitations of the ESXi server. PowerMax supports up to 64k vVols on a PowerMax system.[4] Note that as each vVol uses a host available address, each one counts toward the total number of devices on the array.
vVols will not be covered in detail in this whitepaper as two extensive white papers exist which cover configuration and best practices:
[1] The use of replication on the device does not alter the recommendation.
[2] When it comes to IO tuning, Dell recommends leaving the VMware configuration option DiskMaxIOSize at the default setting of 32 MB.
[3] Therefore, boot from SAN is not supported.
[4] The PowerMax can support up to 64k total devices depending on the configuration (e.g., number of engines). Because of supporting devices such as PEs, the number of vVols would, by necessity, be less than 64k.