Home > Workload Solutions > High Performance Computing > White Papers > Dell Validated Design for HPC pixstor Storage—Joint Solution with Kalray > Overview
The NVMe Tier modules in this release (see Figure 2) were updated to include the PowerEdge R650 and PowerEdge R750 servers, and the NVIDIA-Excelero NVMesh software is no longer a component of the solution. For HA purposes, an alternative based on GPFS replication was implemented for each NVMe server pair. This alternative maintains good performance for the NVMe tier, reduces dependencies on third-party software, reduces complexities from an additional software layer, and reduces the price for this tier (because NVMesh licensing is no longer needed). The servers approved for this tier support NVMe PCIe4 devices. Mixing NVMe PCIe4 devices with lower performant PCIe3 devices is not recommended for the solution and not supported for the same NVMe tier. Performance and capacity for this NVMe tier can be scaled out by additional pairs of NVMe nodes. Increased capacity is provided by selecting the appropriate capacity for the NVMe devices supported on the servers or adding more NVMe servers.
Pairs of PowerEdge servers in HA (failover domains) provide a high-performance flash-based tier for the pixstor solution. Two PowerEdge servers were added and benchmarked as part of the NVMe tier: a PowerEdge R650 server with 10 NVMe direct attached drives and a PowerEdge R750 server with 16 NVMe direct attached devices. The PowerEdge R7525 server with 24 direct attached drives is supported and can also be used, but its performance with PCIe4 devices was not characterized. To maintain homogeneous performance across all the NVMe nodes, allowing striping data across all nodes in the tier, we strongly discourage mixing different server models in the same NVMe tier because differences in performance hold back faster nodes while waiting for stripes on slower nodes. However, having multiple NVMe tiers on a solution is possible to accommodate different sets of servers.
The drives on any NVMe tier server pair are configured as NSD devices; each NSD has a replica on the same pair of servers (failure domain) using GPFS replication. This configuration allows data redundancy not only at the device level but at the server level, though capacity is reduced to 50 percent of total NVMe NSDs capacity. The only restriction for these NVMe tier servers is that they must be used in pairs.
Note that the HDMD used since the initial pixstor release is now replaced by a NVMe tier module based on a pair of PowerEdge R650 servers used to store only metadata. For the rest of the performance characterization, such metadata modules are used and characterized along with other NVMe tier pairs for storing only data. Characterization of this new NVMe HDMD with ME5084-based storage is not included in this document.
Table 6. NVMe tier server components (components for this tier only)
Solution component | At release | Test bed | |
Processor | PowerEdge R650 NVMe nodes | 2x Intel Xeon Gold 6326 2.9 GHz, 16C/32T, 11.2GT/s, 24M Cache, Turbo, HT (185 W) DDR4-3200 | 2x Intel Xeon Gold 6354 3.00 GHz, 18C/36T, 11.2GT/s, 39M Cache, Turbo, HT (205 W) DDR4-3200 |
Optional high-demand metadata | 2x Intel Xeon Gold 6354 3.00 GHz, 18C/36T, 11.2GT/s, 39M Cache, Turbo, HT (205 W) DDR4-3200 | ||
PowerEdge R750 NVMe nodes | 2x Intel Xeon Platinum 8352Y, 2.2 GHz, 32C/64T, 11.2GT/s, 48M Cache, Turbo, HT (205 W) DDR4-3200 | ||
PowerEdge R7525 NVMe nodes | 2x AMD EPYC 7302 3.0 GHz, 16C/32T, 128M L3, (155 W) DDR4-3200 | Not tested with PCIe4 Devices | |
Memory | PowerEdge R650 NVMe nodes | 16x 16 GB RDIMM, 3200MT/s, Dual Rank (256 GiB) | |
Optional high-demand metadata | |||
PowerEdge R750 NVMe nodes | |||
PowerEdge R7525 NVMe nodes | |||
NVMe | PowerEdge R650 NVMe nodes | Supported NVMe devices 10 NSDs replicated across server-pair | 10 Dell AG 1.6 TB (PM1735) PCIe4 |
Optional high-demand metadata | Supported NVMe devices NSDs replicated across server-pair | 10 Dell AG 1.6 TB (PM1735) PCIe4 | |
PowerEdge R750 NVMe nodes | Supported NVMe devices 16 NSDs replicated across server-pair | 16 Dell Intel 1.6 TB (P5600) PCIe4 | |
PowerEdge R7525 NVMe nodes | Supported NVMe devices 24 NSDs replicated across server-pair | Not tested with PCIe4 Devices | |
Operating system | Red Hat Enterprise Linux 8.5 | ||
Kernel version | 4.18.0-348.23.1.el8_5.x86_64 | ||
pixstor software | 6.0.3.1-1 | ||
File system software | Spectrum Scale (GPFS) 5.1.3-1 | ||
High-performance network connectivity | NVMe nodes: 2x ConnectX-6 VPI InfiniBand using HDR (200 Gbps) | ||
High-performance switch | 2x Mellanox QM8700 (or SN3700 for 100 GbE) | ||
OFED version | Mellanox OFED 5.6-1.0.3.3 | ||
Local disks (operating system) | BOSS-S2 with 2x M.2 240 GB in RAID 1 | ||
Systems management | iDRAC9 Enterprise + Dell OpenManage |