NVMe Performance Increases for Next-Generation PowerEdge Servers with PERC11 Controller
Download PDFTue, 17 Jan 2023 05:48:58 -0000
|Read Time: 0 minutes
Summary
Dell Technologies newest RAID iteration, PERC11, has undergone significant change - most notably the inclusion of hardware RAID support for NVMe drives. To better understand the benefits that this will bring, various metrics were tested, including NVMe IOPS, disk bandwidth and latency. This DfD compares NVMe performance readings of the next-generation Dell EMC PowerEdge R650 server, powered by pre-production 3rd Generation Intel® Xeon® Scalable processors, to the prior-generation PowerEdge R640 server, powered by 2nd Generation Intel® Xeon® Scalable processors.
Introduction
With support for NVMe hardware RAID now available on the PERC11 H755N front, H755MX and H755 adapter form factors, we were eager to quanitfy how big of a performance boost next-generation PowerEdge servers with hardware RAID would obtain. Dell Technologies commissioned Principled Technologies to execute various studies that would compare the NVMe Input/Output Per Second (IOPS), disk bandwidth and latency readings of next-geneation PowerEdge servers (15G) with NVMe hardware RAID support against prior-generation PowerEdge servers (14G) without NVMe hardware RAID support.
Test Setup
Two servers were used for this study. The first was a PowerEdge R650 server populated with two 3rd Gen Intel® Xeon® Scalable processors, 1024GB of memory, 3.2TB of NVMe storage and a Dell PERC H755N storage controller. The second was a PowerEdge R640 server populated with two 2nd Gen Intel® Xeon® Gold Scalable processors, 128GB of memory, 1.9TB of SSD storage and a Dell PERC H730P Mini storage controller.
A tool called Flexible Input/Output (FIO) tester was used to create the I/O workloads used in testing. FIO invokes the production of threads or processes to do an I/O action as specified by the user. This test was chosen specifically because it injects the smallest system overhead of all the I/O benchmark tools we use. This in turn allows it to deliver enough data to the storage subsystem to reach 100% utilization. With the tool, five workloads were run at varied thread counts and queue depths on RAID 10, RAID 6, and RAID 5 levels of the Dell EMC PowerEdge R650 server with PERC H755n RAID controller and NVMe drives and the Dell EMC PowerEdge R640 server with a PERC H730P Mini controller and SATA SSD drives.
Read-heavy workloads indicate how quickly the servers can retrieve information from their disks, while write-heavy workloads indicate how quickly the servers can commit or save data to the disk. Additionally, random and sequential in the workload descriptions refer to the access patterns for reading or writing data. Random accesses require the server to pull data from multiple disks in a non-sequential fashion (i.e., visiting multiple websites), while sequential accesses require the server to pull data from a single continuous stream (i.e., streaming a video).
Performance Comparisons
IOPS
IOPS indicates the level of user requests that a server can handle. Based on the IOPS output seen during testing, upgrading from the prior-generation Dell EMC PowerEdge R640 server to the latest-generation Dell EMC PowerEdge R650 server could deliver performance gains for I/O-intensive applications. In all three RAID configurations tested, the PowerEdge R650 with NVMe SSDs delivered significantly more IOPS than the prior-generation server. Figures 1, 2 and 3 show how many average IOPS each configuration handled during testing:
Figure 1: IOPS comparison for RAID 10 configurations
Figure 2: IOPS comparison for RAID 6 configurations
Figure 3: IOPS comparison for RAID 5 configurations
Disk Bandwidth
Disk bandwidth indicates the volume of data a system can read or write. A server with high disk bandwidth can process more data for large data requests, such as streaming video or big data applications. At all three RAID levels, the latest-generation Dell EMC PowerEdge R650 server with NVMe storage transferred significantly more MB per second than the prior-generation server. Figure 4 shows the disk bandwidth that each of the two servers supported for each RAID level:
Figure 4: Disk bandwidth comparison for RAID 10, 6 and 5 configurations
Latency
Latency indicates how quickly the system can respond to a request for an I/O operation. Longer latency can impact application responsiveness and could contribute to a negative user experience. In addition to greater disk bandwidth, the Dell EMC PowerEdge R650 server delivered lower latency at each of the three RAID levels than the prior-generation server. Figure 5 shows the latency that each server delivered while running one workload at each RAID level.
Figure 5: Latency comparison for RAID 10, 6 and 5 configurations
Conclusion
The next-generation PowerEdge R650 server with NVMe HW RAID support increased IOPS by up to 15.7x, disk bandwidth by up to 15.5x, and decreased latency by up to 93%. With the inclusion of NVMe HW RAID support on Dell Technologies’ new PERC11 controllers, now is a great time for PowerEdge customers to migrate their storage medium over to NVMe drives and yield the higher-performance that comes with it!
For more details, please read the full PT report Accelerate I/O with NVMe drives on the New PowerEdge R650 server
Related Documents
NVMe Performance Increases for Next-Generation PowerEdge Servers with PERC11 Controller
Mon, 16 Jan 2023 13:44:21 -0000
|Read Time: 0 minutes
Summary
Dell Technologies newest RAID iteration, PERC11, has undergone significant change - most notably the inclusion of hardware RAID support for NVMe drives. To better understand the benefits that this will bring, various metrics were tested, including NVMe IOPS, disk bandwidth and latency. This DfD compares NVMe performance readings of the next-generation Dell EMC PowerEdge R650 server, powered by pre-production 3rd Generation Intel® Xeon® Scalable processors, to the prior-generation PowerEdge R640 server, powered by 2nd Generation Intel® Xeon® Scalable processors.
Introduction
With support for NVMe hardware RAID now available on the PERC11 H755N front, H755MX and H755 adapter form factors, we were eager to quanitfy how big of a performance boost next-generation PowerEdge servers with hardware RAID would obtain. Dell Technologies commissioned Principled Technologies to execute various studies that would compare the NVMe Input/Output Per Second (IOPS), disk bandwidth and latency readings of next-geneation PowerEdge servers (15G) with NVMe hardware RAID support against prior-generation PowerEdge servers (14G) without NVMe hardware RAID support.
Test Setup
Two servers were used for this study. The first was a PowerEdge R650 server populated with two 3rd Gen Intel® Xeon® Scalable processors, 1024GB of memory, 3.2TB of NVMe storage and a Dell PERC H755N storage controller. The second was a PowerEdge R640 server populated with two 2nd Gen Intel® Xeon® Gold Scalable processors, 128GB of memory, 1.9TB of SSD storage and a Dell PERC H730P Mini storage controller.
A tool called Flexible Input/Output (FIO) tester was used to create the I/O workloads used in testing. FIO invokes the production of threads or processes to do an I/O action as specified by the user. This test was chosen specifically because it injects the smallest system overhead of all the I/O benchmark tools we use. This in turn allows it to deliver enough data to the storage subsystem to reach 100% utilization. With the tool, five workloads were run at varied thread counts and queue depths on RAID 10, RAID 6, and RAID 5 levels of the Dell EMC PowerEdge R650 server with PERC H755n RAID controller and NVMe drives and the Dell EMC PowerEdge R640 server with a PERC H730P Mini controller and SATA SSD drives.
Read-heavy workloads indicate how quickly the servers can retrieve information from their disks, while write-heavy workloads indicate how quickly the servers can commit or save data to the disk. Additionally, random and sequential in the workload descriptions refer to the access patterns for reading or writing data. Random accesses require the server to pull data from multiple disks in a non-sequential fashion (i.e., visiting multiple websites), while sequential accesses require the server to pull data from a single continuous stream (i.e., streaming a video).
Performance Comparisons
IOPS
IOPS indicates the level of user requests that a server can handle. Based on the IOPS output seen during testing, upgrading from the prior-generation Dell EMC PowerEdge R640 server to the latest-generation Dell EMC PowerEdge R650 server could deliver performance gains for I/O-intensive applications. In all three RAID configurations tested, the PowerEdge R650 with NVMe SSDs delivered significantly more IOPS than the prior-generation server. Figures 1, 2 and 3 show how many average IOPS each configuration handled during testing:
Figure 1: IOPS comparison for RAID 10 configurations
Figure 2: IOPS comparison for RAID 6 configurations
Figure 3: IOPS comparison for RAID 5 configurations
Disk Bandwidth
Disk bandwidth indicates the volume of data a system can read or write. A server with high disk bandwidth can process more data for large data requests, such as streaming video or big data applications. At all three RAID levels, the latest-generation Dell EMC PowerEdge R650 server with NVMe storage transferred significantly more MB per second than the prior-generation server. Figure 4 shows the disk bandwidth that each of the two servers supported for each RAID level:
Figure 4: Disk bandwidth comparison for RAID 10, 6 and 5 configurations
Latency
Latency indicates how quickly the system can respond to a request for an I/O operation. Longer latency can impact application responsiveness and could contribute to a negative user experience. In addition to greater disk bandwidth, the Dell EMC PowerEdge R650 server delivered lower latency at each of the three RAID levels than the prior-generation server. Figure 5 shows the latency that each server delivered while running one workload at each RAID level.
Figure 5: Latency comparison for RAID 10, 6 and 5 configurations
Conclusion
The next-generation PowerEdge R650 server with NVMe HW RAID support increased IOPS by up to 15.7x, disk bandwidth by up to 15.5x, and decreased latency by up to 93%. With the inclusion of NVMe HW RAID support on Dell Technologies’ new PERC11 controllers, now is a great time for PowerEdge customers to migrate their storage medium over to NVMe drives and yield the higher-performance that comes with it!
For more details, please read the full PT report Accelerate I/O with NVMe drives on the New PowerEdge R650 server
For Peak Performance, Bare Metal with NVMe® RAID Comes Out on Top
Thu, 14 Mar 2024 16:51:08 -0000
|Read Time: 0 minutes
For Peak Performance, Bare Metal with NVMe® RAID Comes Out on Top
Prowess Consulting testing showed that bare-metal servers with NVMe RAID controllers offer an ideal solution for performance-hungry workloads, such as analytics, AI, ML, and HPC.
Executive Summary
It’s no surprise that bare-metal servers often provide the highest level of performance for demanding workloads. Organizations might be tempted to use bare-metal deployments to maximize performance and value from their platforms, especially for use with demanding workloads such as analytics, artificial intelligence (AI), machine learning (ML), and high-performance computing (HPC). Until recently, this approach has been blocked due to concerns around high availability. For example, although RAID arrays could be used with bare-metal deployments to ensure availability, this option hasn’t been viable in the past due to performance constraints of Serial Attached SCSI (SAS)− and Serial ATA (SATA)−based RAID controllers.
Newer NVM Express® (NVMe®)-based RAID controllers remove that constraint. The NVMe protocol in newer RAID controllers, such as Dell™ PowerEdge™ RAID Controller 11 (PERC 11), can overcome the latency gaps caused by the slower SAS and SATA protocols.
To determine the viability of using platforms with NVMe RAID controllers for demanding workloads, Prowess Consulting ran tests to measure transactional database performance using a Dell™ PowerEdge™ R750 server built with Dell PERC 11 NVMe RAID controllers. We compared performance to a Dell PowerEdge R740xd, configured with Dell PERC 10 controllers using SATA SSDs. In our testing, the PowerEdge R750 with NVMe RAID demonstrated an impressive 2.92x increase in database transaction performance, demonstrating that the newer platform offers a compelling option for businesses looking for high performance with the reliability and availability offered by PERC 11 RAID.
Availability, Reliability, and Performance: Can You Have It All?
The proliferation of high-performance workloads has pushed the performance boundaries of what many traditional deployments can provide. Even with hardware and network performance tuning, organizations might find themselves or their customers growing frustrated while waiting for results. This situation can be particularly apparent when running many of the most common emerging AI, ML, analytics, and database-dependent workloads.
The problem is exacerbated when business applications are dependent on large and ever-increasing quantities of data. For example, a business hosting a customer-accessed website needs to be able to process large amounts of rapidly changing data quickly or risk losing out to the competition. Retail companies also rely on up-to-date sales, inventory, and customer data for processing orders and gathering critical insights. Healthcare organizations process massive quantities of patient data, prescriptions, scheduling information, and other medical records on a daily basis.
For time-critical workloads like these, businesses need a reliable way to ramp up performance without sacrificing the required levels of availability and reliability for apps and data.
The Game-Changing Shift from SAS to NVMe RAID
One option for significantly reducing latency is to eliminate overhead by deploying workloads directly on bare-metal servers instead of virtualized environments. But bare-metal deployments traditionally come with tradeoffs, particularly when it comes to maintaining availability and integrity of data.
RAID is an obvious choice to address availability concerns, but traditional SAS-based or Serial ATA (SATA®)-based RAID controllers and SSDs can’t provide significant (if any) performance gains over traditional deployments. However, those constraints can now be eliminated by newer RAID controllers. Recent RAID controllers from Dell Technologies, for example, support the NVMe interface specification for connecting over a fast PCIe® bus. RAID and storage built on NVMe and PCIe can significantly improve bandwidth and reduce latency over traditional SAS-based or SATA-based controllers. Prowess Consulting decided to put this possibility to the test. Could a bare-metal server with fast NVMe RAID controllers and drives offer a high-performance option for critical workloads? And if so, how much of a performance boost would even newer PERC 11 RAID controllers in Dell PowerEdge R750 servers offer over previous-generation Dell PowerEdge R740xd servers built with PERC 10 controllers?
Putting NVMe RAID to the Test
Prowess testing looked at transactional database performance for a bare-metal PowerEdge R750 server built with Dell PERC 11 RAID controllers that support the NVMe protocol. We then ran comparable tests on a PowerEdge R740xd server built with Dell PERC 10 RAID controllers.
We used Microsoft® SQL Server® for testing, as this application is commonly used with data-intensive workloads that require top performance for businesses and their customers.
The system configurations for this test are shown in Table 1.
Table 1. System configurations used for testing bare-metal/RAID performance
Configuration | Dell™ PowerEdge™ Server Platform | |
Server | 1 x Dell™ PowerEdge™ R740xd | 1 x Dell™ PowerEdge™ R750 |
Processor | Intel® Xeon® Gold 6238 processor | Intel® Xeon® Gold 6338 processor |
Number of CPUs | 2 | 2 |
Cores/threads per CPU | 22/44 | 32/64 |
Cores/threads total | 44/88 | 64/128 |
Frequency (base/SCT/MCT) | 2.10 GHz | 2.00 GHz |
Storage controller 01 | Dell™ PowerEdge™ RAID Controller (PERC) H740P Front Broadcom® LSI | Dell™ PowerEdge™ RAID Controller (PERC) H755N Front Broadcom® LSI |
Disk | 960 GB Samsung MZ7LH960HBJR0D3 SATA SSD | 3.2 TB KIOXIA® Dell™ CM6 MU NVMe® SSD (KCM6XVUL3T20) |
Number of disks | 16 | 8 |
Storage controller 02 | BOSS-S1 | Dell™ PERC H755N Front NVMe® Broadcom® controller |
Disk | 240 GB Intel SSDSCKKB240G8R M.2 SATA | 3.2 TB KIOXIA® Dell™ CM6 MU NVMe® SSD (KCM6XVUL3T20) |
Number of disks | 2 | 8 |
Storage controller 03 | – | Marvell® Technology Group Ltd. Dell™ BOSS-S2 |
Disk | – | 480 GB Micron® MTFDDAV480TDS |
Number of disks | – | 2 |
Installed memory | 256 GB | 256 GB |
Memory DIMM | Hynix® HMA84GR7JJR4N-WM DDR4 | Hynix® HMA82GR7DJR8N-XN 16 GB dual-rank DDR4 |
Memory speed | 2,933 megatransfers per second (MT/s) | 3,200 megatransfers per second (MT/s) |
Number of memory DIMMs | 8 x 32 GB | 16 x 16 GB |
Network | BCM57416 NetXtreme-E Dual-Media 10 Gb RDMA Ethernet Controller NetXtreme® BCM5720 Gb Ethernet PCIe® Ethernet 25G 2P E810-XXV Adapter | 1 x Broadcom® Advanced Dual 25 Gb Ethernet |
Operating system (OS) | Red Hat® Enterprise Linux® 8.5 (Ootpa) | Red Hat® Enterprise Linux® 8.8 (Ootpa) |
OS kernel | Linux® 4.18.0-348.20.1.el8_5.x86_64 | Linux® 4.18.0-477.15.1.el8_8.x86_64 |
BIOS version | 2.13.3 | 1.10.2 |
OS performance profile | Tuned for Microsoft® SQL Server® | Tuned for Microsoft® SQL Server® |
Microsoft® SQL Server® version | Microsoft® SQL Server® 2019 (RTM-CU15) (KB5008996)—15.0.4198.2 (X64) | Microsoft® SQL Server® 2022 (RTM-CU6) (KB5027505)—16.0.4055.4 (X64) |
Measuring Database Performance
The goal of this testing was to generate performance data showing the new orders per minute (NOPM) performance of a SQL Server database running on the Dell PowerEdge server platform. Prowess engineers also collected other performance data to verify that the system was operating as intended.
Prowess Consulting engineers generated performance data showing the NOPM performance of a SQL Server database running on each PowerEdge server platform. For this testing, we used BenchCraft®, a Microsoft benchmarking tool that processes data like a TPC-C® benchmark.[1]
As Figure 1 shows, our testing revealed a significant jump in performance when running a SQL Server workload on the bare-metal PowerEdge R750 server with an NVMe RAID array, compared to a similar workload on the PowerEdge R740xd with a SATA array.
Figure 1. Database performance in new orders per minute (NOPM), comparing a bare-metal Dell™ PowerEdge™ R740xd server with Dell® PERC 10 RAID to a Dell PowerEdge R750 server with Dell PERC 11 RAID
The test results showed that a bare-metal deployment with NVMe RAID controllers can provide exceptional results for users looking to maximize database performance on bare-metal servers, without giving up the availability benefits provided by a RAID array.
Behind the Results
As the results show, eliminating the overhead inherent in a hypervisor-based system resulted in significant performance gains. There’s nothing earth-shattering in measuring higher performance for a bare-metal system alone. What stands out in this test scenario was that these performance gains were apparent even with the use of RAID drives, which traditionally have exhibited unacceptable levels of latency. By incorporating NVMe, the Dell PERC 11 controllers and drives in the tested PowerEdge R750 servers provided the required levels of availability for the platform without contributing measurable latency.
Using bare-metal systems with RAID can also provide other benefits. For example, with SQL Server, the transaction log acts as a buffer for the main database. That means administrators can optimize for performance by configuring the front RAID controllers to use RAID 10, which is the highest performing option. For the back-end database, which is not as performance sensitive, admins can choose to use RAID 5 (or any preferred level) to sacrifice some performance for increased capacity.
For more details on how RAID with support for NVMe drives affects performance for SQL Server transactions, compared to SAS-based drives, see this previous test report from Prowess, which includes details on log disk reads and writes: https://prowessconsulting.com/project/dell-poweredge-r750-pcie-gen4-raid-increase-sql-server-performance/.
NVMe RAID Controllers
As discussed previously, NVMe adapters were the key to the observed performance gains. The PowerEdge R750 server used in the testing paired two Dell PERC 11 H755N Front NVMe adapters with all-NVMe solid-state drives (SSDs) to maximize storage bandwidth and throughput.
The Dell PERC 11 H755N Front NVMe adapter is based on the Broadcom® SAS3916 PCIe to SAS/SATA/PCIe RAID-on-Chip (RoC) controller. These were the first RAID controllers from Dell Technologies to offer both PCIe Gen4 host and PCIe Gen4 storage interfaces, which deliver double the bandwidth and 75 percent more input/output operations per second (IOPS), compared to previous generations.[2]
The Dell PERC 11 H755N Front NVMe adapter provides high PCIe (NVMe) storage-interface data-transfer rates of 16 gigatransfers per second (GT/s), 8 GT/s, 5 GT/s, and 2.5 GT/s per lane, in addition to reliability, high performance, and fault-tolerant disk subsystem management and support for RAID levels 0, 1, 5, 6, 10, 50, and 60.
Ethernet Controllers
For the purposes of this testing, we isolated the test platforms to remove network speed as a variable. However, the PowerEdge R750 server includes the Broadcom® NetXtreme® E-Series BCM57414 50G PCIe 3.0 Ethernet controller, based on Broadcom’s scalable 10/25/50/100/200 gigabit Ethernet (GbE) controller architecture. This network card combines a high-bandwidth Ethernet controller with a unique set of highly optimized hardware-acceleration engines to enhance network performance and improve server efficiency for enterprise and cloud-scale networking and storage applications, including high-performance computing (HPC), telco, ML, storage disaggregation, and data analytics.
Dell™ PowerEdge™ R750 Server Advantages
The Dell™ PowerEdge™ R750 server is an enterprise server designed to deliver high performance for demanding workloads. Powered by 3rd Gen Intel® Xeon® Scalable processors, the PowerEdge R750 server is a dual-socket/2U rack server with support for eight channels of memory per CPU and up to 32 DDR4 DIMMs at 3,200 MT/s speeds. In addition, the PowerEdge R750 server supports PCIe® Gen4 and up to 24 NVM Express® (NVMe®) drives (with an option for four additional rear-mounted drives) with improved air-cooling features and optional Direct Liquid Cooling (DLC) to support increasing power and thermal requirements. This makes the PowerEdge R750 server a compelling option for a wide range of workloads that require performance, extensive storage, and graphics processing unit (GPU) support. These workloads include database and analytics, high-performance computing (HPC), traditional corporate IT, virtual desktop infrastructure (VDI), and artificial intelligence (AI)/machine learning (ML) environments. For more information, see the PowerEdge R750 specification sheet on the Dell Technologies website: https://i.dell.com/sites/csdocuments/Product_Docs/en/poweredge-R750-spec-sheet.pdf.
Bare Metal with RAID Delivers Top Performance with Availability
Bare-metal deployments of SQL Server on platforms built with NVMe-based RAID offer exceptional performance while still providing required levels of reliability for apps and data. As our testing showed, this configuration offers organizations an ideal option for deploying performance-hungry workloads, such as analytics, AI, ML, and HPC.
For detailed testing methodology and configurations used in this study, see “Behind the Report: For Peak Performance, Bare Metal Wins”: https://prowessconsulting.com/project/dell-poweredge-r750-bare-metal-with-NVMe-raid-boosts-performance/.
Learn more about the Dell PowerEdge servers discussed in this study:
- Dell PowerEdge R750: https://i.dell.com/sites/csdocuments/Product_Docs/en/poweredge-R750-spec-sheet.pdf
The analysis in this document was done by Prowess Consulting and commissioned by Dell Technologies.
Results have been simulated and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Prowess and the Prowess logo are trademarks of Prowess Consulting, LLC.
Copyright © 2023 Prowess Consulting, LLC. All rights reserved.
Other trademarks are the property of their respective owners.
Author: Prowess Consulting, LLC
[1] Note that BenchCraft® does not conform to the TPC-C® testing standards, and results between BenchCraft and TPC-C are not comparable.
[2] Broadcom internal data, provided by Dell Technologies.