For Peak Performance, Bare Metal with NVMe® RAID Comes Out on Top
Research Abstract Read the Report See the ScienceThu, 14 Mar 2024 16:51:08 -0000
|Read Time: 0 minutes
For Peak Performance, Bare Metal with NVMe® RAID Comes Out on Top
Prowess Consulting testing showed that bare-metal servers with NVMe RAID controllers offer an ideal solution for performance-hungry workloads, such as analytics, AI, ML, and HPC.
Executive Summary
It’s no surprise that bare-metal servers often provide the highest level of performance for demanding workloads. Organizations might be tempted to use bare-metal deployments to maximize performance and value from their platforms, especially for use with demanding workloads such as analytics, artificial intelligence (AI), machine learning (ML), and high-performance computing (HPC). Until recently, this approach has been blocked due to concerns around high availability. For example, although RAID arrays could be used with bare-metal deployments to ensure availability, this option hasn’t been viable in the past due to performance constraints of Serial Attached SCSI (SAS)− and Serial ATA (SATA)−based RAID controllers.
Newer NVM Express® (NVMe®)-based RAID controllers remove that constraint. The NVMe protocol in newer RAID controllers, such as Dell™ PowerEdge™ RAID Controller 11 (PERC 11), can overcome the latency gaps caused by the slower SAS and SATA protocols.
To determine the viability of using platforms with NVMe RAID controllers for demanding workloads, Prowess Consulting ran tests to measure transactional database performance using a Dell™ PowerEdge™ R750 server built with Dell PERC 11 NVMe RAID controllers. We compared performance to a Dell PowerEdge R740xd, configured with Dell PERC 10 controllers using SATA SSDs. In our testing, the PowerEdge R750 with NVMe RAID demonstrated an impressive 2.92x increase in database transaction performance, demonstrating that the newer platform offers a compelling option for businesses looking for high performance with the reliability and availability offered by PERC 11 RAID.
Availability, Reliability, and Performance: Can You Have It All?
The proliferation of high-performance workloads has pushed the performance boundaries of what many traditional deployments can provide. Even with hardware and network performance tuning, organizations might find themselves or their customers growing frustrated while waiting for results. This situation can be particularly apparent when running many of the most common emerging AI, ML, analytics, and database-dependent workloads.
The problem is exacerbated when business applications are dependent on large and ever-increasing quantities of data. For example, a business hosting a customer-accessed website needs to be able to process large amounts of rapidly changing data quickly or risk losing out to the competition. Retail companies also rely on up-to-date sales, inventory, and customer data for processing orders and gathering critical insights. Healthcare organizations process massive quantities of patient data, prescriptions, scheduling information, and other medical records on a daily basis.
For time-critical workloads like these, businesses need a reliable way to ramp up performance without sacrificing the required levels of availability and reliability for apps and data.
The Game-Changing Shift from SAS to NVMe RAID
One option for significantly reducing latency is to eliminate overhead by deploying workloads directly on bare-metal servers instead of virtualized environments. But bare-metal deployments traditionally come with tradeoffs, particularly when it comes to maintaining availability and integrity of data.
RAID is an obvious choice to address availability concerns, but traditional SAS-based or Serial ATA (SATA®)-based RAID controllers and SSDs can’t provide significant (if any) performance gains over traditional deployments. However, those constraints can now be eliminated by newer RAID controllers. Recent RAID controllers from Dell Technologies, for example, support the NVMe interface specification for connecting over a fast PCIe® bus. RAID and storage built on NVMe and PCIe can significantly improve bandwidth and reduce latency over traditional SAS-based or SATA-based controllers. Prowess Consulting decided to put this possibility to the test. Could a bare-metal server with fast NVMe RAID controllers and drives offer a high-performance option for critical workloads? And if so, how much of a performance boost would even newer PERC 11 RAID controllers in Dell PowerEdge R750 servers offer over previous-generation Dell PowerEdge R740xd servers built with PERC 10 controllers?
Putting NVMe RAID to the Test
Prowess testing looked at transactional database performance for a bare-metal PowerEdge R750 server built with Dell PERC 11 RAID controllers that support the NVMe protocol. We then ran comparable tests on a PowerEdge R740xd server built with Dell PERC 10 RAID controllers.
We used Microsoft® SQL Server® for testing, as this application is commonly used with data-intensive workloads that require top performance for businesses and their customers.
The system configurations for this test are shown in Table 1.
Table 1. System configurations used for testing bare-metal/RAID performance
Configuration | Dell™ PowerEdge™ Server Platform | |
Server | 1 x Dell™ PowerEdge™ R740xd | 1 x Dell™ PowerEdge™ R750 |
Processor | Intel® Xeon® Gold 6238 processor | Intel® Xeon® Gold 6338 processor |
Number of CPUs | 2 | 2 |
Cores/threads per CPU | 22/44 | 32/64 |
Cores/threads total | 44/88 | 64/128 |
Frequency (base/SCT/MCT) | 2.10 GHz | 2.00 GHz |
Storage controller 01 | Dell™ PowerEdge™ RAID Controller (PERC) H740P Front Broadcom® LSI | Dell™ PowerEdge™ RAID Controller (PERC) H755N Front Broadcom® LSI |
Disk | 960 GB Samsung MZ7LH960HBJR0D3 SATA SSD | 3.2 TB KIOXIA® Dell™ CM6 MU NVMe® SSD (KCM6XVUL3T20) |
Number of disks | 16 | 8 |
Storage controller 02 | BOSS-S1 | Dell™ PERC H755N Front NVMe® Broadcom® controller |
Disk | 240 GB Intel SSDSCKKB240G8R M.2 SATA | 3.2 TB KIOXIA® Dell™ CM6 MU NVMe® SSD (KCM6XVUL3T20) |
Number of disks | 2 | 8 |
Storage controller 03 | – | Marvell® Technology Group Ltd. Dell™ BOSS-S2 |
Disk | – | 480 GB Micron® MTFDDAV480TDS |
Number of disks | – | 2 |
Installed memory | 256 GB | 256 GB |
Memory DIMM | Hynix® HMA84GR7JJR4N-WM DDR4 | Hynix® HMA82GR7DJR8N-XN 16 GB dual-rank DDR4 |
Memory speed | 2,933 megatransfers per second (MT/s) | 3,200 megatransfers per second (MT/s) |
Number of memory DIMMs | 8 x 32 GB | 16 x 16 GB |
Network | BCM57416 NetXtreme-E Dual-Media 10 Gb RDMA Ethernet Controller NetXtreme® BCM5720 Gb Ethernet PCIe® Ethernet 25G 2P E810-XXV Adapter | 1 x Broadcom® Advanced Dual 25 Gb Ethernet |
Operating system (OS) | Red Hat® Enterprise Linux® 8.5 (Ootpa) | Red Hat® Enterprise Linux® 8.8 (Ootpa) |
OS kernel | Linux® 4.18.0-348.20.1.el8_5.x86_64 | Linux® 4.18.0-477.15.1.el8_8.x86_64 |
BIOS version | 2.13.3 | 1.10.2 |
OS performance profile | Tuned for Microsoft® SQL Server® | Tuned for Microsoft® SQL Server® |
Microsoft® SQL Server® version | Microsoft® SQL Server® 2019 (RTM-CU15) (KB5008996)—15.0.4198.2 (X64) | Microsoft® SQL Server® 2022 (RTM-CU6) (KB5027505)—16.0.4055.4 (X64) |
Measuring Database Performance
The goal of this testing was to generate performance data showing the new orders per minute (NOPM) performance of a SQL Server database running on the Dell PowerEdge server platform. Prowess engineers also collected other performance data to verify that the system was operating as intended.
Prowess Consulting engineers generated performance data showing the NOPM performance of a SQL Server database running on each PowerEdge server platform. For this testing, we used BenchCraft®, a Microsoft benchmarking tool that processes data like a TPC-C® benchmark.[1]
As Figure 1 shows, our testing revealed a significant jump in performance when running a SQL Server workload on the bare-metal PowerEdge R750 server with an NVMe RAID array, compared to a similar workload on the PowerEdge R740xd with a SATA array.
Figure 1. Database performance in new orders per minute (NOPM), comparing a bare-metal Dell™ PowerEdge™ R740xd server with Dell® PERC 10 RAID to a Dell PowerEdge R750 server with Dell PERC 11 RAID
The test results showed that a bare-metal deployment with NVMe RAID controllers can provide exceptional results for users looking to maximize database performance on bare-metal servers, without giving up the availability benefits provided by a RAID array.
Behind the Results
As the results show, eliminating the overhead inherent in a hypervisor-based system resulted in significant performance gains. There’s nothing earth-shattering in measuring higher performance for a bare-metal system alone. What stands out in this test scenario was that these performance gains were apparent even with the use of RAID drives, which traditionally have exhibited unacceptable levels of latency. By incorporating NVMe, the Dell PERC 11 controllers and drives in the tested PowerEdge R750 servers provided the required levels of availability for the platform without contributing measurable latency.
Using bare-metal systems with RAID can also provide other benefits. For example, with SQL Server, the transaction log acts as a buffer for the main database. That means administrators can optimize for performance by configuring the front RAID controllers to use RAID 10, which is the highest performing option. For the back-end database, which is not as performance sensitive, admins can choose to use RAID 5 (or any preferred level) to sacrifice some performance for increased capacity.
For more details on how RAID with support for NVMe drives affects performance for SQL Server transactions, compared to SAS-based drives, see this previous test report from Prowess, which includes details on log disk reads and writes: https://prowessconsulting.com/project/dell-poweredge-r750-pcie-gen4-raid-increase-sql-server-performance/.
NVMe RAID Controllers
As discussed previously, NVMe adapters were the key to the observed performance gains. The PowerEdge R750 server used in the testing paired two Dell PERC 11 H755N Front NVMe adapters with all-NVMe solid-state drives (SSDs) to maximize storage bandwidth and throughput.
The Dell PERC 11 H755N Front NVMe adapter is based on the Broadcom® SAS3916 PCIe to SAS/SATA/PCIe RAID-on-Chip (RoC) controller. These were the first RAID controllers from Dell Technologies to offer both PCIe Gen4 host and PCIe Gen4 storage interfaces, which deliver double the bandwidth and 75 percent more input/output operations per second (IOPS), compared to previous generations.[2]
The Dell PERC 11 H755N Front NVMe adapter provides high PCIe (NVMe) storage-interface data-transfer rates of 16 gigatransfers per second (GT/s), 8 GT/s, 5 GT/s, and 2.5 GT/s per lane, in addition to reliability, high performance, and fault-tolerant disk subsystem management and support for RAID levels 0, 1, 5, 6, 10, 50, and 60.
Ethernet Controllers
For the purposes of this testing, we isolated the test platforms to remove network speed as a variable. However, the PowerEdge R750 server includes the Broadcom® NetXtreme® E-Series BCM57414 50G PCIe 3.0 Ethernet controller, based on Broadcom’s scalable 10/25/50/100/200 gigabit Ethernet (GbE) controller architecture. This network card combines a high-bandwidth Ethernet controller with a unique set of highly optimized hardware-acceleration engines to enhance network performance and improve server efficiency for enterprise and cloud-scale networking and storage applications, including high-performance computing (HPC), telco, ML, storage disaggregation, and data analytics.
Dell™ PowerEdge™ R750 Server Advantages
The Dell™ PowerEdge™ R750 server is an enterprise server designed to deliver high performance for demanding workloads. Powered by 3rd Gen Intel® Xeon® Scalable processors, the PowerEdge R750 server is a dual-socket/2U rack server with support for eight channels of memory per CPU and up to 32 DDR4 DIMMs at 3,200 MT/s speeds. In addition, the PowerEdge R750 server supports PCIe® Gen4 and up to 24 NVM Express® (NVMe®) drives (with an option for four additional rear-mounted drives) with improved air-cooling features and optional Direct Liquid Cooling (DLC) to support increasing power and thermal requirements. This makes the PowerEdge R750 server a compelling option for a wide range of workloads that require performance, extensive storage, and graphics processing unit (GPU) support. These workloads include database and analytics, high-performance computing (HPC), traditional corporate IT, virtual desktop infrastructure (VDI), and artificial intelligence (AI)/machine learning (ML) environments. For more information, see the PowerEdge R750 specification sheet on the Dell Technologies website: https://i.dell.com/sites/csdocuments/Product_Docs/en/poweredge-R750-spec-sheet.pdf.
Bare Metal with RAID Delivers Top Performance with Availability
Bare-metal deployments of SQL Server on platforms built with NVMe-based RAID offer exceptional performance while still providing required levels of reliability for apps and data. As our testing showed, this configuration offers organizations an ideal option for deploying performance-hungry workloads, such as analytics, AI, ML, and HPC.
For detailed testing methodology and configurations used in this study, see “Behind the Report: For Peak Performance, Bare Metal Wins”: https://prowessconsulting.com/project/dell-poweredge-r750-bare-metal-with-NVMe-raid-boosts-performance/.
Learn more about the Dell PowerEdge servers discussed in this study:
- Dell PowerEdge R750: https://i.dell.com/sites/csdocuments/Product_Docs/en/poweredge-R750-spec-sheet.pdf
The analysis in this document was done by Prowess Consulting and commissioned by Dell Technologies.
Results have been simulated and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Prowess and the Prowess logo are trademarks of Prowess Consulting, LLC.
Copyright © 2023 Prowess Consulting, LLC. All rights reserved.
Other trademarks are the property of their respective owners.
Author: Prowess Consulting, LLC
[1] Note that BenchCraft® does not conform to the TPC-C® testing standards, and results between BenchCraft and TPC-C are not comparable.
[2] Broadcom internal data, provided by Dell Technologies.
Related Documents
Improve Storage Performance for Data-Intensive Workloads on Dell PowerEdge R760 Servers with Dell PERC 12
Wed, 15 May 2024 14:21:58 -0000
|Read Time: 0 minutes
Improve Storage Performance by More than 10x for Data-Intensive Workloads on Dell™ PowerEdge™ R760 Servers with Dell™ PowerEdge RAID Controller 12 (PERC 12)
Prowess Consulting testing demonstrated that next-generation PowerEdge servers with the latest NVM Express® (NVMe®) RAID controllers can accelerate business workloads.
Executive Summary
If your business strategy depends on running analytics, AI, or real-time transaction processing at speed, you depend on high performance from your storage. At the same time, it’s essential to maintain availability for your storage.
RAID can be a powerful tool for ensuring the availability of local storage by combining multiple physical disk drives into a single virtual disk for the purpose of data redundancy, performance improvement, or a combination of both. However, for many years high-performance storage based on the NVM Express® (NVMe®) protocol lacked native RAID controller support and thus could not directly take advantage of RAID’s many benefits. This situation changed with the introduction of newer RAID controllers, such as Dell™ PowerEdge RAID Controller 11 (PERC 11) and PERC 12, which brought native NVMe RAID support, unlocking the ability to directly harness RAID’s benefits for high-performance NVMe storage.
To help organizations evaluate the performance benefits of running data-intensive workloads on servers with NVMe RAID, and to assess the benefits of upgrading to Dell PERC 12 from PERC 11, Prowess Consulting tested a Dell™ PowerEdge™ R760 server with both PERC 12 and PERC 11. Our testing found that using PERC 12 instead of PERC 11 allowed us to improve storage performance more than 10x and increase storage throughput almost 5x. Additionally, with PERC 12, RAID 5 rebuilds were up to 110% faster, RAID 10 rebuilds were up to 56% faster, and Microsoft® SQL Server® database restores were up to 90% faster.
As part of our analysis, we evaluated other features of the PowerEdge R760 server that can also contribute to higher performance and that should be considered as part of an upgrade decision. Finally, taking a holistic view of Dell™ servers, PERC, tools, and services, we determined that—for organizations looking to use next-generation Dell servers—the Dell Technologies portfolio can increase server utilization and simplify management, which can reduce both complexity and total cost of ownership (TCO).
Unleash the Performance of NVMe RAID
RAID enhances the performance, reliability, and data availability of storage systems by combining multiple hard drives into an array. This technology can improve performance by increasing the number of drives used for saving and accessing data. RAID disk subsystems can help improve input/output (I/O) performance and data availability, and they can enhance throughput, as several disks can be accessed simultaneously. RAID systems can also be used to improve storage availability and fault tolerance.
A RAID controller, also known as a RAID card or RAID adapter, is a hardware device or integrated component that manages the RAID array. Although past RAID controllers were not NVMe compatible with the high-speed PCIe® interfaces used by NVMe drives, recent RAID controllers from Dell Technologies (for example) are compatible in that respect.
Quantifying the Benefits of a PERC Upgrade
To help organizations evaluate the performance benefits of running data-intensive workloads on servers with NVMe RAID, we tested a PowerEdge R760 server with PERC 11 and the same server with PERC 12. See the Appendix for server configurations.
Measuring Disk Performance
Maintaining disk throughput is essential for modern businesses. When disk throughput is constrained, storage can become a bottleneck that can impact the speed of business operations and ultimately reduce profitability.
Prowess Consulting engineers tested the disk throughput of the PowerEdge R760 server platform. For this testing, we used fio, RocksDB, and Dbench (part of the Phoronix Test Suite) with one PowerEdge R760 server using PERC 12 (Broadcom®/LSI PERC 12 H965i Front) and the other one using PERC 11 (Broadcom/LSI PERC 11 H755N Front). Across benchmarks, PERC 12 in the PowerEdge R760 server platform outperformed the same platform using PERC 11. These results can serve as a proxy indicator of how PERC 12 can increase storage system performance in other data-intensive workloads including analytics and AI.
Fio
Fio is a versatile I/O testing tool that benchmarks the performance of storage devices such as the solid-state drives (SSDs) in the PowerEdge R760 server platform at a lower level, independent of any specific application like a database. It tests the storage subsystem’s performance by generating various types of I/O workloads, allowing for a detailed analysis of read/write speeds, I/O operations per second (IOPS), and latency under different conditions. Figure 1 and Figure 2 show that PERC 12 delivered more than 10x disk performance for the PowerEdge R760 server platform with both RAID 5 and RAID 10 storage configurations, compared to PERC 11.[i]
RocksDB
The RocksDB benchmark is specifically designed to measure metrics like throughput, latency, and IOPS within the context of database operations. The benchmarks focus on database-specific operations such as read, write, and compaction processes, reflecting how the database performs under various workloads and configurations.
Dbench
The Dbench benchmark is designed to simulate disk and file system load. It generates a series of file I/O operations, mimicking the behavior of varying numbers of simulated clients accessing a server’s file system concurrently (1, 6, 12, 48, 128, and 256 clients across Prowess Consulting’s test runs). The PowerEdge R760 server with the newer PERC 12 achieved more than 4.7 times the disk throughput with RAID 5 and more than 1.5 times the disk throughput with RAID 10, compared to the PowerEdge R760 server with PERC 11 for both RAID configurations (see Figure 6 and Figure 7), as measured with Dbench for one client.
Additional RAID Benefits
Using RAID with NVMe drives can also provide other benefits. For example, with SQL Server, the transaction log acts as a buffer for the main database. That means administrators can optimize servers for performance by configuring the front RAID controllers to use RAID 10, which is the highest-performing option. For the back-end database, which is not as performance-sensitive, admins can choose to use RAID 5 (or any preferred level) to sacrifice some performance for increased capacity.
Rebuild Times Drop
When platforms rely on RAID arrays, rebuild times can come into play after a drive failure or replacement; while a RAID array is being rebuilt, there’s a risk of permanent data loss if another drive fails before full redundancy has been re-established. As a result, rebuild times can be critically important to organizations. To determine whether the newer RAID array offered an advantage in this area, Prowess Consulting compared the rebuild times of PERC 12 to those of PERC 11. To perform this testing, engineers removed one of the drives from the data array and then recorded the time required to rebuild the system and resume SQL Server production use. Figure 8 shows the median of the two runs for each array. For RAID 5, the PERC 12 rebuild was 110% faster than PERC 11. For RAID 10, the PERC 12 rebuild was 56% faster than PERC 11.
Database Restore Times Drop
Like rebuild times, database restore times are important. If a database becomes corrupted, it can be restored from a RAID array, and the quicker, the better. Prowess Consulting compared the Microsoft SQL Server database restore times of PERC 12 to PERC 11. Figure 7 shows that the database restore time for PERC 12 was 90% faster than the restore time for PERC 11.
The Upgrade Decision
Organizations that build their business strategies around analytics, AI, or real-time transaction processing need the right equipment in their data centers. Running workloads on servers with NVMe RAID is one approach to increasing performance, but organizations should also make sure they have the right servers and management software in place. For organizations looking to upgrade to meet business requirements, we researched the qualitative benefits of upgrading. We broke our evaluation down into three areas:
- PowerEdge R760 servers
- PERC 12
- Dell Technologies tools and services
- KIOXIA® CM6 Series Enterprise NVMe SSDs
PowerEdge R760 Servers
For fast and accurate performance, a PowerEdge R760 server makes sense as an upgrade over current systems, especially for workloads like AI and analytics. The PowerEdge R760 server is a 2U server that ships with up to two 4th Gen Intel® Xeon® Scalable processors and with up to 56 cores per socket. It keeps in-memory workloads humming at up to 4,800 megatransfers per second (MT/s) with DDR5 memory. Its Dell Smart Flow chassis helps to air-cool the components. Compared to the Dell PowerEdge R750 server, the PowerEdge R760 server delivers:
- Up to 2.9x greater AI inferencing on 4th Gen Intel Xeon Scalable processors with Intel® Deep Learning Boost (Intel® DL Boost) and Intel® Advanced Matrix Extensions (Intel® AMX)[ii]
- Up to a 20% increase in the number of virtual desktop infrastructure (VDI) users supported on one server[iii]
- Up to 50% more SAP® Sales and Distribution users supported on one server[iv]
Dell PERC 12
Our testing results show that PERC 12 can run transactions up to 190% faster than PERC 11 on a PowerEdge R760 server. But there are other benefits that organizations can realize by upgrading to the latest-generation PERC 12 from PERC 11, or even from the prior-generation PERC 10.
PERC 12 provides a RAID solution that’s powerful and easy to manage. It’s based on the Broadcom SAS4116W processor, a dual-core ARM® A15 1.6 GHz RAID-on-Chip (ROC) designed to offer high I/O performance for data-intensive applications. PERC 12 supports 24 Gbps Serial-Attached SCSI (SAS) drives, increased cache memory speeds of up to 3,200 MHz, and a 16-lane host bus type.
The PERC 12 controller is a tri-mode ROC device. Tri-mode means that a PERC 12 controller can support either NVMe or SAS/Serial ATA (SATA) interfaces—though not both—from a single controller. This feature eliminates the need to use different controllers for SAS- and NVMe-supported servers. The controller has a slimline connector (or a NearStack® connector) for both PCIe and SAS/NVMe interfaces.
Table 1 shows a full list of features in the PERC 12 H965i Front.
Table 1. Dell™ PERC 12 H965i Front features
Dell™ PERC 12 H965i Front | ||
Feature | Description | |
Device interface | 16 PHY SAS/SATA/PCIe® interfaces | |
Form factor | PCIe® adapter card (HHHL) and custom Front and MX form factors | |
Connectors | Two Amphenol® SlimSAS® x8 (SFF-8654 Series) connectors | |
Device support | 12 NVM Express® (NVMe®) devices (x2 connections) 24 SAS/SATA devices (limited by the number of controller back-end PHY slots)[v] | |
Host bus type | 16-lane, PCIe® Gen 4 compliant | |
Data transfer rates | Up to 24 Gbps per PHY/port (SAS)[vi] | |
SAS controller | Dual-core ARM® A15 SAS4116W processor ROC | |
Cache memory | 2 MB shared L2 cache, 15 MB on-chip memory | |
Key RAID and data-protection features | RAID levels 0, 1, 5, and 6, RAID spans 10, 50, and 60, online capacity expansion (OCE),[vii] consistency check for background data integrity, physical disk power management (Dimmer Switch™), patrol read for media scanning and repair, 4K native sector support, self-encrypting drive (SED) support, and auto-resume after power loss during array rebuild | Load balancing, fast initialization for quick array setup, configurable stripe size, NVRAM “wipe” feature protects proprietary data once a card is decommissioned, up to 240 virtual drives,5 DDF-compliant configuration on disk (COD), global and dedicated hot spare with revertible hot-spare, automatic rebuild, enclosure affinity, TRIM/UNMAP support for SSDs in pass-through mode |
RAID management | PERC 12 card management applications include the Comprehensive Embedded Management (CEM), Dell™ OpenManage™ storage management, the Human Interface Infrastructure (HII) configuration utility, and the PERC command-line interface (CLI). | |
Operating temperature | Maximum ambient temperature: 60°C | |
Operating voltage | +3.3 V, +12 V, and +3.3 V aux | |
Optional SSD optimization | Hardware-accelerated I/O feature delivers high IOPS performance on SSD arrays; see User’s Guide for details: www.dell.com/support/manuals/en-us/perc-h965i-front/perc12/dell-technologies-poweredge-raid-controller-12. | |
Operating systems | Windows Server® with Hyper-V®, Red Hat® Enterprise Linux®, Ubuntu® Server, SUSE® Linux Enterprise Server (SLES), and VMware ESXi™ For specifications and interoperability details, see www.dell.com/ossupport. |
Broadcom® SAS4116W
The Broadcom SAS4116W high-performance I/O processor contributes to the higher performance of the Dell™ PERC 12.
The SAS4116W host interface supports 16-wide PCIe® lanes and complies with the PCIe 4.0 specification, offering up to 6.7 million random read IOPS and 1.1 million random write IOPS in RAID configurations.[viii]
This processor is part of the eighth generation of SAS ROCs from Broadcom. It’s based on the industry-leading Fusion-MPT™ architecture, and it features Tri-Mode SerDes technology.
Dell Technologies Tools and Services
Dell Technologies offers a wide range of software tools and services in its Dell™ OpenManage™ systems management portfolio. These tools and services help simplify management and support of PowerEdge servers, including the Dell PowerEdge R760 server. With the help of these services, staff can spend more time on value-add tasks and less time fixing problems. The result is reduced complexity and lower TCO. Some of these services include:
- OpenManage Enterprise console. The Dell OpenManage Enterprise console provides a comprehensive view of Dell Technologies servers and other devices on enterprise networks. It also supports plugins to automate the installation of firmware and drivers for streamlined updates.
- OpenManage Server Administrator (OMSA). OMSA provides tools for managing and monitoring RAID arrays. This includes the ability to create, modify, or delete RAID configurations, in addition to monitoring the status of RAID arrays and perform rebuilds.
- Integrated Dell™ Remote Access Controller (iDRAC). The OpenManage portfolio iDRAC, a remote server-management processor embedded within every PowerEdge server, can help IT administrators deploy, update, and monitor PowerEdge servers locally and remotely. This technology features a telemetry stream that IT admins can use to access sensor data from within the server. This data communicates parameters such as compute usage, aggregate temperature, and power consumption, which IT admins can use to proactively maintain servers and identify issues before they cause downtime.
With iDRAC9, it’s easy to manage the PERC 12 H965 card tested in our research. Without having to deploy an agent, IT admins can configure, deploy, update, and monitor the controller, either via the graphical user interface (GUI), with Redfish®, or through Dell’s CLI, known as RACADM. With iDRAC9, customers can also perform real-time storage operations.
iDRAC9 also supports Secured Component Verification (SCV), a supply-chain assurance offering. SCV enables customers to verify that the PowerEdge servers they receive from the factory match what was manufactured, and that nothing changed along the way.
Customers can get more out of their servers by taking full advantage of Dell Technologies’ tools and services. They can also simplify management and support, which can help reduce TCO.
KIOXIA® CM6 Series Enterprise NVMe SSDs
Dell PowerEdge servers can significantly enhance their Sanitize Instant Erase (SIE) and compliance with Federal Information Processing Standards (FIPS) by using KIOXIA CM6 Series Enterprise NVMe SSDs. The SIE and FIPS security features help ensure data protection and meet stringent security requirements.
Achieving the Performance Edge
PowerEdge R760 servers with NVMe RAID—either PERC 11 or PERC 12—can help meet the performance requirements for workloads such as analytics, AI, and real-time processing. Our testing demonstrated that running transactions on the PowerEdge R760 server with PERC 12 instead of PERC 11 can result in an increase in storage performance of more than 10.8x. PERC 12 also reduced RAID rebuild times and SQL Server database restore times. This data suggests that an upgrade to PERC 12, built on Broadcom SAS4116W silicon, can help organizations achieve a performance edge and higher availability.
Our research also showed us that the PowerEdge R760 server should be considered in upgrade discussions because of its generational performance improvements over the PowerEdge R750 server. Additionally, Dell Technologies’ tools and services offer a large portfolio of solutions that can reduce complexity and lower TCO.
Learn more about Dell PERC 12: www.dell.com/support/manuals/en-us/perc-h965i-mx/perc12/dell-technologies-poweredge-raid-controller-12
Learn more about PowerEdge R760 servers: www.dell.com/en-us/shop/servers-storage-and-networking/new-poweredge-r760-rack-server/spd/poweredge-r760/pe_r760_15724_vi_vp
Appendix
Our test configurations are shown in Table A1.
Table A1. System configurations used for testing Dell™ PERC 12 versus PERC 11 on the Dell™ PowerEdge™ R760 server
Configuration | Single-Server Dell™ PERC 12 | Single-Server Dell™ PERC 11 |
Server | 1 x Dell™ PowerEdge™ R760 | |
Processor | Intel® Xeon® Platinum 8460Y+ processor | |
Number of CPUs | 2 | |
Cores/threads per CPU | 32/64 | |
Cores/threads total | 64/128 | |
Frequency (base/SCT/MCT) | 2.2 GHz (max 4 GHz) | |
Storage controller 01 | Marvell Technology Group Ltd. Dell™ Boot-Optimized Server Storage (BOSS)-N1 Monolithic | |
Disk | 960 GB NVM Express® (NVMe®) Dell SK hynix® PE8010 RI M.2 | |
Number of disks | 2 | |
Storage controller 02 | Broadcom®/LSI Dell™ PERC H965i Front | Broadcom®/LSI Dell™ PERC H755N Front |
Disk | 1.6 TB NVMe® Dell Enterprise KIOXIA® Corporation CM6 MU (firmware: 2.2.0) | |
Number of disks | 8 | |
Storage controller 03 | Broadcom®/LSI Dell™ PERC H965i Front | Broadcom®/LSI Dell™ PERC H755N Front |
Disk | 1.6 TB NVMe® Dell Enterprise KIOXIA® Corporation CM6 MU (firmware: 2.2.0) | |
Number of disks | 8 | |
Installed memory | 256 GB | |
Memory DIMM | SK hynix® HMCG78MEBRA174N Rank: Single-rank Memory type: error-correction code (ECC) DDR5 | |
Memory speed | 4,800 megatransfers per second (MT/s) | |
Number of memory DIMMs | 16 | |
Network | 10/25 gigabit Ethernet (GbE) remote direct memory access (RDMA) Broadcom® NetXtreme® E-Series BCM57508 quad-port Ethernet controller 2 x 100 GbE PCIe® Broadcom® NetXtreme® E-Series P2100D BCM57508 QSFP | |
Operating system (OS) | Red Hat® Enterprise Linux® 9.2 (Plow) | |
OS version | 9.2 | |
OS kernel | Linux® 5.14.0-284.30.1.el9_2.x86_64 | |
Microsoft® SQL Server® version | Microsoft® SQL Server® 2022 (RTM-CU10) (KB5031778)—16.0.4095.4 (x64) | |
BIOS version | 1.5.6 | 1.5.6 |
The analysis in this document was done by Prowess Consulting and commissioned by Dell Technologies.
Results have been simulated and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance.
Prowess and the Prowess logo are trademarks of Prowess Consulting, LLC.
Copyright © 2024 Prowess Consulting, LLC. All rights reserved.
Other trademarks are the property of their respective owners.
Author: Prowess Consulting, LLC
[i] These test results obtained by Prowess Consulting using the fio benchmark confirm and extend similar testing conducted in 2023 by Tolly. For details on those test results, see: Tolly, “Dell Technologies Dell PowerEdge RAID Controller 12 (PERC 12) 16th Generation (16G) Server Performance vs PERC 11 & PERC 10.” Commissioned by Dell Technologies. January 2023. https://infohub.delltechnologies.com/en-US/section-assets/tolly223103delltechnologiespoweredgeraidcontroller12performance.
[ii] Testing commissioned by Dell Technologies in December 2022 and performed by Scalers AI demonstrated greater AI inferencing for object detection using INT8 on 4th Gen Intel® Xeon® Scalable processors for Dell™ PowerEdge™ R760 servers using 4th Gen Intel Xeon Scalable processors with updated Intel® Deep Learning Boost (Intel® DL Boost) instructions. Actual results will vary.
[iii] Based on internal Dell Technologies testing results provided by Dell Technologies, in which a Dell™ PowerEdge™ R760 server with two Intel® Xeon® Gold 6454S processors hosted 220 VDI sessions, compared to a PowerEdge R750 server with two Intel Xeon Gold 6348 processors, which was able to host 183 VDI sessions.
[iv] Based on test results approved by SAP under certification number 2023005, in which the Dell™ PowerEdge™ R760 server with two Intel® Xeon® Platinum 8480 processors with a total of 112 cores, 224 threads, and 2,048 GB DRAM hosted 72,250 SD users using SUSE® Linux® Enterprise Server (SLES) 15, compared to published results under certification number 2021026 for the PowerEdge R750 server with two Intel Xeon Platinum 8380 processors with a total of 80 cores, 160 threads, and 1,024 GB DRAM, which demonstrated support for up to 48,000 SD users using Red Hat® Enterprise Linux 8.2. Source: SAP. SAP Standard Application Benchmarks web page: https://www.sap.com/dmc/exp/2018-benchmark-directory/#/sd. Accessed August 2023.
[v] Silicon supports up to 240 virtual drives.
[vi] SAS4 “22.5 Gbps” speed is used synonymously with “24G” and “24 Gbps” in some documents and applications; 22.5 Gbps is the data rate, whereas 24 Gbps is the link speed.
[vii] OCE is supported from Dell™ PERC 12.1 onward.
[viii] Broadcom. “SAS4116W Tri-Mode RAID-on-Chip Product Brief.” October 2022. https://docs.broadcom.com/doc/SAS4116W-ROC-PB1XX.
NVMe Performance Increases for Next-Generation PowerEdge Servers with PERC11 Controller
Tue, 17 Jan 2023 05:48:58 -0000
|Read Time: 0 minutes
Summary
Dell Technologies newest RAID iteration, PERC11, has undergone significant change - most notably the inclusion of hardware RAID support for NVMe drives. To better understand the benefits that this will bring, various metrics were tested, including NVMe IOPS, disk bandwidth and latency. This DfD compares NVMe performance readings of the next-generation Dell EMC PowerEdge R650 server, powered by pre-production 3rd Generation Intel® Xeon® Scalable processors, to the prior-generation PowerEdge R640 server, powered by 2nd Generation Intel® Xeon® Scalable processors.
Introduction
With support for NVMe hardware RAID now available on the PERC11 H755N front, H755MX and H755 adapter form factors, we were eager to quanitfy how big of a performance boost next-generation PowerEdge servers with hardware RAID would obtain. Dell Technologies commissioned Principled Technologies to execute various studies that would compare the NVMe Input/Output Per Second (IOPS), disk bandwidth and latency readings of next-geneation PowerEdge servers (15G) with NVMe hardware RAID support against prior-generation PowerEdge servers (14G) without NVMe hardware RAID support.
Test Setup
Two servers were used for this study. The first was a PowerEdge R650 server populated with two 3rd Gen Intel® Xeon® Scalable processors, 1024GB of memory, 3.2TB of NVMe storage and a Dell PERC H755N storage controller. The second was a PowerEdge R640 server populated with two 2nd Gen Intel® Xeon® Gold Scalable processors, 128GB of memory, 1.9TB of SSD storage and a Dell PERC H730P Mini storage controller.
A tool called Flexible Input/Output (FIO) tester was used to create the I/O workloads used in testing. FIO invokes the production of threads or processes to do an I/O action as specified by the user. This test was chosen specifically because it injects the smallest system overhead of all the I/O benchmark tools we use. This in turn allows it to deliver enough data to the storage subsystem to reach 100% utilization. With the tool, five workloads were run at varied thread counts and queue depths on RAID 10, RAID 6, and RAID 5 levels of the Dell EMC PowerEdge R650 server with PERC H755n RAID controller and NVMe drives and the Dell EMC PowerEdge R640 server with a PERC H730P Mini controller and SATA SSD drives.
Read-heavy workloads indicate how quickly the servers can retrieve information from their disks, while write-heavy workloads indicate how quickly the servers can commit or save data to the disk. Additionally, random and sequential in the workload descriptions refer to the access patterns for reading or writing data. Random accesses require the server to pull data from multiple disks in a non-sequential fashion (i.e., visiting multiple websites), while sequential accesses require the server to pull data from a single continuous stream (i.e., streaming a video).
Performance Comparisons
IOPS
IOPS indicates the level of user requests that a server can handle. Based on the IOPS output seen during testing, upgrading from the prior-generation Dell EMC PowerEdge R640 server to the latest-generation Dell EMC PowerEdge R650 server could deliver performance gains for I/O-intensive applications. In all three RAID configurations tested, the PowerEdge R650 with NVMe SSDs delivered significantly more IOPS than the prior-generation server. Figures 1, 2 and 3 show how many average IOPS each configuration handled during testing:
Figure 1: IOPS comparison for RAID 10 configurations
Figure 2: IOPS comparison for RAID 6 configurations
Figure 3: IOPS comparison for RAID 5 configurations
Disk Bandwidth
Disk bandwidth indicates the volume of data a system can read or write. A server with high disk bandwidth can process more data for large data requests, such as streaming video or big data applications. At all three RAID levels, the latest-generation Dell EMC PowerEdge R650 server with NVMe storage transferred significantly more MB per second than the prior-generation server. Figure 4 shows the disk bandwidth that each of the two servers supported for each RAID level:
Figure 4: Disk bandwidth comparison for RAID 10, 6 and 5 configurations
Latency
Latency indicates how quickly the system can respond to a request for an I/O operation. Longer latency can impact application responsiveness and could contribute to a negative user experience. In addition to greater disk bandwidth, the Dell EMC PowerEdge R650 server delivered lower latency at each of the three RAID levels than the prior-generation server. Figure 5 shows the latency that each server delivered while running one workload at each RAID level.
Figure 5: Latency comparison for RAID 10, 6 and 5 configurations
Conclusion
The next-generation PowerEdge R650 server with NVMe HW RAID support increased IOPS by up to 15.7x, disk bandwidth by up to 15.5x, and decreased latency by up to 93%. With the inclusion of NVMe HW RAID support on Dell Technologies’ new PERC11 controllers, now is a great time for PowerEdge customers to migrate their storage medium over to NVMe drives and yield the higher-performance that comes with it!
For more details, please read the full PT report Accelerate I/O with NVMe drives on the New PowerEdge R650 server