
U.2 – Still the Industry Standard in 2.5” NVMe SSDs
Download PDFMon, 16 Jan 2023 13:44:22 -0000
|Read Time: 0 minutes
Summary
This DfD is an informative technical paper meant to educate readers about the initial intentions around the U.3 interface standard, how it proceeded to fall short upon development, and why server users may want to continue using U.2 SSDs for their server storage needs.
Introduction
In our world of technology, we expect to see multiple generations of devices and standards, with each successive generation being faster and more feature-rich than the previous. We have seen this pattern so often that we expect version N+1 of anything to be better than version N in nearly all respects.
So, what about the new U.3 interface standard compared to U.2? Surprisingly, U.3 is not the next generation since it was not intended to replace U.2. It was originally conceived as a low-cost NVMe replacement for the SATA SSD. A lot has changed since the inception of U.3 and eventually the standard required U.3 SSDs to be backwards compatible to existing x4 U.2 SSDs. This requirement forced SSD vendors to either develop flash controller silicon with 6 PCIe lanes or to add mux chips to steer the existing PCIe lanes on the SSD. By doing so, U.3 SSDs have the following disadvantages:
- U.3 SSDs do not hit the same cost points as SATA SSDs
- U.3 SSDs do not have any cost advantage over U.2 SSDs (and may end up being more expensive)
- U.3 SSDs lost their ability to differentiate themselves from U.2 SSDs
U.3 has been touted as a way to enable a tri-mode backplane that will support SAS, SATA and NVMe drives to work across multiple use-cases. The tri-mode backplane claim was to reduce system costs, while providing an upgrade path so that users can later replace their existing SAS and SATA drives with higher performance NVMe SSDs. While a tri-mode backplane can technically support SAS, SATA, and NVMe drives, mixing SAS and SATA virtual disks behind a single controller is rarely done. Adding NVMe to the mix makes even less sense because NVMe SSDs are much higher performing than SAS or SATA drives.
Even an upgrade path from SAS or SATA drives to all NVMe SSDs is severely limited by the tri-mode controller. A high-performance controller has 16 lanes that can support, for example, 16 x1 devices. Replacing 16 x1 SATA SSDs with 16 x1 NVMe SSDs as originally envisioned by U.3 would make sense. However, because U.3 matched U.2 and with support for up to a x4 link, customers will likely not want to give up the higher performance the additional lanes provide. A 16-lane tri-mode controller could support only 4 x4 U.3 SSDs – not very many for such an expensive controller.
A SAS expander would normally be used to increase the number of SAS devices, but it cannot support PCIe as there are no tri-mode expanders. Additionally, a PCIe switch would normally be used to increase the number of NVMe devices, but it cannot support SAS or SATA devices. The result is that the system either suffers poor
U.3 performance or must incur the cost of additional tri-mode controllers. Thus, because U.3 combines the SAS and SATA lanes with the NVMe lanes, it is much more difficult and expensive to scale out the tri-mode solution to achieve high performance. The argument that U.3 allows system designers to develop a common set of backplanes that work across multiple use cases does not hold, as the difference in link widths and the inability to scale will push users to continue adopting solutions tailored to their specific needs.
U.2 keeps the SAS and SATA lanes separate from the NVME lanes, allowing system designers to scale solutions independently with readily available SAS expanders and PCIe switches. Dell Technologies recognizes a wide range of customer requirements and provides solutions that are tailored to each market as opposed to a one-size-fits-all solution. To that end, Dell Technologies has developed high-performance, universal x4 drive bays that have been shipping on Dell PowerEdge servers for the last two generations. Dell Technologies also provides SAS and SATA-only solutions to reduce cost in entry-level systems. Next generation backplanes enable NVMe HWRAID which connects up to 8 NVMe SSDs at PCIe Gen4 x2. Direct connect solutions remain at PCIe Gen4 x4.
Conclusion
U.3 enables a tri-mode backplane which allows simple upgrades from SAS or SATA to NVMe, yet increases the base cost of a SAS or SATA solution. Moreover, unless the system is designed with sufficient lanes for the NVMe SSDs, the performance will be poor. The additional hardware required to obtain full NVMe performance negates any system cost benefits of the U.3 architecture.
Dell Technologies has demonstrated that the wide range of customer requirements can be met with SAS, SATA and U.2 drives, using designs targeted individually for performance or cost. Dell’s universal U.2 backplane takes advantage of the separation of SAS and SATA lanes from NVMe lanes to maximize NVMe performance, while maintaining SAS and SATA compatibility in a universal bay. This high-performance, universal U.2 backplane avoids the confusion and complexity brought by U.3. However, it is important to remember that the key is the backplane architecture, not the drive type. Dell’s backplane will work with U.3 SSDs, as well as U.2 SSDs, since U.3 SSDs are required to be compatible. Dell has also designed next generation backplanes to enable NVMe HWRAID which connects up to 8 NVMe SSDs at PCIe Gen4 x2. All of Dell’s direct-connect solutions remain at PCIe Gen4 x4.
Related Documents

New PCIe Gen4 Data Center NVMe Drives Offer Unmatched Value for PowerEdge Servers
Mon, 16 Jan 2023 13:44:29 -0000
|Read Time: 0 minutes
Summary
PCIe Gen4 Data Center NVMe drives challenge the existence of traditional SAS and SATA drives. This entry-level NVMe offering outperforms both SAS and SATA while retaining an affordable price that directly competes with SATA.
The purpose of this DfD is to educate readers on our new Gen4 Data Center NVMe offerings, including a brief history on the technology, a performance data comparison to SAS and SATA, and their value proposition in the market today. With this knowledge we hope our customers can make the best storage investment decision to optimize their PowerEdge servers.
NVMe Market Positioning
The NVMe host controller interface has been rapidly evolving since its inception less than a decade ago. By including high-performance, scalable technologies absent from both SAS and SATA interfaces, such as non-volatile (persistent) memory and the high-speed PCIe bus, NVMe was originally designed only as a premium storage offering and was priced to reflect that for several years. However, this novelty technology has become conventional. The shift inclined suppliers to optimize their processes to create more competitive price points. Now that the Gen4 Data Center NVMe drive has been introduced with both the high-performance and a more affordable price – is it time to start transitioning completely to NVMe?
Figure 1 – Dell U.2 PCIe Gen4 NVMe SSD
Enterprise vs. Data Center NVMe
There are two classes of NVMe drives used in servers – Enterprise NVMe and Data Center NVMe SSD. Enterprise NVMe is the premium drive made for enterprise environments that run 24/7. This won’t be discussed, as its performance edge and enterprise features drive the price too high to compete with SAS and SATA at this time.
Instead, we will be focusing on the Data Center NVMe SSD. This vSAN-certified NVMe drive is tailored for scale-out/hyperscale environments where enterprise features, such as dual port and FIPS support, are not needed for the Data Center customer. This more targeted feature set, coupled with a lower-cost eight-channel controller, enables a more attractive price comparable to SATA. This high performance (see Figure 2) and lower price-point creates a clear and distinctive value proposition for this class of NVMe drive.
Interface | Capacity | Class | Seq. Read GB/s | Seq. Write GB/s | Random Reads - (4K) IOPs | Random Writes (4K)- IOPs |
RI NVMe | 1.92TB | Enterprise | 6.2 | 2.3 | 920 | 110 |
RI NVMe | 1.92TB | Data Center | 5.9 | 2.5 | 870 | 120 |
RI NVMe | 3.84TB | Enterprise | 6.2 | 3.45 | 1360 | 130 |
RI NVMe | 3.84TB | Data Center | 5.9 | 3 | 1050 | 150 |
RI NVMe | 7.68TB | Enterprise | 6.2 | 3.45 | 1360 | 130 |
RI NVMe | 7.68TB | Data Center | 5.9 | 3.5 | 1050 | 140 |
Figure 2 – Performance comparison of PCIe Gen4 Enterprise and Data Center NVMe SSDs
When comparing identical capacities of enterprise and data center NVMe drives, we can see that most performance readouts are very similar to one another, at around ±10%. Outliers do seem to exist, but they favor both sides. This indicates that the DC NVMe SSD does not sacrifice any significant amount of performance for its ‘entry-level’ price tag, but primarily its enterprise features. Thus, Gen4 DC NVMe is an excellent NVMe option for users who do not require the enterprise features.
Comparing Performance and Pricing
Here is where it gets really interesting. Pricing for identical NVMe drives remain relatively constant with each new generation. In this case, both PCIe Gen3 and PCIe Gen4 DC NVMe are priced nearly one-to- one, despite seeing significant performance gains with support for PCIe Gen4. This begs for some due diligence – with higher performance at cost parity, will Gen4 DC NVMe now expunge any remaining value proposition that may justify still using SAS or SATA in your PowerEdge servers?
To answer this question, we must first scrutinize the performance data. Figure 3 below shows how the performance readouts stack up and helps us understand the variances (highlighted in orange).
Swimlane |
Interface |
Capacity |
PCIe |
Model | Endurance (DWPD) | Seq. Reads (GB/s) |
| Seq. Writes (GB/s) |
| Random Reads - 4K (IOPs) |
| Random Writes - 4K (IOPs) |
|
Read Intensive | NVMe | 960GB | Gen4 x4 | Data Center Agnostic | 1 | 5.90GB/s |
| 1.40GB/s |
| 550K |
| 50K |
|
Read Intensive | SAS | 960GB | N/A | Dell Brand Agnostic | 1 | 1.02GB/s | 5.8x | 0.84GB/s | 1.7x | 184K | 3.0x | 34K | 1.5x |
Read Intensive | SATA | 960GB | N/A | Dell Brand Agnostic | 1 | 0.48GB/s | 12.3x | 0.44GB/s | 3.2x | 79K | 7.0x | 23K | 2.2x |
Read Intensive | NVMe | 1920GB | Gen4 x4 | Data Center Agnostic | 1 | 5.90GB/s |
| 2.50GB/s |
| 870K |
| 120K |
|
Read Intensive | SAS | 1920GB | N/A | Dell Brand Agnostic | 1 | 1.02GB/s | 5.8x | 0.95GB/s | 2.6x | 186K | 4.7x | 56K | 2.1x |
Read Intensive | SATA | 1920GB | N/A | Dell Brand Agnostic | 1 | 0.47GB/s | 12.6x | 0.44GB/s | 5.7x | 78K | 11.2x | 26K | 4.6x |
Read Intensive | NVMe | 3840GB | Gen4 x4 | Data Center Agnostic | 1 | 5.90GB/s |
| 3.00GB/s |
| 1050K |
| 150K |
|
Read Intensive | SAS | 3840GB | N/A | Dell Brand Agnostic | 1 | 1.02GB/s | 5.8x | 0.96GB/s | 3.1x | 189K | 5.6x | 57K | 2.6x |
Read Intensive | SATA | 3840GB | N/A | Dell Brand Agnostic | 1 | 0.48GB/s | 12.3x | 0.44GB/s | 6.8x | 79K | 13.3x | 25K | 6.0x |
Read Intensive | NVMe | 7680GB | Gen4 x4 | Data Center Agnostic | 1 | 5.90GB/s |
| 3.50GB/s |
| 1050K |
| 140K |
|
Read Intensive | SAS | 7680GB | N/A | Dell Brand Agnostic | 1 | 1.01GB/s | 5.8x | 0.96GB/s | 3.6x | 188K | 5.6x | 47K | 3.0x |
Figure 3 – Table comparing read and write performance for three storage mediums (Gen4 DC NVMe, SAS, and SATA)
Gen4 DC NVMe outperforms its competitors by a longshot for every metric, with a performance increase multiplier ranging from:
- 1.7x - 5.8x when compared to SAS
- 2.2x - 13.3x when compared to SATA
The sequential and random read numbers for Gen4 DC NVMe are excellent as expected, and the write IOPs have significantly improved and are even approaching Enterprise Mixed Use (MU) NVMe IOPs. Additionally, the endurance has also doubled from 0.5 to 1 DWPD (Drive Writes Per Day).
Pricing comparisons are a bit more complex to discuss accurately because they are always shifting and very sensitive. So, for this exercise, we will determine relative pricing percentages for vendor-agnostic models from our Q4 2021 price list. If Gen4 DC NVMe drives are the baseline (1.0x), then for the same capacity you will see an average price multiplier of:
- 1.43x for SAS
- 0.77x for SATA
Based on this performance and pricing we can conclude that there is no benefit in choosing SAS over Gen4 DC NVMe, because it is both more expensive and has much lower performance.
All that is left to determine is the value proposition of SATA. As mentioned earlier, SATA drives are still the most affordable form of storage medium on the market, at around 0.77x the price of Gen4 DC NVMe. However, there is still a bigger picture to craft when the data is analyzed at a deeper level. By calculating the performance-per-dollar (IOPs/$) for each capacity of Gen4 DC NVMe and SATA, we can get a better grasp on how effective each dollar spent really is. See Figure 4 below:
When the data is analyzed at this level, it becomes very apparent that each dollar spent on Gen4 DC NVMe goes way farther than it does with SATA. In the case of random read performance, each dollar spent on SATA will produce 0.1x - 0.2x as many IOPs as Gen4 DC NVMe would. In layman’s terms, this means that every dollar spent on SATA is nearly an order of magnitude less effective than Gen4 DC NVMe!
Final Thoughts
Now that the data has been presented and analyzed we can circle back to the original question - is it time to start transitioning completely to NVMe? Based on the high performance and very attractive price point, we believe that the Data Center NVMe drive has a clear and compelling value proposition for PowerEdge customers looking to invest in a scalable, forward-looking storage medium. However, customers that prioritize the lowest price possible will also find SATA to be a valuable solution.
- PCIe Gen4 Data Center NVMe is strongly recommended for most customer user cases. Budget-conscious customers should also consider that greater performance can be achieved at a lower price than SATA simply by scaling down the total number of Gen4 DC NVMe drives. With a performance output of up to 13.3x more than SATA while only being around 1.3x more expensive, customers can scale down the quantity of NVMe drives for the most cost-effective solution.
- SATA is recommended to customers where the lowest price is most important. As long as SATA holds its extremely low price, there will always be a target audience for this storage type. Customers that neither intend to scale, nor require high-performance, may certainly find this to be the best solution for their needs.
In conclusion, while SATA technology still brings value to the market through its extremely low price point, PCIe Gen4 Data Center NVMe technology has demonstrated that it is certainly the most cost-effective storage solution from a price-per-performance metric. Customers also have more flexibility with the option to increase performance and lower price by scaling down the total number of NVMe drives! Regardless, PCIe Gen4 Data Center NVMe technology has proven that for the time being it offers unmatched value to PowerEdge servers.

NVMe Performance Increases for Next-Generation PowerEdge Servers with PERC11 Controller
Tue, 17 Jan 2023 05:48:58 -0000
|Read Time: 0 minutes
Summary
Dell Technologies newest RAID iteration, PERC11, has undergone significant change - most notably the inclusion of hardware RAID support for NVMe drives. To better understand the benefits that this will bring, various metrics were tested, including NVMe IOPS, disk bandwidth and latency. This DfD compares NVMe performance readings of the next-generation Dell EMC PowerEdge R650 server, powered by pre-production 3rd Generation Intel® Xeon® Scalable processors, to the prior-generation PowerEdge R640 server, powered by 2nd Generation Intel® Xeon® Scalable processors.
Introduction
With support for NVMe hardware RAID now available on the PERC11 H755N front, H755MX and H755 adapter form factors, we were eager to quanitfy how big of a performance boost next-generation PowerEdge servers with hardware RAID would obtain. Dell Technologies commissioned Principled Technologies to execute various studies that would compare the NVMe Input/Output Per Second (IOPS), disk bandwidth and latency readings of next-geneation PowerEdge servers (15G) with NVMe hardware RAID support against prior-generation PowerEdge servers (14G) without NVMe hardware RAID support.
Test Setup
Two servers were used for this study. The first was a PowerEdge R650 server populated with two 3rd Gen Intel® Xeon® Scalable processors, 1024GB of memory, 3.2TB of NVMe storage and a Dell PERC H755N storage controller. The second was a PowerEdge R640 server populated with two 2nd Gen Intel® Xeon® Gold Scalable processors, 128GB of memory, 1.9TB of SSD storage and a Dell PERC H730P Mini storage controller.
A tool called Flexible Input/Output (FIO) tester was used to create the I/O workloads used in testing. FIO invokes the production of threads or processes to do an I/O action as specified by the user. This test was chosen specifically because it injects the smallest system overhead of all the I/O benchmark tools we use. This in turn allows it to deliver enough data to the storage subsystem to reach 100% utilization. With the tool, five workloads were run at varied thread counts and queue depths on RAID 10, RAID 6, and RAID 5 levels of the Dell EMC PowerEdge R650 server with PERC H755n RAID controller and NVMe drives and the Dell EMC PowerEdge R640 server with a PERC H730P Mini controller and SATA SSD drives.
Read-heavy workloads indicate how quickly the servers can retrieve information from their disks, while write-heavy workloads indicate how quickly the servers can commit or save data to the disk. Additionally, random and sequential in the workload descriptions refer to the access patterns for reading or writing data. Random accesses require the server to pull data from multiple disks in a non-sequential fashion (i.e., visiting multiple websites), while sequential accesses require the server to pull data from a single continuous stream (i.e., streaming a video).
Performance Comparisons
IOPS
IOPS indicates the level of user requests that a server can handle. Based on the IOPS output seen during testing, upgrading from the prior-generation Dell EMC PowerEdge R640 server to the latest-generation Dell EMC PowerEdge R650 server could deliver performance gains for I/O-intensive applications. In all three RAID configurations tested, the PowerEdge R650 with NVMe SSDs delivered significantly more IOPS than the prior-generation server. Figures 1, 2 and 3 show how many average IOPS each configuration handled during testing:
Figure 1: IOPS comparison for RAID 10 configurations
Figure 2: IOPS comparison for RAID 6 configurations
Figure 3: IOPS comparison for RAID 5 configurations
Disk Bandwidth
Disk bandwidth indicates the volume of data a system can read or write. A server with high disk bandwidth can process more data for large data requests, such as streaming video or big data applications. At all three RAID levels, the latest-generation Dell EMC PowerEdge R650 server with NVMe storage transferred significantly more MB per second than the prior-generation server. Figure 4 shows the disk bandwidth that each of the two servers supported for each RAID level:
Figure 4: Disk bandwidth comparison for RAID 10, 6 and 5 configurations
Latency
Latency indicates how quickly the system can respond to a request for an I/O operation. Longer latency can impact application responsiveness and could contribute to a negative user experience. In addition to greater disk bandwidth, the Dell EMC PowerEdge R650 server delivered lower latency at each of the three RAID levels than the prior-generation server. Figure 5 shows the latency that each server delivered while running one workload at each RAID level.
Figure 5: Latency comparison for RAID 10, 6 and 5 configurations
Conclusion
The next-generation PowerEdge R650 server with NVMe HW RAID support increased IOPS by up to 15.7x, disk bandwidth by up to 15.5x, and decreased latency by up to 93%. With the inclusion of NVMe HW RAID support on Dell Technologies’ new PERC11 controllers, now is a great time for PowerEdge customers to migrate their storage medium over to NVMe drives and yield the higher-performance that comes with it!
For more details, please read the full PT report Accelerate I/O with NVMe drives on the New PowerEdge R650 server