Accelerating Intra-Host Data Movement with VMware PVRDMA on a Dell AMD PowerEdge Server
Download PDFMon, 16 Jan 2023 23:18:32 -0000
|Read Time: 0 minutes
Summary
PVRDMA is an innovative new technology to help accelerate intra-host data movement (i.e. data between VMs on a host). The ESXi driver takes advantage of DMA engines in the AMD Epyc 2nd and 3rd Generation of processors to provide significant performance improvements for this use case. Please contact Dell if you need more information.
Introduction
As CPU cores scale, the usage and demand for ever larger data exchanges within a host scale. It scales among kernels, applications, VMs, and I/O devices. At the same time, VM density is on the rise. Hyperconverged Storage appliances are enabling new workloads on server class systems where Data locality is important. Each intra-host exchange can comprise multiple memory buffer copies (or transformations) and are generally implemented with layers of software stacks. While Kernel-to-I/O can leverage I/O- specific hardware memory copy, SW-to-SW usually relies on per- core synchronous software (CPU-only) memory copies
Intra-host SW memory copy and transformation needs are not keeping up with scaling of cores & workloads. It is difficult for complex processors to ensure a single-core attains full processor- to memory-bandwidth and therefore, noticeable core utilization is needed to satisfy synchronous copy needs. Further, I/O intensive workloads can take away compute CPU cycles available. Network and Storage workloads can take compute cycles for Data Movement, Encryption, Decryption, Compression etc.
Hypervisors want a well-defined capability to quiesce, suspend, and resume the architectural state of a per-address-space data mover that is independent of the actual hardware-offload and the various accelerations that it can perform. This allows “live” workload (VM migration) between different servers with different HW-offload implementations of standard data-movement operations
What is PVRDMA
PVRDMA is one such well-architected, virtualizable interface to workload VMs. The interface is now standard in ESXi and is implemented across CPUs, Platforms, and NICs. Acceleration can occur transparent to the workload. PVRDMA emulation uses RDMA or TCP/IP NICs for inter-host VM-to-VM transfers. However, PVRDMA uses SW Memory Copy for intra-host VM-to-VM transfers if no RDMA NIC and this is ideal for a DMA engine.
Using the AMD EPYC 7002 or 7003 Processors with built-in PTDMA DMA Engines and an AMD ESXi PTDMA PVSP1 Driver, Dell AMD PowerEdge Servers can accelerate Intra-host VM-to-VM Data Movement. PTDMA engines can perform memory copies without CPU intervention
Fig.1 shows the results of PVRDMA testing with SW memory copies for a copy size of 8KiB to 256KiB and PTDMA engine memory copies from 256KiB to 64M.
Fig.1 PVRDMA* Socket-to-Socket Memory Copy Throughput Using AMD PTDMA+ Driver
We observe that SW copy performance is good for small copy sizes. However, as copy sizes fall out of the L1 cache(2*32KiB), SW copy performance drops yet continues to outperform PTDMA copy performance before it falls out of the L2 cache (1/2 * 512KiB). At this L2 cache threshold, PTDMA copies become more performant and outperform SW copy performance. We tested it upto 8M and we expect that the performance will vary with different workloads and application cache interaction
PVRDMA in vSphere 7.0 u1 can now use this PVSP1 DMA compliant driver to perform intra-host copy. PVRDMA and the prototype AMD PTDMA (PVSP) driver support a hybrid approach. By default, the threshold is 256 KiB and can be adjusted for various application tunings.
Next Steps
A PVRDMA technology demo is available on demand from Dell using Dell PowerEdge Servers with this technology. Interested customers can contact their Dell account executive to schedule an engagement request with a Dell Customer Solution Center 2
Also, a VMworld presentation titled “Accelerated, Virtualized, and Standardized Intra-host Data Movement” is available on demand here3 that provides more info on this technology and Dell’s vision into the future in this space.
References
- Partner Verified and Supported Program
- https://www.delltechnologies.com/en-us/what-we-do/customer-engagement- programs/customer_solution_centers.htm
- https://www.vmworld.com/en/video-library/search.html#text=%22OCTO2592%22&year=2020
Related Documents
Understanding Confidential Computing with Trusted Execution Environments and Trusted Computing Base models
Tue, 17 Jan 2023 00:35:08 -0000
|Read Time: 0 minutes
Summary
As the value of data increases, it becomes essential to protect data in- use from unauthorized access. Confidential Computing provides various levels of protection options to mitigate different kinds of threat vectors.
Introduction
Data is the new oil. As the value of data increases, it becomes increasingly important to protect data in-use to perform computations. Data in use is often stored in the clear in memory (DRAM) and accessed via unencrypted memory buses. Whether data in use is a machine learning data set or relates to keeping a secret in memory, data in-use can be vulnerable to threats vectors that can snoop on the contents of memory or the access bus. Data- in-use protection is necessary to secure computations that are increasingly operating on large data sets in memory. Additionally, code executing on the data must be trusted, tamper-free with facilities to separate trusted and non-trusted code execution environments with respect to data in-use.
Trusted Execution Environments and Trusted Computing Base models
With per country regulation requirements on data confidentiality increasing, data generators and users need secure TEEs (Trusted Executions Environments) to satisfy data privacy and protection regulations. Hosting and Infrastructure providers must enable trusted execution environment to guarantee data confidentiality of client data. This requires that entities outside the trust boundary should not be able to access the data in-use
To mitigate against increasing threat vectors combined with usage models that range from multi-tenant environments to edge deployments, trust boundaries need to shrink. Data owners and clients should prefer to keep a small TCB (Trusted Computing Base) to minimize attack coordinates and data misuse by untrusted elements. They should look closely at what TCB levels they can trust for their usage model. A TCB level informs the code footprint that can be trusted
While a reduced TCB can be achieved using software techniques, silicon-aided features can greatly aid the creation, separation and protection of TEEs with reduced TCBs. Silicon features are needed to minimize TCB to a Trusted Host Execution Environment, Trusted Virtual Machine Execution environment, and a Trusted Application Execution Environment for new and emerging deployments
Picking an appropriate TCB footprint level
To consider an appropriate TCB footprint level, one should determine if the entity hosting the code and data execution environment can be trusted and has the facility to separate trusted and non- trusted components. For e.g., a data center level TCB can imply a data center administrator is a trusted operator for the data in use. This means the entire data center execution environment is trusted and applications users can employ a data center wide application/workload deployment policy. A Platform/Host level TCB requirement can imply a system administrator is a trusted operator for the data and the code running on the platform and can deploy a trusted Host execution environment for the workloads. A VM level TCB footprint requirement implies a trusted guest machine user for data in use running in a trusted Guest Execution Environment. An App level TCB footprint requirement can imply only the App owner is trusted with data in use access. See Figure 1 for a representation of various TCB footprint levels. If you observe carefully, as TCB footprint shrinks, the application owner has fewer layers of trusted software.
Figure 1 A view of various TCB footprint levels
These levels come with varying degree of usability to application deployments. They have unique advantages and tradeoffs when it comes performance, application mobility, trust granularity and integration with management stacks.
In general, to enable these TEEs, silicon enables memory encryption such that trusted, differentiated and secure memory access is possible for data in use. Data/app owners must be able to independently attest to the integrity of the platform and the TCB levels supported by the underlying infrastructure.
Dell believes in the power of choice when it comes to offering a trusted execution environment with a level of TCB needed to run your applications. Dell’s breadth of technologies including the enhanced cyber resilient architecture that is part of the latest generations of PowerEdge servers enables usages at the edge, core and the cloud.
Conclusion
To maximize protection of data in-use, consideration should be given to the TCB footprint that is appropriate for the use case. Dell EMC PowerEdge servers are loaded with top notch security features to provide maximum protection for your data. In addition, Dell Technologies is pleased to partner with key vendors to support features like SME, SEV-ES, and SGX, etc. with various levels of confidential computing usage models that cater to various Trusted Execution Environments
Save Time, Rack Space, and Money—5:1 Server Consolidation Made Possible with the Latest AMD EPYC Processors
Thu, 20 Apr 2023 17:41:37 -0000
|Read Time: 0 minutes
Summary
The latest Dell PowerEdge servers with AMD EPYC 4th Generation processors, each with up to 96 cores, deliver exceptional value to our customers. The large number of cores coupled with the high-speed DDR5 memory and very high-speed PCIe Gen5 devices makes for servers that can run almost any workload with ease. These servers are especially well suited for virtualization workloads. These unprecedented performance enhancements enabled Dell Technologies to achieve multiple virtualization world records. The cluster-level benchmarks for virtualized workloads are an excellent example of the performance and power-performance world record gains that are achievable.
Running a mixture of architectures in your data center can be cause for some concern—especially if you are looking to upgrade to the latest AMD servers and you are currently running the workloads on legacy Intel® based servers. Even with the greatest level of planning, there is always the fear that some unexpected variable might turn everything upside down during the migration process. Now, there is a new tool for your toolbox to make such migrations easier. The VMware Architecture Migration Tool1 is a PowerShell script that uses VMware PowerCLI to eliminate the guesswork and complexity involved in migrating a virtual machine from one hardware architecture to another.
To fully test the tool, Dell ran a full migration scenario. We were able to consolidate 380 VMs running on five legacy Intel platform servers into one Dell PowerEdge R7625 with AMD EPYC 4th Gen processors. We describe our testing in more detail later in this paper.
Why migrate?
In today’s IT departments, workloads are always evolving. There is increasing pressure to support new workloads while keeping existing workloads to support existing business needs—all while also trying to reduce costs and meet corporate goals.
The latest technology tends to bring multiple advantages, driving the need to upgrade. Some of these advantages are:
- Higher performance
The latest Dell PowerEdge servers with 4th Gen AMD EPYC processors have class-leading performance with up to 121 percent higher scores than prior generations.2
- Better efficiency
The Dell PowerEdge servers with 4th Gen AMD EPYC processors are some of the first to achieve the EPEAT silver rating, indicating the highest level of environmental responsibility and efficiency. Dell has achieved 159 percent higher performance per kilowatt on the VMmark benchmark with the R7625 compared to the prior-generation model server.3
- More security
With Dell’s Cyber Resilient Architecture and AMD’s Infinity Guard, the PowerEdge servers with 4th Gen AMD EPYC processors offer top-class security to ensure that your data and infrastructure are protected.4
- Workload optimizations
The 4th Generation AMD EPYC processors have several optimizations, such as support for AVX-512, INT8, and BFLOAT16. The processors can deliver exceptional performance for workloads that can take advantage of such optimizations.
VMware Architecture Migration Tool
The VMware Architecture Migration Tool (VAMT) was developed jointly by AMD and VMware to automate the migration of legacy VMs from Intel architecture to AMD architecture, with the goal of delivering a better user experience and better business value. Freely available on GitHub, VAMT offers several key features:
- Architecture agnostic and open source
- Fully automated cold migration
- VM success validation
- Process throttling
- Change window support
- Email and syslog support
- Audit trail
- Rollback
The tool streamlines and simplifies the migration process in a trustworthy fashion.
Benchmarking
Dell leveraged the VAMT tool and the VMmark benchmark to achieve some remarkable consolidation on the PowerEdge R7625.
The VMmark benchmark allowed us to set up a workload in the form of tiles within each hardware cluster. Each tile consisted of 19 different VMs running a workload internally. The benchmark was deployed across five legacy Intel based servers and eventually migrated to a single AMD based PowerEdge server. A Dell PowerMax 2000 SAN was used for data storage. The following table shows the configuration details:
Table 1. Configuration of source and target servers
Component/specification | Source | Target |
---|---|---|
Number of servers | 5 | 1 |
Processor | Intel 8180 | AMD EPYC 9654 |
Cores per server | 56 | 192 |
Memory | 768 GB | 3 TB |
Tiles | 4 | 20 |
VMs per server | 76 | 380 |
Server | Server vendor A | Dell PowerEdge R7625 |
Storage | PowerMax 2000; 30 TB spread across 6 LUNs | |
Network | 32 GB FC network for storage, 25 GbE for data network on VMs through a 4-way splitter, 100 Gb switch |
We were able to run four tiles per legacy server for a total of 380 VMs. The VAMT was then used to migrate the VMs across to the target PowerEdge server.
The tool completed a cold migration of all 380 VMs to the target server in 57 minutes!
Achieving value
The Dell PowerEdge R7625 with 4th Gen AMD EPYC processors delivers significant technology advancements that can deliver value in any virtualized deployment. Consolidating from five servers to a single server is an example of the extent of savings possible. This kind of consolidation allows for significant license cost savings and fewer hours on system management. Decommissioning the five legacy systems also reduces power draw and operational costs by as much 64 percent,5 even while also running workloads on the latest architecture with security features like Secure Memory Encryption (SME) and Secure Encrypted Virtualization (SEV). AMD SEV helps safeguard privacy and integrity by encrypting each virtual machine.
1 https://github.com/vmware-samples/vmware-architecture-migration-tool
2 Based on Dell analysis of submitted SPECFPRate score of 1410 achieved on a Dell PowerEdge R7625 with AMD EPYC 9654s compared to the previous high score of 636 on a Dell PowerEdge R7525 with AMD EPYC 7763 processors as of 11/3/2022. Actual performance might vary.
3 Based on Dell analysis of published VMmark Server Power-Performance score of 21.0179@21 tiles achieved on a Dell PowerEdge R7615 cluster with AMD EPYC 9654P processors compared to the score of 8.1263@12 tiles achieved on a Dell PowerEdge R7515 cluster with the AMD EPYC 7763 processors as of 4/13/2023. Actual performance might vary.
5 Based on Dell internal analysis comparing the total CPU TDP of 2,050 W from five dual-socket servers with the Intel Xeon 8180 processors compared to the total CPU TDP of 720 W from a single dual-socket Dell PowerEdge server with AMD EPYC 9654 processors as of 4/13/2023. Actual performance might vary.