Accelerating Intra-Host Data Movement with VMware PVRDMA on a Dell AMD PowerEdge Server
Download PDFMon, 16 Jan 2023 23:18:32 -0000
|Read Time: 0 minutes
Summary
PVRDMA is an innovative new technology to help accelerate intra-host data movement (i.e. data between VMs on a host). The ESXi driver takes advantage of DMA engines in the AMD Epyc 2nd and 3rd Generation of processors to provide significant performance improvements for this use case. Please contact Dell if you need more information.
Introduction
As CPU cores scale, the usage and demand for ever larger data exchanges within a host scale. It scales among kernels, applications, VMs, and I/O devices. At the same time, VM density is on the rise. Hyperconverged Storage appliances are enabling new workloads on server class systems where Data locality is important. Each intra-host exchange can comprise multiple memory buffer copies (or transformations) and are generally implemented with layers of software stacks. While Kernel-to-I/O can leverage I/O- specific hardware memory copy, SW-to-SW usually relies on per- core synchronous software (CPU-only) memory copies
Intra-host SW memory copy and transformation needs are not keeping up with scaling of cores & workloads. It is difficult for complex processors to ensure a single-core attains full processor- to memory-bandwidth and therefore, noticeable core utilization is needed to satisfy synchronous copy needs. Further, I/O intensive workloads can take away compute CPU cycles available. Network and Storage workloads can take compute cycles for Data Movement, Encryption, Decryption, Compression etc.
Hypervisors want a well-defined capability to quiesce, suspend, and resume the architectural state of a per-address-space data mover that is independent of the actual hardware-offload and the various accelerations that it can perform. This allows “live” workload (VM migration) between different servers with different HW-offload implementations of standard data-movement operations
What is PVRDMA
PVRDMA is one such well-architected, virtualizable interface to workload VMs. The interface is now standard in ESXi and is implemented across CPUs, Platforms, and NICs. Acceleration can occur transparent to the workload. PVRDMA emulation uses RDMA or TCP/IP NICs for inter-host VM-to-VM transfers. However, PVRDMA uses SW Memory Copy for intra-host VM-to-VM transfers if no RDMA NIC and this is ideal for a DMA engine.
Using the AMD EPYC 7002 or 7003 Processors with built-in PTDMA DMA Engines and an AMD ESXi PTDMA PVSP1 Driver, Dell AMD PowerEdge Servers can accelerate Intra-host VM-to-VM Data Movement. PTDMA engines can perform memory copies without CPU intervention
Fig.1 shows the results of PVRDMA testing with SW memory copies for a copy size of 8KiB to 256KiB and PTDMA engine memory copies from 256KiB to 64M.
Fig.1 PVRDMA* Socket-to-Socket Memory Copy Throughput Using AMD PTDMA+ Driver
We observe that SW copy performance is good for small copy sizes. However, as copy sizes fall out of the L1 cache(2*32KiB), SW copy performance drops yet continues to outperform PTDMA copy performance before it falls out of the L2 cache (1/2 * 512KiB). At this L2 cache threshold, PTDMA copies become more performant and outperform SW copy performance. We tested it upto 8M and we expect that the performance will vary with different workloads and application cache interaction
PVRDMA in vSphere 7.0 u1 can now use this PVSP1 DMA compliant driver to perform intra-host copy. PVRDMA and the prototype AMD PTDMA (PVSP) driver support a hybrid approach. By default, the threshold is 256 KiB and can be adjusted for various application tunings.
Next Steps
A PVRDMA technology demo is available on demand from Dell using Dell PowerEdge Servers with this technology. Interested customers can contact their Dell account executive to schedule an engagement request with a Dell Customer Solution Center 2
Also, a VMworld presentation titled “Accelerated, Virtualized, and Standardized Intra-host Data Movement” is available on demand here3 that provides more info on this technology and Dell’s vision into the future in this space.
References
- Partner Verified and Supported Program
- https://www.delltechnologies.com/en-us/what-we-do/customer-engagement- programs/customer_solution_centers.htm
- https://www.vmworld.com/en/video-library/search.html#text=%22OCTO2592%22&year=2020