Direct from Development - PowerEdge MX7000 Direct Orthogonal Connectors
Tue, 10 Nov 2020 16:54:16 -0000|
Read Time: 0 minutes
The trend of increasingly higher- power processors and DIMMs creates challenges for traditional midplane architectures, particularly with regard to airflow and cooling, high speed signal transmission, and power delivery.
The new PowerEdge MX7000 modular platform eliminates the midplane and instead uses Direct Orthogonal Connectors for internal interconnection of compute, storage and switch modules.
Elimination of the midplane greatly optimizes airflow and enhances cooling. Signal transmission and power delivery are also improved by this new, innovative design.
Modular systems (such as the Dell EMC PowerEdge M1000e blade enclosure) integrating both compute/storage modules and switch (I/O) modules require an intricate method of interconnection within the chassis. A traditional interconnect architecture contains vertical compute/storage modules and horizontal switch modules, and a midplane enabling interconnection between these modules, as shown in Figure 1 below.
Figure 1: A traditional interconnect architecture using a midplane for interconnection of compute and switch (I/O) modules
A traditional midplane architecture such as the above has been an effective means of interconnect for many generations of servers, but a number of design challenges have been apparent and need to be overcome. Airflow has been a key challenge. Proper airflow is critical in order to ensure cooling of the compute modules, especially with current and future systems with high-power processors and DIMM’s. One attempt to resolve this has been to incorporate vent holes into the midplane to allow airflow. However, due to signal and power routing within the midplane, properly sized and effective placement of the vent holes is not always feasible. Moreover, signal integrity (SI) may be impacted when routing high speed signal traces long distances within the midplane. Even the use of Low Loss PCB material (such as Megtron 6) for high speed signal traces has limitations.
Consequently, in the development of the PowerEdge MX7000 Modular Chassis, the elimination of the midplane was a key design goal to improve airflow and signal integrity. To accomplish this, the implementation of Direct Orthogonal Connectors was selected to provide the interconnection between compute modules and switch modules. Fundamentally, Orthogonal Connectors provide a direct, right angle interconnection between a vertical PWA (Printed Wiring Assembly, also known as a Printed Circuit Board Assembly, PCBA) such as a compute module, and a horizontal PWA, i.e. a switch module, without a midplane. This direct interconnection is shown in Figure 2 below:
Figure 2: The PowerEdge MX7000 Orthogonal Connectors enable direct interconnection of compute/storage and switch (I/O) modules, liberating the architecture from the encumbrances and cost of a midplane.
The use of direct orthogonal connectors in the MX7000 eliminates airflow impedance caused by a midplane, and enhances overall airflow and cooling. The schematic in Figure 3 below illustrates how eliminating the midplane opens up internal areas of the PowerEdge MX7000 and improves airflow:
Figure 3: Elimination of the midplane in the PowerEdge MX7000 architecture opens up internal areas to improve airflow.
In addition to improving airflow and enhancing cooling, use of Direct Orthogonal Connectors in the PowerEdge MX7000 enables the formation of a comprehensive interconnection for high speed signal transmission between compute/storage modules and switch (I/O) modules, as well as for effective power delivery to these modules. Figure 4 below illustrates direct orthogonal interconnections between a vertical Compute Module and horizontal Switch Module, as well as power delivery using both Vertical and Horizontal Power Distribution PWA’s to those modules:
Figure 4: Schematic showing orthogonal power connectors for horizontal switch modules and vertical compute/storage modules.
While earlier servers and modular systems relied on a traditional midplane architecture, implementation of innovative Direct Orthogonal Connectors in the new PowerEdge MX7000 eliminates the midplane and greatly optimizes airflow through the compute and switch modules. This enhances cooling and ensures that current and future high-power processors and DIMM’s can be cooled more effectively, even at high ambient temperatures. Additionally, signal integrity will be optimal without being compromised by long Printed Circuit Board (PCB) traces, allowing for high speed signals (including PCIe Gen 4).
- For more information about airflow and cooling in the new PowerEdge MX7000, including its Multiple Airflow Zone design, see the Direct from Development tech note, PowerEdge MX7000 Chassis Thermal Airflow Architecture.
Related Blog Posts
Direct from Development – PowerEdge MX and Intel QAT
Wed, 11 Nov 2020 12:36:28 -0000|
Read Time: 0 minutes
PowerEdge MX is the first Dell EMC server to offer a software licensing option to enable Intel® QuickAssist Technology. It provides a software-enabled foundation for security, authentication, and compression, and significantly increases the performance and efficiency of standard platform solutions. Intel QAT on PowerEdge MX servers offer performance across applications. That includes symmetric encryption and authentication, asymmetric encryption, digital signatures, RSA, DH, and ECC, and lossless data compression.
Encryption and Key Generation
Many users will be familiar with the “https” prefix on frequently-visited websites. Behind all of these secure websites is an implementation of TLS (transport layer security) or its predecessor SSL (secure sockets layer). Each protocol entails a “handshake” between the client and server that establishes authenticity of the server and creates a session key for encrypting the exchanged data. These Public Key Encryption (PKE) algorithms, historically performed by software, can be offloaded from the CPU into the Intel® QAT engine for providing significant performance gains for Web Server, eCommerce, VPN, Firewall or Security Load Balancer and Wan Acceleration solutions.
Data Compression and Decompression
Users of “zip” files will be familiar with the benefit of another common software function, data compression. Like cryptography, compression and decompression can be compute-intensive functions. Intel® QAT is comprised of acceleration engines for data compression as well, yielding faster performance and higher throughput for software and systems that rely on compressed data such as storage, web compression, big data, or high performance computing (HPC).
Benefit of Intel® QAT
It really boils down to the TCO, or total cost of ownership. A web server, cloud load balancer, or security gateway that can handle significantly more secure connections per second and provide high performance encrypted data throughput for reduced infrastructure cost. A storage system that uses accelerated compression to decrease the total required capacity vastly reduces storage footprint and subsequent costs. Application efficiency also reduces the thermal footprint of a datacenter or computing cluster, lowering energy costs. Improved efficiency and reduced active power for security and compression translate to reduced infrastructure.
- Symmetric (Bulk) Cryptography
- Ciphers (AES, 3DES/DES, RC4, KASUMI*, ZUC, Snow 3G)
- Message digest/hash (MD5, SHA1, SHA2, SHA3) and authentication (HMAC, AES-XCBC)
Supported Operations (cont)
- Algorithm chaining (one cipher and one hash in a single operation)
- Authenticated encryption (AES-GCM, AES-CCM)
- KASUMI, Snow 3G and ZUC in encryption and authentication modes
- Asymmetric (Public Key) Cryptography
- Modular exponentiation for Diffie-Hellman (DH)
- RSA key generation, encryption/decryption and digital signature generation/verification
- DSA parameter generation and digital signature generation/verification
- Elliptic Curve Cryptography: ECDSA, ECDHE, Curve25519, SM2
- Compression/Decompression DEFLATE (Lempel-Ziv77) & Huffman.
Introducing Optional Software Licenses for Intel® QAT in PowerEdge MX
Intel® QAT has a long history with the deliveries of the 8920 model and the subsequent 8955 on PCIe cards. In the Intel® Xeon® Processor Scalable Family, Intel® is making the next generation of Intel® QAT available with significantly improved performance in a chipset-integrated version. Dell EMC is offering hardware-enabling licenses for chipset Intel® QAT on the MX series blade servers (MX740c and MX840c). These licenses can be installed without the need to add hardware to the system and occupy slots. Depending on the license level installed and the performance level desired, the chipset based Intel® QAT will be programmed to offer the bandwidth performance as defined below, mimicking the performance of the latest model 8960 and model 8970 PCIe cards. The licenses are installed through the iDRAC license manager.
Software is provided through the Intel open source site https://01.org/intel-quickassist-technology. The applicable drivers are associated with the C62x chipset. Application and library examples are posted here along with the API reference manuals, allowing users to build upon these open source libraries and examples or build their own applications. Release notes identify operating system compatibility.
Openssl is a software library that implements cryptographic functions that secure communications over computer networks. It implements the aforementioned protocols SSL and TLS. OpenSSL versions 1.1.0 and beyond now have asynchronous support for hardware accelerators, which helps achieve power, performance, cost, capacity and efficiency benefits discussed above. Prior to this support, all cryptographic function calls were performed in a synchronous manner, which meant that any given CPU thread was “blocked” awaiting the result of an operation. With asynchronous operation, several operations can be queued for Intel® QAT engine, and the responses can be collected and consumed as soon as they are completed in rapid succession. The following resources describe how to get Intel® QAT working with openssl:
Instructions to use openssl to integrate with applications such as NGINX web server and HAProxy, a load balancer and proxy, can be found on https://01.org/intel-quickassist-technology. NGINX has been demonstrated to handle more connections per second with the benefit of Intel® QAT.
DPDK (Data Plane Development Kit)
An open source project consisting of a set of libraries and drivers for fast packet processing, DPDK employs PMDs (Poll Mode Drivers) to interact with user space software, avoiding latency expensive context switches between kernel and user space. Instructions on installing the Intel® QAT PMD can be found at DPDK GUIDES LINK. Using DPDK, performance benefit has been demonstrated for IPsec (Internet Protocol Security), which provides security at a lower level in the protocol stack than TLS. For further reading on IPSEC, see the links Getting Started Guidehttps://software.intel.com/en-us/articles/get-started-with-ipsec-acceleration-in-the-fdio-vpp-project Sample Application Usage https://doc.dpdk.org/guides-16.04/sample_app_ug/ipsec_secgw.html.
Compression and Decompression
The primary vehicle for delivering sample code for data compression and decompression for Linux is QATZip, which is a user space library that produces data in standard gzip format. See the most recent release notes for the drivers and the API application guides for more information on data compression.
Intel® Key Protection Technology (Intel® KPT)
Inside the Intel chipset, there is a path for delivering keys directly from the key store in the chipset to the Intel® QAT engines. Software applications can utilize Intel® KPT to manage secure asymmetric and private key transactions for applications such as Hardware Security Modules(HSM) or Security Middle Box solutions.
Server workload performance is dependent on a wide variety of factors. The amount of CPU load on the system, the number of cores, the amount of memory, packet sizes, and compression levels are among many of such factors. Dell recommends specific testing to determine the exact improvements realizable by this offload. Below are some expected performance enhancements according to testing conducted Intel(r) Xeon Processor Scalable Family & Intel(r) C627 Chipset.
NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2
Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMs Intel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS:
24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total 96 GB), 6 Channels, 1 x Intel® Corporation Red Rock Canyon 100GbE Ethernet Switch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO.
Direct from Development - PowerEdge MX7000 Acoustical Options
Wed, 11 Nov 2020 00:11:49 -0000|
Read Time: 0 minutes
For the majority of PowerEdge MX7000 deployments, the acoustical experience meets customer expectations. For customers deploying MX7000 in noise-sensitive areas, a three-pillar strategy can help reduce the acoustical noise output. These pillars are: Configuration selection; Software settings; and Acoustical hardware.
Today’s server market is a challenging place to build quieter servers. Virtually every new generation of components require more power to drive incredible new features. Increased power means increased heat generation, stimulating increased airflow to achieve required cooling. For technology-dense data center products like the Dell EMC PowerEdge MX7000, increasing fan speed is the prescribed approach to deliver new features, though it comes with some acoustical output tradeoffs.
Leveraging the new efficient thermal design of the MX70001, the acoustical design of MX7000 fits well within the Dell EMC metrics for standard unattended modular data center products. However, Dell EMC acoustical engineers are aware of unique permanent or temporary applications where customers show increased acoustical noise sensitivity. For these applications, Dell EMC recommends a three-pillar strategy to achieve the desired level of noise for your application:
- Configuration selection;
- Software settings
- Acoustical hardware
Note: The MX7000 is not appropriate for office or general-use space deployments with or without the following pillars.
Image 1: The PowerEdge MX7000 modular platform
The most effective strategy for reducing acoustical output starts at the point of purchase. Though specific configuration recommendations are difficult to provide due to the wide range of workloads and applications that the MX7000 system supports, the following guidelines can be used to understand tradeoffs and optimize a system for a specific application space.
- Typically sled fans (rear fan modules) are the loudest component in the system, therefore reducing the total power consumption on individual sleds is the most successful approach to reducing acoustics. Choose lower wattage components, especially CPUs, and optimize DIMM counts to reduce sled power consumption.
- For compute sled configurations (MX740c & MX840c), CPU thermal design power (TDP) drives cooling requirements of the sled for most workloads. Choose the lowest TDP required achieve workload requirements. Where possible choose general purpose processors over low core-count or frequency optimized models to achieve lower acoustical output.
- For IOM-A/B options, 10 GbT and 25 GbE pass through, fabric expander (MX7116n) module and the switching module (MX5108n) provide better acoustical experience. Fabric switching engine (MX9116n) requires higher fan speeds to cool, which may compromise efforts to reduce acoustics.
- For IOM-C options, SAS storage IOM (MX5000s) requires lower fan speeds than the fibre channel module (MXG610s).
- When sled or module slots are empty, blanks must be installed to achieve efficient cooling and keep fan speeds from increasing.
The following table lists three configurations designed for specific workloads and deployment in attended data center applications.
Table 1: Select configurations that are designed for deployment in attended data center spaces.
- Computational MX740c sled configured with 2 145W CPUs, 12 32GB DIMMs, 4 1.6TB NVME SSD drives, 2 25Gb Mezzanine cards, and an H740 PERC.
- Transactional MX740c sled configured with 2 135W CPUs, 12 32GB DIMMS, 6 1.6TB 12Gb/s SAS SSD drives, 2 25Gb Mezzanine cards, 1 Fiber Channel MMZ, 2 M.2 Drives
- Virtualization MX740c configured with 2 135W CPUs, 12 32GB DIMMS, 6 1.6TB NVME SSD drives, 2 25Gb Mezzanine cards, 1 H745P PERC. MX5016s configured with 16 1.6TB SAS SSD drives.
For some MX7000 deployments, noise sensitivity may be situational and/or temporary. For these applications Dell EMC developed a software-based solution that can be enabled on demand. Sound cap is a custom thermal profile available in the BIOS and iDRAC GUI on MX740c and MX840c sleds. The sound cap feature limits acoustical output by applying a percentage-based power cap to the CPU. Therefore, acoustical output reduction comes at some cost to system performance.
Currently, sound cap must be enabled manually in each compute sled installed in an MX7000 chassis to be most effective. Sled reboot is required to enable or disable sound cap. Currently, sound cap can only be enabled in a sled iDRAC interface or in the BIOS options during sled boot up, sound cap cannot be enabled through MSM.
Table 2: Sound power1 impact for typical and feature rich configurations of PowerEdge MX7000 chassis when all CPUs are stressed to maximum power.
Sound Power with All CPUs @ Max Stress, Sound Cap Off, (bels)
Sound Power with All CPUs @ Max Stress, Sound Cap On, (bels)
- Sound power reported in this table represent engineering measurements collected during the course of development and are not official declared sound power measurements for MX7000. For official MX7000 sound power output data, refer to the MX7000 environmental data sheet.
- Typical A configuration includes 4 MX740c sleds, 2 MX840c sleds, 4 MX5108n IOMs and 2 MXG610 IOMS. MX740c sleds configured with 2 140 W TDP CPUs, 12 32 GB DIMMS, 6 1.6 TB SAS SSD Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ. H740+ PERC. MX840c sleds configured with 4 165 W TDP CPUs, 48 16 GB DIMMS, 6 1.6 TB NVME Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ.
- Typical B configuration includes 6 MX740c sleds, 4 MX5108n IOMs and 2 MXG610 IOMs. MX740c sleds configured with 2 140 W TDP CPUs, 12 32 GB DIMMS, 6 1.6 TB SAS SSD Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ. H740+ PERC.
- Feature Rich configuration includes 6 MX740c sleds, 2 MX5016s sleds, 2 MX9116n IOMs, 2 MX7116n IOMs, and 2 MX5000s SAS Switches. MX740c sleds configured with 2 165W TDP CPUs, 24 32 GB DIMMs, 6 1.6 TB NVME Drives, 2 25 Gb Mezzanine Cards, H745p PERC. MX5016s sleds configured with 16 1.6 TB SAS SSD,
Finally, for persistent acoustically-sensitive deployments, Dell EMC has developed a hardware baffle solution, available as an optional add-on package to the MX7000 chassis. The baffle fits behind the MX7000 chassis and is designed to reduce the acoustical contribution of the rear fan modules. The baffle features a tool-less install; and fits within a standard rack depth without impacting cable management or rack door operation.
During product development, the MX7000 acoustical baffle and sound cap were tested under iterative usability studies. 26 IT professionals provided their experiential insights and acceptable performance tradeoffs for the MX7000 acoustical baffle and sound cap under simulated MX7000 workloads. The baffle alone was reportedly effective in reducing some shrill tones, even at 100% CPU utilization. Usability testing resulted in resoundingly positive testing scores, as the baffle scored the highest grade averaging an ‘A’. IT Professionals reported the acoustical benefit of shrill tones being blocked, making the MX7000 an acceptably quiet chassis to work around. Thus, the sound cap coupled with the acoustical baffle was worth the acoustic-to-performance trade-off in certain work environments. In these unique work scenarios, peer communication and employee discomfort-to-noise can be managed where employees may be mandated to work around exceptionally loud blade servers.
The new PowerEdge MX7000 chassis is a versatile and dense modular infrastructure that comes with acoustical noise tradeoffs. For the majority of MX7000 deployments in unattended data centers, the acoustical experience will meet customer expectations. For customers deploying MX7000 in noise sensitive areas, these three pillars can help reduce the acoustical noise output of the PowerEdge MX7000.
1. See the Direct from Development tech note, “PowerEdge MX7000 Chassis Thermal Airflow Architecture”