Direct from Development - PowerEdge MX7000 Chassis Thermal Airflow Architecture
Tue, 10 Nov 2020 23:06:50 -0000|
Read Time: 0 minutes
Creating a modular infrastructure that can efficiently cool the high-power, high- density workloads of today and tomorrow requires intelligent, scalable design.
Dell EMC’s PowerEdge MX7000 modular infrastructure meets these thermal challenges through an innovative, patent- pending chassis architecture design.
The Multiple Airflow Zone architecture of the new MX7000 chassis enables enhanced energy efficiency in the cooling of higher power and denser system configurations, extends the lifecycle of the chassis to accommodate multiple generations of future IT technology, and delivers excellent investment protection.
Modular infrastructures enable customers to be flexible in configuration, agile in management, and efficient in design, over several generations of compute, storage, and networking needs. The unique thermal design of the new Dell EMC PowerEdge MX7000 chassis ensures that customers can confidently grow over many generations of IT technology, satisfying forthcoming demands for scalable expansion and performance while providing excellent investment protection. The multiple airflow zone architecture of the MX7000 is engineered to ensure fresh air is delivered to each component of the chassis, enabling enhanced cooling of higher power and dense system configurations.
The overall chassis architecture of the PowerEdge MX7000 uses dedicated, independent airflow paths for each critical subsystem (Compute, I/O, and Power Supplies) to provide fresh air to each individual zone. This design allows for expanded feature support, improved cooling efficiency, and the flexibility to scale with future needs. Figure 1 below provides a view of the front and rear of the MX7000 chassis with each zone highlighted.
- Zone 1 (Blue): Cooling air for the eight vertically-arranged Compute and Storage sled slots at the front of the chassis is drawn thru the chassis by five horizontally-arranged fans at the rear
- Zone 2 (Yellow): Four vertically arranged fans at the front of the chassis push cooling air into the I/O Modules (IOMs) at the rear of the chassis
- Zone 3 (Green): The bottom of the chassis is populated with up to six power supply units (PSUs) each with a dedicated fan and exhaust airflow path out the rear of the chassis
Figure 1: Front and rear view of PowerEdge MX7000 chassis with dedicated airflow zones highlighted
The orthogonal layout of these zones allows for simple, front-to-back airflow thru the chassis. By avoiding overly complex airflow paths to each subsystem, the overall chassis minimizes its impedance, allowing for increased airflow capability compared to traditional modular chassis architectures. Additionally, the independent airflow zones allow for more granular fan control algorithms which increase fan speeds only when and where it is needed to further increase efficiency.
Compute and Storage Sled Cooling
Compared with previous modular chassis from Dell and competitors, the MX7000 chassis contains no vertical midplane that would restrict the airflow through the chassis. Instead, sleds within the chassis mate directly with rear IOMs in A1/A2/B1/B2 slots through direct orthogonal interconnects. In the space between the A1/A2 and B1/B2 IOMs, the MX7000 80mm fan modules are directly ducted to the sleds to pull air through compute and/or storage sleds in Zone 1. Multiple chassis seals ensure that the low pressure generated by the rear 80mm fans stays within Zone 1. Containing the low pressure is critical in enabling the increased sled storage density with support for up to six 2.5” HDDs in the front end of the Dell EMC PowerEdge MX740c.
Figure 2 below provides a cross-sectional view of the chassis to highlight the Zone 1 airflow path thru a compute sled. Cool air enters thru the hard disk drive (HDD) bay of the sled before cooling the CPUs, memory, and peripheral components. The 80mm fan modules at the rear of the chassis pull air through the sleds and exhaust hot air out of the system. Fans are positioned in line with the critical heat loads, the CPUs, to avoid complex ducting and provide efficient cooling. Since there are no downstream components in this flow path, a large temperature difference is allowed across CPUs to maximize processor TDP support with headroom to scale with future generations.
Figure 2: Cross-sectional view of airflow through Zone 1 of the MX7000 chassis, showing cooling of Compute sleds. Air enters at front of chassis (on left) and exhausts thru rear of chassis (on right). Other airflow zones have been greyed out in this graphic.
Networking and I/O
Due to their internal layout, modular infrastructures have traditionally been forced to cool networking fabrics with preheated air from upstream components or through tortuous pathways that limit the amount of cooling airflow available. With limited airflow capability, the ability to support future networking and I/O technologies becomes more difficult as these technologies continue to advance at a rapid pace. In contrast, the MX7000 chassis has been designed for fresh inlet air delivery to all rear IOMs and Chassis Management Modules (CMMs) via four dedicated fan modules in the front of the chassis. The direct airflow path is shown in the cross-sectional graphic in Figure 3 (at the top of Page 3), which highlights airflow Zone 2. The four 60mm fan modules arranged vertically share a large plenum down the center of the chassis. An air ducts splits the airflow into an upper path which cools IOMs A1/A2 and a lower path to cool IOMs B1/B2/C1/C2 and CMMs. The fresh air entering these modules ensures the MX7000 chassis can run fans at lower speeds and operate over a wider range of environmental conditions as well as providing sufficient airflow capability for future networking technologies.
Figure 3: Airflow through Zone 2 of the MX7000 chassis, showing cooling of IOMs. Other airflow zones have been greyed out in this graphic.
Power supplies in the MX7000 chassis benefit from a simple, independent airflow path at the bottom of the chassis, as shown in the cross sectional view of Figure 4 below. PSUs intake fresh air at the front of the chassis. Dedicated fans in the front of each PSU push airflow through dense electrical components and under the power connector at the back of the PSU. Since there are no downstream components to cool, PSUs can operate with very high exhaust temperatures of 60°C or higher. This thermal design helps ensure reliable delivery of up to 3000W supplied power in the dense form factor of each MX7000 PSU.
Figure 4: Airflow through Zone 3 of the MX7000 chassis, illustrating cooling of PSUs. Other airflow zones have been greyed out in this graphic
Traditional modular architectures have been characterized by complex airflow pathways that restrict airflow available to cool internal components, or require downstream components to operate within the constraints of preheated air from upstream components. This results in sub-optimal energy efficiency as well as places limitations on feature support, which in turn adversely impacts scalable growth for higher performance and/or capacity expansion.
The innovative multiple airflow zone architecture of the new PowerEdge MX7000 chassis overcomes these challenges and ensures that fresh air is delivered to each component in the chassis, enhancing cooling for higher power and denser system configurations. This design enables increased energy efficiency, extends the lifecycle of the chassis to accommodate multiple generations of future IT technology, and delivers excellent investment protection.
Related Blog Posts
Direct from Development – PowerEdge MX and Intel QAT
Wed, 11 Nov 2020 12:36:28 -0000|
Read Time: 0 minutes
PowerEdge MX is the first Dell EMC server to offer a software licensing option to enable Intel® QuickAssist Technology. It provides a software-enabled foundation for security, authentication, and compression, and significantly increases the performance and efficiency of standard platform solutions. Intel QAT on PowerEdge MX servers offer performance across applications. That includes symmetric encryption and authentication, asymmetric encryption, digital signatures, RSA, DH, and ECC, and lossless data compression.
Encryption and Key Generation
Many users will be familiar with the “https” prefix on frequently-visited websites. Behind all of these secure websites is an implementation of TLS (transport layer security) or its predecessor SSL (secure sockets layer). Each protocol entails a “handshake” between the client and server that establishes authenticity of the server and creates a session key for encrypting the exchanged data. These Public Key Encryption (PKE) algorithms, historically performed by software, can be offloaded from the CPU into the Intel® QAT engine for providing significant performance gains for Web Server, eCommerce, VPN, Firewall or Security Load Balancer and Wan Acceleration solutions.
Data Compression and Decompression
Users of “zip” files will be familiar with the benefit of another common software function, data compression. Like cryptography, compression and decompression can be compute-intensive functions. Intel® QAT is comprised of acceleration engines for data compression as well, yielding faster performance and higher throughput for software and systems that rely on compressed data such as storage, web compression, big data, or high performance computing (HPC).
Benefit of Intel® QAT
It really boils down to the TCO, or total cost of ownership. A web server, cloud load balancer, or security gateway that can handle significantly more secure connections per second and provide high performance encrypted data throughput for reduced infrastructure cost. A storage system that uses accelerated compression to decrease the total required capacity vastly reduces storage footprint and subsequent costs. Application efficiency also reduces the thermal footprint of a datacenter or computing cluster, lowering energy costs. Improved efficiency and reduced active power for security and compression translate to reduced infrastructure.
- Symmetric (Bulk) Cryptography
- Ciphers (AES, 3DES/DES, RC4, KASUMI*, ZUC, Snow 3G)
- Message digest/hash (MD5, SHA1, SHA2, SHA3) and authentication (HMAC, AES-XCBC)
Supported Operations (cont)
- Algorithm chaining (one cipher and one hash in a single operation)
- Authenticated encryption (AES-GCM, AES-CCM)
- KASUMI, Snow 3G and ZUC in encryption and authentication modes
- Asymmetric (Public Key) Cryptography
- Modular exponentiation for Diffie-Hellman (DH)
- RSA key generation, encryption/decryption and digital signature generation/verification
- DSA parameter generation and digital signature generation/verification
- Elliptic Curve Cryptography: ECDSA, ECDHE, Curve25519, SM2
- Compression/Decompression DEFLATE (Lempel-Ziv77) & Huffman.
Introducing Optional Software Licenses for Intel® QAT in PowerEdge MX
Intel® QAT has a long history with the deliveries of the 8920 model and the subsequent 8955 on PCIe cards. In the Intel® Xeon® Processor Scalable Family, Intel® is making the next generation of Intel® QAT available with significantly improved performance in a chipset-integrated version. Dell EMC is offering hardware-enabling licenses for chipset Intel® QAT on the MX series blade servers (MX740c and MX840c). These licenses can be installed without the need to add hardware to the system and occupy slots. Depending on the license level installed and the performance level desired, the chipset based Intel® QAT will be programmed to offer the bandwidth performance as defined below, mimicking the performance of the latest model 8960 and model 8970 PCIe cards. The licenses are installed through the iDRAC license manager.
Software is provided through the Intel open source site https://01.org/intel-quickassist-technology. The applicable drivers are associated with the C62x chipset. Application and library examples are posted here along with the API reference manuals, allowing users to build upon these open source libraries and examples or build their own applications. Release notes identify operating system compatibility.
Openssl is a software library that implements cryptographic functions that secure communications over computer networks. It implements the aforementioned protocols SSL and TLS. OpenSSL versions 1.1.0 and beyond now have asynchronous support for hardware accelerators, which helps achieve power, performance, cost, capacity and efficiency benefits discussed above. Prior to this support, all cryptographic function calls were performed in a synchronous manner, which meant that any given CPU thread was “blocked” awaiting the result of an operation. With asynchronous operation, several operations can be queued for Intel® QAT engine, and the responses can be collected and consumed as soon as they are completed in rapid succession. The following resources describe how to get Intel® QAT working with openssl:
Instructions to use openssl to integrate with applications such as NGINX web server and HAProxy, a load balancer and proxy, can be found on https://01.org/intel-quickassist-technology. NGINX has been demonstrated to handle more connections per second with the benefit of Intel® QAT.
DPDK (Data Plane Development Kit)
An open source project consisting of a set of libraries and drivers for fast packet processing, DPDK employs PMDs (Poll Mode Drivers) to interact with user space software, avoiding latency expensive context switches between kernel and user space. Instructions on installing the Intel® QAT PMD can be found at DPDK GUIDES LINK. Using DPDK, performance benefit has been demonstrated for IPsec (Internet Protocol Security), which provides security at a lower level in the protocol stack than TLS. For further reading on IPSEC, see the links Getting Started Guidehttps://software.intel.com/en-us/articles/get-started-with-ipsec-acceleration-in-the-fdio-vpp-project Sample Application Usage https://doc.dpdk.org/guides-16.04/sample_app_ug/ipsec_secgw.html.
Compression and Decompression
The primary vehicle for delivering sample code for data compression and decompression for Linux is QATZip, which is a user space library that produces data in standard gzip format. See the most recent release notes for the drivers and the API application guides for more information on data compression.
Intel® Key Protection Technology (Intel® KPT)
Inside the Intel chipset, there is a path for delivering keys directly from the key store in the chipset to the Intel® QAT engines. Software applications can utilize Intel® KPT to manage secure asymmetric and private key transactions for applications such as Hardware Security Modules(HSM) or Security Middle Box solutions.
Server workload performance is dependent on a wide variety of factors. The amount of CPU load on the system, the number of cores, the amount of memory, packet sizes, and compression levels are among many of such factors. Dell recommends specific testing to determine the exact improvements realizable by this offload. Below are some expected performance enhancements according to testing conducted Intel(r) Xeon Processor Scalable Family & Intel(r) C627 Chipset.
NGINX* and OpenSSL* connections/second. Conducted by Intel Applications Integration Team. Claim is actual performance measurement. Intel® microprocessor. Processor: Intel® Xeon® processor Scalable family with C6xxB0 ES2
Performance tests use cores from a single CPU, Memory configuration:, DDR4–2400. Populated with 1 (16 GB) DIMM per channel, total of 6 DIMMs Intel® QuickAssist Technology driver: QAT1.7.Upstream.L.0.8.0-37 Fedora* 22 (Kernel 4.2.7) BIOS:
24 Core Intel(r) Xeon Scalable Platform -SP @1.8GHz, Single (UP) Processor configuration. Intel(r) C627 PCH with crypto acceleration capability (in x16 mode) Neon City platform. DDR4 2400MHz RDIMMs 6x16GB(total 96 GB), 6 Channels, 1 x Intel® Corporation Red Rock Canyon 100GbE Ethernet Switch in the x16 PCIe slot on Socket 0. 8 cache ways allocated for DDIO.
Direct from Development - PowerEdge MX7000 Acoustical Options
Wed, 11 Nov 2020 00:11:49 -0000|
Read Time: 0 minutes
For the majority of PowerEdge MX7000 deployments, the acoustical experience meets customer expectations. For customers deploying MX7000 in noise-sensitive areas, a three-pillar strategy can help reduce the acoustical noise output. These pillars are: Configuration selection; Software settings; and Acoustical hardware.
Today’s server market is a challenging place to build quieter servers. Virtually every new generation of components require more power to drive incredible new features. Increased power means increased heat generation, stimulating increased airflow to achieve required cooling. For technology-dense data center products like the Dell EMC PowerEdge MX7000, increasing fan speed is the prescribed approach to deliver new features, though it comes with some acoustical output tradeoffs.
Leveraging the new efficient thermal design of the MX70001, the acoustical design of MX7000 fits well within the Dell EMC metrics for standard unattended modular data center products. However, Dell EMC acoustical engineers are aware of unique permanent or temporary applications where customers show increased acoustical noise sensitivity. For these applications, Dell EMC recommends a three-pillar strategy to achieve the desired level of noise for your application:
- Configuration selection;
- Software settings
- Acoustical hardware
Note: The MX7000 is not appropriate for office or general-use space deployments with or without the following pillars.
Image 1: The PowerEdge MX7000 modular platform
The most effective strategy for reducing acoustical output starts at the point of purchase. Though specific configuration recommendations are difficult to provide due to the wide range of workloads and applications that the MX7000 system supports, the following guidelines can be used to understand tradeoffs and optimize a system for a specific application space.
- Typically sled fans (rear fan modules) are the loudest component in the system, therefore reducing the total power consumption on individual sleds is the most successful approach to reducing acoustics. Choose lower wattage components, especially CPUs, and optimize DIMM counts to reduce sled power consumption.
- For compute sled configurations (MX740c & MX840c), CPU thermal design power (TDP) drives cooling requirements of the sled for most workloads. Choose the lowest TDP required achieve workload requirements. Where possible choose general purpose processors over low core-count or frequency optimized models to achieve lower acoustical output.
- For IOM-A/B options, 10 GbT and 25 GbE pass through, fabric expander (MX7116n) module and the switching module (MX5108n) provide better acoustical experience. Fabric switching engine (MX9116n) requires higher fan speeds to cool, which may compromise efforts to reduce acoustics.
- For IOM-C options, SAS storage IOM (MX5000s) requires lower fan speeds than the fibre channel module (MXG610s).
- When sled or module slots are empty, blanks must be installed to achieve efficient cooling and keep fan speeds from increasing.
The following table lists three configurations designed for specific workloads and deployment in attended data center applications.
Table 1: Select configurations that are designed for deployment in attended data center spaces.
- Computational MX740c sled configured with 2 145W CPUs, 12 32GB DIMMs, 4 1.6TB NVME SSD drives, 2 25Gb Mezzanine cards, and an H740 PERC.
- Transactional MX740c sled configured with 2 135W CPUs, 12 32GB DIMMS, 6 1.6TB 12Gb/s SAS SSD drives, 2 25Gb Mezzanine cards, 1 Fiber Channel MMZ, 2 M.2 Drives
- Virtualization MX740c configured with 2 135W CPUs, 12 32GB DIMMS, 6 1.6TB NVME SSD drives, 2 25Gb Mezzanine cards, 1 H745P PERC. MX5016s configured with 16 1.6TB SAS SSD drives.
For some MX7000 deployments, noise sensitivity may be situational and/or temporary. For these applications Dell EMC developed a software-based solution that can be enabled on demand. Sound cap is a custom thermal profile available in the BIOS and iDRAC GUI on MX740c and MX840c sleds. The sound cap feature limits acoustical output by applying a percentage-based power cap to the CPU. Therefore, acoustical output reduction comes at some cost to system performance.
Currently, sound cap must be enabled manually in each compute sled installed in an MX7000 chassis to be most effective. Sled reboot is required to enable or disable sound cap. Currently, sound cap can only be enabled in a sled iDRAC interface or in the BIOS options during sled boot up, sound cap cannot be enabled through MSM.
Table 2: Sound power1 impact for typical and feature rich configurations of PowerEdge MX7000 chassis when all CPUs are stressed to maximum power.
Sound Power with All CPUs @ Max Stress, Sound Cap Off, (bels)
Sound Power with All CPUs @ Max Stress, Sound Cap On, (bels)
- Sound power reported in this table represent engineering measurements collected during the course of development and are not official declared sound power measurements for MX7000. For official MX7000 sound power output data, refer to the MX7000 environmental data sheet.
- Typical A configuration includes 4 MX740c sleds, 2 MX840c sleds, 4 MX5108n IOMs and 2 MXG610 IOMS. MX740c sleds configured with 2 140 W TDP CPUs, 12 32 GB DIMMS, 6 1.6 TB SAS SSD Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ. H740+ PERC. MX840c sleds configured with 4 165 W TDP CPUs, 48 16 GB DIMMS, 6 1.6 TB NVME Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ.
- Typical B configuration includes 6 MX740c sleds, 4 MX5108n IOMs and 2 MXG610 IOMs. MX740c sleds configured with 2 140 W TDP CPUs, 12 32 GB DIMMS, 6 1.6 TB SAS SSD Drives, 2 25 Gb Mezzanine Cards, 1 Fibre Channel MMZ. H740+ PERC.
- Feature Rich configuration includes 6 MX740c sleds, 2 MX5016s sleds, 2 MX9116n IOMs, 2 MX7116n IOMs, and 2 MX5000s SAS Switches. MX740c sleds configured with 2 165W TDP CPUs, 24 32 GB DIMMs, 6 1.6 TB NVME Drives, 2 25 Gb Mezzanine Cards, H745p PERC. MX5016s sleds configured with 16 1.6 TB SAS SSD,
Finally, for persistent acoustically-sensitive deployments, Dell EMC has developed a hardware baffle solution, available as an optional add-on package to the MX7000 chassis. The baffle fits behind the MX7000 chassis and is designed to reduce the acoustical contribution of the rear fan modules. The baffle features a tool-less install; and fits within a standard rack depth without impacting cable management or rack door operation.
During product development, the MX7000 acoustical baffle and sound cap were tested under iterative usability studies. 26 IT professionals provided their experiential insights and acceptable performance tradeoffs for the MX7000 acoustical baffle and sound cap under simulated MX7000 workloads. The baffle alone was reportedly effective in reducing some shrill tones, even at 100% CPU utilization. Usability testing resulted in resoundingly positive testing scores, as the baffle scored the highest grade averaging an ‘A’. IT Professionals reported the acoustical benefit of shrill tones being blocked, making the MX7000 an acceptably quiet chassis to work around. Thus, the sound cap coupled with the acoustical baffle was worth the acoustic-to-performance trade-off in certain work environments. In these unique work scenarios, peer communication and employee discomfort-to-noise can be managed where employees may be mandated to work around exceptionally loud blade servers.
The new PowerEdge MX7000 chassis is a versatile and dense modular infrastructure that comes with acoustical noise tradeoffs. For the majority of MX7000 deployments in unattended data centers, the acoustical experience will meet customer expectations. For customers deploying MX7000 in noise sensitive areas, these three pillars can help reduce the acoustical noise output of the PowerEdge MX7000.
1. See the Direct from Development tech note, “PowerEdge MX7000 Chassis Thermal Airflow Architecture”