Defense in-depth: Comprehensive Security on PowerEdge AMD EPYC Generation 2 (Rome) Servers
Download PDFTue, 17 Jan 2023 00:11:16 -0000
|Read Time: 0 minutes
Summary
Security in servers is no longer an afterthought – it is a key consideration in the choice of a server provider and platform. Dell EMC approaches security in multiple layers to best protect customer assets and data. This includes not just security built into the system and components, but also to manufacturing processes and ensuring a secure supply chain.
Introduction
In the wake of Spectre and Meltdown and endless other side-channel issues, and with predictive indicators showing that new forms of attack are likely – security is a critical requirement for servers. And it is important to ensure that server security is at layers within the systems so that malicious activity can be mitigated in numerous ways. PowerEdge servers with AMD Rome processors use a multi-layer, end-to-end approach of security to help ensure that users’ data and assets are protected, see Figure 1.
Figure 1: Layers of security in PowerEdge AMD Rome-based servers
Layer 1: AMD EPYC-based System Security for Processor, Memory and VMs on PowerEdge
The first generation of the AMD EPYC processors have the AMD Secure Processor – an independent processor core integrated in the CPU package alongside the main CPU cores. On system power-on or reset, the AMD Secure Processor executes its firmware while the main CPU cores are held in reset. One of the AMD Secure Processor’s tasks is to provide a secure hardware root-of-trust by authenticating the initial PowerEdge BIOS firmware. If the initial PowerEdge BIOS is corrupted or compromised, the AMD Secure Processor will halt the system and prevent OS boot. If no corruption, the AMD Secure Processor starts the main CPU cores, and initial BIOS execution begins.
The very first time a CPU is powered on (typically in the Dell EMC factory) the AMD Secure Processor permanently stores a unique Dell EMC ID inside the CPU. This is also the case when a new off-the-shelf CPU is installed in a Dell EMC server. The unique Dell EMC ID inside the CPU binds the CPU to the Dell EMC server. Consequently, the AMD Secure Processor may not allow a PowerEdge server to boot if a CPU is transferred from a non-Dell EMC server (and CPU transferred from a Dell EMC server to a non-Dell EMC server may not boot).
AMD EPYC Generation 2 processors also offer the AMD Secure Processor --- for cryptographic functionality for secure key generation and key management. This provides full stack encryption without any overhead for the processor. In addition, for hardware-accelerated memory encryption for data-in-use protection, the security components in Rome processors include the AES-128 encryption engine, which is embedded in the memory controller and automatically encrypts and decrypts data in main memory with an appropriate key.
The AMD EPYC processors also include these two unique security features:
Secure Memory Encryption (SME):
SME uses a single key to encrypt system memory, which is generated by the AMD Secure Processor at boot. SME requires enablement in the system BIOS or operating system; when enabled in the BIOS, memory encryption is transparent and can be run with any operating system
Secure Encrypted Virtualization (SEV):
In addition to what SME capabilities, SEV provides Virtual Machine (VM) level encryption, which protects against hypervisor corruption with hardware protection – a more robust solution than software protection. The EPYC Generation 2 (Rome) processors also offer 509 keys per hypervisor for SEV, versus 16 in EPYC (Naples)-based servers
Secure Encrypted Virtualization – Encrypted State (SEV ES):
Encrypts all CPU register contents when a VM stops running, preventing leakage of information in CPU registers to components like the hypervisor, and it can detect malicious modifications to a CPU register state. Some technical details:
- Guest register state is encrypted with guest encryption key and integrity protected
- Only the guest can modify its register state
- Guest must explicitly share register state with the hypervisor
- Guest-Hypervisor Communication Block (GHCB)
- Protects the guest register state from the hypervisor
- Adds additional protection against VM state related attacks (exfiltration, control flow, rollback)
For more information, see this technical brief on EPYC first generation security: AMD CPU Security Features in PowerEdge 14G Servers.
Layer 2: PowerEdge Systems Security
All Dell EMC PowerEdge servers offer built-in security that supports customers with compliance, preventive security, and fast means to recover in the event of errors or breaches. This includes FIPs/Common Criteria Compliance, immutable silicon root of trust (PowerEdge CPUs have a Dell signature: once it is used in a Dell system if cannot be used in another server), digitally signed firmware updates, automatic BIOS recovery, firmware rollback, and more.
In addition, Dell EMC offers differentiated security features in every PowerEdge system:
- Dell EMC OpenManage Secure Enterprise Key Manager –embedded in Dell EMC PowerEdge servers and works in conjunction with leading Key Management Servers for enabling keys at scale
- System Lockdown – Locks down the configuration and firmware, protecting the server(s) from inadvertent or malicious changes, and is enabled or disabled by the IT Administrator… and prevents system/firmware “drift”
- System erase of all user drives, including NVMe – through a process that is not only fast, but enables the drives to be reused and meets NIST recommendations for data erasure
- Rapid OS Recovery – Allows users to boot a trusted backup OS image from a hidden boot device
- Enhanced UEFI secure boot with custom certificates – with UEFI Secure Boot, each component in the chain is validated and authorized against a specific certificate before it is allowed to load or run
- Dynamically-enabled USB ports
- This feature allows administrators to disable all USB ports and then enable them dynamically to allow local crash cart usage (to let a local technician have temporary access)
- The USB ports can be dynamically enabled and disabled without rebooting the server; normally changing the USB port state requires a reboot and takes down the workloads
- Intrusion-switch included – detection of chassis intrusion at no extra expense
- Domain Isolation - an important feature for multi-tenant hosting environments, hosting providers may want to block any re-configuration by tenants. Domain isolation is a configuration option that ensures that management applications in the host OS have no access to the out-of-band iDRAC or to Intel chipset functions
For more information, see this technical brief: Security in Server Design
And this video for further information:
Server Security – Dell EMC PowerEdge Servers
Layer 3: Dell Technologies Factory Security
Factories where Dell products are built must meet specified Transported Asset Protection Association (TAPA) facility security requirements, including the use of closed-circuit cameras in key areas, access controls, and continuously guarded entries and exits. Additional controls are applied at Dell and supplier- managed facilities and for air, rail, and ocean shipments to address the variety of risks faced across transportation modes and regions. Some of these protections include tamper-evident packaging, security reviews of shipping lanes, locks or hardware meeting required specifications, and container integrity requirements. GPS tracking devices may also be placed on any container and monitored 24x7 until confirmation of delivery.
Dell also maintains certification with the United States Customs and Border Patrol’s Customs-Trade Partnership Against Terrorist (C-TPAT). This logistics security program is recognized as compatible with similar programs around the world, including the Authorized Economic Operator (AEO), Canada’s Supply Chain Assurance v4.0 | Dell Inc., 2018 4 Partners in Protection, and Singapore’s Secure Trade Partnership programs. While the primary focus of these programs is to prevent contraband, the required protections also guard against tampering with products being imported.
Layer 4: Dell Technologies Supply Chain Security
The goal of Dell’s supply chain security processes is to provide continuous security risk assessment and improvement. Dell’s Supply Chain Risk Management framework mirrors that of the comprehensive risk management framework of the National Infrastructure Protection Plan (NIPP), which outlines how government and the private sector can work together to mitigate risks and meet security objectives. Dell’s framework incorporates an open feedback loop (see Figure 2) that allows for continuous improvement.
Risk mitigation plans are prioritized and implemented as appropriate throughout the entire solution life cycle.
Figure 2 Managing the supply chain for Dell Technologies products
The process includes these safeguards by Dell Technologies for the supply chain:
- Supplier governance by Dell
- Audits
- Global Inventory Control Policy
- Measure suppliers’ security practices against industry best practices for physical security and for mitigating counterfeit components, tainted software and firmware, and intellectual property theft
- Quarterly Reviews
- Supply Chain Security
- Physical (factory/manufacturing) – factories where Dell products are built must meet specified Transported Asset Protection Association (TAPA) facility security requirements. Dell also maintains certification with the United States Customs and Border Patrol’s Customs-Trade Partnership Against Terrorist (C-TPAT).
- Personnel – Dell policy requires employees throughout the supply chain, including those at contract suppliers, to go through a pre-employment suitability screening process.
- Information – Dell collects and uses sensitive information about products, solutions, customers, suppliers and partners throughout the supply chain lifecycle. Dell uses numerous measures to guard this sensitive information against exposure and exploitation.
- Supply Chain Integrity
Dell has developed baseline specifications that are securely preserved and later used as a reference to verify that no unauthorized modifications have been made to hardware or software. The goal is to ensure that the products received by customers are the products customers expected and will operate as intended.
For hardware, this includes processes to minimize the opportunity for counterfeit components to infiltrate our supply chain. For software, Industry software engineering best practices include security throughout the development process for any code, including operating systems, applications, firmware, and device drivers. Dell reduces opportunities for the exploitation of software security flaws by incorporating Secure Development Lifecycle (SDL) measures throughout the development process. These measures are tightly aligned with Software Assurance Forum for Excellence in Code (SAFECode) guidelines and ISO 27034.
- Stronger together
Dell participates in supply chain risk management activities with trusted industry groups and public/private partnerships. Dell has been actively engaged in the Open Group Trusted Technology Forum (O-TTPF), the Software and Supply Chain Assurance Forum, SAFECode, the Supply Chain Risk Leadership Council, the Internet Security Alliance, and the IT Sector Coordinating Council. Dell is also an active member of the Government Information Data Exchange Program (GIDEP). Dell has participated in the development of numerous standards and best-practice guidelines related to supply chain integrity including the Open Group Trusted Technology Provider Standard (O-TTPS) which is also recognized as ISO 20243, SAFECode, ISO 27036, and National Institute of Science and Supply Chain Assurance v4.0 | Dell Inc., 2018 6 Technology (NIST) Interagency Report (IR) 7622, NIST Special Publication (SP) 800-161, NIST SP800-53, and the NIST Cybersecurity Framework. To address customer concerns about product tampering and supply chain assurance, Dell continues to monitor and influence the development and potential impact of legislation, regulations, voluntary standards, and contract language
For more details on Dell supply chain security please refer to this white paper: https://i.dell.com/sites/csdocuments/CorpComm_Docs/en/supply-chain-assurance.pdf?newtab=true
In Conclusion
Security must be designed within the architecture of the server to effectively withstand sophisticated cyber - crime: phishing attacks that harvest credentials, advanced persistent threats (taking control of firmware), data exfiltration (stealing data). Yet it’s not just the server features that need to support customer security – it is also necessary to provide protection against the possibility of corruption in manufacturing and within the server supply chain. These layers of security must be considered as critical criteria for user decisions on integrating technical equipment into their environments.
As Dell EMC designs products, it will always be to protect, protect, and protect customer data and assets – and in consideration of worst-case scenarios, ensure that users of Dell EMC solutions can recover quickly, and resume production with as little disruption as possible. With these goals, Dell EMC is constantly evaluating new ways within each security layer to protect customers.
Related Documents
Dell PowerEdge 16G Server BIOS Settings for Optimized Performance: R7625, R6625, R7615, R6615, C6615
Tue, 26 Mar 2024 22:46:05 -0000
|Read Time: 0 minutes
BIOS setting recommendations
The following tables provide the BIOS setting recommendations for the latest generation of PowerEdge servers:
Table 1. BIOS setting recommendations - System profile settings
System setup screen | Setting | BIOS Defaults | SPEC cpu2017 int rate (General Purpose Performance) | SPEC cpu2017 fp rate | SPEC cpu2017 int speed | SPEC cpu2017 fp speed | Memory Throughput | HPC | Latency |
System profile setting | System profile | Performance Per Watt | Custom | Custom | Custom | Custom | Custom | Custom | Custom |
System profile setting[*] | CPU Power Management | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM | Max Performance | Max Performance |
System profile setting | Memory Frequency | Max Performance | Max Performance | Max Performance | Max Performance | Max Performance | Max Performance | Max Performance | Max Performance |
System profile setting | Turbo Boost | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
System profile setting | C-States | Enabled | Enabled | Enabled | Disabled | Disabled | Disabled | Disabled | Disabled |
System profile setting | Write Data CRC | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Enabled | Disabled |
System profile setting | Memory Patrol Scrub | Standard | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
System profile setting | Memory Refresh Rate | 1x | 1x | 1x | 1x | 1x | 1x | 1x | 1x |
System profile setting | Workload Profile | not configured | not configured | not configured | not configured | not configured | not configured | HPL | not configured |
System profile setting | PCI ASPM L1 Link Power Management | Enabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
System profile setting | Determinism Slider | Performance Determinism | Power Determinism | Power Determinism | Power Determinism | Power Determinism | Power Determinism | Power Determinism | Power Determinism |
System profile setting | Power Profile Select | High Performance Mode | High Performance Mode | High Performance Mode | High Performance Mode | High Performance Mode | High Performance Mode | High Performance Mode | High Performance Mode |
System profile setting | PCIE Speed PMM Control | Auto | Auto | Auto | Auto | Auto | Auto | Auto | (GEN 5) |
System profile setting | EQ Bypass To Highest Rate | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
System profile setting | DF PState Frequency Optimizer | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
System profile setting | DF PState Latency Optimizer | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
System profile setting | DF CState | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
System profile setting | Host System Management Port (HSMP) Support | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
System profile setting | Boost FMax | 0-Auto | 0-Auto | 0-Auto | 0-Auto | 0-Auto | 0-Auto | 0-Auto | 0-Auto |
System profile setting | Algorithm Performance Boost Disable (ApbDis) | Disabled | Disabled | Disabled | Enabled | Enabled | Disabled | Disabled | Enabled |
System profile setting | ApbDis Fixed Socket P-State[2] | P0 | P0 | P0 | |||||
System profile setting | Dynamic Link Width Management (DLWM) | Unforced | Unforced | Unforced | Unforced | Unforced | Unforced | Unforced | Forced x16 |
[*] For C6615, apply setting from Table 3.
[1] Depends on how system was ordered. Other System Profile defaults are driven by this choice and may be different than the examples listed. Select Performance Profile first, and then select Custom to load optimal profile defaults for further modification.
[2] Pstate field is dependent on Algorithm Performance Boost Disable (ApbDis) and is visible only when it is enabled.
Table 2. BIOS setting recommendations – Memory, processor, and iDRAC settings
System setup screen | Setting | BIOS Defaults | SPEC cpu2017 int rate (General Purpose Performance) | SPEC cpu2017 fp rate | SPEC cpu2017 int speed | SPEC cpu2017 fp speed | Memory Throughput | HPC | Latency |
Memory settings | System Memory Testing | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Memory settings | DRAM Refresh Delay | Minimum | Performance | Performance | Performance | Performance | Performance | Performance | Performance |
Memory settings | Correctable memory ECC SMI | Enabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Memory settings | Uncorrectable Memory Error (DIMM Self healing on uncorrectable memory) | Enabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Memory settings | Correctable Error Logging | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | Logical Processor | Enabled | Enabled | Disabled[1] | Disabled[1] | Disabled[1] | Disabled[1] | Disabled[1] | Disabled[1] |
Processor settings | Virtualization Technology | Enabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | IOMMU Support | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | Kernel DMA Protection | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | L1 Stream HW Prefetcher | Enabled | Enabled | Disabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | L2 Stream HW Prefetcher | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | L1 Stride Prefetcher | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | L1 Region Prefetcher | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | L2 Up Down Prefetcher | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | MADT Core Enumeration | Linear | Linear | Linear | Linear | Linear | Linear | Linear | Linear |
Processor settings[*] | NUMA Node Per Socket | 1 | 4 | 4 | 4 | 1 | 4 | 4 | 4 |
Processor settings | L3 cache as NUMA | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | Secure Memory Encryption | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | Minimum SEV no-ES ASID | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 1 |
Processor settings | SNP Memory Coverage | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | Secure Nested Paging | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | Transparent Secure Memory Encryption | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled | Disabled |
Processor settings | ACPI CST C2 Latency | 800 | 18 | 18 | 18 | 18 | 800 | 18 | 800 |
Processor settings | Configurable TDP | Maximum | Maximum | Maximum | Maximum | Maximum | Maximum | Maximum | Maximum |
Processor settings | x2APIC Mode | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled | Enabled |
Processor settings | Number of CCDs per Processor | All | All | All | All | All | All | All | All |
Processor settings | Number of Cores per CCD | All | All | All | All | All | All | All | All |
iDRAC settings | Thermal Profile | Default | Maximum Performance | Maximum Performance | Maximum Performance | Maximum Performance | Maximum Performance | Maximum Performance | Maximum Performance |
[*] For C6615, apply setting from Table 3.
[1] Logical Processor (Hyper Threading) tends to benefit throughput-oriented workloads such as SPEC CPU2017. Many HPC workloads disable this option.
Table 3. BIOS setting recommendations specific to C6615 (apply remaining settings from Table 1 and 2)
System setup screen | Setting | BIOS Defaults | SPEC cpu2017 int rate (General Purpose Performance) | SPEC cpu2017 fp rate | SPEC cpu2017 int speed | SPEC cpu2017 fp speed | Memory Throughput | HPC | Latency |
Processor settings | NUMA Node per Socket | 1 | 1 | 1 | 1 | 1 | 2 | 1 | 2 |
System profile setting | CPU Power Management | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM | OS DBPM |
iDRAC recommendations
Following are what we would recommend for an iDRAC environment:
- Thermally challenged environments should increase fan speed through iDRAC Thermal Section.
- All Power Capping should be removed in performance-sensitive environments.
Glossary
System profile: (Default="Performance Per Watt")
To assist the average customer in setting each individual power/performance feature for their specific environment, a menu option is provided that can help a customer optimize the system for factors such as minimum power usage/acoustic levels, maximum efficiency, Energy Star optimization, and maximum performance.
Performance Per Watt OS mode optimizes the performance/watt efficiency with a bias towards performance. It is the favored mode for Energy Star. Note that this mode is slightly different than Performance Per Watt DAPC mode. In this mode, no bus speeds are derated, leaving the OS in charge of making those changes.
Custom allows the user to individually modify any of the low-level settings that are preset and unchangeable in any of the other four preset modes.
C-States
C-states reduce CPU idle power. There are three options in this mode: Legacy, Autonomous, and Disable.
Enabled: When “Enabled” is selected, the operating system initiates the C-state transitions. Some OS SW may defeat the ACPI mapping, such as intel_idle driver.
Autonomous: When "Autonomous" is selected, HALT and C1 requests get converted to C6 requests in hardware.
Disable: When "Disable" is selected, only C0 and C1 are used by the OS. C1 gets enabled automatically when an OS autohalts.
CPU Power Management
CPU Power Management allows the selection of CPU power management methodology. Maximum Performance is typically selected for performance-centric workloads where it is acceptable to consume additional power to achieve the highest possible performance for the computing environment. This mode drives processor frequency to the maximum across all cores (although idled cores can still be frequency-reduced by C-States enforcement through BIOS or OS mechanisms if enabled). This mode also offers the lowest latency of the CPU Power Management Mode options, so it is always preferred for latency-sensitive environments. OS DBPM is another Performance Per Watt option that relies on the operating system to dynamically control individual core frequency. Both Windows and Linux can take advantage of this mode to reduce the frequency of idled or underutilized cores in order to save power. This will be Read-only unless System Profile is set to Custom.
Memory Frequency
Memory Frequency governs the BIOS memory frequency. The variables that govern maximum memory frequency include the maximum rated frequency of the DIMMs, the DIMMs per channel population, the processor choice, and this BIOS option. Additional power savings can be achieved by reducing the memory frequency at the expense of reduced performance. This will be Read-only unless System Profile is set to Custom.
Turbo Boost
Turbo Boost governs the Boost Technology. This feature allows the processor cores to be automatically clocked up in frequency beyond the advertised processor speed. The amount of increased frequency (or 'turbo upside') one can expect from an EPYC processor depends on the processor model, thermal limitations of the operating environment, and in some cases power consumption. In general terms, the fewer cores being exercised with work, the higher the potential turbo upside. The potential drawbacks for Boost are mainly centered on increased power consumption and possible frequency jitter that can affect a small minority of latency-sensitive environments. This will be Read-only unless System Profile is set to Custom.
Memory Patrol Scrub
Memory Patrol Scrubbing searches the memory for errors and repairs correctable errors to prevent the accumulation of memory errors. When set to Disabled, no patrol scrubbing will occur. When set to Standard Mode, the entire memory array will be scrubbed once in a 24-hour period. When set to Extended Mode, the entire memory array will be scrubbed more frequently to further increase system reliability. This will be Read-only unless System Profile is set to Custom.
Memory Refresh Rate
The memory controller will periodically refresh the data in memory. The frequency at which memory is normally refreshed is referred to as 1X refresh rate. When memory modules are operating at a higher-than-normal temperature or to further increase system reliability, the refresh rate can be set to 2X, however this may have a negative impact on memory subsystem performance under certain circumstances. This will be Read-only unless System Profile is set to Custom.
PCI ASPM L1 Link Power Management
When enabled, PCIe Advanced State Power Management (ASPM) can reduce overall system power while slightly reducing system performance.
Note: Some devices may not perform properly (they may hang or cause the system to hang) when ASPM is enabled; for this reason, L1 will only be enabled for validated qualified cards.
Determinism Slider
The Determinism Slider controls whether BIOS will enable determinism to control performance.
Performance Determinism: BIOS will enable 100% deterministic performance control.
Power Determinism: BIOS will not enable deterministic performance control.
Power Profile Select
High Performance Mode (default): Favors core performance. All DF P-States are available in this mode, and the default DF P-State and DLWM algorithms are active.
Efficiency Mode: Configures the system for power efficiency. Limits boost frequency available to cores and restricts DF P-States available in the system. Maximum IO.
Performance Mode: Sets up Data Fabric to maximize IO sub-system performance.
Algorithm Performance Boost Disable (ApbDis)
When enabled, a specific hard-fused Data Fabric (SoC) P-state is forced for optimizing workloads sensitive to latency or throughput. For higher performance and power savings, when disabled, P-states will be automatically managed by the Application Power Management, allowing the processor to provide maximum performance while remaining within a specified power-delivery and thermal envelope.
ApbDis Fixed Socket P-State
This value defines the forced P-state when ApbDis is enabled.
Dynamic Link Width Management (DLWM)
DLWM reduces the XGMI link width between sockets from x16 to x8 (default) when no traffic is detected on the link. As with Data Fabric and Memory P-states, this feature is optimized to trade power between core and high IO/memory bandwidth workloads.
Forced: Force link width to x16, x8, or x2.
Unforced: Link width will be managed by DLWM engine.
System Memory Testing
System Memory Testing indicates whether or not the BIOS system memory tests are conducted during POST. When set to Enabled, memory tests are performed.
Note: Enabling this feature will result in a longer boot time. The extent of the increased time depends on the size of the system memory.
Dram Refresh Delay
By enabling the CPU memory controller to delay running the REFRESH commands, you can improve the performance for some workloads. By minimizing the delay time, it is ensured that the memory controller runs the REFRESH command at regular intervals. For Intel-based servers, this setting only affects systems configured with DIMMs which use 8 Gb density DRAMs.
Correctable Memory ECC SMI
Allows the system to log ECC-corrected DRAM errors into the SEL log. Logging these rare errors can help identify marginal components, however the system will pause for a few milliseconds after an error while the log entry is created. Latency-conscious customers may want to disable the feature. Spare Mode and Mirror mode require this feature to be enabled.
DIMM Self Healing (Post Package Repair) on Uncorrectable Memory Error Enable/Disable Post Package Repair (PPR) on Uncorrectable Memory Error.
Correctable Error Logging
Enable/Disable logging of correctable memory threshold error.
Logical Processor
Each processor core supports up to two logical processors. When set to Enabled, the BIOS reports all logical processors. When set to Disabled, the BIOS only reports one logical processor per core. Generally, a higher processor count results in increased performance for most multi-threaded workloads, and the recommendation is to keep this enabled. However, there are some floating point/scientific workloads, including HPC workloads, where disabling this feature may result in higher performance.
Virtualization Technology
When set to Enabled, the BIOS will enable processor Virtualization features and provide the virtualization support to the Operating System (OS) through the DMAR table. In general, only virtualized environments such as VMware(r) ESX(tm), Microsoft Hyper-V(r), Red Hat(r) KVM, and other virtualized operating systems will take advantage of these features. Disabling this feature is not known to significantly alter the performance or power characteristics of the system, so leaving this option Enabled is advised for most cases.
IOMMU Support
Enable or Disable IOMMU support. Required to create IVRS ACPI Table.
Kernel DMA Protection
When set to Enabled, using IOMMU, BIOS & Operating System will enable direct memory access protection for DMA-capable peripheral devices. Enable IOMMU Support to use this option.
L1 Stream HW Prefetcher
When set to Enabled, the processor provides advanced performance tuning by controlling the L1 stream HW prefetcher setting. Use the recommended setting, and this option will allow for optimizing overall workloads.
L2 Stream HW Prefetcher
When set to Enabled, the processor provides advanced performance tuning by controlling the L2 stream HW prefetcher setting. Use the recommended setting, and this option will allow for optimizing overall workloads.
L1 Stride Prefetcher
When set to Enabled, the processor provides additional fetch to the data access for an individual instruction for performance tuning by controlling the L1 stride prefetcher setting. Use the recommended setting, and this option will allow for optimizing overall workloads.
L1 Region Prefetcher
When set to Enabled, the processor provides additional fetch to data along with the data access to the given instruction for performance tuning by controlling the L1 region prefetcher setting. Use the recommended setting, and this option will allow for optimizing overall workloads.
L2 Up Down Prefetcher
When set to Enabled, the processor uses memory access to determine whether to fetch next or previous for all memory accesses for advanced performance tuning by controlling the L2 up/down prefetcher setting. Use the recommended setting, and this option will allow for optimizing overall workloads.
MADT Core Enumeration
This field determines how BIOS enumerates processor cores in the ACPI MADT table. When set to Round Robin, processor cores are enumerated in a Round Robin order to evenly distribute interrupt controllers for the OS across all Sockets and Dies. When set to Linear, processor cores are enumerated across all Dies within a Socket before enumerating additional Sockets for a linear distribution of interrupt controllers for the OS.
NUMA Nodes Per Socket
This field specifies the number of NUMA nodes per socket. The Zero option is for 2 socket configurations.
L3 cache as NUMA Domain
This field specifies that each CCX within the processor will be declared as a NUMA Domain.
Secure Memory Encryption
This field enables or disables AMD secure encryption features such as Secure Memory Encryption (SME) and Secure Encrypted Virtualization (SEV). In addition to enabling this option, SME must be supported and activated by the operating system. Similarly, SEV must be supported and activated by the hypervisor. This option also determines if other secure encryption feature such as TSME and SEV-SNP features can be enabled.
Minimum SEV non-ES ASID
This field determines the number of Secure Encrypted Virtualization (SEV) Encrypted States (ES) and non-ES available Address Space IDs. The number specified is the dividing line between ES and non-ES ASIDs. The register save state area is also encrypted along with the entire guest memory area. The maximum number of ASIDs available depends on installed CPU and memory configuration which can either be 15, 253, or 509. The default value is 1, and the value entered by user means the number of non-ES ASIDs starts from the value entered and ends at the maximum number of ASIDs available. A value of 1 means there are only non-ES ASIDs available. For example, if the maximum number of ASIDs is 15, the default value 1 means there are 15 SEV non-ES ASIDs and 0 SEV ES ASIDs. Alternatively, if the maximum number of ASIDs is 15, the value 4 means there are 12 SEV non-ES ASIDs and 3 SEV ES ASIDs. Further, if the maximum number of ASIDs is 509, the value 40 means there are 470 SEV non-ES ASIDs and 39 SEV ES ASIDs.
Secure Nested Paging
This option enables or disables SEV-SNP, a set of additional security protections.
SNP Memory Coverage
This option selects the operating mode of the Secure Nested Paging (SNP) Memory and the Reverse Map Table (RMP). The RMP is used to ensure a one-to-one mapping between system physical addresses and guest physical addresses.
Transparent Secure Memory Encryption
This field enables or disables Transparent Secure Memory Encryption (TSME). TSME is always-on memory encryption that does not require operating system or hypervisor support. If the operating system supports SME, this field does not need to be enabled. If the hypervisor supports SEV, this field does not need to be enabled. Enabling TSME affects system memory performance.
ACPI CST C2 Latency
Enter in 18 - 1000 microseconds (decimal value). Larger C2 latency values will reduce the number of C2 transitions and reduce C2 residency. Fewer transitions can help when performance is sensitive to the latency of C2 entry and exit. Higher residency can improve performance by allowing higher frequency boost and reduce idle core power. With Linux kernel 6.0 or later, the C2 transition cost is significantly reduced. The best value will be dependent on kernel version, use case, and workload.
Configurable TDP
Configurable TDP allows the reconfiguration of the processor Thermal Design Power (TDP) levels based on the power and thermal delivery capabilities of the system. TDP refers to the maximum amount of power the cooling system is required to dissipate.
Note: This option is only available on certain SKUs of the processors, and the number of alternative levels varies as well.
x2APIC Mode
Enable or Disable x2APIC mode. Compared to the traditional xAPIC architecture, x2APIC extends processor addressability and enhances interrupt delivery performance.
Number of CCDs per Processor
This field enables the number of CCDs per Processor.
Number of Cores per CCD
This field enables the number of Cores per CCD.
Authors: Charan Soppadandi, Chris Cote, Donald Russell, Kavya AR
13% Better Performance in Financial Trading with PowerEdge R7615 and AMD EPYC 9374F
Wed, 16 Aug 2023 15:41:36 -0000
|Read Time: 0 minutes
Summary
Dell PowerEdge R7615 with 4th Generation AMD EPYC 9374F provides up to a 13 percent performance gain over Dell PowerEdge R7615 with 4th Generation AMD EPYC 9354P for financial trading benchmarks.[1] This Direct to Development (DfD) document looks at CPU benchmarks for three R7615 32-core based CPU configurations and highlights key features that enable businesses enterprises to host different workloads.
Dell PowerEdge R7615
Dell PowerEdge R7615 is a 2U, single-socket rack server. It is designed to be the best investment per dollar for your data center. This server provides performance, and flexible low-latency storage options in an air or Direct Liquid Cooling (DLC) configuration by using an AMD EPYC 4th generation processor to deliver up to 50% more core count per single socket platform in an innovative air-cooled chassis. It delivers breakthrough innovation for traditional and emerging workloads, including software-defined storage, data analytics, and virtualization, using the latest performance and density.
Figure 1. Side angle of the extremely scalable R7615
4th Generation AMD EPYC processors
PowerEdge R7615 is the latest single socket AMD server supporting 4th Generation AMD EPYC 9004 Series processors, the latest generation of the AMD64 System-on-Chip (SoC) processor family. It is based on the Zen 4 microarchitecture introduced in 2022, supporting up to 128 cores (256 threads) and 12 memory channels per socket, a 50% increase over the previous generation. This series includes three different CPU(s) with 32 cores:
Processor | CPU Cores | Threads | Max. Boost Clock | All core boost speed | Base clock | L3 Cache | Default TDP |
AMD EPYC 9374F | 32 | 64 | Up to 4.3GHz | 4.1GHz | 3.85GHz | 256MB | 320W |
AMD EPYC 9354P | 32 | 64 | Up to 3.8GHz | 3.75GHz | 3.25GHz | 256MB | 280W |
AMD EPYC 9334 | 32 | 64 | Up to 3.9GHz | 3.85GHz | 2.7GHz | 128MB | 210W |
The Base Clock, also known as Base Frequency, refers to the minimum operational clock speed of an AMD processor's cores when running under normal conditions. It serves as the foundational clock speed for the processor's overall performance. During tasks that do not require intense processing power, the processor operates at or around this speed, conserving energy and minimizing heat generation.
The Max Boost Clock, often called Max Turbo Frequency or Max Turbo Boost, signifies the upper limit of a processor's clock speed. This clock speed is achieved when specific cores of the AMD processor dynamically increase their frequency to deliver peak performance. The Max Boost Clock is typically applied to a subset of cores and is triggered when the workload demands require a burst of processing power, such as for gaming, video editing, financial trading, and other intensive applications.
The All-Core Boost Speed refers to the clock speed that all cores of an AMD processor can achieve simultaneously when under load. Unlike the Max Boost Clock, which is applicable to only a select number of cores, the All-Core Boost Speed ensures that all cores are operating at an elevated clock speed for optimized multi-threaded performance. This feature is particularly advantageous for tasks that rely heavily on parallel processing, such as rendering, simulations, and content creation.
AMD EPYC 9374F is the frequency/core optimized offering which provides up to a 13 percent increase in all core boost speed over AMD EPYC 9354P, the basic 32 core 1-socket offering. The series also includes AMD EPYC 9334 which has half the L3 Cache but offers over 52 percent drop in Default TDP over AMD EPYC 9374F, making it the most energy efficient of the three CPUs.
Performance data
We captured four benchmarks:
- Sockperf is a network benchmarking tool designed to measure network latency and throughput performance using the Socket Direct Protocol (SDP) for high-performance computing clusters and data centers.
- The QuantLib benchmark is a software library used in quantitative finance and derivatives pricing for modeling and analyzing financial instruments, providing tools for pricing, risk management, and quantitative research. It is widely used by financial professionals and institutions for accurate and efficient financial calculations.
- High Performance Conjugate Gradient measures the computational efficiency of solving a sparse linear system using conjugate gradient methods, providing insights into HPC system performance and optimization. It complements the traditional HPL benchmark, reflecting real-world application characteristics.
- The dav1d benchmark is a performance testing tool used to assess the decoding speed and efficiency of the AV1 video codec, helping to evaluate its real-time playback capabilities and effectiveness in video streaming applications. It aids in optimizing AV1 codec implementations for improved video compression and playback performance.
To compare performance across three R7615 4th Generation AMD EPYC processors, let us first consider the Sockperf benchmark. This benchmark reports throughput in terms of messages per second, the speed at which queries are processed and data is retrieved or stored. It also reports latency overload in usec, measuring the system's response time (latency) under different load conditions.
Figure 1. Three CPU comparison demonstrating Throughput performance using the Sockperf benchmark
Figure 2. CPU comparison showing Latency under Load performance using the Sockperf benchmark
In PowerEdge R7615 with AMD EPYC 9374F, we see up to 14 percent better throughput performance and an 11 percent drop-in time taken for the Latency Under Load subtest to complete using the Sockperf benchmark over AMD EPYC 9354P.
We also report dav1d results in Frames per second. This test measures the time taken to decode AV1 video content and QuantLib results in MFLOPS, a benchmark for quantitative finance for modeling, trading, and risk management scenarios.
Figure 3. A three CPU comparison demonstrating dav1d performance
Figure 4. A three CPU comparison demonstrating performance using the QuantLib benchmark
Performance in PowerEdge R7615 with AMD EPYC 9374F is better for the dav1d and QuantLib benchmarks than for the other tested configurations. We find an up to 8 percent performance uptake for video decoding and an additional 13 percent better performance for financial modelling and trading workloads in Dell Technologies PowerEdge R7615 with the frequency optimized AMD EPYC 9374F.
Conclusion
Some workloads benefit from more cores and some benefit from higher frequency. Here we have shown examples of workloads that take advantage of the higher boost frequencies.
Like most industries, the financial trading industry continues to evolve. Firms are pushing workloads harder and with larger datasets, all while expecting immediate or real-time results. These organizations must be confident that they are investing in the right platforms to support computational requirements. With PowerEdge R7615 with AMD EPYC 9374F, Dell Technologies delivers the systems to address the current and expanding needs for high-performance quantitative trading modelling and risk management scenarios.
References
- Dell PowerEdge R7615 Spec Sheet
- AMD EPYC™ 9374F Processors | AMD
- DDR5 Memory Bandwidth for Next-Generation PowerEdge Servers Featuring 4th Gen AMD EPYC Processors | Dell Technologies Info Hub
[1] Tests were performed in August 2023 at the Solutions and Performance Analysis Lab at Dell Technologies.