16th Generation Dell PowerEdge R760 server with Broadcom NICs: Save money, do more work, and use less energy
See the Science View the InfographicMon, 15 Apr 2024 20:10:10 -0000
|Read Time: 0 minutes
Principled Technologies testing showed that a 16th Generation Dell PowerEdge R760 server featuring a Broadcom BCM57508-P2100G NIC delivered all three vs. previous-generation PowerEdge servers
The last 20 years have seen a dramatic increase in the amount of data businesses generate, analyze, and store. During this period of data proliferation, businesses gained access to a wealth of new sources of data from the rise of the internet, the widespread adoption of smart phones, and the growth of the internet of things (IoT). More powerful processors have made it possible for businesses to analyze this data in new ways using tools such as machine learning (ML) and artificial intelligence (AI). These developments mean many organizations need to add servers to their data centers to keep up with the growth in demand for compute capacity and storage, but it can be difficult to balance the need for higher performance and the challenge of reducing costs. Additionally, energy conservation has become increasingly important as energy costs have risen and businesses have made goals to reduce their carbon footprint to fight climate change.
We tested the performance and power efficiency of three generations of Dell PowerEdge servers to help businesses understand which of these servers to choose to meet the demands of data proliferation, reducing costs, and energy conservation. This report outlines our test methodologies and findings in the context of an increasingly cost-sensitive and conservation-conscious business landscape. We measured the Ops/s and the performance per watt ratio using the Memtier benchmark utility. We ran write and read commands against the popular open-source, in-memory data structure store Redis. Common use cases of Redis include message broker software[1] and real-time streaming analytics [2], among others.
Our testing showed that by choosing the 16th Generation Dell PowerEdge R760 featuring a Broadcom BCM57508-P2100G NIC, organizations can save more money, do more work, and use less energy.
Spend less to do the same amount of work with Dell PowerEdge R760 servers with Broadcom NICs
Figure 1: The Ops/s that the servers processed, the price of the servers in US dollars, and the Ops/s per US dollar of the servers. Higher Ops/s per US dollar is better. Source: Principled Technologies.
Figure 2: The Ops/s that the servers processed, the price of the servers in US dollars, and the Ops/s per US dollar of the servers. Higher Ops/s per US dollar is better. Source: Principled Technologies.
When looking at adding capacity, using the latest servers can boost performance, but as with any business decision, organizations should weigh the costs as well as the benefits associated with purchasing the latest equipment. Perhaps an organization has reasons they may consider equipment that is one or two generations back, say, to match existing equipment. However, this decision can be critical to both purchase and ongoing costs. To understand how much you can reduce costs by using the latest more powerful and more energy efficient servers, we obtained list pricing for the configurations we tested.[3] We configured the servers in these price comparisons as closely as possible to those we tested, but some minor differences were unavoidable because of product availability.
The PowerEdge R760 with a Broadcom NIC processed 2.29 times as many Ops/s as the PowerEdge R740 at a cost that is 13.7 percent lower. As Figure 1 shows, the PowerEdge R760 processed 166.1 percent more Ops/s per Dollar as the PowerEdge R740. In addition to the lower purchase price, the PowerEdge R760 was 24.2 percent more energy efficient in our tests, which could further reduce costs by requiring less energy for operation and cooling.
The PowerEdge R760 with a Broadcom NIC processed 30.6 percent more Ops/s than the PowerEdge R750 at a cost that is only 3.9 percent higher. The cost difference between the PowerEdge R750 and the PowerEdge R760 is relatively small compared to the difference in performance, and as Figure 2 shows, the PowerEdge R760 server offered 25.4 percent more Ops/s per US dollar than the PowerEdge R750.
The impact of these cost, energy efficiency, and performance numbers will vary depending on the needs of a given data center. Let’s examine what the performance and value of these servers would look like in several scenarios.
If an organization needed the additional database processing power of 10 PowerEdge R760 servers, they could meet those needs with any of the servers we tested, but they would spend the least ($318,236.20) if they bought 10 PowerEdge R760 servers. To meet those needs with the PowerEdge R750 servers, they’d spend $399,181.54, and with the PowerEdge R740 servers, they’d pay $847,046.62.
Figure 3: A comparison of how many servers it would take to do the same work as 10 Dell PowerEdge R760 servers and the cost in USD. Source: Principled Technologies.
The savings scale with the size of an organization’s database processing needs: 500 PowerEdge R760 servers would cost $15,911,810.00 and process 32,141,262,500 Ops/s. To do the same amount of work, the organization would need to purchase approximately 653 PowerEdge R750 servers, which would cost $19,956,907.00: over $4 million US dollars more than choosing the PowerEdge R760. Using this same hypothetical, an organization would need to purchase approximately 1,147 PowerEdge R740 servers to do the work of 500 PowerEdge R760 servers, at a cost of $42,319,539.95: over $26 million US dollars more than choosing the PowerEdge R760.
When you consider the higher performance, the savings discussed above, and the lower energy costs of the more efficient PowerEdge R760 with a Broadcom NIC, it becomes clear that organizations will save the most money by choosing the latest-generation Dell PowerEdge R760 server.
Better performance with the Dell PowerEdge R760 server
Figure 4: The Ops/s the servers processed. Higher is better. Source: Principled Technologies.
Figure 5: The throughput of the servers in MB/s. Higher is better. Source: Principled Technologies.
Figure 5: The performance per watt of the servers in Ops/s per watt. Higher is better. Source: Principled Technologies.
Bytes per second and operations per second (Ops/s) are both measures of throughput, or rates. Bytes per second are a system level statistic that typically come from either disk or network. We often refer to this metric in terms of a multiplier, such as megabytes per second (MB/s) or gigabytes per second (GB/s). In the case of our testing, we measured MB/s that passed through the ethernet controller. Ops/s typically refers to an application measure, in this case the operations per second that Redis processed during our memtier tests. Both measures of throughput are important to consider depending on what you are trying to measure. An increase in either number shows improvement in the amount of data (MB/s) or operations (Ops/s) that the server or application processed in a fixed amount of time.
When we compared the throughput of the three Dell PowerEdge servers, we found that the PowerEdge R760 server offered up to 126.9 percent more MB/s throughput and processed up to 129.5 percent more Ops/s than the PowerEdge R740 server. The PowerEdge R760 offered up to 30.8 percent more MB/s throughput and processed up to 30.6 percent more Ops/s than the previous-generation PowerEdge R750 server. Organizations that want the highest performing server should consider the PowerEdge R760, based on these findings.
Save money by getting more performance per watt with the PowerEdge R760 server featuring a Broadcom NIC
To understand the power efficiency of the servers, we measured the work a server did (Ops/s), and divided it by the power the server used (watts). We found that the PowerEdge R760 with a Broadcom NIC offered up to 24.2 percent better operations per second per watt than the 14th Generation Dell PowerEdge R740 server, and up to 9.2 percent better operations per second per watt than the 15th Generation Dell PowerEdge R750 server. Organizations that select these new servers with Broadcom ethernet controllers can reduce the amount of power necessary to process a given unit of work, which will help reduce their energy costs and carbon footprint.
Our hands-on testing of the Dell PowerEdge R760
To help organizations understand which PowerEdge server best meets their needs, we ran a series of tests comparing the performance and power efficiency of three servers that Dell is currently selling:
- 16th Generation Dell PowerEdge R760
- 15th Generation Dell PowerEdge R750
- 14th Generation Dell PowerEdge R740
These results might also be helpful for organizations considering upgrading their older servers to the latest-generation PowerEdge R760.
For testing, we used the database Redis, a widely-used, in-memory, open-source key-value store—a type of NoSQL database that uses pairs of keys and associated values to quickly access information. Companies including Twitter, GitHub, Snapchat, Craigslist, and StackOverflow rely on Redis for their operations.[4]
In the case of traditional relational databases, the architecture is such that the designers of the database must model, or structure, the data in a certain way. When applications request data and the data is not in memory, the database must retrieve the data from disk, causing a small delay. This architecture works well for some applications, but not all. In many types of applications, a developer may need to store data with less structure—perhaps strings, hashes, lists, or arrays. Redis works well for this, as its architecture is based on a key and a value. A key can have multiple values, resulting in databases that are comprised of a long list of keys and their associated values, with the potential for fewer complexities and slowdowns of relational databases. This data design makes Redis useful for web application user session data (such as cookies) and other small elements of data that web server applications need frequently. Caching that data in Redis allows web applications to return data very quickly.
Memtier, the benchmark utility we used, is a utility that runs set (write) and get (read) commands against a Redis database. Those commands simulate different request patterns. According to Redis, the memtier_benchmark is “a high-throughput benchmarking tool” that can “generate various traffic patterns against both Memcached and Redis instances.”[5] Various parameters that the operator can use, such as data size and key-pattern, can alter the traffic pattern that memtier returns. By operating the memtier utility on a client server and targeting a Redis server, we utilized the Redis server’s processor and ethernet controller subsystems heavily.
We configured all three of the servers we tested with Red Hat Enterprise Linux 8.7 and the most recent firmware as of March 14th, 2023 for the PowerEdge R740 and PowerEdge R750, and April 6th, 2023 for the PowerEdge R760. We recorded the operations per second (Ops/s) and throughput (MB/s) during the test and used those readings for our results. We used an external power meter to measure the power draw by the power supply units (PSUs) while the servers were under load and recorded the average power draw for the entire duration of the test. Then, we calculated the power efficiency (Ops/s/watt) of the devices using the results of our testing. We wanted to utilize close to 100 percent of the database server’s CPU capacity to see how the systems performed under stress. We used the memtier_benchmark to create a roughly 100GB database and ran a constant size, 32 byte (default/small), 100 percent read workload to create enough load to reach nearly 100 percent CPU usage. Read-only operations constitute the majority of the workload in applications such as content streaming and data analysis. Additionally, testing with read-only operations provides comparable performance metrics without the additional complexity of write operations.
Dell PowerEdge R760 servers with Broadcom BCM57508-P2100G NICs
The Dell PowerEdge R760 server is a 2U, two-socket server that features a Broadcom BCM57508-P2100G Dual-Port 100GbE PCl Ethernet Controller. Dell designed the PowerEdge R760 to handle “demanding workloads including artificial intelligence, database analytics, and high-density virtualization.”[6] A wide range of businesses can use the PowerEdge R760 for workloads that demand high performance, such as:
- Artificial intelligence
- Virtualized workloads
- Machine learning
- Database analysis
According to Dell, users of the PowerEdge R760 server can:
- “Add up to two 4th generation Intel® Xeon® scalable processors with up to 56 cores for fast and accurate performance
- Accelerate in-memory workloads with up to 32 DDR5 RDIMMS up to 4800 MT/s(1DPC)
- Maximize performance with up to 2x double-wide or 6x single-wide GPUs for finishing workloads faster
- [Use the] new Smart Flow chassis to air-cool some of the highest core count CPUs within current IT infrastructure”[7]
The PowerEdge R760 server includes built-in security features such as secured component verification, signed firmware and drift detection, dynamic system lockdown, and BIOS recovery.[8] The PowerEdge R760 also features intelligent automation features such as OpenManage Enterprise, CloudIQ solutions, and PowerEdge iDRAC. The OpenManage systems management portfolio enables administrators to deploy, configure, monitor, and manage PowerEdge servers, including easily obtaining power data and capping power as necessary. For more information about the Dell PowerEdge R760 server, visit: https://www.dell.com/en-us/shop/productdetailstxn/poweredge-r760.
About the Broadcom BCM57508-P2100G Dual-Port 100GbE PCle 4.0 ethernet controller
A network interface controller (NIC) is a hardware component in a server that connects the server to a network. The NIC translates outgoing data from the server into a format that the network can read and translates incoming data from the network into a format that the server can read. A higher performing NIC can reduce latency, increase throughput, and allow the server to transmit and receive a great volume of data.
The Dell PowerEdge R760 features the Broadcom BCM57508-P2100G Dual-Port 100GbE PCle 4.0 ethernet controller, which supports speeds of up to 200 Gigabits per second. Broadcom designed the BCM57508-P2100G “to build highly-scalable, feature-rich networking solutions in servers for enterprise and cloud-scale networking and storage applications, including high-performance computing, telco, machine learning, storage disaggregation, and data analytics.”[9]
The BCM57508-P2100G features TruFlow™, which Broadcom asserts can “increase VM density up to 50 percent.”[10] The BCM57508-P2100G also features BroadSAFE® technology, “to provide unparalleled platform security” and a “unique set of highly-optimized hardware acceleration engines to enhance network performance and improve server efficiency.”[11]
Efficiency and expansion: Unraveling the challenges of the growing data center market
We’ve shown that the PowerEdge R760 is more energy-efficient and offers better performance than previous-generation PowerEdge servers, but why does that matter? Organizations realize the value of their data and the insights they provide. Despite economic uncertainty, spending on servers has remained consistent with previous years, and many experts expect it to increase.[12] According to one industry analyst, “despite the general tightening of spending by governments and corporations..., organizations are still spending on infrastructure even if they are, in some cases, cutting back on personnel.”[13] Increasingly, to deal with the ever growing importance of their data, businesses are aware that adopting new technologies such as ML and AI is “both a necessity and a race.”[14]
While the new 16th Generation server we tested uses more power, it does so more efficiently—in this example resulting in an approximately 20 percent lower annual cost, based on the following calculations. In our example if we factor in power usage, our tested 16th Generation server used approximately 831W (.831kW), while the 14th Generation server used approximately 450W (.450kW). In an earlier section we established that an organization would need 1,147 PowerEdge R740 servers to do the work of 500 PowerEdge R760 servers. Thus we can calculate the data center power cost per year as follows:
Figure 7: Calculations showing the annual power cost of the servers in US dollars. Lower is better. Source: Principled Technologies.[15]
Sustainability challenges
As new technologies and data proliferation have increased the demands on data center performance and capacity, sustainability concerns have begun to put pressure on companies to increase the efficiency and reduce the carbon footprint of their data centers. By some estimates, the carbon footprint of data centers worldwide equals that of the airline industry.[16] According to S&P Global Market Intelligence, “the data center industry faces pressure from multiple sources— including government, financial markets, and corporate clients—to improve sustainability and reduce carbon emissions.”[17] Over the last few years, businesses have contended with volatile energy prices that have trended higher.[18] For many companies, the energy consumption of their servers is a significant expense.[19]
The sustainability challenges related to data centers are not limited to carbon emissions. Many data centers use large amounts of water to cool their servers. According to the U.S. Department of Energy, an average data center that uses evaporative cooling can consume 3 to 5 million gallons of water per day— similar to the daily capacity of a city of 30,000-50,000 people.[20] Roughly 20 percent of data centers in the U.S. rely on watersheds that are under stress from drought.[21] Choosing more efficient servers can help organizations meet internal and external climate goals and reduce their water usage and carbon footprints.
Managing costs over time
So far in this report we have explained why choosing the PowerEdge R760 to meet new capacity needs can save money and provide better performance, but what about organizations considering upgrading to newer, more efficient servers? In addition to the savings associated with reducing cooling costs, businesses that upgrade older servers may also save money by avoiding some of the other costs associated with delaying their server replacements. Older servers become more expensive to operate as they age. According to analysis by IDC, businesses reported an average annual operating cost of $7,541 per server during the first three years of use; during years 4 to 6, those servers reached a high of an average of $22,511 per server, a 199 percent increase.[22] Upgrading aging servers can reduce the amount of time IT spends on compliance, maintenance, and security, as newer servers often include the most recent manageability and security features. Aging servers may also contribute to slower response times and a diminished customer experience. Research by Deloitte found that “with a 0.1s improvement in site speed…consumers spent almost 10 percent more” on retail sites.[23] Upgrading to the latest-generation server technology can provide a better user experience.
Historically, server manufacturers have improved the performance of their systems in parallel with the growing demand for data processing, but the ever increasing reliance on data processing plus the rapid growth of resource-hungry technologies such as ML and AI have combined to make data center processing and power consumption challenges even more urgent. ML and AI workloads require a large amount of processing power, and therefore can lead to data centers running at higher power densities, which creates more demand on cooling systems. Some analysts estimate the use of ML will grow at a compound annual growth rate of 35.6 percent until 2030.[24] Data centers will need more powerful, more energy-efficient systems to keep up with this growing demand.
According to the International Monetary Fund (IMF), “energy prices surged to near historic highs in recent years amid the pandemic and the war in Ukraine.”[25] In their latest World Economic Outlook forecast, the IMF predicted “that growth will slow from 3.4 percent last year to 2.8 percent this year.”[26] Faced with volatile energy prices and growing economic uncertainty, many organizations are looking for ways to lower computing costs without sacrificing performance. Organizations can potentially gain performance and reduce cooling costs by upgrading their servers to more powerful models that consume less energy under load. Upgrading to more efficient servers can also help businesses meet their climate change goals.
Server power consumption
In the last two decades, server capabilities have increased dramatically, becoming more powerful and more efficient. While businesses have benefited from the advanced analytics and data processing capabilities of these new servers, they have come at a cost due to the energy required to run them. The design of a server and its components determine how much power it will consume during operation.
Thermal design power
Thermal design power (TDP) refers to the maximum theoretical amount of power, measured in watts, a computer chip (typically a processor or peripheral component) uses.
TDP can be a helpful starting point in considering what the total power draw of a server could be or what power supplies the server should be equipped with. Using these wattage approximations and other server energy usage information, data center designers can calculate the approximate British Thermal Units (BTUs) a data center will need for cooling. Because of advancements in technology and performance such as increased core count, the TDP of many processors has doubled in the last few years, making power considerations even more significant.[27]
Key factors in server design that affect power consumption
Multiple factors in server design affect power consumption. In many servers, the central processor uses more power than other components. Based on some models, the processor can consume as much as 32 percent of the server’s power.[28] A study by the Swedish Energy Agency examined data center and server energy consumption and found, “the central processing unit (CPU) is the largest contributor to the total server power consumption, followed by peripheral slots (including network card slot and input and output devices (I/O) devices), conduction losses, memory, motherboard, disk/storage, and cooling fan.”[29]
CPUs are, of course, not the only components that require power. Every component in a server consumes power, from disks to graphics processing units (GPUs) to power supply units (PSUs), and can therefore impact the server’s power efficiency. Adding more disks or GPUs might increase the server’s power consumption, while using fewer disks or no GPUs might decrease it. Similarly, using a higher efficiency power supply can reduce a server’s power consumption—according to one source, by 3 to 4 percent.[30]
The NIC also has an impact on the power consumption of a server. Based on the specifications Broadcom lists for the NICs in the servers we tested, the 100Gb Broadcom BCM957508-P2100G NIC in the PowerEdge R760 typically draws 16.0W and the 25Gb BCM957414-P225p NIC in the PowerEdge R740 typically draws 10.9W with Passive DAC at 55°C ambient temperature.[31] This means that choosing the PowerEdge R760 instead of the PowerEdge R750 would provide a NIC with four times the raw capacity that draws less than 50 percent more power. Based on the specifications Broadcom lists for a 10Gb BCM57712 NIC with the same configuration as the one in the PowerEdge R740 we tested, we estimate that selecting the PowerEdge R760 over the PowerEdge R740 would provide a NIC with ten times the raw capacity that draws less than 80 percent more power.[32]
Data center cooling
Cooling is critical to a data center’s operation because poorly managed heat can lead to “downtime, damage to critical components, and a shorter lifespan for equipment.”[33] A data center’s cooling generally falls into two categories: direct cooling of the servers and cooling the data center floor. Inside the servers, cooling technologies focus on moving the heat that the server generates away from the devices. On the data center floor, cooling technologies focus on maintaining a comfortable—if warm—ambient room temperature.[34]
The most common cooling system data centers use is a computer room air conditioner (CRAC). A CRAC works the same way as a residential air conditioner, but is inside of a server room. The CRAC method is energy inefficient but one of the most affordable methods of data center cooling.[35] A more energy-efficient method of cooling a data center is evaporative cooling. However, this requires a large amount of water, so it is not always the most environmentally friendly option depending on the watershed in the location of the data center.
Dell incorporated several cooling technologies into 16th Generation PowerEdge servers to help organizations meet the demands of reducing energy costs and environmental impact. These include high-performing fans and redesigned heatsinks to improve airflow and thermal management.[36] According to Dell, high airflow “is achieved using new impedance airflow paths, new storage backplane ingredients, and optimized mechanical structures all tuned to provide up to a 15% higher airflow compared to traditional designs.”[37] Software solutions such as OME Power Manager will automatically adjust power usage based on policies that admins can customize. To learn more about the power management benefits of OME, visit https://www.principledtechnologies.com/Dell/OpenManage-Power-Manager-1122.pdf.
Conclusion
Organizations don’t have to choose between saving money, doing more work, and using less energy; the new 16th Generation Dell PowerEdge R760 servers with Broadcom NICs do all three. In our hands-on testing, the new 16th Generation Dell PowerEdge R760 server featuring a Broadcom NIC offered 166.1 percent more Ops/s per US dollar, 129.5 percent more Ops/s, and 24.2 percent better performance per watt than the 14th Generation Dell PowerEdge R740 server. This means the PowerEdge R760 with a Broadcom NIC did over twice as much work at a lower cost with nearly 25 percent better power efficiency than the PowerEdge R740. When compared to the 15th Generation Dell PowerEdge R750, the PowerEdge R760 processed 29.8 percent more Ops/s at a lower cost in terms of Ops/s per US dollar and offered 3.8 percent better performance per watt.
In addition to lowering cooling costs and carbon emissions, choosing the latest-generation servers can help businesses lower other operating costs, such as maintenance and security compliance. Because the latest-generation Dell PowerEdge R760 servers offer more compute capacity, fewer servers can do more work, potentially reducing licensing costs and the physical footprint of a data center. Newer servers often feature the most recent manageability and security features, which can make it easier for businesses to manage their servers and protect their data. By choosing servers with better performance, businesses can reduce the response time of their systems and provide a better user experience.
Increasingly, businesses are under pressure to reduce the energy consumption and cost of their data centers while maintaining or increasing performance. Key web technologies such as Redis can offer businesses a competitive edge, but they also require a large amount of data processing. Businesses can meet these challenges with Dell PowerEdge R760 servers with Broadcom NICs that offer better cost-efficiency, performance, and power-efficiency.
This project was commissioned by Dell Technologies.
September 2023
Principled Technologies is a registered trademark of Principled Technologies, Inc.
All other product names are the trademarks of their respective owners.
[1] Redis, “Low-latency message queue & broker software,” accessed June 20, 2023, https://redis.com/solutions/use-cases/messaging/.
[2] Redis, “What is Redis?,” accessed June 20, 2023, https://aws.amazon.com/redis/.
[3] Prices obtained from Dell Marketing, US only. We used list pricing because it was the only pricing readily available for all three servers and because discounts vary.
[4] Redis, “Low-latency message queue & broker software.”
[5] Redis, “memtier_benchmark: A High-Throughput Benchmarking Tool for Redis and Memcached,” accessed June 15, 2023, https://redis.com/blog/memtier_benchmark-a-high-throughput-benchmarking-tool-for-redis-memcached/.
[6] Dell, “PowerEdge R760 Rack Server,” accessed June 7, 2023, https://www.dell.com/en-us/shop/productdetailstxn/poweredge-r760.
[7] Dell, “PowerEdge R760 Rack Server.”
[8] Dell.
[9] Broadcom, “BCM57508 – 200GbE,” accessed June 5, 2023, https://www.broadcom.com/products/ethernet-connectivity/network-adapters/bcm57508-200g-ic.
[10] Broadcom, “BCM57508 – 200GbE.”
[11] Broadcom.
[12] Timothy Prickett Morgan, “The Interesting Years Ahead for Servers,” accessed June 11, 2023, https://www.nextplatform.com/2023/01/04/the-interesting-years-ahead-for-servers/.
[13] Timothy Prickett Morgan, “The Interesting Years Ahead for Servers.”
[14] OliverWyman, “Machine Learning: Competitive Advantage and Strategic Imperative,” accessed June 11, 2023, https://www.oliverwyman.com/our-expertise/insights/2016/nov/Machine-Learning-Competitive-Advantage-and-Strategic-Imperative.html.
[15] Mary Zhang, “United States Data Centers: Top 10 Locations,” accessed July 27, 2023, https://dgtlinfra.com/united-states-data-centers/.
[16] S&P Global Market Intelligence, “Sustainability is no longer a ‘nice to have’ goal for the data center industry,” accessed June 14, 2023, https://www.spglobal.com/marketintelligence/en/news-insights/research/sustainability-is-no-longer-a-nice-to-have-goal-for-the-data-center-industry.
[17] S&P Global Market Intelligence, “Sustainability is no longer a ‘nice to have’ goal for the data center industry.”
[18] Lori Aniti, “Wholesale U.S. electricity prices were volatile in 2022,” accessed June 14, 2023, https://www.eia.gov/todayinenergy/detail.php?id=55139.
[19] InfoQ, “The Problem of Power Consumption in Servers,” accessed June 14, 2023, https://www.infoq.com/articles/power-consumption-servers/.
[20] Marcus Law, “Energy efficiency predictions for data centres in 2023,” accessed June 14, 2023, https://datacentremagazine.com/articles/efficiency-to-loom-large-for-data-centre-industry-in-2023.
[21] Michael Copley, “Data centers, backbone of the digital economy, face water scarcity and climate risk,” accessed June 14, 2023, https://www.npr.org/2022/08/30/1119938708/data-centers-backbone-of-the-digital-economy-face-water-scarcity-and-climate-ris.
[22] IDC, “Optimizing Performance with Frequent Server Replacements for Enterprise,” accessed June 12, 2023, https://www.delltechnologies.com/asset/en-us/products/servers/industry-market/server-infrastructure-resiliency-enterprise-whitepaper.pdf.
[23] Deloitte, “Milliseconds make Millions,” accessed June 12, 2023, https://www2.deloitte.com/content/dam/Deloitte/ie/Documents/Consulting/Milliseconds_Make_Millions_report.pdf.
[24] GVR, “Machine Learning Industry Data Book,” accessed June 14, 2023, https://www.grandviewresearch.com/sector-report/machine-learning-industry-data-book.
[25] Adil Mohommad, Mehdi Raissi, Kyuho Lee, Chapheng Fizzarotti, “IMF Blog: Volatile Commodity Prices Reduce Growth and Amplify Swings in Inflation,” accessed May 17, 2023, https://www.imf.org/en/Blogs/Articles/2023/03/28/volatile-commodity-prices-reduce-growth-and-amplify-swings-in-inflation.
[26] Mehdi Benatiya Andaloussi, Chiara Maggi, “IMF Blog: World Economic Outlook Shows Economies Facing High Uncertainty,” accessed June 21, 2023, https://www.imf.org/en/Blogs/Articles/2023/04/19/world-economic-outlook-shows-economies-facing-high-uncertainty.
[27] Robert Curtis et al, The Future of Server Cooling,” accessed June 8, 2023, https://infohub.delltechnologies.com/p/the-future-of-server-cooling-part-2-new-it-hardware-features-and-power-trends-1/.
[28] Kazi Main Uddin Ahmed, Manuel Alvarez, Math Bollen, “A Review of Data Centers Energy Consumption And Reliability Modeling,” accessed June 14, 2023, https://www.researchgate.net/publication/355862079_A_Review_of_Data_Centers_Energy_Consumption_And_Reliability_Modeling#pf8.
[29] Kazi Main Uddin Ahmed, Manuel Alvarez, Math Bollen, “A Review of Data Centers Energy Consumption And Reliability Modeling.”
[30] Patrick Kennedy, “Deep Dive into Lowering Server Power Consumption,” accessed June 8, 2023, https://www.servethehome.com/deep-dive-into-lowering-server-power-consumption-intel-inspur-hpe-dell-emc/.
[31] Broadcom, “Broadcom Eternet NICs,” accessed July 25, 2023, https://docs.broadcom.com/doc/PCIe-NIC-Ethernet-Adapters-Specification-Sheet.
[32] Allied Telesis, “Broadcom NetXtreme II® Network Adapter User Guide,” accessed July 25, 2023, https://www.alliedtelesis.com/sites/default/files/documents/installation-guides/brcm_nx2_server_t7-10.pdf.
[33] Datacenters.com, “Data Center Cooling: Future of Cooling Systems, Methods and Technologies,” accessed June 6, 2023, https://www.datacenters.com/news/data-center-cooling-future-of-cooling-systems-methods-and-technologies.
[34] Brien Posey, “Data center temperature and humidity guidelines,” accessed June 20, 2023, https://www.techtarget.com/searchdatacenter/tip/Data-center-temperature-and-humidity-guidelines.
[35] LDP Associates, “What are the Different Types of Data Center Cooling?,” accessed June 21, 2023, https://www.ldpassociates.com/what-are-the-different-types-of-data-center-cooling/.
[36] Matt Ogle and Hasnain Shabbir, “Multi Vector Cooling 2.0 for Next-Generation PowerEdge Servers,” accessed September 8, 2023, https://infohub.delltechnologies.com/p/multi-vector-cooling-2-0-for-next-generation-poweredge-servers/.
[37] Dell, “The Future of Server Cooling - Part 2: New IT hardware Features and Power Trends,” accessed July 25, 2023, https://infohub.delltechnologies.com/p/the-future-of-server-cooling-part-2-new-it-hardware-features-and-power-trends-1/.
Author: Principled Technologies
Related Documents
Improve performance by easily migrating to a modern OpenShift environment on PowerEdge R7615 servers
Tue, 14 May 2024 20:15:19 -0000
|Read Time: 0 minutes
Improve performance and gain room to grow by easily migrating to a modern OpenShift environment on Dell PowerEdge R7615 servers with 4th Generation AMD EPYC processors and high-speed 100GbE Broadcom NICs
We deployed this modern environment, then migrated database VMs from legacy servers and saw performance improvements that support consolidation.
Transactional databases are the backbone of many business operations, powering ecommerce and order fulfillment, human resources and payroll, and a host of other activities. If your company is running these kinds of workloads on server infrastructure that is several years old, you might believe that performance is adequate and that you have little reason to consider upgrading to new servers with modern processors, networking, and a Red Hat® OpenShift® container-based environment. In fact, by continuing to use this older gear, you could be incurring higher than necessary operating expenditures by maintaining and powering more servers than you need to perform a given volume of work. You could also be risking downtime with aging hardware that is likelier to break down. By upgrading to a modern environment, you could mitigate these issues and future-proof your infrastructure. A 2019 Forrester Consulting report recommended that organizations refresh their servers at least every three years to maximize agility and productivity.[1] The report states not only that modern servers allow organizations to adopt more emerging technologies at a faster rate, but also “modern hardware has a profound impact on business benefits such as better customer experience, employee productivity, and innovation.”[2]
We explored the process of migrating VMs from a legacy environment and conducted testing to quantify the resulting improvements in network and database performance. We started with a legacy environment consisting of MySQL™ virtual machines (VMs) running on a cluster of three Dell™ PowerEdge™ R7515 servers with 3rd Generation AMD EPYC™ processors and 25Gb Broadcom® NICs. We then deployed a modern OpenShift container-based environment comprising three Dell PowerEdge R7615 servers with 4th Generation AMD EPYC processors and high-speed 100Gb Broadcom NICs. While the primary application of OpenShift is typically for containerized workloads, we used OpenShift Virtualization, which presents a familiar VM layer to administrators while utilizing the containerized technology on the underlying layer. Both environments used a Dell PowerStore 1200T for external storage that the servers accessed using iSCSI. We measured database performance using the HammerDB TPROC-C benchmark.
We found that the modern cluster environment of Dell PowerEdge R7615 servers with 4th Generation AMD EPYC processors and high-speed 100Gb Broadcom NICs outperformed the legacy cluster environment, delivering 44 percent greater database performance. These improvements mean that companies that upgrade can enjoy savings by meeting their workload requirements with fewer servers to license, maintain, power, and cool. Selecting 100Gb Broadcom NICs also positions companies well to take advantage of increasingly popular network-intensive technologies such as artificial intelligence (AI).
The benefits of containerization and Red Hat OpenShift Virtualization
Many organizations choose containers for DevOps due to their easy scalability and portability. Because a container encapsulates an application as well as everything necessary to run that application, it’s simple to move the container from development to test and production environments, adding instances of the application by replicating the container. Containers can also be useful for microservices, data streaming, and other use cases.[3]
Containers aren’t necessarily ideal for every use case, however, and for some infrastructures, IT teams may wish to incorporate both containers and VMs. Red Hat OpenShift Virtualization, which we used in our testing, enables organizations to run both VMs and containers on the same platform by bringing VMs into containers.[4] This lets IT reap the benefits of both containers and VMs with the efficiency benefit of relying on one management tool, rather than having to maintain two distinct infrastructures.
About our testing
We explored the process of deploying a modern data center environment and migrating VMs to it from a legacy environment. We also measured the database performance the VMs achieved in both environments:
Legacy environment
- Three Dell PowerEdge R7515 servers with 3rd Generation AMD EPYC 7663 56-core processors and Broadcom Advanced Dual 25Gb Ethernet NICs
- External storage using Dell PowerStore 1200T over iSCSI
- VMware® vSphere® 8
Modern environment
- Three Dell PowerEdge R7615 servers with 4th Generation AMD EPYC 9554 64-core processors and Broadcom NetExtreme-E BCM57508 100GB NICs
- External storage using Dell PowerStore 1200T over iSCSI
- Red Hat OpenShift 4.14
Figure 1 presents a diagram of our test configuration. In addition to our test server clusters, we needed three servers to host infrastructure VMs, workload client VMs, and the OpenShift control node VMs. We configured a Dell PowerEdge R7525 to serve as the host for our infrastructure VMs for services such as AD, DHCP, and DNS, as well as HammerDB client VMs. We also configured a Dell PowerEdge R7625 to host additional HammerDB client VMs. For the OpenShift environment, we deployed a Dell PowerEdge R540 to host the OCP control nodes. We virtualized the control nodes to reduce the number of servers needed for the test bed.
Figure 1: Our test configuration. Source: Principled Technologies.
To test the MySQL database performance of each environment, we used the TPROC-C workload from the HammerDB benchmark. HammerDB developers derived their OLTP workload from the TPC-C benchmark specifications; however, as this is not a full implementation of the official TPC-C standards, the results in this paper are not directly comparable to published TPC-C results. For more information, please visit https://www.hammerdb.com/docs/ch03s01.html.
Each VM had a single MySQL instance with a TPROC-C database. We targeted the maximum transactions per minute (TPM) each environment could achieve by increasing the user count until performance degraded.
What we learned
Finding 1: Deploying OpenShift in the modern environment was easy
For our environment, the OpenShift installation process using the Red Hat Assisted Installer to install an OpenShift Installer-Provisioned Cluster was straightforward and simple. We started by setting up the prerequisites for the environment, which included a VM for Active Directory, DNS, and DHCP. We created a domain for our private network and added the API and ingress routes as DNS A records. Next, we set up a VM as a router so that our OpenShift environment could access the internet from our private network. Finally, we created three blank VMs to serve as our OpenShift controller nodes. Once we had met the pre-requisite requirements, we logged into the Red Hat Hybrid Console and navigated to the Assisted Installer to create the cluster.
The Assisted Installer streamlined the process by walking us through configuration menus for storage, network, and access to the cluster. We started the cluster creation by assigning it a name, providing the domain, and selecting an OpenShift version. From there the installer guided us through the process of providing an installer image using the SSH public key of the server running the installer. After downloading the ISO, we booted each of the controller and worker nodes into the image and the Assisted Installer discovered each node. After discovering the controller and worker nodes, the installer walked us through the rest of the configuration process and then began the installation. The Assisted Installer made the process very simple with only six configuration tabs to advance through, and with our total install time after configuration taking around three hours. Once the installation was complete, each node rebooted into the OpenShift OS and the Assisted Installer provided us with a cluster console fully qualified domain name (FQDN) to connect to and manage the cluster from. For detailed steps on the OpenShift deployment process, see the science behind the report.
Finding 2: Migrating VMs from the legacy VMware environment to the modern OpenShift environment was easy
Migrating a VM from the VMware environment to OpenShift was also a straightforward process and quick to set up. While the actual migration time will vary depending on VM size and hardware speed, the setup consists of only a few steps and took us less than 10 minutes. We first installed the Migration Toolkit for Virtualization from the OpenShift OperatorHub. We then entered the IP address and credentials for the vCenter as a new provider. Next, we created a NetworkMap and a StorageMap to connect the respective resources between the environments. We then created a new migration plan to map the VMs to a namespace in OCP. We ran the migration plan on a single VM, and confirmed that we were able to enter the VM console once the migration was complete. For detailed steps on the process of migrating VMs from the legacy environment to the modern environment, see the science behind the report.
About 4th Gen AMD EPYC 9554 processors
According to AMD, EPYC 9554 processors deliver fast performance “for cloud, enterprise, and HPC workloads—helping accelerate your business.”[5] EPYC processors include AMD Infinity Guard, which per AMD is “a set of layered, cutting-edge security features that help you protect sensitive data and avoid the costly downtime cause by security breaches.”[6]
In addition to performance and security features, AMD claims their processors are energy-efficient, which can reduce energy costs and “minimize environmental impacts from data center operations while advancing your company’s sustainability objectives.”[7]
When comparing SPECCPU Floating Point peak rates and the default thermal design power (TDP) of the AMD EPYC 9554 and the AMD EPYC 7663, the 9554 has 54 percent better performance per watt, which demonstrates the improved power efficiency with the new 4th Gen AMD EPYC process.[8],[9]
For more information about 4th Gen AMD EPYC processors visit: https://www.amd.com/en/processors/epyc-server-cpu-family.
Finding 3: Database performance improved by 44 percent in the new environment
Figure 2 shows the results of our database performance testing using the TPROC-C workload from the HammerDB benchmark suite. The modern OpenShift cluster of Dell PowerEdge R7615 servers outperformed the legacy cluster by 44 percent. This extra capability could benefit companies upgrading to the new environment in several ways. The company could provide a better user experience, perform more work—or support more users—with a given number of servers, or reduce the number of servers necessary to execute a given workload.
Figure 2: Performance in transactions per minute using the TPROC-C workload of the HammerDB benchmark suite. Higher is better. Source: Principled Technologies.
Finding 4: Performance improved in the modern cluster, supporting consolidation, which leads to savings
Based on the results of our performance tests (see Figure 3), a company could consolidate the database workloads of a four-node Dell PowerEdge 7515 cluster with some additional headroom into three modern Dell PowerEdge R7615 servers with 4th Generation AMD EPYC processors and high-speed 100Gb Broadcom NICs.
The cluster of three modern servers delivered a total of 9,674,180 transactions per minute (3,224,726 TPMs per server). The cluster of three legacy servers delivered a total of 6,714,712 TPM (2,238,237 per server). Based on these results, four legacy servers would achieve a total of 8,952,948 TPM, which would leave 721,231 additional TPM room for growth on the modern three-node cluster.
Reducing the number of servers you need means that operational expenditures such as data center power and cooling and administrator time for maintenance also decrease, leading to ongoing savings.
Figure 3: Performance in transactions per minute that three modern servers and four legacy servers could achieve, based on our hands-on testing. Higher is better. Source: Principled Technologies.
About Dell PowerEdge R7615 servers
The Dell PowerEdge R7615 is a 2U, single-socket rack server. Dell states that it has designed this server to provide “performance and flexible, low-latency storage options in an air or Direct Liquid Cooling (DLC) configuration.”[10]
According to Dell, this server uses the AMD EPYC 4th generation processor to deliver up to 50 percent higher core count per single-socket platform in an innovative air-cooled chassis.[11] It also supports DDR5 at 4800 MT/s memory and PCIe® Gen5 with double the speed of previous Gen4 for faster access and transport of data, optimizing application output.[12] It supports up to six single-wide full-length GPUs or three double-wide full-length GPUs, to improve responsiveness or reduce app load time for power users, plus lower-latency, high-performance NVMe SSDs to help maximize compute performance.[13]
Learn more at https://www.delltechnologies.com/asset/en-us/products/servers/technicalsupport/poweredge-r7615-spec-sheet.pdf.
How high-speed 100Gb Broadcom NICs can help your organization
Even if a 25Gb NIC is sufficient to meet a company’s current networking needs, opting to equip new servers with the high-speed 100Gb Broadcom NIC can be a smart move. Future-proofing your network can allow you to meet the increasing demands of emerging technologies.
Advanced technologies such as artificial intelligence and machine learning, which can require the processing and transmission of large amounts of data, are becoming increasingly prevalent across businesses of all sizes. In a June 2023 survey of small business decision-makers, 74 percent were interested in using AI or automation in their business and 55 percent said their interest in these technologies had grown in the first half of 2023.[14] Upgrading to a modern environment with a highspeed 100Gb Broadcom NIC positions companies to take advantage of AI applications for social media, content creation, marketing, customer support, and many other use cases.
Another way that investing in the high-speed 100Gb Broadcom NIC can help your company is through improved efficiency. You might be tempted to go with a 25Gb NIC, thinking that as your networking needs increase, you can simply add more NICs of this size. However, consider a 2023 Principled Technologies study that compared the performance of a server solution with a 100Gb Broadcom 57508 NIC and a solution with four 25Gb NICs.[15] Testing revealed that the 100Gb NIC solution achieved up to 2.3 times the throughput of the solution with 25Gb NICs. It also delivered greater bandwidth consistency, which can translate to providing a better user experience; the report states that applications using the 25Gb NICs network configuration “would experience significant variation in available bandwidth, potentially causing jittery or interrupted service to multiple streams.”[16]
About the Broadcom BCM57508-P2100G Dual-Port 100GbE PCle 4.0 ethernet controller
A higher performing NIC can reduce latency, increase throughput, and allow the server to transmit and receive a great volume of data. The Dell PowerEdge R7615 we tested features the Broadcom BCM57508-P2100G DualPort 100GbE PCle 4.0 ethernet controller, which supports speeds of up to 200 Gigabits per second. Broadcom designed the BCM57508-P2100G “to build highlyscalable, feature-rich networking solutions in servers for enterprise and cloud-scale networking and storage applications, including high-performance computing, telco, machine learning, storage disaggregation, and data analytics.”[17]
The BCM57508-P2100G features BroadSAFE® technology, “to provide unparalleled platform security” and a “unique set of highly-optimized hardware acceleration engines to enhance network performance and improve server efficiency.”[18]
BCM57508-P2100G Dual-Port 100GbE PCle 4.0 ethernet controller. Image provided by Dell.
Conclusion
If your organization’s transactional databases are running on gear that is several years old, you have much to gain by upgrading to modern servers with new processors and networking components and an OpenShift environment. In our testing, a modern OpenShift environment with a cluster of three Dell PowerEdge R7615 servers with 4th Generation AMD EPYC processors and high-speed 100Gb Broadcom NICs outperformed a legacy environment with MySQL VMs running on a cluster of three Dell PowerEdge R7515 servers with 3rd Generation AMD EPYC processors and 25Gb Broadcom NICs. We also easily migrated a VM from the legacy environment to the modern environment, with only a few steps required to set up and less than ten minutes of hands-on time. The performance advantage of the modern servers would allow a company to reduce the number of servers necessary to perform a given amount of database work, thus lowering operational expenditures such as power and cooling and IT staff time for maintenance. The high-speed 100Gb Broadcom NICs in this solution also give companies better network performance and networking capacity to grow as they embrace emerging technologies such as AI that put great demands on networks.
This project was commissioned by Dell Technologies.
May 2024
Principled Technologies is a registered trademark of Principled Technologies, Inc.
All other product names are the trademarks of their respective owners.
Read the report on the PT site at https://facts.pt/2V6p3FG and see the science at https://facts.pt/Dj53ZJb.
Author: Principled Technologies
[1] Forrester, “Why Faster Refresh Cycles and Modern Infrastructure Management are Critical to Business Success,” accessed May 1, 2024, www.techrepublic.com/resource-library/casestudies/forrester-why-faster-refresh-cycles-and-modern-infrastructure-management-are-critical-to-business-success/.
[2] Forrester, “Why Faster Refresh Cycles and Modern Infrastructure Management are Critical to Business Success,” accessed May 1, 2024, www.techrepublic.com/resource-library/casestudies/forrester-why-faster-refresh-cycles-and-modern-infrastructure-management-are-critical-to-business-success/.
[3] Red Hat, “Understanding containers,” accessed April 12, 2024, https://www.redhat.com/en/topics/containers.
[4] Red Hat, “Red Hat OpenShift Virtualization,” accessed April 12, 2024,
https://www.redhat.com/en/technologies/cloud-computing/openshift/virtualization.
[5] AMD, “AMD EPYC Processors,” accessed April 12, 2024, https://www.amd.com/en/processors/epyc-server-cpu-Family.
[6] AMD, “AMD EPYC Processors.”
[7] AMD, “AMD EPYC Processors.”
[8] SPEC, “SPEC CPU®2017 Floating Point Rate Result for Dell PowerEdge R6615 (AMD EPYC 9554 64-Core Processor),” accessed May 2, 2024, https://www.spec.org/cpu2017/results/res2024q1/cpu2017-20240212-41481.html.
[9] SPEC, “SPEC CPU®2017 Floating Point Rate Result for Dell PowerEdge R6515 (AMD EPYC 7663 56-Core Processor),” accessed May 2, 2024, https://www.spec.org/cpu2017/results/res2021q3/cpu2017-20210913-29288.html.
[10] Dell, “PowerEdge R7615 Specification Sheet,” accessed April 12, 2024, https://www.delltechnologies.com/asset/en-us/products/servers/technical-support/poweredge-r7615-spec-sheet.pdf.
[11] Dell, “PowerEdge R7615 Specification Sheet.”
[12] Dell, “PowerEdge R7615 Specification Sheet.”
[13] Dell, “PowerEdge R7615 Specification Sheet.”
[14] Constant Contact, “AI Stats and Trends Small Businesses Need to Know Now,” accessed April 12, 2024, https://news.constantcontact.com/small-business-now-ai-2023.
[15] Principled Technologies, “Opt for modern 100Gb Broadcom 57508 NICs in your
Dell PowerEdge R750 servers for improved networking performance,” accessed April 12, 2024,
https://www.principledtechnologies.com/Dell/PowerEdge-R750-networking-iPerf-1023.pdf.
[16] Principled Technologies, “Opt for modern 100Gb Broadcom 57508 NICs in your
Dell PowerEdge R750 servers for improved networking performance,” accessed April 12, 2024,
https://www.principledtechnologies.com/Dell/PowerEdge-R750-networking-iPerf-1023.pdf.
[17] Broadcom, “BCM57508 – 200GbE,” accessed April 12, 2024,
https://www.broadcom.com/products/ethernet-connectivity/network-adapters/bcm57508-200g-ic.
[18] Broadcom, “BCM57508 – 200GbE.”
How Dell and Broadcom can help you make the transition to IPv6
Thu, 14 Mar 2024 16:56:04 -0000
|Read Time: 0 minutes
IPv4 vs. IPv6: How we got here
As the internet grew and commercialized late last century, it became increasingly clear that Internet Protocol version 4 (IPv4) limitations would eventually present issues. Enter Internet Protocol version 6 (IPv6) in the 1990s. Despite the technology’s age, its adoption has been slow in the US. Until very recently, many companies and other entities still primarily used IPv4, as shown by IPv6 adoption trackers such as one from Google.[1] Recently, however, the transition to IPv6 has been ramping, as some of those IPv4 limitations—such as the dwindling pool of available IPv4 addresses—are quickly becoming reality. One effort to encourage this transition includes a 2020 mandate from the U.S. Office of Management and Budget (OMB) requiring federal government agency devices be at least 80 percent IPv6-only by 2025.[2]
Regardless of mandates or address pools, there are a host of other reasons to choose IPv6 over IPv4. IPv6 includes features such as support for larger packets and multicasting, simpler header formats, smaller routing tables, and the elimination of the network address translation (NAT) process—all of which can increase performance over IPv4 in certain use cases. IPv6 also has built-in end-to-end encryption and name resolution protocol enhancements that contribute to better base security than IPv4.
Despite the advantages of using IPv6, some companies have resisted transitioning because it’s not a small undertaking. In this paper, we explain why making the transition can be worth the investment and introduce a solution to help make the transition easier: Dell and Broadcom™ combine to have one of the first IPv6-only compliant end- to-end solutions. We also present the results of our testing, including performance advantages for IPv6 over IPv4 on read workloads and larger performance increases for IPv4 and IPv6 available by enabling the Offload feature in the Broadcom network interface cards (NIC).
Why organizations are shifting to IPv6
The transition to IPv6 has been a long, slow process that is complicated by the fact that IPv4 and IPv6 are not compatible, requiring companies to either choose just one or manage two networks via dual stack. For many companies, however, fully abandoning IPv4 is not an option. Doubling the number of networks you deploy means doubling the security concerns and hardware expenses. Additionally, applications built on IPv4 may need rebuilding or updating to work with IPv6. Despite these factors, we believe these complications to be worth the benefits you gain from taking advantage of the IPv6 landscape. Not only could organizations use the features we mention above, but as more companies and users move to IPv6, it will also be easier for others to follow. Thus, IPv6 will grow more valuable over time. In time, we hope this can lead to companies being able to shed their IPv4 network, leaving the single, more efficient IPv6 network in place.
Below, we detail some of the reasons to transition to IPv6 including current issues with IPv4, specific industries or government agencies with particular IPv6 requirements, and the benefits of IPv6.
The shortage of IPv4 addresses
One of the most important reasons for the push to transition to IPv6 is the limited number of possible IPv4 addresses. The IPv4 address space is a 32-bit field, meaning there are a total of 232, or roughly 4.3 billion, possible IPv4 addresses. As of November 2019, this address space was officially depleted, meaning there are no new IPv4 addresses to obtain.[3] This has created a premium on IPv4 addresses, allowing companies to sell or lease their addresses, increasing the costs of buying an IPv4 address. Amazon Web Services (AWS), for example, is adding a charge to every IPv4 address on its platform, citing a 300 percent cost increase of IPv4 addresses over the past 5 years.[4] Several workarounds for this problem exist, such as NAT, which allows organizations to map several private addresses within a local network to a single public address before transferring information to the internet. However, NAT comes with its own share of problems that can affect the performance and reliability of network applications. By adding an extra layer of translation and processing, NAT can introduce latency, errors, or packet loss.[5]
IPv6, by utilizing a 128-bit address field, increases available IP addresses to roughly 2128, or ~3.4*1028, essentially solving the address limitation for the foreseeable future. Companies that work with the Internet of Things (IoT), virtual reality (VR), self-driving vehicles, telecom, and other technologies requiring many IP addresses could avoid the IPv4 address market and limited address availability by moving to IPv6.
The federal government gets involved
US federal government agencies also find themselves impacted by the Office of Management and Budget (OMB) mandate, which claims that “full transition to IPv6 is the only viable option to ensure future growth and innovation in Internet technology and services.”[6] The latest version of the mandate states that running dual stack IPv4 and IPv6 networks, as previous versions of the mandate dictated, is too complex and no longer necessary. Instead, this new mandate requires IPv6-only networking, outlining four actions agencies must take:
- Create an IPv6 project team.
- Create and publish an agency-wide policy that states their intentions to phase out all IPv4 use and make all federal IT systems IPv6 enabled by the end of 2023.
- Identify and test at least one IPv6 pilot by the end of 2021.
- Develop a plan by 2021 for implementing IPv6-only networking, with milestones including at least 50 percent of IP-enabled assets, transitioned to IPv6-only by the end of 2023, and 80 percent on IPv6-only networks by the end of 2025.[7]
This means that by the end of the first quarter of 2024, federal agencies should already have half of their systems converted to IPv6-only, and the rest fully transitioned in just two more years.
Telecom and ISP industries are leading the way
With the development of the 5G cellular network and its need for high speeds and low latencies, much of the telecom industry has already converted to IPv6. As more and more devices connect over cellular networks with 5G, the increased address pool of IPv6 provides another benefit for internet service providers (ISPs) and cellular network companies. Additionally, the built-in quality of service (QoS) field in the IPv6 header allows ISPs to prioritize voice traffic over other traffic less vulnerable to latency such as http, SSH, and more. According to Akamai, a content delivery network and cloud computing company, ISPs and telecoms such as Comcast Cable, Verizon Business, AT&T, and T-Mobile have all reached IPv6 adoption above 70 percent—T-Mobile has as much as 92.7 percent IPv6 adoption.[8]
As these network providers continue to expand the IPv6 backbone, and mobile app developers continue to embrace IPv6 advantages, the rest of the world’s industries will lag behind if they continue to rely on IPv4.
How IPv6 helps your business
Even if you don’t need a wide range of IP addresses or aren’t part of the government mandate or the telecom industry, there are still benefits you can see from transitioning from IPv4 to IPv6. First, IPv6 could increase performance in several ways, mostly by increasing network efficiency. Second, IPv6 can offer some additional security benefits over IPv4. While IPv4 has had more security upgrades and patches to existing networks—simply by virtue of existing longer—the features that come with IPv6 offer stronger baseline security, which we examine in the following pages. Increased use and investment in IPv6 security enhancements should quickly close any existing gap between it and current IPv4 network security.
IPv6 can also improve network performance over IPv4 by using a more simplified header that takes less time and fewer resources to process. All IP packets include headers that contain the necessary information for proper route allocation and delivery. Much like the parts of a physical address tell postal workers the house, street, city, and country the letter originated from and is destined for, the IP header includes such information as the IP addresses of the source and destination devices, the version indicator, the total length of the packet, and other important information. Instead of requiring routers to perform a header checksum to ensure data integrity of every packet, IPv6 relies on Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and other existing protocol checksums. Additionally, this header allocates optional and non-essential header fields to header extensions, leaving only the most vital information in the header for processing. This practice increases the efficiency of processing data packets across the network by getting the metaphorical “letter” to the right “building” quickly and letting the extension headers then direct it more specifically, to the equivalent of the correct suite, floor, or person.
IPv6 also eliminates the need for NATs as each device on an IPv6 network can have its own unique IP address. If devices aren’t having to share IP addresses, then the routers do not need to translate the network addresses to send the packet to the correct device, thus eliminating a step in the data transmission process. Finally, while the Maximum Transmission Unit (MTU) technology limits packet sizes in many networks to 1,500 bytes, IPv6 networks are capable of much larger packet sizes, up to 4 GB. IPv4 packets, on the other hand, are limited to much smaller theoretical maximums of 64K bytes.[9]
Another way that IPv6 increases network efficiency is with its ability to multicast instead of broadcast. On IPv4, data transmission is broadcast: When a packet leaves a source device, the information is sent to every host connected to the network. Every single host, in turn, checks the packet to see if the data is meant for it. IPv4 is like a kid yelling out, “Mom!” at a crowded playground, causing every mother in the vicinity to stop and check if their child is the one in need. IPv6, on the other hand, uses multicast, which is the ability to transmit a packet only to the device or devices for which it is intended. Now, instead of a child yelling, “Mom!” into a crowd, IPv6 taps their parent on the shoulder and talks directly to them, allowing the other moms to focus on their own children.
IPv4 routing tables, the list of networks and other links a router consults to determine where a packet should go, are quite large—and they continue to grow. The larger the table, the longer it takes to search and find the relevant data. While IPv4 addresses and networks can be difficult to aggregate or simplify, the structure of the IPv6 address allows for just that. The IPv6 address contains three parts: the network or site prefix, the subnet ID, and the host or interface ID.[10] Routers use a site prefix to route the packet through the internet. Network creators and ISPs can also use site prefixes to create packet groups that aggregate packets going in the same general direction. Aggregating the IPs like this allows the internet and routers to act similarly to public transportation. A person boards the train at a specific stop, rides it to another stop, then leaves the train and follows the maps to the exit nearest the actual destination they have in mind. With IPv6, packets behave similarly, where several packets with completely different final destinations can all “exit” the internet at the same stop and travel more granularly from there. This behavior allows for smaller, more efficient routing tables, speeding up the routing process and lowering the overhead on router hardware.
As we mentioned above, comparing IPv4 and IPv6 security isn’t completely straightforward, as professionals have invested more time and effort into the older IPv4 network. However, IPv6 has the potential to be more secure than IPv4 due to at least two built-in advantages. First, IPv6 has end-to-end encryption and authentication built in via default Internet Protocol Security (IPsec) inclusion. While enabling IPsec on IPv6 networks may not be mandatory in some places, implementing it provides better security.[11] The second way IPv6 enhances security is simply by being much larger. With its more numerous IP addresses, IPv6 networks are nearly impossible to brute-force scan.[12]
One more benefit of IPv6 is the ability for users to implement stateful or stateless configurations. While IPv4 network devices rely heavily on dynamic host configuration protocol (DHCP) devices to assign IP addresses, IPv6 networks can use the stateless address auto configuration (SLAAC) technology to let devices generate their own IPs without manual application or the use of a third-party device such as DHCP. If users prefer to use a stateful network, IPv6’s version of DHCP is available.
The NIST and transitioning to IPv6
According to the National Institute of Standards and Technology (NIST), “the IPv6 protocol
suite offers a vastly greater address space than IPv4 and supports significant new capabilities necessary to enable modern network environments.”[13] To facilitate this transition, the NIST National Cyber-security Center of Excellence (NCCoE) “is planning a project to provide guidance and reference architecture that address operational, security, and privacy issues associated with the evolution to IPv6-only network infrastructures.”[14]
The project aims to provide enterprise organizations attempting to transition to IPv6 with guidance and tools that will “ensure that evolving enterprise IT environments to be IPv6-only can be accomplished in a secure and robust manner.”[15]
Regardless of the size of your network, the NIST Cybersecurity Practice Guide will provide best-practices and documentation to ensure that your transition to IPv6 is secure. For more information about the transition resources available from the NIST, visit https://www.nccoe.nist.gov/projects/IPv6-transition.
How Dell servers and Broadcom NICs can help on your journey to IPv6
The National Institute of Standards and Technology (NIST) is part of the U.S. Department of Commerce and serves to support and promote technology and science innovation and investment.[16] In addition to mandating that government agencies switch to IPv6-only networks, the OMB has mandated the NIST to create a set of standards and tools to support the transition. The resulting U.S. Government IPv6 (USGv6) Program develops, tests, and maintains IPv6 standards to help companies and government agencies ensure successful IPv6 transitions and deployments. Vendors can use the IPv6 tests and certifications that NIST developed to certify that their products meet the requirements and standards defined by NIST’s program.[17] In response to the 2020 OMB mandate for all government agencies, NIST revised their USGv6 program to include several objectives including updating specifications to add new and remove old technologies and streamlining their testing program based on previous experience.[18] With a set of agreed upon standards, definitions, and requirements, NIST and the USGv6-r1 provide OEMs a way to ensure their customers that their products are ready for IPv6 implementation.
Dell Technologies™ is the first company to offer a full USGv6-r1 certified server and storage stack for IPv6-only networking.[19] Dell’s certifications include:
- Dell™ PowerEdge™ servers – first in the industry to be fully IPv6 Ready Logo 5.1.2 compliant with RedHat 8.4 and Windows 2019 and 2022.
- Dell PowerEdge iDRAC9 with FW version 5.10.0.00 – first baseboard management controller (BMC) validated by USGv6-r1 as IPv6-only compliant.
- Unity-XT storage array – first storage product validated by USGv6-r1 as IPV6-only compliant meeting the requirements in the IPv6-Only Functional v1.1 (36277) profile.[20]
- Additionally, PowerStore, PowerEdge with VMware 8.0.1, and PowerEdge with SUSE SLES15 SP4 are on the USGv6-r1 registry.
Additionally, Dell servers and storage leverage Broadcom NICs and Adapters to ensure network performance and security for IPv6 customers. Broadcom NICs such as the Broadcom BCM957508-P2200G dual-port 100GbE NIC include several IPv6 offloads that can boost network performance.[21] These offloads allow the NIC to directly handle some of the computational needs of the network rather than use the OS stack, which can provide lower latencies and lower CPU utilization dedicated to network traffic.[22] Broadcom BCM957508-P2200G dual-port 100GbE NICs also offer features such as NVME over Fabrics (NVMe-oF) capabilities that allow NVMe storage traffic to travel through network instead of directly through PCIe channels. NVMe-oF allows users to connect storage via Ethernet (TCP), Fibre, and RDMA.[23] This rerouting of storage network allows for extremely low latency to get the most out of NVMe-based storage.
To show how Dell and Broadcom can provide one great hardware stack option for your IPv6 needs, we conducted some testing to highlight the performance you can expect with IPv6-connected Dell PowerEdge R660 servers to a PowerStore 1200T storage array using 100GbE Broadcom 57508 NICs.
Broadcom NICs
Broadcom NICs can serve most networking needs because they offer speeds ranging from 1G to 200G. According to Broadcom, their network cards feature:
- “Low power adapters and controllers with outstanding thermal performance
- Low latency and high throughput RoCEv2 [for] ground-breaking performance
for machine learning, HPC and
storage applications - Broadsafe™ embedded security [for] Silicon Root of Trust and attestation delivering industry’s most secure Ethernet controller
- Modern architecture [that] delivers industry’s lowest latency and lowest CPU utilization for real-world network conditions
- TruFlow™ engine [to accelerate] virtual switch processing, reduces server CPU usage
- TruManage™ [for] end-user manageability needs to allow fine-tuning of networks for maximum performance
- On-chip tunneling protocol processing for Geneve, VXLAN, and NVGRE [that] provides up to a 5x throughput increase
- Acceleration engines for SDN and NFV [to] enable leading-edge service provider solutions”[24]
Measuring performance
The goal of our performance testing was to show the benefits of the USGv6-r1 IPv6-only certified Dell PowerEdge server and Broadcom NIC solution. This included investigating the performance differences between IPv4 and IPv6 in a real-world environment. Most North American users continue to rely on IPv4, which typically requires NAT or packet fragmentation support from a network router.[25] In a typical scenario, a routing device need only read an IPv4 packet to determine its destination and send it on its way. In the case of IPv4 using NAT or requiring packet fragmentation, the routing device must modify the packet before it can send it along, which requires overhead. We wanted to quantify the impact of this overhead on network performance.
Other than the Layer 3 protocol, every aspect of the test scenario was the same. We used Linux standard tools and NVMe/TCP and/or NFS transport protocols for this test. We did not attempt to enable the best speed of each of these protocols; rather, we used those protocols to drive the tests to compare any differences in speed based on the journey of that data provided by Layer 3 (IP).
We configured two Dell PowerEdge servers as SUSE Linux Enterprise Server 15 SP4 hosts, sending data of diverse sizes to a Dell PowerStore storage array using the transport protocols we identified earlier. The data traversed multiple switches we configured to provide Border Gateway Protocol (BGP) routing and packet fragmentation within a heterogeneous multi-hop network.
We configured the host networks using an MTU of 9,000, with a 1,500 MTU on the switches emulating the core network (which forced packet fragmentation). The edge switches used BGP routing to communicate with the core network.
Comparing IPv6 and IPv4 performance without the Broadcom Offload feature
First, we tested the relative performance of IPv6 and IPv4 on a write workload with the Broadcom Offload feature off. Table 1 presents the results. In terms of both performance (IOPS and throughput in MB per second) and CPU utilization, we observed approximate parity between the two IP versions at both block sizes we tested.
Table 1. IPv6 vs. IPv4 performance on a write workload with Offload off. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Write workload, Offload off |
|
|
| |
IP version | Block size | IOPS | MB/sec | Percentage CPU utilization |
IPv4 | 256K | 8,696.1 | 2,174.01 | 4.9 |
IPv6 | 256K | 8,752.1 | 2,188.02 | 4.9 |
IPv6 % improvement | 0.64% | 0.64% | 0.00% | |
IPv4 | 64K | 34,862.7 | 2,178.92 | 6.6 |
IPv6 | 64K | 34,972.1 | 2,185.76 | 6.5 |
IPv6 % improvement | 0.31% | 0.31% | 1.51% |
Next, we ran the same test using a read workload. As Table 2 shows, in contrast to the comparable performance we observed on the write workload, IPv6 had a performance advantage over IPv4 on the read workload. At the larger block size of 256K, IPv6 delivered 13.83 percent greater performance. At the smaller block size of 64 K, IPv6 delivered 9.83 percent greater performance. These results indicate that users in a real-world setting would enjoy better performance by using IPv6. We also observed a CPU utilization improvement for IPv6.
Table 2. IPv6 vs. IPv4 performance on a read workload with Offload off. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Read workload, Offload off |
|
|
| |
IP version | Block size | IOPS | MB/sec | Percentage CPU utilization |
IPv4 | 256K | 19,987.8 | 4,996.95 | 14.1 |
IPv6 | 256K | 22,752.4 | 5,688.09 | 13.6 |
IPv6 % improvement | 13.83% | 13.83% | 3.54% | |
IPv4 | 64K | 73,194.1 | 4,574.63 | 13.9 |
IPv6 | 64K | 80,392.4 | 5,024.53 | 12.5 |
IPv6 % improvement | 9.83% | 9.83% | 10.07% |
Measuring the impact of the Broadcom Offload feature on IPv6 performance
A secondary component of our testing was investigating the capabilities of the Broadcom IP Offload feature. IP Offloading is a feature Broadcom has implemented in its NIC (Layer 2) to process IP (Layer 3) data to offload the processing of this data from the OS/CPU, leaving those clock cycles to process user data rather than managing the flow control of the protocol. We refer to this feature as Offload.
Table 3 presents IPv6 performance on a write workload with Offload off and with Offload on. While performance was comparable under both conditions, CPU utilization was lower with Offload on.
Table 3. IPv6 performance on a write workload with Offload off and Offload on. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Write workload, IPv6 |
|
|
| |
Block size | IOPS | MB/sec | Percentage CPU utilization | |
Offload off | 256K | 8,752.1 | 2,188.02 | 4.9 |
Offload on | 256K | 8,615.5 | 2,153.88 | 2 |
Offload on % improvement | -1.56% | -1.56% | 59.18% | |
Offload off | 64K | 34,972.1 | 2,185.76 | 6.5 |
Offload on | 64K | 34,895.6 | 2,180.97 | 3.4 |
Offload on % improvement | -0.21% | -0.21% | 47.69% |
Table 4 presents IPv6 performance on a read workload with Offload off and with Offload on. In contrast to the approximate parity we saw on the write workload, performance improved greatly with the use of Offload, particularly at the larger block size, where IPv6 delivered 58.15 percent greater performance than with Offload off. At the 64K block size, enabling Offload improved performance by 25.43 percent. Figure 1 illustrates these advantages. As we saw with the write workload, CPU utilization was lower with Offload on.
Table 4. IPv6 performance on a read workload with Offload off and Offload on. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Read workload, IPv6 |
|
|
| |
Block size | IOPS | MB/sec | Percentage CPU utilization | |
Offload off | 256K | 22,752.4 | 5,688.09 | 13.6% |
Offload on | 256K | 35,983.4 | 8,995.86 | 8.3% |
Offload on % improvement | 58.15% | 58.15% | 38.97% | |
Offload off | 64K | 80,392.4 | 5,024.53 | 12.5% |
Offload on | 64K | 100,840.9 | 6,302.55 | 7.3% |
Offload on % improvement | 25.43% | 25.43% | 41.60% |
Figure 1. Performance improvement of IPv6 using Offload feature on a read workload. Higher is better. Source: Principled Technologies.
Measuring the impact of the Broadcom Offload feature on IPv4 performance
Table 5 presents IPv4 performance on a write workload with Offload off and with Offload on. As we saw with IPv6, performance was comparable under both conditions and CPU utilization improved with Offload on.
Table 5. IPv4 performance on a write workload with Offload off and Offload on. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Write workload, IPv4 |
|
|
| |
Block size | IOPS | MB/sec | Percentage CPU utilization | |
Offload off | 256K | 8,696.1 | 2,174.01 | 4.9 |
Offload on | 256K | 8,596.0 | 2,148.99 | 2.0 |
Offload on % improvement | -1.15% | -1.15% | 59.18% | |
Offload off | 64K | 34,862.7 | 2,178.92 | 6.6 |
Offload on | 64K | 34,727.8 | 2,170.49 | 3.6 |
Offload on % improvement | -0.38% | -0.38% | 45.45% |
As Table 6 shows, the impact of enabling Offload on IPv4 read performance followed the same pattern we saw with IPv6. Using Offload dramatically improved read performance, by 77.90 percent at the larger block size and by 38.73 percent at the 64K block size. Figure 2 highlights these performance improvements. Once again, using Offload improved CPU utilization.
Table 6. IPv4 performance on a read workload with Offload off and Offload on. Higher IOPS and MB/sec and lower CPU utilization are better. Source: Principled Technologies.
Read workload, IPv4 |
|
|
| |
Block size | IOPS | MB/sec | Percentage CPU utilization | |
Offload off | 256K | 19,987.8 | 4,996.95 | 14.1 |
Offload on | 256K | 35,559.3 | 8,889.81 | 8.1 |
Offload on % improvement | 77.90% | 77.90% | 42.55% | |
Offload off | 64K | 73,194.1 | 4,574.63 | 13.9 |
Offload on | 64K | 101,545.6 | 6,346.60 | 7.4 |
Offload on % improvement | 38.73% | 38.73% | 46.76% |
Figure 2. Performance improvement of IPv4 using the Broadcom Offload feature on a read workload. Higher is better. Source: Principled Technologies.
While we have discussed many advantages to making the shift to IPv6, our test results demonstrate that companies who opt not to do so immediately could reap performance benefits on read workloads—and CPU utilization benefits on both read and write workloads—by using the Dell-Broadcom solution we tested and enabling the Broadcom NIC Offload feature.
Conclusion
With IPv4 address pools rapidly disappearing and a federal mandate for government agency devices to begin shifting to IPv6-only and telecom 5G with IoT and edge devices, it’s clear that IPv6 is the future. Transitioning from IPv4 to IPv6 can be a challenge, so organizations may be interested to learn that switching to IPv6 has the potential to improve performance. In our testing without the Broadcom Offload feature, IPv6 delivered comparable performance to IPv4 on write workloads and better performance on read workloads while also reducing CPU utilization. When we enabled the Broadcom Offload feature on both IPv6 and IPv4, read workload performance increased dramatically and CPU utilization on both read and write workloads improved. Whether your organization is transitioning to IPv6 right away or choosing to delay the shift, this feature can boost performance on read workloads, which can improve the experience for users, reduce backup windows, and allow databases to load more quickly.
This project was commissioned by Dell Technologies.
January 2024
Principled Technologies is a registered trademark of Principled Technologies, Inc.
All other product names are the trademarks of their respective owners.
[1] Google, “IPv6,” accessed December 21, 2023, https://www.google.com/intl/en/ipv6/statistics.html.
[2] Russell T. Vought, “MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES,” accessed December 21, 2023, https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-07.pdf.
[3] RipeNCC, “What is IPv4 Run Out?” accessed December 21, 2023,
[4] Jeff Barr, “New – AWS Public IPv4 Address Charge + Public IP Insights,” accessed December 21, 2023,
https://aws.amazon.com/blogs/aws/new-aws-public-ipv4-address-charge-public-ip-insights/.
[5] Linkedin, “How can NAT affect the performance and reliability of network applications?” accessed December 21,
2023, https://www.linkedin.com/advice/0/how-can-nat-affect-performance-reliability-network.
[6] Russell T. Vought, “MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES,” accessed December 21, 2023, https://www.whitehouse.gov/wp-content/uploads/2020/11/M-21-07.pdf.
[7] Russell T. Vought, “MEMORANDUM FOR HEADS OF EXECUTIVE DEPARTMENTS AND AGENCIES.”
[8] Akamai, “IPv6 Adoption Visualization,” accessed December 21, 2023,
[9] Network Academy, “IPv4 vs IPv6 - Understanding the differences,” accessed December 21, 2023,
[10] Diego Asturias, “Migration to IPv6: Benefits and Techniques,” accessed December 21, 2023,
[11] Diego Asturias, “Migration to IPv6: Benefits and Techniques.”
[12] Diego Asturias, “Migration to IPv6: Benefits and Techniques.”
[13] “IPv6 Transition,” accessed December 21, 2023, https://www.nccoe.nist.gov/projects/IPv6-transition.
[14] “IPv6 Transition.”
[15] “IPv6 Transition.”
[16] NIST, “About NIST,” accessed December 21, 2023, https://www.nist.gov/about-nist.
[17] NIST, “USGv6,” accessed December 21, 2023, https://www.nist.gov/programs-projects/usgv6-program/usgv6.
[18] NIST, “USGv6 Revision 1,” accessed December 21, 2023,
https://www.nist.gov/programs-projects/usgv6-program/usgv6-revision-1.
[19] George Dilger, “Industry First IPv6-only Support on Servers and Storage,” accessed December 21, 2023,
https://www.dell.com/en-us/blog/industry-first-ipv6-only-support-on-servers-and-storage/.
[20] George Dilger, “Industry First IPv6-only Support on Servers and Storage.”
[21] Broadcom, “Broadcom Ethernet NICs PCIe NIC Ethernet Adapters Specification Sheet,” accessed December 21,
2023, https://docs.broadcom.com/doc/PCIe-NIC-Ethernet-Adapters-Specification-Sheet.
[22] Eden Kim and Fred Zhang, “Optimizing NVMe® over Fabrics (NVMe-oFTM),” accessed December 21, 2023,
https://www.snia.org/sites/default/files/education/snia-optimizing-nvme-over-fabrics-nvme-of.pdf.
[23] Juan Mulford, “What is NVMe-oF?,” accessed December 21, 2023, https://www.storagereview.com/review/nvme-nvme-of-background-overview.
[24] “Ethernet Network Adapters,” accessed December 21, 2023,
https://www.broadcom.com/products/ethernet-connectivity/network-adapters.
[25] Google, “IPv6,” accessed December 21, 2023, https://www.google.com/intl/en/ipv6/statistics.html.