Kafka with Kubernetes

At the top of this webpage are 3 PDF files outlining test results and reference configurations for Dell PowerEdge servers using both the 3rd Generation Intel^® Xeon^® processors and 4th Generation Intel Xeon processors. All testing was conducted in Dell Labs by Intel and Dell Engineers in October and November of 2023.

• “Dell DfD Kafka ICX” – highlights the recommended configurations for Dell PowerEdge servers using 3rd generation Intel^® Xeon^® processors.
• “Dell DfD Kafka SPR” – highlights the recommended configurations for Dell PowerEdge servers using 4th generation Intel^® Xeon^® processors.
• “Dell DfD Kafka Kubernetes Test Report” – Highlights the results of performance testing on both configurations with comparisons that demonstrate the performance differences between them.   

Solution Overview

The Apache^® Software Foundation developed Kafka as an Open Source solution to provide distributed event store and stream processing capabilities. Apache Kafka uses a publish-subscribe model to enable efficient data sharing across multiple applications. Applications can publish messages to a pool of message brokers, which subsequently distribute the data to multiple subscriber applications in real time.

Kafka is often deployed for mission-critical applications and streaming analytics along with other use cases. These types of workloads require leading-edge performance which places significant demand on hardware.

There are five major APIs in Kafka[i]:

• Producer API – Permits an application to publish streams of records.
• Consumer API – Permits an application to subscribe to topics and process streams of records.
• Connect API – performs the reusable producer and consumer APIs that can link the topics to the existing applications.
• Streams API – This API converts the input streams to output and produces the result.
• Admin API – Used to manage Kafka topics, brokers, and other Kafka objects.

Kafka with Dell PowerEdge and Intel processor benefits

The introduction of new server technologies allows customers to deploy solutions using the newly introduced functionality, but it can also provide an opportunity for them to review their current infrastructure and determine if the new technology might increase performance and efficiency.  Dell and Intel recently conducted testing of Kafka performance in a Kubernetes environment and measured the performance of two different compression engines on the new Dell PowerEdge R760 with 4th generation Intel^® Xeon^® Scalable processors and compared the results to the same solution running on the previous generation R750 with 3rd generation Intel^® Xeon^® Scalable processors to determine if customers could benefit from a transition. 

Some of the key changes incorporated into 4th generation Intel® Xeon® Scalable processors include:

• Quick Assist Technology (QAT) to accelerate data compression and encryption.  
• Support for 4800 MT/s DDR5 memory

Raw performance: As noted in the report, our tests showed a 72% producers’ latency decrease with gzip compression and a 62% producers’ latency decrease with zstd compression.

Conclusion

Choosing the right combination of Server and Processor can increase performance and reduce time, allowing customers to react faster and process more data.  As this testing demonstrated, the Dell PowerEdge R760 with 4th Generation Intel® Xeon® CPUs significantly outperformed the previous generation.

• The Dell PowerEdge R760 with 4th Generation Intel® Xeon® Scalable processors delivered:
  • 62% faster processing using zstd compression
  • 72% faster procession using gzip compression
• 4th Generation Intel^® Xeon^® Scalable processors benefits are the results of:
  • Innovative CPU microarchitecture providing a performance boost
  • Introduction of DDR5 memory support