The Tiered Storage architecture augments Kafka brokers with a Dell EMC ECS object store, storing data on ECS instead of local storage on brokers. Brokers now contain less state locally, making them more lightweight and rebalancing operations by orders-of-magnitude faster. Producers and consumers connect to brokers and allow adoption. This architecture is transparent to the producers and consumers, which connect to brokers as always.

Tiered Storage simplifies the operation and scaling of a Kafka cluster enough so that it is easy to scale individual Kafka clusters to petabytes of data. With the ECS system as the back end, Tiered Storage has the performance to make all Kafka data accessible for both streaming consumers and historical queries.

Figure 1.   Kafka architecture with Tiered Storage and Dell EMC ECS

ECS design principles map well to disaggregated Kafka clusters; both ECS and Kafka use scale-out architectures with linear and predictable scaling of performance and capacity. As an S3 back end for Tiered Storage, ECS provides simple, performant, reliable, and scalable storage for on-premises data pipelines.