Deploy Splunk Enterprise Efficiently with Red Hat OpenShift on Dell Servers with Intel® Technologies
Download PDFTue, 17 Jan 2023 05:13:39 -0000
|Read Time: 0 minutes
Summary
Splunk Enterprise containerized deployments with Red Hat OpenShift can deliver substantial business benefits. In this brief, Intel and Dell technologists discuss key considerations to successfully deploy Splunk based containers, with recommendations on configurations based on the most recent 15th Generation PowerEdge Server portfolio offerings.
Integrating data strategy into business strategy is key to digital transformation. To harness the value of untapped data, many organizations are turning to Splunk Enterprise, a high-performance data-analytics platform that enables decision makers to bring data to every question, decision, and action.
To deploy workloads like Splunk Enterprise more efficiently, IT architects are choosing containerization. Red Hat OpenShift, an enterprise-ready Kubernetes container platform, is a popular choice. By using Red Hat OpenShift, architects don’t need to separate dedicated nodes for each Splunk Enterprise function, and they can add more nodes and scale them separately from storage.
Intel and Splunk have partnered to develop recommended hardware configurations for deploying Splunk Enterprise with Red Hat OpenShift on Dell PowerEdge servers. Organizations that use these configurations can benefit from the high performance enabled by Intel® compute, storage, and network technologies.
Key Considerations
Key considerations for deploying Splunk Enterprise with Red Hat OpenShift successfully include:
- Size local NVM Express (NVMe) storage. Size local NVMe storage for persistent volumes according to the expected ingestion rate and hot-data-retention period. Size by adding more or larger drives.
- Deploy additional storage nodes. Deploy additional storage nodes to scale object-store capacity for Splunk SmartStore independently of worker nodes. Optimize for the expected ingestion rate and warm or cold data retention using high-performance (with NVMe solid-state drives [SSDs]) or high-capacity (with rotational hard-disk drives [HDDs]) configurations with a set of storage policies. An in- cluster solution based on MinIO is recommended. Alternatively, an external S3-compatible object store like Dell PowerScale (Isilon) or Elastic Cloud Storage (ECS) can be used. For dedicated network storage, an additional network interface controller (NIC) can be used. Contact Dell Technologies for help choosing the proper storage solution.
- Manage NVMe drives. Manage NVMe drives for local and object storage using the Local Storage Operator. If necessary, partition the drives to get more volumes with smaller size. Configure proper redundancy and replication on the application layer.
Available Configurations
| Red Hat OpenShift Control Plane (Master) Nodes-3 Nodes Required | Splunk Worker Nodes | Optional Dedicated Storage Node for Object Storage | ||||
Platform | Dell PowerEdge R650 server supporting 10 x 2.5” drives with NVMe backplane | Dell PowerEdge R750 server supporting 16 x 2.5” drives with NVMe backplane (direct) | Dell PowerEdge R650 server supporting 10 x 2.5” drives with NVMe backplane (direct) | Dell PowerEdge R750 server supporting 12 x 3.5” drives with Serial- Attached SCSI (SAS)/Serial ATA (SATA) backplane | |||
Node type |
| Base configuration | Plus configuration | High performance | High capacity | ||
CPU | 2 x Intel® Xeon® Gold 6326 processor (16 cores at 2.9 GHz) or better | 2 x Intel® Xeon® Gold 6348 processor (28 cores at 2.6 GHz) | 2 x Intel® Xeon® Platinum 8360Y processor (36 cores at 2.4 GHz) | 2 x Intel® Xeon® Gold 6342 processor (24 cores at 2.8 GHz) | 2 x Intel® Xeon® Gold 6326 processor (16 cores at 2.9 GHz) | ||
DRAM | 128 GB (16 x 8 GB DDR4-3200) | 256 GB (16 x 16 GB DDR4-3200) | 512 GB (16 x 32 GB DDR4- 3200) | 128 GB (16 x 8 GB DDR4-3200) | |||
Storage controller | Not applicable (N/A) | HBA355i adapter | |||||
Persistent memory | Not applicable N/A | Optional | N/A | ||||
Boot device | Dell Boot Optimized Server Storage (BOSS)-S2 with 2 x 480 GB M.2 SATA SSD (RAID1) | ||||||
Ephemeral storagei |
1 x 1.6 TB Intel® SSD P5600 NVMe |
1 x 1.6 TB Intel® SSD P5600 (PCIe Gen4, mixed-use) |
N/A | ||||
Local storageii | N/A | 1 x (up to 5 x) 1.6 TB or 3.2 TB Intel® SSD P5600 (PCIe Gen4, mixed-use) | N/A | ||||
Object storageiii |
N/A |
4 x (up to 10 x) 2 TB, 4 TB or 8 TB Intel® SSD P5500 (PCIe Gen4, read-intensive) |
Up to 10 x 2 TB, 4 TB, 8 TB Intel® SSD P5500 (PCIe Gen4, read-intensive) |
Up to 12 x 8 TB, 12 TB, 18 TB 3.5-in 12 Gbps SAS HDD 7.2K rotations per minute (RPM) | |||
Network interface controller (NIC)iv | Intel® Ethernet Network Adapter E810-XXVDA2 for OCP3 (dual-port 25 gigabit Ethernet [GbE]) | Intel® Ethernet Network Adapter E810-XXV for OCP3 (dual-port 25 Gb) or Intel Ethernet Network Adapter E810-CQDA2 PCIe add-on card (dual-port 100 Gb) | Intel® Ethernet Network Adapter E810-CQDA2 PCIe add-on card (dual- port 100 Gb) | Intel® Ethernet Network Adapter E810-XXV for OCP3 (dual-port 25 Gb) | |||
Additional NIC for external storagev |
N/A | Intel® Ethernet Network Adapter E810-XXV PCIe add-on card (dual- port 25 Gb) or Intel® Ethernet Network Adapter E810-CQDA2 PCIe add-on card (dual-port 100 Gb) |
N/A | ||||
Learn More
Contact your dedicated Dell or Intel account team for a customized quote. 1-877-289-3355 “Build High Performance Splunk SmartStores with MinIO”
“Harness the Power of Splunk with Dell Storage”
i Ephemeral storage is used only for container images and ephemeral volumes.
ii Local storage for persistent volumes includes Splunk® hot tier.
iii The number of drives and capacity for MinIO® object storage depends on the dataset size and performance requirements.
iv 100 Gb NICs recommended for higher throughput.
v Optional; required only if dedicated storage network for external storage system is necessary.
Note: This document may contain language from third-party content that is not under
Dell Technologies’ control and is not consistent with current guidelines for Dell Technologies’ own content. When such third-party content is updated by the relevant third parties, this document will be revised accordingly.
Related Documents
Achieve Real-Time Data Processing with Confluent® Platform and Apache Kafka®
Tue, 17 Jan 2023 07:15:23 -0000
|Read Time: 0 minutes
Summary
Enabling mission critical application, system and connecting data to the entire organization with real-time data flow and process means that the system and software stack must be optimized. In this document Intel and Dell discuss key considerations and sample configurations for PowerEdge server deployments to ensure your Confluent Kafka architecture is robust and takes advantage of the most recent advancements in server technology.
Mission-critical applications need to analyze large amounts of data in real time, but this requires refined tools built on scalable platforms.
Originally developed at LinkedIn by the founders of Confluent, Apache Kafka® is an open-source, high-throughput message broker that fills this need. It quickly decouples, queues, processes, stores and consumes high-volume streams of event data. With Apache Kafka, enterprises can acquire data once and consume it multiple times.
Confluent continues to enhance the Kafka platform with tools like cluster management, additional security, and more connectors. Companies like Square, Bosch and The Home Depot use Confluent’s distribution of Apache Kafka to identify actionable patterns within business datai. Intel created an Apache Kafka data pipeline based on Confluent® Platform for faster security threat detection and response for its Cyber Intelligence Platform (CIP). Data flows to a Kafka message bus and then into the Splunk® platform.
Organizations that are looking for a solution to enable real-time processing of massive data streams should consider Confluent Platform and Apache Kafka running on Dell EMC™ PowerEdge™ servers with high-performing Intel compute, storage and networking technologies.
Key Considerations
- Compute. 3rd Generation Intel® Xeon® Scalable processors ingest and analyze massive quantities of data fast in the decoupling work common to Apache Kafka broker nodes.
- Storage. The Intel SSD P5500 is recommended for storage for all node types. Architected with 96-layer TLC and Intel 3D NAND Technology, it optimizes performance and capacity. The Dell™ PowerEdge RAID Controller (PERC) H755N is recommended for Brokers + Apache ZooKeeper™ nodes. It offers expandable storage capacity to improve performance.
- Networking. Network speed is one of the most important factors in Kafka performance. Intel Ethernet 800 Series network adapters enable scaling from 10 gigabit Ethernet (GbE) to 100 GbE for accelerated packet processing.
Available Configurations
Configurations for the control center node, ksqlDB + Kafka Connect + Schema Registry, and Brokers + Apache ZooKeeper are shown below.
| Control Center Node (One Node Required) | ksqlDB + Apache Kafka® Connect + Schema Registry (Minimum of Two Nodes Required) | Brokers + Apache ZooKeeper™ (Minimum of Three Nodes Required) |
Platform | Dell EMC™ PowerEdge™ R650 or R750 chassis supporting NVM Express® (NVMe®) drives | ||
CPUii | 2 x Intel® Xeon® Silver 4316 processor (20 cores at 2.3 GHz) | 2 x Intel® Xeon® Gold 6330 processor (28 cores at 2.0 GHz) | 2 x Intel® Xeon® Silver 4316 (20 cores at 2.3 GHz)—small throughput clusters 2 x Intel® Xeon® Gold 6338 (32 cores at 2.0 GHz)—medium throughput clusters 2 x Intel® Xeon® Platinum 8368 (38 cores at 2.4 GHz)—high throughput clusters with full encryption enabled |
DRAMiii | 64 GB (4 x 16 GB) | 128 GB (8 x 16 GB) | 128 GB (8 x 16 GB) or more |
Boot device | Dell EMC™ Boot Optimized Server Storage (BOSS)-S2 with 2 x 480 GB Intel® SSD D3-S4510 M.2 Serial ATA (SATA) | ||
Storage controlleriv | None | Dell™ PERC H755N Front NVMe | |
Storagev | 2 x 3.84 TB Intel® SSD P5500 | 4 x 3.84 TB Intel® SSD P5500 | |
Network interface controller (NIC) | Intel® Ethernet Network Adapter E810-XXVDA2 for OCP3 (dual-port 25 Gb) | Intel® E810-XXVDA2 for OCP3 (dual-port 25 Gb) or Intel® E810- CQDA2 PCIe® (dual-port 100 Gb) for high-throughput clusters |
Learn More
Contact your dedicated Dell or Intel account team. 1-877-289+-3355
Download the solution briefs and white papers below:
- Enabling Real-Time Processing of Massive Data Streams
- IT@Intel: Building a Modern, Scalable Cyber Intelligence Platform with Apache Kafka
- IT@Intel: Transforming Intel’s Security Posture with Innovations in Data Intelligence
The information in this publication is provided as is. Dell Inc. makes no representations or warranties of any kind with respect to the information in this publication, and specifically disclaims implied warranties of merchantability or fitness for a particular purpose.
Use, copying, and distribution of any software described in this publication requires an applicable software license.
Copyright © 2021 Dell Inc. or its subsidiaries. All Rights Reserved. Dell, EMC, PowerEdge and other trademarks are trademarks of Dell Inc. or its subsidiaries. Other trademarks may be the property of their respective owners.
Dell Inc. believes the information in this document is accurate as of its publication date. The information is subject to change without notice.
i Confluent. “Set Your Data in Motion.” 2021. www.confluent.io/.
ii Small throughput: less than 10 gigabits per second (Gbps), medium throughput: less than 25 Gbps, high throughput: more than 25 Gbps
iii Brokers and Apache ZooKeeper™: More memory might be required to accommodate traffic bursts.
iv Brokers and Apache ZooKeeper™: An NVMe® RAID controller is optional for small- and medium-throughput clusters.
v Brokers and Apache ZooKeeper™: Add more drives or add higher capacity drives as needed for higher throughput, extended data-retention periods or desired (optional) RAID configurations.
Scaling and Optimizing ML in Enterprises
Tue, 16 May 2023 19:53:46 -0000
|Read Time: 0 minutes
Summary
This joint paper, written by Dell Technologies, in collaboration with Intel®, describes the key hardware considerations when configuring a successful MLOps deployment and recommends configurations based on the most recent 15th Generation Dell PowerEdge Server portfolio offerings.
Today’s enterprises are looking to operationalize machine learning to accelerate and scale data science across the organization. This is especially the case as their needs grow to deploy, monitor, and maintain data pipelines and models. Cloud native infrastructure, such as Kubernetes, offers a fast and scalable means to implement Machine Learning Operations (MLOps) by using Kubeflow, an open source platform for developing and deploying Machine Learning (ML) pipelines on Kubernetes.
Dell PowerEdge R650 servers with 3rd Generation Intel® Xeon® Scalable processors deliver a scalable, portable, and cost-effective solution to implement and operationalize machine learning within the Enterprise organization.
Key Considerations
- Portability. A single end-to-end platform to meet the machine learning needs of various use cases, including predictive analytics, inference, and transfer learning.
- Optimized performance. High-performance 3rd Generation Intel® Xeon® Scalable processors optimize performance for machine learning algorithms using AVX-512. Intel® performance optimizations that are built into Dell PowerEdge servers can help fine-tune large Transformers models across multi-node systems. These work in conjunction with open-source cloud native MLOps tools. Optimizations include Intel® and open-source software and hardware technologies such as Kubernetes stack, AVX-512, Horovod for distributed training, and Tensorflow 2.10.0.
- Scalability. As the machine learning workload grows, additional compute capacity needs to be added to the cloud native infrastructure. Dell PowerEdge R750 servers with 3rd Generation Intel® Xeon® Scalable processors deliver an efficient and scalable approach to MLOps.
Recommended Configurations
Cluster | ||
| Control Plane Nodes (Three Nodes Required) | Data Plane Nodes (4 Nodes or More) |
Functions | Kubernetes services | Develop, Deploy, Run Machine Learning (ML) workflows |
Platform | Dell PowerEdge R650 up to 10x 2.5” NVMe Direct Drives | |
CPU | 2x Intel® Xeon® Gold 6326 processor (16 cores @ 2.9GHz), or better | 2x Intel® Xeon® Platinum 8380 processor (40 cores at 2.3 GHz), or 2x Intel® Xeon® Platinum 8368 processor (38 cores @ 2.4GHz), or Intel® Xeon® Platinum 8360Y processor (36 cores @ 2.4GHz) |
DRAM | 128 GB (16x 8 GB DDR4-3200) | 512 GB (16x 32 GB DDR5-4800) |
Boot device | Dell Boot Optimized Server Storage (BOSS)-S2 with 2x 240GB or 2x 480 GB Intel® SSD S4510 M.2 SATA (RAID1) | |
Storage adapter | Not required for all-NVMe configuration. | |
Storage (NVMe) | 1x 1.6TB Enterprise NVMe Mixed- Use AG Drive U.2 Gen4 | 1x 1.6TB (or larger) Enterprise NVMe Mixed-Use AG Drive U.2 Gen4 |
NIC | Intel® E810-XXVDA2 for OCP3 (dual-port 25GbE) | Intel® E810-XXVDA2 for OCP3 (dual-port 25GbE), or Intel® E810-CQDA2 PCIe (dual-port 100Gb) |
Resources
Visit the Dell support page or contact your Dell or Intel account team for a customized quote 1-877-289-3355.