Dell Technologies infrastructure blocks enable telco customers to adopt telco-centric AIOps to improve operations.

Communication service providers (CSPs) are racing towards fully autonomous networks and consider automation and artificial intelligence (AI) adoption in telecom networks to be of great value. According to the latest industry insights report published by TM Forum® (New-generation intelligent operations: the service-centric transformation path), most CSPs aim to achieve Level-3 automation (conditional autonomous networks) and Level-4 automation (highly autonomous networks) by 2025. There is increased interest in accelerating Level-5 automation (AI-driven automation) using AIOps solutions.

However, telco adoption of AI-centric automation is not easy primarily because most CSPs operate a geographically distributed brownfield network and manage a multi-generation fleet of infrastructure and resources. CSPs also operate at different scales which means there is no simple, “cookie-cutter” approach towards AI-driven operations (AIOps).

In addition, CSPs adopt solutions based on clearly defined, standard telecom architectures like ETSI® (European Telecommunications Standards Institute), TM Forum® (Telecom Management Forum), 3GPP® (3^rd Generation Partnership Project), and O-RAN (Open RAN alliance). CSPs also source solutions that can interwork and interoperate at a global scale. Finally, CSPs expect these solutions to fully integrate into their brownfield environments.

Hence, there is a requirement to build an outcome-based solution that supports existing operations. At the same time, the solution must enable them to accelerate the adoption of the next era of operations (based on data-driven insights and artificial intelligence).

How to adopt AIOps-based, intelligent operations for networks

CSPs are working alongside many standards bodies (especially the TM Forum®) to accelerate automation towards Level-4 (full-service orchestration and automation) and Level-5(AI-driven automation). However, there still lacks a clear path for applying these principles to large-scale networks. Building the right architecture starts from clear requirements quantification.

The right AIOps solution that is designed for CSPs must align with unique telco-specific requirements on: 

• Distributed topology maps. Telco networks are being purpose-built and deployed to deliver critical and differentiated services for many decades. These networks are not like data centers but instead a fleet of resources—such as home networking, fiber, transport, radio, core, cloud, and WAN services. Topology alignment (like A/B plane, Ring) and service resilience are key requirements.
• Multi-vendor and multi-generation. Typically, CSPs operate a brownfield network over multiple generations. Most of these systems have an extended lifetime of 10 to 15 years. So, the solution should not only be future-proof but also cater to the requirements of existing deployed solutions.
• Data models. CSP networks, by nature, are highly protected—with network data existing in many silos. Network operations also follow a hierarchical, process-based delivery that is defined by Network Operations Centers (NOCs). In addition, data knowledge is based on tools and systems that vendors provide. 

Given that AIOps systems are already proven and prevalent in the Cloud and IT industries, these systems and solutions must be adapted to meet CSP requirements. 

AIOps in networks should strictly align with both telecom standards and network-specific needs—delivering the following capabilities:

• Process alignment. The current operational model of a telco cloud heavily relies on an operational team knowledge base and expertise. So, it is not just about data but also the unique experience of CSP operations—which are important. 
• Data access. CSPs follow strict security and privacy requirements where customer data and information cannot be exposed. So, in order to adopt AIOps, data access models must be standardized to ensure AIOps use cases can retrieve data as per approved policies. 
• Tuning. Because CSP-deployed networks must operate for extended periods of time, current solutions—which follow strict AI rules—cannot meet their future requirements. Therefore, AIOps systems must be adaptive. 
• Scalability. CSPs operate at different scales starting from Tier-1 (many geographies) to Tier-2 (small scale). Therefore, telco-specific AIOps systems should offer a T-shirt sizing approach. 

Accelerate the network AIOps journey

Today, many CSPs have already deployed small-scale AIOps solutions. However, most of these solutions are not highly aligned with telco-specific requirements—resulting in many silos that are hard to manage and scale. Further, CSPs must invest heavily in terms of time and cost to do Life Cycle Management (LCM) of these solutions. It all translates to barriers towards cloud native transformation.

Just as telco cloud CSPs have adopted standards like ETSI®, LFN® (Linux Foundation Networking) and ORAN® (Open RAN), there is a requirement to adopt a standard architecture for the Telco Multicloud AI foundation that can smoothly integrate with brownfield networks. Below are the key capabilities of an AI-centric telco platform that can enable AIOps use cases:

• Horizontal AI platform. The telco-centric AI platform should enable a composable platform that consists of:
  • AIOps application layer: hosting various AIOps tasks
  • Machine Learning (ML) layer: adopting specific ML models suitable for AIOps
  • Knowledge layer: integrating the NOC processes and knowledge of CSPs 
  • Data layer: resolving any data silos in networks
  • Physical layer: managing telecom networks using fully decoupled infrastructure automation
• Distributed data ingestion. The telco-centric AI platform should ingest data from fully distributed networks—delivering both reactive (respond after event occurs) and proactive (predictive) use cases on:
  • MOP integration: Existing MOP and workflows must be integrated. 
  • Operational processes integration: Existing NOC processes must be integrated in data pipelines.
• Cloud native MLOps and AIOps capabilities. Telcos must supplement operational in-house knowledge with ML models and find a way to tune and extend it. Different models must be integrated. A systematic integration of knowledge systems with ML models (in a use case-driven approach) is required for success in network operations.

Figure: Reference architecture for AIOps-driven telecom network

How to adopt AIOps operations using telecom infrastructure blocks

Dell Technologies has worked closely with leading cloud partners, including Wind River® and Red Hat®, to bring forward an operationally ready telco cloud platform. This platform is thoroughly tested, validated, and automated to deliver telco AIOps use cases. This platform also accelerates a CSP’s adoption of zero-touch operations while consistently aligning to telecom standards and frameworks.

The Dell Technologies Telecom Multicloud Foundation flexibly transforms network operations towards programmable infrastructure using a consistent tooling and AIOps capabilities approach.

Because the platform supports multiple versions and offers with various partners, CSPs can operate all such foundational infrastructure blocks as one. Through the following key capabilities, our solution can quickly transform operational models and processes and enables agile MLOps (required in a telco environment).  

Figure: Solution architecture for AIOps Foundation

To support the unique CSP requirements to adopt AIOps and a cloud operating model, our Telecom Infrastructure Blocks provide the following key capabilities: 

• Consistent platform. The first challenge is to deliver a standard and consistent platform that can integrate all layers above—abstracting the complexities of multiple technologies and components from different vendors.
• Cloud native MLOps. AIOps use cases require cloud-type agility towards data and ML. Our current version of infrastructure blocks delivers a ready platform. In future releases, we plan to enable AI enhancements (like Openshift® AI) on top of this platform. This means CSPs can build, program, and manage all their ML models and capabilities in the same way they manage cloud resources. 
• Autonomous operations. Adopting data-centric architectures and ML approaches provides CSPs a smooth evolution path from their current automation approaches to an AI-centric automation that is aligned with the telco future mode of operations (FMO).
• Data-driven architecture: The automation architecture is data-driven and distributed, so data can be tapped from edge and regional sites—enabling real-time use cases and data-driven operations.
• Automated fault management: The FMO follows zero-touch and intent-driven networks. Our solution is fully aligned with this vision that enables all cloud platforms to use declarative workflows. The solution also enables all northbound integration towards orchestration and assurance systems.
• Single pane for DevOps and MLOps operations: As CSPs adopt ML/AI frameworks to deliver AIOps use cases, there is an increasing requirement to integrate and operate both DevOps and MLOPs as one. In addition, AIOps platform capabilities must be enabled in telco cloud platforms. Doing so provides a single management and observation platform.

Figure: DevOps and MLOps workflow using AIOps platform capabilities

Dell Technologies developed Telecom Multicloud Foundation and Telecom Infrastructure Blocks to accelerate telco cloud transformation. Our engineered and factory-integrated system delivers a consistent platform. This platform is ready to deliver telco-specific AIOps use cases that are fully aligned with telecom architectures—enabling our customers to accelerate AIOps solutions in networks.

Visit the Dell Telecom Multicloud Foundation site to learn more about our solution.

Welcome back to our 5G Core blog series.  In the second blog post of the series, we discussed the 5G Core, its architecture, and how it stands apart from its predecessors, the role of cloud-native architectures, the concept of network slicing, and how these elements come together to define the 5G Network Architecture.

In this third blog, we look at Dell Technologies’ and Red Hat's collaboration with their latest offering of Dell Telecom Infrastructure Blocks for Red Hat. We explore how Infrastructure Blocks streamline Communications Service Providers’ (CSPs) processes for a Telco cloud used with 5G core from initial technology onboarding at day 0/1 to day 2 life cycle management.

Helping CSPs transition to a cloud-native 5G core 

Building a cloud-native 5G core network is not easy. It requires careful planning, implementation, and expertise in cloud-native architectures. The network needs to be designed and deployed in a way that ensures high availability, resiliency, low latency, efficient resource utilization, and flawless component interoperability. CSPs may feel overwhelmed when considering the transition from legacy architectures to an open, best-of-breed cloud-native architecture. This can lead to delays in design, deployment, and life cycle management processes that stall projects and reduce a CSP’s ability to effectively deploy and manage their disaggregated cloud-native network.   

Automation plays a critical role in managing deployment and life cycle management processes.  Many projects stall or fail due to poorly defined automation strategies that make it difficult to ensure compatibility between hardware and software configurations across a large, distributed network.  This is especially true when trying to deploy and manage a cloud platform running on bare metal.   

Dell Telecom Infrastructure Blocks for Red Hat are foundational building blocks for creating a Telco cloud that is based on Red Hat OpenShift. They aim to reduce the time, cost, and risk of designing, deploying, and maintaining 5G networks using open software and industry standard infrastructure. The current release of Telecom Infrastructure Blocks for Red Hat supports the creation of management and workload clusters for 5G core network functions running Red Hat OpenShift on bare metal servers.

There are a number of challenges to build and maintain Kubernetes clusters on bare metal to run 5G network functions:

• Ensuring interoperability and fault tolerance in a disaggregated network is not an easy task. Deploying and managing Kubernetes clusters on bare metal requires extensive design, planning, and interoperability testing to ensure a reliable, fault tolerant, and performant system.  
• Automating the deployment and life cycle management of hardware resources and cloud software in a bare metal environment can be complex. It involves deploying and updating a fleet of bare metal servers at scale.
• There is a lack of pre-built software integrations specifically designed for deploying Kubernetes clusters on bare metal servers and bringing those cluster configurations to a state where they are ready to run workloads. This means that configuring and deploying Kubernetes on bare metal frequently requires more manual effort to build and maintain the automation needed to manage deployments and upgrades at scale.  This manual effort can be time-consuming and add complexity that introduces risk to the process. 
• This lack of consistent, easy-to-manage automation to deploy and update the cloud stack to meet workload requirements also make it harder to implement a unified cloud platform across all workloads.   This leads to infrastructure silos that limit the ability to pool resources to improve infrastructure utilization rates, which in turn reduces network TCO efficiency.     

These challenges are amplified when running 5G network functions, which require low latency and high reliability to meet carrier-grade service level agreement (SLAs). This collaboration between Dell and Red Hat aims to offer a comprehensive solution for CSPs that addresses the challenges associated with building and maintaining carrier-grade cloud infrastructure for 5G core network functions.

Key objectives of Dell Telecom Infrastructure Blocks for Red Hat

Implement a “shift left” approach

In  software development, the term “shift left” refers to the ability to move tasks to an earlier stage in the development or production process to reduce time to value for those processes.  The shift left approach being offered with Infrastructure Blocks moves much of the testing and integration work performed by the CSP into the supply chain prior to onboarding the new technology. This method provides CSPs with a speedy path to value by shortening the preparation and validation phase for a new network deployment. It also simplifies the procurement process by reducing the number of suppliers the CSP needs to work with and simplifies support by providing one call support for the full cloud stack.  Proactive problem-solving, reduction of field touch points, risk minimization, and operational simplification are byproducts of the Infrastructure Block approach that hasten the introduction of new technology into a CSP’s network. By adopting this approach, CSPs can obtain faster rollout times and reduced operational costs. Dell does three things to help CSPs shift the technology onboarding processes left:

Engineering

Telecom Infrastructure Blocks are foundational building blocks that are co-designed with Red Hat to help CSPs build and scale out their network. These building blocks are purpose-built to meet specific workload requirements.  Dell collaborates with Red Hat to maintain a roadmap of feature enhancements and perform continuous design and integration testing to accelerate the adoption of new technologies and software upgrades. The design planning and extensive interoperability testing performed by Dell simplifies the processes of building and maintaining a fault tolerant and performant cloud platform to run 5G core workloads.  

Automation

Many CSPs today rely on procedural automation that they build and maintain on their own to automate the deployment and life cycle management of their cloud platform at scale.  Procedural automation requires an understanding of the current state of their cloud stack and the maintenance of scripts or playbooks to define the steps needed to update the configuration to the desired state. When deploying Kubernetes on bare metal to support 5G core workloads, there are a number of items with dependencies that must be configured appropriately, including the following properties:

• Cloud platform software version
• BIOS version and settings 
• Firmware versions for network interface cards (NICs) and other Peripheral Component Interconnect Express (PCIe) cards
• Single root I/O virtualization (SR-IOV) / Data Plane Development Kit (DPDK) configurations
• RAID configurations 
• Site-specific data

Building and maintaining these scripts and automation playbooks is no easy task. It requires an up-to-date view of the current configuration of the infrastructure, an understanding of the dependencies between hardware and software that must be met to perform an update, and people with specialized skills that include knowledge of server hardware and the tools to manage them, the cloud software, and how to write or update playbooks to execute deployments and upgrades. 

Managing this across a large, distributed network with a range of workloads that frequently require unique configurations of the cloud stack is a difficult and time-consuming process.  Also consider that, in an open ecosystem environment, there is always a new version of software, BIOS, or firmware coming and the people with the needed skill sets are in short supply, resulting in a herculean effort with mixed results.

Telecom Infrastructure Blocks include purpose-built automation software that is easy to use and maintain.  The blocks integrate with Red Hat Advanced Cluster Management and Red Hat OpenShift to automate deployment and life cycle management of the hardware and software stack used in a Telco cloud.  This software uses declarative automation to simplify the deployment and upgrade of the cloud platform hardware and software to align with approved configuration profiles.  With declarative automation, the CSP simply defines the desired state of the cloud stack, and the automation software determines the steps required to achieve the desired state and executes those steps to align the system with the approved configuration.      

This infrastructure automation software uses a declarative data model that defines the desired state of the system, the resources properties, and keeps a list of the current state of the cloud configuration and inventory, which significantly simplifies the deployment and life cycle management of the cloud stack.  Infrastructure Blocks come with Topology and Orchestration Specification for Cloud Applications (TOSCA) workflows that define the configurations needed to update the system based on the extensive design and validation testing performed by Dell and Red Hat.  Dell provides regular updates that simplify the process of upgrading to the latest release. CSPs simply update a Customer Input Questionnaire and the automation software updates the hardware and the cloud platform software, and brings them to a workload-ready state with a single click of a button.

Integration

Dell ships fully integrated systems that are designed, optimized, and tested to meet the requirements of a range of telecom use cases and workloads. They include all the hardware, software, and licenses needed to build and scale out a Telco cloud. Delivering fully integrated building blocks from Dell’s factory significantly reduces the time spent configuring infrastructure onsite or in a network configuration center.  They are also backed by Dell Technologies one-call support for the full cloud stack that meets telecom SLAs.  Dell has established escalation paths with Red Hat to ensure the highest levels of support for customers.       

Streamlining Day 0 through Day 2 tasks with Telecom Infrastructure Blocks

In a typical CSP network operating model, there are four stages an CSPs goes through from initial technology onboarding through managing ongoing operations.  These stages are: 

• Stage 1: Technology onboarding (Day 0)
• Stage 2: Pre-production (Day 0)
• Stage 3: Production (Day 1)
• Stage 4: Operations and lifecycle management (Day 2+)

Dell Telecom Infrastructure Blocks were built to streamline each stage of the processes to reduce the time and risk of building and maintaining a Telco cloud.  We do this by proactively working with Red Hat to create an engineered system that meets telecom SLAs, includes automation that delivers zero-touch provisioning, and simplifies life cycle management through continuous design and integration testing.

Let's look at how Dell Telecom Infrastructure blocks affects CSP processes from Day 0 to Day 2+.

Stage 1: Day 0 technology onboarding 

Dell and Red Hat collaborate to design an engineered system that is validated through an extensive array of test cases to ensure its reliability and performance. These test cases cover aspects such as functionality, interoperability, security, reliability, scalability, and infrastructure-specific requirements for 5G Core workloads. This testing is aimed at ensuring optimal performance across a diverse array of performance metrics and scale points, thereby guaranteeing performant system operation. 

Some of our design and validation test cases include:

Cloud Infrastructure Cluster testing:  Cloud Infrastructure Cluster testing refers to the process of testing the infrastructure components of a cloud cluster to ensure their proper functioning, performance, and scalability. It involves validating the networking, storage, compute resources, and other infrastructure elements within a cluster. These test cases include:

• Installation and validation of Infrastructure Block plugins and automation.
• Cluster storage (Red Hat OpenShift AI) validation and testing.
• Validation of cluster network configurations 
• Verification of Container Network Interface (CNI) plugin configurations in the cluster.
• Validation of high availability configurations
• Scalability and performance testing 

Performance Benchmarking testing: Performance Benchmarking testing usually includes several steps. First, test scenarios are created to simulate real-world usage. Second, performance tests are conducted and the resulting performance data is collected and analyzed. Finally, the results are compared to established benchmarks. Some of the testing performed with Infrastructure Blocks includes:

• CPU benchmarking 
• Memory benchmarking 
• Storage benchmarking 
• Network benchmarking
• Container as a Service (CaaS) benchmarking
• Interoperability tests between the Infrastructure and CaaS layers

At this step, we define the automation workflows and configuration templates that will be used by the automation software during the deployment. Dell works proactively with its customers and partners to understand their best practices and incorporate those into the blueprints included with every Telecom Infrastructure Block. 

This process produces an engineered system that streamlines the CSP’s reference architecture design, benchmarking, and proof of concept (POC) processes to reduce engineering costs and accelerate the onboarding of new technology.  To further streamline the design, validation, and certification process, we provide Dell Open Telecom Ecosystem Lab (OTEL), which can act as an extension of a CSP’s lab to validate workloads on Infrastructure Blocks that meet the CSP’s defined requirements.

Stage 2: Pre-production

The main objective in stage 2 is to onboard network functions onto the cloud infrastructure, define the network golden configuration, and to prepare it for production deployment. Infrastructure Blocks eliminates some of the touch steps in the onboarding by:

• Delivering an integrated and validated building block direct from Dell’s factory
• Delivering a deployment guide that simplifies onboarding 
• Providing Customer Input Questionnaires that are configuration input templates used by the automation software to streamline deployment

CSPs can also leverage the Red Hat test line in Dell’s Open Telecom Ecosystem Lab (OTEL) to validate and certify the CSP’s workloads on Infrastructure Blocks.  OTEL can play a significant role in enabling CSPs and partners by developing custom blueprints or performing custom tests required by the CSP using Dell’s Solution Integration Platform (SIP) which is part of OTEL.  SIP is an advanced automation and service integration platform developed by Dell. It supports multi-vendor system integration and life cycle management testing at scale.   It uses industry standard components, toolkits, and solutions such as GitOps.  

 Dell Services also offers tailor-made configurations to cater to specific operator needs. These are carried out at a Dell second-touch configuration facility. Here, configurations customized to the customer's specifications are pre-installed and then dispatched directly to the customer's location or configuration facility.

Stage 3: Production

At the production stage, Dell integrates and configures all hardware and settings to support the discovery and installation of the validated version of the cloud platform software, which eliminates the need to configure hardware on site or in a configuration center. Dell’s infrastructure automation software then deploys the validated versions of Red Hat Advanced Cluster Manager and Red Hat Openshift on the servers used in the management and workload clusters and brings those clusters to a workload ready state.  This process ensures a consistent and reliable installation process that reduces the risk of configuration errors or compatibility issues. Dell's automation enables zero-touch provisioning that configures hundreds of servers at the same time with full visibility into the health and status of the server infrastructure before and after deployment.  Should the CSP need assistance with deployment, Dell's professional services team is standing by to assist. Dell ProDeploy for Telecom provides on-site support to rack, stack, and integrate servers into their network or  remote support for deployment. 

Stage 4: Operations and lifecycle management

In Day 2+ operations, CSPs must sustain network performance while adapting to changes to the network over time.  This includes ensuring software and infrastructure compatibility as updates are made to the network, performing rolling updates, fault and performance management, scaling resources to meet network demands, ensuring efficient use of network resources, and adapting to technology evolution. 

Infrastructure Blocks simplify Day 2+ operations in several ways:

• Dell works with Red Hat, its customers, and other partners to capture updates and new requirements necessary to evolve Infrastructure Blocks to support new technologies and software enhancements over time. This requires pro-active collaboration to ensure continuous roadmap alignment across parties.  Dell then performs extensive design and validation testing on these enhancements before integrating them into Infrastructure Blocks to deliver a resilient and performant design. This helps CSPs stay on the leading edge of the technology curve while minimizing the risk of encountering faults and performance issues in production.
• Today, Telecom Infrastructure Blocks offers support for three releases per year. In every release, we prioritize the introduction of new capabilities, features, components, and solution enhancements. In addition, there are six patch releases per year that prioritize sub features to ensure compatibility across different releases. Long Term Support releases are provided at the end of the twelve-month release cycle, with a focus on fixing any solution defects that may arise.
• The out-of-the box automation provided with Infrastructure Blocks ensures a consistent, carrier-grade deployment or upgrade of the hardware and cloud platform software each time. This eliminates configuration errors to further reduce issues found in production.
• When bringing together various hardware and software components, CSPs frequently manage different release cycles to support a range of workload requirements. To address any difficulties with software compatibility and life cycle management, Dell Technologies has created a system release cadence process.  It includes testing, validating, and locking the release compatibility matrixes for all Infrastructure Block components. This helps to resolve deployment problems affecting software compatibility and Day 2+ life cycle management procedures.
• Dell Professional Services can also provide custom integrations into a CSP’s CI/CD pipeline, providing the CSP with validated updates to the cloud infrastructure that pass directly into the CSP’s CI/CD tool chains to enhance DevOps processes.
• In addition, Dell offers single-call, carrier grade support that meets telecom grade SLAs with guaranteed response and service restoration times for the entire cloud stack (hardware & software).
• The declarative automation provided with Infrastructure Blocks eliminates the time spent updating scripts and playbooks to push out system updates and minimizes the risk of configuration errors that lead to fault or performance issues.   

Summary

Dell Telecom Infrastructure Blocks for Red Hat offers a streamlined and efficient way to build and manage Telco cloud infrastructure. From initial technology onboarding to Day 2+ operations, they simplify every step of the process.  This makes it easier for CSPs to transition from their vertically integrated, legacy architectures of today to an open cloud-native software platform running on industry standard hardware that delivers reliable and high-quality services to their customers.

This blog post is a collaborative effort from the staff of Dell Technologies and Red Hat.