After we had the physical infrastructure layer deployed and updated, we installed AKS hybrid. We used Windows Admin Center to install AKS hybrid in the lab, but IT administrators can also use PowerShell or the Azure portal to accomplish this. We confirmed that we met all the requirements in the System requirements for Azure Kubernetes Service on Azure Stack HCI and Windows Server documentation before stepping through the setup. The automated deployment of AKS hybrid using Windows Admin Center removed all the complexity of a typical Kubernetes deployment using predefined configurations from Microsoft.

The deployment process involved the creation of two types of clusters:

• Management cluster (or AKS host) – dbaas-mgmt-cluster-1 – This cluster consisted of a single Linux VM with the core orchestration components of the AKS hybrid service installed. Core components included the API server, which provided an access point for tools like Windows Admin Center, PowerShell modules, Azure CLI, and kubectl. A load balancer directed traffic to the API server.
• Workload clusters (or target clusters) – Software developers deploy and run containerized applications in these clusters. We built the following two workload clusters in the lab:
  • dbaas-databases-1 – Dedicated Azure Arc-enabled SQL Managed Instances and SQL databases.
  • dbaas-applications-1 – Dedicated to containerized HammerDB instances for synthetic benchmarking of the SQL databases. This cluster simulated the application tier of cloud-native applications.

  Each workload cluster consisted of the following:

• 1 x Linux VM – Control plane API server that also ran etcd for life cycle management.
• 1 x Linux VM – Ran HAProxy and KeepAlive to provide load-balancing services.
• 3 x Linux-based (Mariner) worker node VMs – This node pool ran Kubernetes system and user pods and the Azure Arc-enabled Kubernetes agents. It also hosted the Hammer database and SQL MI pods in the dbaas-applications-1 and dbaas-databases-1 workload clusters, respectively. One namespace in each cluster was dedicated to pods running the Prometheus and Grafana monitoring tools.  

The following figure illustrates the AKS hybrid setup in the lab.

Figure 10.    AKS hybrid logical architecture

Windows Admin Center provides some basic AKS hybrid administrative functionality like the ability to download workload cluster’s kubeconfig files and perform non-disruptive updates of the Kubernetes environment.

Figure 11.    AKS hybrid environment managed using Windows Admin Center

The AKS hybrid deployment automatically distributed each worker node in the node pools across the three physical Azure Stack HCI cluster nodes as depicted in the next figure.

Figure 12.    AKS hybrid VMs automatically distributed across Azure Stack HCI cluster

For dbaas-database-1, we selected one of the larger AKS hybrid worker node VM sizes, Standard_D16s_v3, to accommodate multiple Azure Arc-enabled SQL MI pods per node. In our performance testing, we targeted a cluster-wide CPU and memory average utilization of 80%. This left a 20% buffer for Kubernetes operations. We also reserved enough free capacity on each physical Azure Stack HCI cluster node for autoscaling node pools and for VM provisioning.

The dbaas-applications-1 workload cluster did not require as large an AKS hybrid worker node VM size since it was only running containerized instances of HammerDB. In a production environment, the sizing of both clusters would depend on many variables including the number of applications being created by the software development team and their anticipated performance and capacity demands. The following table summarizes all the configuration details about the environment we deployed in the lab.

Table 4.      AKS hybrid configuration details

Management and operations

The following table lists the tools we used to perform the deployment, life cycle management, and monitoring of the AKS hybrid layer.

Note: We always found it beneficial to ensure we were running the latest versions of all the tools listed – especially for Helm, Azure CLI, and the Azure CLI extensions.  

Table 5.      Tools used with AKS hybrid layer

We onboarded the workload clusters to Azure Arc-enabled Kubernetes to make use of Azure management and governance services like Azure Monitor Container Insights and Azure Policy. We also heavily leveraged the ability to explore the deployed Kubernetes resources from within the Azure portal as shown in the following figure.  

Figure 13.    AKS hybrid resource explorer in the Azure portal

We opted to install the monitoring and logging solutions described in the AKS hybrid documentation section Use on-premises monitoring. Prometheus is an open-source application used for monitoring and alerting of containerized workloads. Prometheus can be deployed during initial creation of the workload cluster or deployed on an existing workload cluster using a single PowerShell command. Grafana is a tool for visualizing metrics, logs, and other data in a wide variety of formats. Microsoft provides guidance on installing Grafana for AKS hybrid and configuring it to use Prometheus as the data source using a helm chart in the AKS-HCI-Apps GitHub Repo. This repository also provides details for on-premises logging based on Elasticsearch, Fluent Bit, and Kibana (EFK).

Figure 14.    AKS hybrid Grafana dashboard