Kubernetes Node Non-Graceful Shutdown and Remediation: Insights from Dell Technologies
Tue, 12 Dec 2023 18:16:57 -0000
|Read Time: 0 minutes
Introduction
Kubernetes has become a pivotal technology in managing containerized applications, but it's not without its challenges, particularly when dealing with Stateful Apps and non-graceful shutdown scenarios. This article delves into the intricacies of handling such situations, drawing insights from Dell Technologies' expertise and more importantly, how to enable it.
Understanding Graceful vs. Non-Graceful Node Shutdowns in Kubernetes
A 'graceful' node shutdown in Kubernetes is an orchestrated process. When kubelet detects a node shutdown event, it terminates the pods on that node properly, releasing resources before the actual shutdown. This orderly process allows critical pods to be terminated after regular pods, ensuring an application continues operating as long as possible. This process is vital for maintaining high availability and resilience in applications.
However, issues arise with a non-graceful shutdown, like a hard stop or node crash. In such cases, kubelet fails to detect a clean shutdown event. This leads to Kubernetes marking the node ‘NotReady', and Pods in a Stateful Set can remain stuck in 'Terminating' mode indefinitely!
Kubernetes adopts a cautious approach in these scenarios since it cannot ascertain if the issue is a total node failure, a kubelet problem, or a network glitch. This distinction is critical, especially for stateful apps, where rescheduling amidst active data writing could lead to severe data corruption.
Role of Dell's Container Storage Module (CSM) for Resiliency
Dell's CSM for Resiliency plays a crucial role in automating decision-making in these complex scenarios, aiming to minimize manual intervention and maximize uptime. The module's functionality is highlighted through a typical workflow:
- Consider a pod with two mounted volumes, annotated for protection with CSM resiliency.
- Upon an abrupt node power-off, the Kubernetes API detects the failure, marking the node as 'Not Ready'.
- The podmon controller of CSM Resiliency then interrogates the storage array, querying its status regarding the node and volumes.
- Depending on its findings and a set heuristic, the module determines whether it's safe to reschedule the pod.
- If rescheduling is deemed safe, the module quickly fences off access for the failed node, removes the volume attachment, and force-deletes the pod, enabling Kubernetes to reschedule it efficiently.
The following tutorial allow to test the functionality live: https://dell.github.io/csm-docs/docs/interactive-tutorials/
How to enable the module ?
To take advantage of the CSM resiliency you need two things:
- Enable it for your driver, for example with PowerFlex
- With the CSM wizard, just check the resiliency box
- With the Operator just set enable: true in the section .spec.modules.name['resiliency']
- With the helm chart set enable: true in the section .csi-vxflexos.podmon
- Then protect you application by adding the magic label podmon.dellemc.com/driver: csi-vxflexos
Conclusion:
Managing non-graceful shutdowns in Kubernetes, particularly for stateful applications, is a complex but essential aspect of ensuring system resilience and data integrity.
Tools like Dell's CSM for Resiliency are instrumental in navigating these challenges, offering automated, intelligent solutions that keep applications running smoothly even in the face of unexpected failures.
Sources
Stay informed of the latest updates of Dell CSM eco-system by subscribing to:
* The Dell CSM Github repository
* Our DevOps & Automation Youtube playlist
* The Slack