By default, alert and metrics data are stored in an empty directory. When you delete the pods, the data is also deleted. We recommend that you save the data in persistent volumes.
Add PV storage to Prometheus and Alert Manager pods:
[core@csah ~] oc create -f <YAML file>
Note: In the following YAML file, all the pods are 40 G and the storage class is powermax-bronze.
[core@csah pmax]$ oc edit configmap cluster-monitoring-config -n openshift-monitoring
Note: This step terminates existing Prometheus and Alert Manager pods. New pods with storage added are created and are in the Running state, as shown in the following example.
[core@csah pvcs]$ oc get pods -n openshift-monitoring | grep -i -E "alertmanager|prometheus-k8s"
alertmanager-main-0 3/3 Running 0 4m
alertmanager-main-1 3/3 Running 0 4m
alertmanager-main-2 3/3 Running 0 4m
prometheus-k8s-0 7/7 Running 1 3m50s
prometheus-k8s-1 7/7 Running 1 3m50s