By default, alert and metrics data are stored in an empty directory. When you delete the pods, the data is also deleted. We recommend that you save the data in persistent volumes.
Add PV storage to Prometheus and Alert Manager pods:
- Create a YAML file to create a config map (see this sample file in GitHub):
[core@csah ~] oc create -f <YAML file>
- Edit the config map and specify storage options by using volumeClaimTemplate for both Prometheus and Alert Manager pods:
Note: In the following YAML file, all the pods are 40 G and the storage class is powermax-bronze.
[core@csah pmax]$ oc edit configmap cluster-monitoring-config -n openshift-monitoring
apiVersion: v1
data:
config.yaml: |
prometheusk8s:
volumeClaimTemplate:
metadata:
name: prometheusdb
spec:
storageClassName: powermax-bronze
resources:
requests:
storage: 40Gi
alertmanagerMain:
volumeClaimTemplate:
metadata:
name: alertmanager
spec:
storageClassName: powermax-bronze
resources:
requests:
storage: 40Gi
kind: ConfigMap
metadata:
creationTimestamp: "2020-12-10T19:06:05Z"
name: cluster-monitoring-config
namespace: openshift-monitoring
resourceVersion: "4150896"
selfLink: /api/v1/namespaces/openshift-monitoring/configmaps/cluster-monitoring-config
uid: 4796aecd-e188-40a6-bfd0-6d069c5d01e5
- Save the configmap using :wq.
Note: This step terminates existing Prometheus and Alert Manager pods. New pods with storage added are created and are in the Running state, as shown in the following example.
[core@csah pvcs]$ oc get pods -n openshift-monitoring | grep -i -E "alertmanager|prometheus-k8s"
alertmanager-main-0 3/3 Running 0 4m
alertmanager-main-1 3/3 Running 0 4m
alertmanager-main-2 3/3 Running 0 4m
prometheus-k8s-0 7/7 Running 1 3m50s
prometheus-k8s-1 7/7 Running 1 3m50s
