Home > DevOps > Kubernetes > K8s Monitoring Setup

K8s Monitoring Setup

Setting Up Monitoring Using kube-prometheus-stack Helm Chart

The kube-prometheus-stack Helm chart provides an easy and comprehensive way to deploy a complete Prometheus-based monitoring solution in your Kubernetes cluster. It automates the deployment of several critical components for collecting, storing, visualizing, and alerting on metrics. Below is a step-by-step guide to understanding and deploying the stack.

Components of kube-prometheus-stack

Prometheus
- A time-series database that scrapes, stores, and exposes metrics from your Kubernetes environment and applications.
Node-Exporter
- Collects resource utilization data (CPU, memory, disk) from the nodes in your Kubernetes cluster.
- Automatically deployed and configured to be scraped by Prometheus.
Kube-State-Metrics
- Exposes information about Kubernetes API objects (e.g., Pods, containers, nodes) to Prometheus.
Grafana
- An observability platform that connects with Prometheus to create and display dashboards for visualizing metrics.
Alertmanager
- Provides notifications when metrics breach predefined thresholds (e.g., email alerts for high CPU usage or Slack notifications for Pod failures).

Installation Steps

1. Add the Helm Chart Repository

To begin, register the Prometheus community Helm chart repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

2. Update the Repository List

Fetch the latest charts available in the repository:

helm repo update

3. Install kube-prometheus-stack

Deploy the Helm chart into a new namespace in your Kubernetes cluster:

helm install kube-prometheus-stack \
  --create-namespace \
  --namespace kube-prometheus-stack \
  prometheus-community/kube-prometheus-stack

4. Verify the Installation

After running the installation command, you will see an output similar to this:

NAME: kube-prometheus-stack
LAST DEPLOYED: Tue Jan  3 14:26:18 2023
NAMESPACE: kube-prometheus-stack
STATUS: deployed
REVISION: 1

Check the Status of Deployed Components

Run the following command to see the status of the components:

kubectl -n kube-prometheus-stack get pods

Expected output:

NAME                                                       READY   STATUS    RESTARTS      AGE
alertmanager-kube-prometheus-stack-alertmanager-0          2/2     Running   1 (66s ago)   83s
kube-prometheus-stack-grafana-5cd658f9b4-cln2c             3/3     Running   0             99s
kube-prometheus-stack-kube-state-metrics-b64cf5876-52j8l   1/1     Running   0             99s
kube-prometheus-stack-operator-754ff78899-669k6            1/1     Running   0             99s
kube-prometheus-stack-prometheus-node-exporter-vdgrg       1/1     Running   0             99s
prometheus-kube-prometheus-stack-prometheus-0              2/2     Running   0             83s

Ensure all Pods show as Running to confirm successful deployment.

Accessing and Using the Monitoring Stack

Prometheus
- Prometheus starts scraping metrics from the exporters automatically after deployment.
Grafana
- Use Grafana to visualize Prometheus data:
  - Obtain the Grafana URL:
```
kubectl -n kube-prometheus-stack get svc
```
  - Use the credentials admin/prom-operator (default) to log in.
Alertmanager
- Configure alerts for metrics changes (e.g., high CPU utilization, Pod evictions).

Challaneges and Limitations

By default, Kubernetes uses the emptyDir volume type, which is ephemeral in nature. This means that any data stored in an emptyDir volume is tied to the lifecycle of the pod and will be lost if the pod restarts or is deleted.

To ensure data persistence, especially for critical components like Prometheus, it is essential to configure Persistent Volumes (PVs) and Persistent Volume Claims (PVCs). By doing so, Prometheus can retain its data across pod restarts or deletions, maintaining the continuity and reliability of its metrics and historical data. This approach decouples the storage from the pod lifecycle, allowing for long-term storage and improved fault tolerance.

In this context, we will set up a Persistent Volume using an NFS-backed storage solution to provide a robust and reliable foundation for Prometheus data storage. This ensures that even in scenarios of node failures or pod rescheduling, the data remains intact and accessible.

Dynamic NFS Storage Provisioning in Kubernetes

Dynamic NFS (Network File System) storage provisioning in Kubernetes allows automatic provisioning and management of NFS volumes for applications on-demand. This approach enables creating Persistent Volumes (PVs) and Persistent Volume Claims (PVCs) dynamically, eliminating the need for pre-provisioned storage or manual intervention. The NFS provisioner handles the creation and binding of PVs to PVCs by interacting with the NFS server.

Prerequisites

Pre-installed Kubernetes Cluster
A user with admin rights on the Kubernetes cluster
Internet connectivity

Steps for Configuring Dynamic NFS Storage Provisioning

Step 1: Prepare the NFS Server

Install the NFS server on the Kubernetes master node (Ubuntu 22.04):
```
sudo apt update
sudo apt install nfs-kernel-server -y
```

Create and share a folder for dynamic storage:

sudo mkdir /data/k8mondata
sudo chown -R nobody:nogroup /data/k8mondata
sudo chmod 2770 /data/k8mondata

Edit the exports file to configure NFS shares:
```
sudo vi /etc/exports
/data/k8mondata 172.21.0.0/16(rw,sync,no_subtree_check)
```
Replace 172.21.0.0/16 with the appropriate network for your deployment.

Apply the changes:

sudo exportfs -a
sudo systemctl restart nfs-kernel-server
sudo systemctl status nfs-kernel-server

Install NFS client utilities on worker nodes:
```
sudo apt install nfs-common -y
```

Step 2: Install and Configure NFS Client Provisioner

Install Helm (if not already installed):

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3
chmod 700 get_helm.sh
./get_helm.sh

Add the Helm repository for the NFS provisioner:

helm repo add nfs-subdir-external-provisioner https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner

Deploy the NFS provisioner:

helm install -n nfs-provisioning --create-namespace nfs-subdir-external-provisioner \
nfs-subdir-external-provisioner/nfs-subdir-external-provisioner \
--set nfs.server=172.21.0.36 --set nfs.path=/data/k8mondata

Verify the installation:
```
kubectl get all -n nfs-provisioning
kubectl get sc -n nfs-provisioning
```
Ensure the provisioner pod, deployment, and storage class (e.g., nfs-client) are created successfully.

Step 3: Create Persistent Volume Claims (PVCs)

Create a PVC configuration file (e.g., demo-pvc.yml):

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: demo-claim
  namespace: nfs-provisioning
spec:
  storageClassName: nfs-client
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 10Mi

Deploy the PVC:
```
kubectl create -f demo-pvc.yml
```
Verify the PVC and PV:
```
kubectl get pv,pvc -n nfs-provisioning
```

It creates the pv and pvc status bound.

Step 4: Test and Verify Dynamic NFS Provisioning

Create a test pod configuration file (e.g., test-pod.yml):

kind: Pod
apiVersion: v1
metadata:
  name: test-pod
  namespace: nfs-provisioning
spec:
  containers:
  - name: test-pod
    image: busybox:latest
    command:
      - "/bin/sh"
    args:
      - "-c"
      - "touch /mnt/SUCCESS && sleep 600"
    volumeMounts:
      - name: nfs-pvc
        mountPath: "/mnt"
  restartPolicy: "Never"
  volumes:
    - name: nfs-pvc
      persistentVolumeClaim:
        claimName: demo-claim

Deploy the test pod:
```
kubectl create -f test-pod.yml
```

Verify the pod status and NFS volume mount:

kubectl get pods -n nfs-provisioning
kubectl exec -it test-pod -n nfs-provisioning -- /bin/sh

Check if the volume is mounted and accessible.

Cleanup resources:

kubectl delete -f test-pod.yml
kubectl delete -f demo-pvc.yml
kubectl get pv,pvc -n nfs-provisioning

Ensure the PV is deleted automatically upon PVC deletion.

This setup ensures that your Kubernetes applications can dynamically provision and manage NFS storage effectively, with data persistence across pod restarts and deletions.

Configuring the Storage Class for Persistent Retention

By default, the reclaim policy for a Kubernetes Storage Class is set to Delete. This means that when a Persistent Volume (PV) is released, the data stored on it will be deleted. For scenarios where data persistence is critical, such as with Prometheus, this behavior is not ideal. To address this, we need to modify the reclaim policy to Retain, ensuring that the data remains intact even after the PV is released.

Steps to Configure the Storage Class:

Extract Current Helm Values
Retrieve the current configuration values for the NFS Subdir External Provisioner:
```
helm get values nfs-subdir-external-provisioner --namespace nfs-provisioning --all > nfs_values.yml
```

Edit the Configuration
Open the nfs_values.yml file and locate the reclaimPolicy field under the storageClass section. Update its value from Delete to Retain:

storageClass:
  ...
  reclaimPolicy: Retain
  ...

Example snippet after modification:

storageClass:
  accessModes: ReadWriteOnce
  allowVolumeExpansion: true
  annotations: {}
  archiveOnDelete: true
  create: true
  defaultClass: false
  name: nfs-client
  onDelete: null
  pathPattern: null
  reclaimPolicy: Retain
  volumeBindingMode: Immediate

Uninstall the Existing Release
Uninstall the current release of the NFS Subdir External Provisioner to apply the updated configuration:
```
helm uninstall <release-name> -n nfs-provisioning
```

Reinstall the Provisioner with Updated Values
Reinstall the provisioner using the modified configuration file:

helm install -f nfs_values.yml nfs-prov-retain-sc --namespace nfs-provisioning nfs-subdir-external-provisioner/nfs-subdir-external-provisioner

Verify the Changes
After reinstalling, ensure the Storage Class is created with the updated Retain reclaim policy:
```
kubectl get sc
```
Look for the ReclaimPolicy column to confirm it is set to Retain.

This configuration ensures that your persistent data remains secure, even when the associated Persistent Volume Claim (PVC) is deleted, providing a robust solution for data-critical applications.

Reconfiguring the Kube-Prometheus-Stack to Use Dynamic Volume Provisioning

To enable dynamic volume provisioning for the kube-prometheus-stack, follow these steps to configure it to utilize the NFS-based storage class (nfs-client):

Step 1: Extract the Default Values

Use the following command to retrieve the default configuration values of the kube-prometheus-stack Helm chart:

helm show values prometheus-community/kube-prometheus-stack >> values_persistent.yaml

Step 2: Modify the Configuration File

Open the generated values_persistent.yaml file and locate the prometheus section. Update the volumeClaimTemplate configuration under the prometheus settings as follows:

prometheus:
  volumeClaimTemplate:
    spec:
      ReclaimPolicy: Retain
      storageClassName: nfs-client
      accessModes: 
        - ReadWriteMany
      resources:
        requests:
          storage: 30Gi

storageClassName: Specifies the dynamic storage class to use (nfs-client).
accessModes: Set to ReadWriteMany to allow multiple pods to access the storage.
resources.requests.storage: Set the desired storage capacity (e.g., 30Gi).

Step 3: Deploy the Updated Configuration

Upgrade your kube-prometheus-stack release using the modified configuration file:

helm upgrade <release-name> prometheus-community/kube-prometheus-stack -f values_persistent.yaml -n kube-prometheus-stack

Uninstall all the previous releases of kube-prometheus-stack in helm.

Step 4: Verify the Persistent Volume

After deploying the changes, ensure that the Persistent Volume (PV) and Persistent Volume Claim (PVC) have been created dynamically. Use the following commands:

Check the PVCs:

kubectl get pvc -n kube-prometheus-stack

Check the corresponding PVs:
```
kubectl get pv
```
Confirm that the storageClassName for the PVC matches nfs-client and the storage capacity reflects the requested size (e.g., 30Gi).

Step 5: Verify Prometheus is Using the PVC

Ensure that Prometheus is running and utilizing the dynamically provisioned storage:

kubectl get pods -n kube-prometheus-stack -o wide

You should see the Prometheus pods running and mounted with the dynamically provisioned volume.

This configuration ensures that Prometheus data persists across pod restarts and redeployments, leveraging the dynamic NFS storage provisioned via the nfs-client storage class.

Accessing Prometheus and Grafana Dashboards Outside the Kubernetes Cluster

To enable external access to Prometheus and Grafana dashboards deployed in the kube-prometheus-stack namespace, we utilize HAProxy as a load balancer. HAProxy routes external traffic to the appropriate ClusterIP services within the Kubernetes cluster. Here’s the setup and configuration process:

Step 1: Check Services in the `kube-prometheus-stack` Namespace

Run the following command to list the services in the kube-prometheus-stack namespace:

kubectl get svc -n kube-prometheus-stack

Example output:

NAME                                                 TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
alertmanager-operated                                ClusterIP   None             <none>        9093/TCP,9094/TCP,9094/UDP   5d2h
nfs-configured-prom-stack-alertmanager               ClusterIP   10.109.165.184   <none>        9093/TCP,8080/TCP            5d2h
nfs-configured-prom-stack-grafana                    ClusterIP   10.104.88.100    <none>        80/TCP                       5d2h
nfs-configured-prom-stack-prometheus                 ClusterIP   10.96.250.81     <none>        9090/TCP,8080/TCP            5d2h

Key services for external access:

Grafana: nfs-configured-prom-stack-grafana with ClusterIP 10.104.88.100:80
Prometheus: nfs-configured-prom-stack-prometheus with ClusterIP 10.96.250.81:9090

Step 2: Install HAProxy

Install HAProxy on one of the servers within the Kubernetes cluster (e.g., 172.21.0.36). Configure HAProxy to route traffic to the respective ClusterIP services. The advantage of using ClusterIP is that it remains stable even after pod restarts or deletions.

Step 3: Configure HAProxy

Use the following HAProxy configuration to route traffic for Grafana and Prometheus dashboards:

frontend fe_grafana_server
  bind 0.0.0.0:3030
  mode tcp
  default_backend be_grafana_server
  http-request set-header X-Forwarded-Proto https


frontend fe_prometheus_server
  bind 0.0.0.0:9091
  mode tcp
  default_backend be_prometheus_server
  http-request set-header X-Forwarded-Proto https

backend be_grafana_server
  mode tcp
  option tcp-check
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server grafana_server 10.104.88.100:80 check

backend be_prometheus_server
  mode tcp
  option tcp-check
  default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  server prometheus_server 10.96.250.81:9090 check

Accessing Dashboards Using Private IP

Once the HAProxy configuration is set up, you can access the dashboards using the private IP of the server hosting HAProxy:

Grafana Dashboard: http://172.21.0.36:3030
Prometheus Dashboard: http://172.21.0.36:9091

This setup ensures that you can monitor and manage the Prometheus and Grafana dashboards externally without directly exposing your Kubernetes cluster services.