Gremoire

Monitoring Kubernetes with Prometheus & Grafana ​

Kubernetes is a powerful container orchestration platform, but monitoring your cluster is essential for stability, performance, and troubleshooting. In this guide, we will set up Prometheus to collect metrics from Kubernetes nodes and cAdvisor, and visualize them in Grafana. We’ll use Docker Compose to run the monitoring stack.

Step 1: Prerequisites ​

Before you begin, make sure you have:

  • A running Kubernetes cluster (v1.32+)
  • Docker & Docker Compose installed
  • kubectl configured with cluster access
  • Basic understanding of Kubernetes RBAC

Step 2: Create a Service Account for Prometheus ​

Prometheus needs permission to scrape Kubernetes metrics. Create a service account and generate a token:

bash
kubectl create serviceaccount prometheus-viewer -n kube-system
kubectl create clusterrolebinding prometheus-viewer \
  --clusterrole=view \
  --serviceaccount=kube-system:prometheus-viewer
kubectl -n kube-system create token prometheus-viewer > token.txt

This token.txt will be used by Prometheus to authenticate when scraping metrics.

Step 3: Set Up Docker Compose ​

Create a project directory with this structure:

.
├── docker-compose.yml
├── prometheus/
│   └── prometheus.yml
└── grafana/
    └── provisioning/

docker-compose.yml ​

yaml
version: "3.9"
services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    restart: always
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - ./token.txt:/etc/prometheus/token.txt:ro
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    restart: always
    ports:
      - "3000:3000"
    volumes:
      - grafana_data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning
    environment:
      - GF_SECURITY_ADMIN_USER=admin
      - GF_SECURITY_ADMIN_PASSWORD=admin
      - GF_USERS_ALLOW_SIGN_UP=false

volumes:
  grafana_data:

Prometheus Configuration (prometheus/prometheus.yml) ​

yaml
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "kubernetes-nodes"
    scheme: https
    metrics_path: /metrics
    static_configs:
      - targets:
          - 192.168.1.118:10250
          - 192.168.1.142:10250
          - 192.168.1.143:10250
          - 192.168.1.144:10250
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token.txt

  - job_name: "cadvisor"
    scheme: https
    metrics_path: /metrics/cadvisor
    static_configs:
      - targets:
          - 192.168.1.118:10250
          - 192.168.1.142:10250
          - 192.168.1.143:10250
          - 192.168.1.144:10250
    tls_config:
      insecure_skip_verify: true
    bearer_token_file: /etc/prometheus/token.txt

Step 4: Launch the Monitoring Stack ​

Run the following command to start Prometheus and Grafana:

bash
docker-compose up -d

Access your services:

  • Prometheus: http://localhost:9090
  • Grafana: http://localhost:3000 (username: admin, password: admin)

Step 5: Configure Grafana ​

  1. Go to Configuration → Data Sources → Add → Prometheus
  2. URL: http://prometheus:9090
  3. Click Save & Test

Step 6: Import Dashboards ​

Grafana has ready-made dashboards for Kubernetes:

  • Kubernetes Dashboard (ID: 19713)

These dashboards display node CPU, memory, pod status, and container metrics.

Step 7: Verify Metrics ​

Prometheus queries examples:

promql
node_cpu_seconds_total
container_memory_usage_bytes
kube_node_status_condition

You should see live metrics from all nodes and cAdvisor scraping container-level stats.

Tips & Best Practices ​

  • Mount the token.txt as read-only in Prometheus for security.
  • Use insecure_skip_verify: true only for internal clusters.
  • If you get 401 Unauthorized, verify RBAC and the token file.
  • Scrape intervals can be adjusted based on cluster size and performance.

✅ Now you have a fully functional monitoring stack for Kubernetes with Prometheus and Grafana!