Kubernetes Monitoring and Troubleshooting

# Kubernetes Monitoring and Troubleshooting

Introduction

Effective monitoring and troubleshooting are critical for operating Kubernetes clusters in production. This tutorial covers monitoring with Prometheus and Grafana, logging with ELK stack, debugging techniques, and common Kubernetes issues with resolutions.

Learning Objectives

By the end of this tutorial, you will be able to:

• Deploy Prometheus and Grafana for cluster monitoring
• Configure alerting with Alertmanager
• Aggregate logs with ELK stack
• Debug pods, deployments, and services
• Diagnose common Kubernetes issues
• Implement health checks and readiness probes

Technical Background

Monitoring Architecture

flowchart TD
    Pods[Pods] -->|Metrics| Kubelet[Kubelet]
    Kubelet -->|Metrics API| MetricsServer[Metrics Server]
    MetricsServer --> Prometheus[Prometheus]
    Prometheus --> Alertmanager[Alertmanager]
    Prometheus --> Grafana[Grafana]
    Logs[Logs] --> Fluentd[Fluentd]
    Fluentd --> Elasticsearch[Elasticsearch]
    Elasticsearch --> Kibana[Kibana]

Troubleshooting Flow

Issue Detected --> Check Pod Status --> Check Events --> Check Logs --> Check Network --> Check Storage --> Fix Issue

Prerequisites

• Running Kubernetes cluster
• kubectl configured
• Helm installed

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Kubernetes Cluster | v1.25+ | | Prometheus | v2.40+ | | Grafana | v10+ |

Step-by-Step Implementation

Phase 1: Install Monitoring Stack

# Add Prometheus Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack
helm install prometheus prometheus-community/kube-prometheus-stack \
  --namespace monitoring --create-namespace

# Verify
kubectl get pods -n monitoring

Phase 2: Configure Alerting

# alert-rules.yaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: kubernetes-alerts
  namespace: monitoring
spec:
  groups:
  - name: kubernetes-alerts
    rules:
    - alert: PodCrashLooping
      expr: rate(kube_pod_container_status_restarts_total[15m]) > 5
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "Pod crash looping detected"
    - alert: HighCPUUsage
      expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
      for: 5m
      labels:
        severity: warning

Phase 3: Install Logging Stack

# Add Elastic Helm repo
helm repo add elastic https://helm.elastic.co
helm repo update

# Install Elasticsearch, Kibana, Fluentbit
helm install elasticsearch elastic/elasticsearch \
  --namespace logging --create-namespace
helm install kibana elastic/kibana --namespace logging
helm install fluent-bit fluent/fluent-bit --namespace logging

Phase 4: Debugging Techniques

#### Check Pod Status

kubectl get pods
kubectl get pods -n <namespace>
kubectl describe pod <pod-name>

#### Check Events

kubectl get events --sort-by=.lastTimestamp
kubectl get events -n <namespace> --sort-by=.lastTimestamp

#### Check Logs

kubectl logs <pod-name>
kubectl logs <pod-name> -c <container-name>
kubectl logs --previous <pod-name>  # Previous container
kubectl logs -f <pod-name>  # Follow logs

#### Exec into Pod

kubectl exec -it <pod-name> -- /bin/sh
kubectl exec -it <pod-name> -c <container-name> -- /bin/sh

#### Port Forward

kubectl port-forward pod/<pod-name> 8080:80

#### Check Service Endpoints

kubectl get endpoints <service-name>
kubectl describe service <service-name>

Phase 5: Common Issues and Resolutions

#### ImagePullBackOff

**Cause:** Wrong image name, missing credentials, or network issue.

**Resolution:**

# Check image name and tag
kubectl describe pod <pod-name>

# Configure image pull secret
kubectl create secret docker-registry regcred \
  --docker-server=myregistry.com \
  --docker-username=user \
  --docker-password=pass \
  --docker-email=email@example.com

# Add to pod spec
imagePullSecrets:
- name: regcred

#### CrashLoopBackOff

**Cause:** Application crashing on startup.

**Resolution:**

# Check logs
kubectl logs <pod-name> --previous

# Check events
kubectl describe pod <pod-name>

# Fix application configuration or image

#### Pending Pod

**Cause:** Insufficient resources, taints, or scheduling constraints.

**Resolution:**

# Check events
kubectl describe pod <pod-name>

# Add resources or remove taints
kubectl taint nodes <node-name> <key>-  # Remove taint

#### DNS Resolution Failure

**Cause:** CoreDNS issues or misconfiguration.

**Resolution:**

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Restart CoreDNS
kubectl rollout restart deployment/coredns -n kube-system

Phase 6: Health Checks

# health-probes.yaml
apiVersion: v1
kind: Pod
metadata:
  name: health-check-pod
spec:
  containers:
  - name: app
    image: myapp:latest
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 30
      periodSeconds: 10
    readinessProbe:
      httpGet:
        path: /ready
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5
    startupProbe:
      httpGet:
        path: /startup
        port: 8080
      initialDelaySeconds: 5
      periodSeconds: 5

Expected Output

• Prometheus and Grafana monitoring cluster
• Alertmanager configured for alerts
• Logging stack aggregating logs
• Debugging procedures documented
• Common issues resolved

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Prometheus targets | `kubectl get servicemonitors -A` | ServiceMonitors listed | | Grafana access | `kubectl get svc -n monitoring` | Grafana service accessible | | Alert firing | Check Alertmanager | Alerts visible | | Log aggregation | Check Kibana | Logs searchable |

Troubleshooting Commands Cheat Sheet

# General
kubectl get all
kubectl get events --sort-by=.lastTimestamp
kubectl cluster-info

# Pods
kubectl describe pod <name>
kubectl logs <name>
kubectl exec -it <name> -- sh
kubectl logs --previous <name>

# Deployments
kubectl rollout status deployment/<name>
kubectl rollout history deployment/<name>
kubectl rollout undo deployment/<name>

# Nodes
kubectl get nodes
kubectl describe node <name>
kubectl top node

# Resources
kubectl top pods
kubectl top nodes

Best Practices

• **Use health probes:** Configure liveness, readiness, startup probes
• **Centralize logs:** Aggregate logs from all pods
• **Set resource requests and limits:** Prevent resource exhaustion
• **Monitor cluster components:** Track etcd, API server, scheduler
• **Create dashboards:** Visualize key metrics
• **Test alerting:** Verify alerts fire correctly
• **Document runbooks:** Create procedures for common issues

Security Considerations

• **Restrict monitoring access:** Limit who can view dashboards
• **Sanitize logs:** Remove PII and credentials
• **Secure Alertmanager:** Protect notification endpoints
• **Audit logging:** Enable Kubernetes audit logging

Conclusion

Kubernetes monitoring and troubleshooting provide the observability needed to operate reliable clusters. By implementing Prometheus, Grafana, logging, and debugging techniques, teams can detect issues early and resolve them quickly.

References

• Prometheus: https://prometheus.io/docs/
• Grafana: https://grafana.com/docs/
• Kubernetes Debugging: https://kubernetes.io/docs/tasks/debug/
• Kubernetes Troubleshooting: https://kubernetes.io/docs/tasks/debug/debug-cluster/
• Logging Architecture: https://kubernetes.io/docs/concepts/cluster-administration/logging/