DevOps Monitoring and Logging

# DevOps Monitoring and Logging

Introduction

Monitoring and logging are essential for operating reliable software systems. Observability—the ability to understand a system's internal state from its external outputs—enables teams to detect issues early, diagnose problems quickly, and maintain service reliability. This tutorial covers metrics collection, dashboards, alerting, log aggregation, distributed tracing, and incident response practices.

Learning Objectives

By the end of this tutorial, you will be able to:

• Design monitoring architecture with metrics, logs, and traces
• Configure Prometheus for metrics collection
• Build Grafana dashboards for visualization
• Implement alerting with Alertmanager
• Aggregate logs with ELK stack
• Implement distributed tracing
• Define SLIs, SLOs, and SLAs
• Respond to incidents effectively

Technical Background

Observability Pillars

flowchart TD
    Metrics[Metrics] --> Observability[Observability]
    Logs[Logs] --> Observability
    Traces[Traces] --> Observability
    
    Metrics -->|Numerical data| TSDB[Time Series Database]
    Logs -->|Event records| LogStore[Log Storage]
    Traces -->|Request paths| TraceStore[Trace Storage]
    
    TSDB --> Dashboard[Dashboards]
    LogStore --> Search[Log Search]
    TraceStore --> TraceView[Trace Visualization]
    
    Dashboard --> Alerts[Alerting]
    Search --> Alerts
    TraceView --> Alerts
    
    subgraph Pillars
    Metrics
    Logs
    Traces
    end

SLIs, SLOs, and SLAs

| Term | Definition | Example | |------|------------|---------| | SLI | Service Level Indicator (metric) | Request latency < 200ms | | SLO | Service Level Objective (target) | 99% of requests < 200ms | | SLA | Service Level Agreement (contract) | 99.9% availability monthly |

Four Golden Signals

| Signal | Description | |--------|-------------| | Latency | Time to service a request | | Traffic | Demand on the system | | Errors | Rate of failed requests | | Saturation | How full the system is |

Prerequisites

• Linux servers or containers to monitor
• Understanding of application architecture
• Network access to targets
• Storage for metrics and logs
• Knowledge of PromQL or similar query language

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Monitoring Server | 4+ CPU, 8+ GB RAM, 100+ GB disk | | Prometheus | v2.40 or later | | Grafana | v10 or later | | Elasticsearch | v8 or later (for ELK) | | Logstash/Fluentd | For log collection | | Alertmanager | v0.25 or later |

Step-by-Step Implementation

Phase 1: Install Prometheus

# Create prometheus user
sudo useradd --no-create-home --shell /bin/false prometheus

# Create directories
sudo mkdir /etc/prometheus /var/lib/prometheus

# Download and install Prometheus
wget https://github.com/prometheus/prometheus/releases/download/v2.47.0/prometheus-2.47.0.linux-amd64.tar.gz
tar xvf prometheus-2.47.0.linux-amd64.tar.gz
sudo cp prometheus-2.47.0.linux-amd64/prometheus /usr/local/bin/
sudo cp prometheus-2.47.0.linux-amd64/promtool /usr/local/bin/

# Configure Prometheus
sudo tee /etc/prometheus/prometheus.yml << 'EOF'
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']
      
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']
EOF

sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheus

# Create systemd service
sudo tee /etc/systemd/system/prometheus.service << 'EOF'
[Unit]
Description=Prometheus
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.path=/var/lib/prometheus \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start prometheus
sudo systemctl enable prometheus

Phase 2: Install Node Exporter

# Download node exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-1.7.0.linux-amd64.tar.gz
sudo cp node_exporter-1.7.0.linux-amd64/node_exporter /usr/local/bin/

# Create systemd service
sudo tee /etc/systemd/system/node_exporter.service << 'EOF'
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start node_exporter
sudo systemctl enable node_exporter

Phase 3: Install Grafana

# Add Grafana repository
sudo apt install -y apt-transport-https software-properties-common
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
echo "deb https://packages.grafana.com/oss/stable stable main" | sudo tee /etc/apt/sources.list.d/grafana.list

# Install Grafana
sudo apt update
sudo apt install grafana -y

# Start Grafana
sudo systemctl start grafana-server
sudo systemctl enable grafana-server

Phase 4: Configure Prometheus Alertmanager

# alertmanager.yml
global:
  smtp_smtp_host: localhost
  smtp_from: alertmanager@example.com

route:
  group_by: ['alertname']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'team-email'
  routes:
    - match:
        severity: critical
      receiver: 'team-critical'

receivers:
  - name: 'team-email'
    email_configs:
      - to: devops-team@example.com
  - name: 'team-critical'
    email_configs:
      - to: devops-oncall@example.com
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/xxx'
        channel: '#alerts'
        title: 'Critical Alert'

# Install Alertmanager
wget https://github.com/prometheus/alertmanager/releases/download/v0.26.0/alertmanager-0.26.0.linux-amd64.tar.gz
tar xvf alertmanager-0.26.0.linux-amd64.tar.gz
sudo cp alertmanager-0.26.0.linux-amd64/alertmanager /usr/local/bin/

# Create service
sudo tee /etc/systemd/system/alertmanager.service << 'EOF'
[Unit]
Description=Alertmanager
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/alertmanager \
    --config.file=/etc/alertmanager/alertmanager.yml \
    --storage.path=/var/lib/alertmanager

[Install]
WantedBy=multi-user.target
EOF

sudo systemctl daemon-reload
sudo systemctl start alertmanager
sudo systemctl enable alertmanager

Phase 5: Create Alerting Rules

# alerts.yml
groups:
  - name: example_alerts
    rules:
      - alert: HighCPUUsage
        expr: 100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is above 80% (current value: {{ $value }}%)"
      
      - alert: DiskSpaceLow
        expr: (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is below 10% (current value: {{ $value }}%)"
      
      - alert: ServiceDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Service {{ $labels.job }} is down"
          description: "Service {{ $labels.job }} on {{ $labels.instance }} has been down for more than 1 minute"

Phase 6: Implement Logging with ELK Stack

#### Filebeat Configuration

# filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log
    - /var/log/myapp/*.log

filebeat.modules:
- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

output.elasticsearch:
  hosts: ["elasticsearch:9200"]
  username: "elastic"
  password: "${ELASTIC_PASSWORD}"

setup.kibana:
  host: "kibana:5601"

#### Logstash Pipeline

# logstash.conf
input {
  beats {
    port => 5044
  }
}

filter {
  grok {
    match => { "message" => "%{COMBINEDAPACHELOG}" }
  }
  date {
    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
    index => "myapp-logs-%{+YYYY.MM.dd}"
  }
}

Phase 7: Implement Distributed Tracing

#### OpenTelemetry Collector

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger]

Phase 8: Build Grafana Dashboards

#### Dashboard JSON Example

{
  "dashboard": {
    "title": "System Overview",
    "panels": [
      {
        "title": "CPU Usage",
        "type": "stat",
        "targets": [
          {
            "expr": "100 - (avg by(instance) (rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
            "legendFormat": "{{instance}}"
          }
        ]
      },
      {
        "title": "Memory Usage",
        "type": "graph",
        "targets": [
          {
            "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100",
            "legendFormat": "{{instance}}"
          }
        ]
      }
    ]
  }
}

Phase 9: Incident Response

#### Incident Response Workflow

Alert Detected --> Triage --> Investigate --> Mitigate --> Resolve --> Postmortem

#### Incident Severity Levels

| Severity | Definition | Response Time | |----------|------------|---------------| | SEV-1 | Complete service outage | 15 minutes | | SEV-2 | Major feature degraded | 1 hour | | SEV-3 | Minor issue, workaround available | 4 hours | | SEV-4 | Cosmetic issue | Next business day |

#### Blameless Postmortem Template

1. **Summary:** What happened? 2. **Timeline:** When did it start, when was it resolved? 3. **Impact:** How many users affected, what was the business impact? 4. **Root Cause:** What caused the incident? 5. **Contributing Factors:** What made it worse? 6. **Action Items:** What will we do to prevent recurrence?

Expected Output

• Prometheus collecting metrics from targets
• Grafana dashboards visualizing system health
• Alertmanager routing alerts to appropriate channels
• ELK stack aggregating and searching logs
• Distributed tracing for microservices
• SLIs, SLOs, and alerts defined
• Incident response procedures documented

Verification Procedures

| Check | Method | Expected Result | |-------|--------|-----------------| | Prometheus targets | http://prometheus:9090/targets | All targets UP | | Grafana datasource | http://grafana:3000 | Prometheus connected | | Alert firing | Trigger test alert | Notification received | | Log aggregation | Search Kibana | Logs visible and searchable | | Dashboard | Open Grafana dashboard | Metrics displayed correctly |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Targets DOWN in Prometheus | Network or exporter issue | Check connectivity; verify exporter running | | No data in Grafana | Wrong query or datasource | Verify PromQL query; check datasource config | | Alert not firing | Wrong threshold or expression | Test expression in Prometheus UI | | Logs not appearing | Filebeat not shipping | Check Filebeat logs; verify Elasticsearch connectivity | | Dashboard slow | Too many queries or high cardinality | Optimize queries; reduce time range |

Best Practices

• **Monitor the four golden signals:** Latency, traffic, errors, saturation
• **Set meaningful SLOs:** Base SLOs on user expectations, not technical capabilities
• **Alert on symptoms, not causes:** Alert when users are affected, not when CPU is high
• **Use check mode:** Test alerting rules before deploying
• **Centralize logs:** Aggregate logs from all services in one place
• **Retain data appropriately:** Keep metrics for 15 days, logs for 30 days
• **Document runbooks:** Clear procedures for common incidents
• **Test alerting:** Regularly verify alerts fire correctly

Security Considerations

• **Restrict monitoring access:** Limit who can view dashboards and alerts
• **Encrypt log data:** Use TLS for log transmission
• **Sanitize logs:** Remove PII and credentials from logs
• **Secure Alertmanager:** Protect notification endpoints
• **Audit dashboard access:** Log who views and modifies dashboards
• **Separate monitoring network:** Isolate monitoring traffic from production

Conclusion

Monitoring and logging provide the observability needed to operate reliable systems. By collecting metrics, aggregating logs, implementing distributed tracing, and defining meaningful alerts, teams can detect issues before they impact users and respond quickly when problems occur. The combination of Prometheus, Grafana, ELK, and Alertmanager provides a comprehensive observability stack for modern DevOps teams.

References

• Prometheus Documentation: https://prometheus.io/docs/
• Grafana Documentation: https://grafana.com/docs/
• ELK Stack: https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
• SRE Book: https://sre.google/sre-book/
• Four Golden Signals: https://sre.google/sre-book/monitoring-distributed-systems/
• OpenTelemetry: https://opentelemetry.io/docs/