Docker Logging, Monitoring and Troubleshooting

# Docker Logging, Monitoring and Troubleshooting

Introduction

Effective logging, monitoring, and troubleshooting are essential for operating Docker containers in production. This tutorial covers Docker log drivers, container monitoring, debugging techniques, and common issues with resolutions.

Learning Objectives

By the end of this tutorial, you will be able to:

• Configure Docker log drivers
• View and manage container logs
• Monitor container resource usage
• Debug running containers
• Diagnose common Docker issues
• Implement centralized logging

Technical Background

Docker Log Drivers

| Driver | Description | |--------|-------------| | json-file | Default; stores logs as JSON | | syslog | Sends to syslog | | journald | Sends to systemd journal | | fluentd | Sends to Fluentd | | awslogs | Sends to CloudWatch | | splunk | Sends to Splunk |

Monitoring Commands

| Command | Description | |---------|-------------| | docker stats | Real-time resource usage | | docker events | Real-time events | | docker inspect | Detailed container info | | docker logs | Container logs | | docker exec | Run command in container |

Prerequisites

• Docker installed and running
• Running containers to monitor

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Docker | v24+ |

Step-by-Step Implementation

Phase 1: Configure Logging

#### Default json-file Logging

# View container logs
docker logs <container-name>

# Follow logs in real-time
docker logs -f <container-name>

# View logs with timestamps
docker logs -t <container-name>

# View last N lines
docker logs --tail=50 <container-name>

# View logs since timestamp
docker logs --since="2024-01-01T00:00:00" <container-name>

#### Configure Log Driver in daemon.json

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3",
    "compress": "true"
  }
}

#### Configure Per-Container Logging

# Run with specific log driver and options
docker run -d \
  --log-driver syslog \
  --log-opt syslog-address=tcp://192.168.1.100:514 \
  --log-opt tag="{{.Name}}/{{.ID}}" \
  --name myapp \
  myapp:latest

Phase 2: Monitor Container Resources

# Real-time resource usage
docker stats

# Specific container
docker stats <container-name>

# No stream, single output
docker stats --no-stream

# Format output
docker stats --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}"

Phase 3: Debug Running Containers

#### Inspect Container

# Detailed container information
docker inspect <container-name>

# Inspect specific field
docker inspect --format='{{.NetworkSettings.IPAddress}}' <container-name>

# Inspect container config
docker inspect --format='{{.Config.Env}}' <container-name>

#### Execute Commands in Container

# Run interactive shell
docker exec -it <container-name> /bin/bash

# Run command without shell
docker exec <container-name> ps aux

# Check running processes
docker top <container-name>

#### View Container Events

# Real-time events
docker events

# Filter events
docker events --filter container=<container-name>

# Events since timestamp
docker events --since="2024-01-01T00:00:00"

#### View Container Processes

# List processes in container
docker top <container-name>

# Detailed process info
docker exec <container-name> ps -ef

Phase 4: Troubleshooting Common Issues

#### Container Exits Immediately

# Check exit code
docker ps -a --filter name=<container-name>

# Check logs
docker logs <container-name>

# Run with entrypoint override to debug
docker run --rm -it --entrypoint /bin/sh myapp:latest

#### Cannot Connect to Container

# Check container is running
docker ps --filter name=<container-name>

# Check published ports
docker port <container-name>

# Check container network
docker inspect <container-name> | grep -A 10 Networks

# Test connectivity from host
curl http://localhost:<published-port>

#### High CPU or Memory Usage

# Monitor container
docker stats <container-name>

# Check processes inside container
docker exec <container-name> top

# Limit resources
docker run --memory="256m" --cpus="0.5" --name myapp myapp:latest

#### Disk Space Issues

# Check Docker disk usage
docker system df

# Detailed disk usage
docker system df -v

# Clean up unused data
docker system prune

# Remove all unused data
docker system prune -a --volumes

# Prune specific types
docker container prune
docker image prune
docker volume prune
docker network prune

Phase 5: Centralized Logging

#### Configure Fluentd Log Driver

# Run container with Fluentd logging
docker run -d \
  --log-driver fluentd \
  --log-opt fluentd-address=localhost:24224 \
  --log-opt tag=docker.{{.Name}} \
  --name myapp \
  myapp:latest

#### Docker Logging with ELK Stack

# docker-compose.yml with logging
version: '3.8'
services:
  app:
    image: myapp:latest
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "3"
  fluentd:
    image: fluent/fluentd:latest
    volumes:
      - ./fluentd.conf:/fluentd/etc/fluentd.conf
    ports:
      - "24224:24224"
    depends_on:
      - elasticsearch
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.10.0
    environment:
      - discovery.type=single-node
    ports:
      - "9200:9200"
  kibana:
    image: docker.elastic.co/kibana/kibana:8.10.0
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

Phase 6: Health Checks

# docker-compose.yml with healthchecks
version: '3.8'
services:
  app:
    image: myapp:latest
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s
    restart: unless-stopped

# Check health status
docker inspect --format='{{.State.Health.Status}}' <container-name>

# View health check history
docker inspect --format='{{json .State.Health}}' <container-name>

Expected Output

• Container logs accessible and configurable
• Container resource usage monitorable
• Debugging techniques working
• Common issues diagnosed and resolved
• Centralized logging configured

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Logs viewable | `docker logs <container>` | Application output | | Resource usage | `docker stats` | CPU and memory shown | | Container inspect | `docker inspect <container>` | Detailed JSON output | | Exec works | `docker exec <container> echo test` | Output: test | | Health check | `docker inspect --format='{{.State.Health.Status}}'` | healthy |

Troubleshooting Commands Cheat Sheet

# View all containers (including stopped)
docker ps -a

# View container logs
docker logs <container>

# Follow logs
docker logs -f <container>

# Inspect container
docker inspect <container>

# Execute command in container
docker exec -it <container> /bin/bash

# View running processes
docker top <container>

# Monitor resources
docker stats

# View events
docker events

# Check disk usage
docker system df

# Clean up unused resources
docker system prune -a --volumes

Best Practices

• **Configure log rotation:** Prevent disk space exhaustion
• **Use structured logging:** JSON format for easier parsing
• **Implement health checks:** Detect unhealthy containers
• **Monitor resource usage:** Set alerts for high CPU/memory
• **Centralize logs:** Use ELK or similar for production
• **Regular cleanup:** Remove unused containers, images, volumes

Security Considerations

• **Sanitize logs:** Remove PII and credentials from container output
• **Restrict log access:** Limit who can view container logs
• **Secure log transmission:** Use TLS for remote log shipping
• **Audit container access:** Log exec and attach commands

Conclusion

Docker logging, monitoring, and troubleshooting provide the observability needed to operate containerized applications in production. By configuring log drivers, monitoring resources, implementing health checks, and mastering debugging techniques, you can quickly diagnose and resolve issues.

References

• Docker Logging: https://docs.docker.com/config/containers/logging/
• Docker Monitoring: https://docs.docker.com/config/containers/monitoring/
• Docker Troubleshooting: https://docs.docker.com/config/daemon/troubleshoot/
• Docker Stats: https://docs.docker.com/engine/reference/commandline/stats/