Linux Performance Monitoring and Troubleshooting

# Linux Performance Monitoring and Troubleshooting

Introduction

Linux performance monitoring and troubleshooting are critical skills for maintaining responsive, reliable systems. Understanding CPU, memory, disk, and network performance enables administrators to identify bottlenecks, plan capacity, and resolve issues before they impact users.

This tutorial covers performance monitoring tools (top, htop, vmstat, iostat, sar, perf), performance analysis methodology, CPU/memory/disk/network troubleshooting, and capacity planning.

By the end of this guide, you will be able to monitor Linux system performance, identify bottlenecks, and resolve performance issues.

Technical Background

Performance Metrics

| Resource | Key Metrics | Tools | |----------|-------------|-------| | CPU | Utilization, load average, context switches, CPU steal | top, mpstat, perf | | Memory | Used, free, buffered, cached, swap, page faults | free, vmstat, sar | | Disk | Utilization, throughput, IOPS, await, service time | iostat, iotop, sar | | Network | Bandwidth, packets, errors, drops, latency | sar, iftop, netstat, ss |

Load Average

Load average represents the average number of processes waiting for CPU or I/O over 1, 5, and 15 minutes. A load average equal to the number of CPU cores indicates full utilization.

CPU States

| State | Description | |-------|-------------| | us (user) | CPU time in user space | | sy (system) | CPU time in kernel space | | ni (nice) | CPU time for low-priority processes | | id (idle) | CPU idle time | | wa (IO wait) | CPU waiting for I/O | | st (steal) | CPU time stolen by hypervisor (virtualized) |

Prerequisites

• Linux system with root or sudo access
• Performance monitoring tools installed
• Understanding of system resources (CPU, memory, disk, network)

Architecture

Performance Monitoring Stack

Application → Kernel → Hardware
   ↓           ↓         ↓
perf, eBPF   /proc     CPU, Disk, NIC
   ↓           ↓         ↓
top, vmstat  /sys       iostat, sar
   ↓           ↓         ↓
Analysis → Visualization → Alerting

Step-by-Step Implementation

Step 1: Install Performance Monitoring Tools

**Purpose**: Install essential performance monitoring packages.

**Commands** (Ubuntu/Debian):

sudo apt update
sudo apt install sysstat htop iotop perf bpfcc-tools

**Commands** (RHEL/CentOS/Fedora):

sudo dnf install sysstat htop iotop perf bcc-tools

**Enable sar data collection**:

# Enable sysstat data collection
sudo sed -i 's/ENABLED="false"/ENABLED="true"/' /etc/default/sysstat
sudo systemctl restart sysstat

**Expected output**: Performance tools installed; sar data collection enabled.

**Verification**:

which top htop vmstat iostat sar perf
# All tools found

Step 2: Monitor CPU Performance

**Purpose**: Identify CPU bottlenecks and high-load processes.

**Commands**:

# Real-time process monitoring
top
htop

# Per-CPU statistics
mpstat -P ALL 1

# CPU utilization
top -b -n 1 | head -20

# Load average
uptime
cat /proc/loadavg

# Context switches
vmstat 1 5

# Detailed CPU profiling
sudo perf top

**Expected output**: CPU utilization, load average, and top CPU-consuming processes.

**Verification**:

uptime
# Shows: load average: 0.50, 0.30, 0.20

Step 3: Monitor Memory Performance

**Purpose**: Identify memory pressure, swapping, and cache usage.

**Commands**:

# Memory overview
free -h

# Detailed memory info
cat /proc/meminfo

# Virtual memory statistics
vmstat -s

# Memory usage by process
ps aux --sort=-%mem | head -10

# Check swap usage
swapon --show
cat /proc/swaps

# Monitor memory over time
vmstat 1 10

**Key memory indicators**:

• `free`: Available memory
• `buff/cache`: Memory used by kernel buffers and page cache
• `swap`: Swap space used (high swap = memory pressure)
• `si/so` (vmstat): Swap in/swap out (high values = thrashing)

**Expected output**: Memory usage, swap status, and memory pressure indicators.

**Verification**:

free -h
# Shows: total, used, free, shared, buff/cache, available

Step 4: Monitor Disk Performance

**Purpose**: Identify disk I/O bottlenecks and high disk utilization.

**Commands**:

# Disk I/O statistics
iostat -x 1 5

# I/O by process
sudo iotop -o

# Disk usage
df -h

# Directory sizes
du -sh /var/log /home/*

# Block device statistics
iostat -d -x /dev/sda 1

# Check disk queue length
cat /proc/diskstats

**Key disk metrics**:

• `%util`: Percentage of time device is busy (> 80% = bottleneck)
• `await`: Average wait time for I/O (ms)
• `svctm`: Average service time (ms)
• `r/s`, `w/s`: Read/write operations per second

**Expected output**: Disk I/O statistics and utilization.

**Verification**:

iostat -x 1 5
# Shows %util, await, svctm for each device

Step 5: Monitor Network Performance

**Purpose**: Identify network bottlenecks, errors, and high-bandwidth connections.

**Commands**:

# Network interface statistics
ip -s link show eth0

# Network connections
ss -tanp

# Network bandwidth by connection
sudo iftop

# Network statistics
sar -n DEV 1 5

# Check for errors
netstat -i

# Ping and latency
ping -c 4 8.8.8.8
mtr -rwc 10 8.8.8.8

**Expected output**: Network interface statistics and connection details.

**Verification**:

ip -s link show eth0
# Shows RX/TX packets, bytes, errors, drops

Step 6: Use sar for Historical Performance Data

**Purpose**: Review historical performance data collected by sysstat.

**Commands**:

# CPU usage today
sar -u

# CPU usage on specific day
sar -u -f /var/log/sysstat/sa15

# Memory usage
sar -r

# Disk I/O
sar -d

# Network I/O
sar -n DEV

# Load average
sar -q

# View all data for specific time range
sar -s 09:00:00 -e 17:00:00

**Expected output**: Historical performance data for trend analysis.

**Verification**:

sar -u
# Shows CPU utilization percentages over time

Step 7: Use perf for Detailed CPU Profiling

**Purpose**: Profile CPU usage at function level.

**Commands**:

# System-wide profiling
sudo perf top

# Record profile for later analysis
sudo perf record -a -g sleep 30

# View recorded profile
sudo perf report

# Profile specific command
sudo perf record -g ./myapp
sudo perf report

**Expected output**: Function-level CPU usage profile.

**Verification**:

sudo perf top
# Shows top CPU-consuming functions

Configuration Examples

sysstat Configuration (`/etc/default/sysstat`)

# Enable data collection
ENABLED="true"

# Compress old data files
COMPRESSAFTER="10"

# Directory for data files
DIR="/var/log/sysstat"

/etc/sysstat.conf (I/O statistics configuration)

# Devices to monitor (empty = all)
# OPTIONS="-x"

Performance Monitoring Script (`/usr/local/bin/perf-check.sh`)

#!/bin/bash
echo "=== CPU Load ==="
uptime
echo ""
echo "=== Memory ==="
free -h
echo ""
echo "=== Top CPU Processes ==="
ps aux --sort=-%cpu | head -10
echo ""
echo "=== Top Memory Processes ==="
ps aux --sort=-%mem | head -10
echo ""
echo "=== Disk I/O ==="
iostat -x 1 2
echo ""
echo "=== Network ==="
ip -s link show | grep -A 5 "RX"

Cron Job for Performance Monitoring

# /etc/cron.d/perf-monitoring
*/5 * * * * root /usr/local/bin/perf-check.sh >> /var/log/perf-monitor.log 2>&1

Verification

Performance Verification Commands

uptime                    -- Load average
free -h                   -- Memory usage
mpstat -P ALL 1           -- Per-CPU usage
iostat -x 1 5             -- Disk I/O
sar -u                    -- Historical CPU
sar -r                    -- Historical memory
sar -d                    -- Historical disk I/O
ss -tanp                  -- Network connections
top -b -n 1               -- Snapshot of processes

Expected Results

• Load average below number of CPU cores
• Memory available > 20% of total
• Disk %util below 80%
• Network errors and drops near zero
• No processes consuming excessive CPU or memory

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | High CPU load | CPU-intensive process | `top`, `ps aux --sort=-%cpu` | Kill or optimize process; add CPU | Load normalizes | | High memory usage | Memory leak or insufficient RAM | `free -h`, `ps aux --sort=-%mem` | Add RAM; fix memory leak; add swap | Memory available | | Swapping | Insufficient RAM | `vmstat`, `swapon --show` | Add RAM; reduce memory usage | Swap usage low | | High disk %util | Disk bottleneck | `iostat -x` | Add SSD; optimize I/O; add disks | %util below 80% | | High disk await | Slow disk or I/O queue | `iostat -x` | Check disk health; reduce I/O | await normalizes | | Network errors | Bad cable, NIC issue | `ip -s link show` | Replace cable; check NIC | Errors drop to 0% | | Slow DNS | DNS server issue | `dig`, `resolvectl status` | Use faster DNS; check DNS server | DNS resolves quickly | | Application slow | Resource contention | `top`, `perf top` | Optimize application; add resources | Application responsive | | High context switches | Too many threads/processes | `vmstat` | Reduce thread count; tune application | Context switches lower |

Best Practices

1. **Monitor continuously**: Use sar or monitoring tools for historical data. 2. **Establish baselines**: Record normal performance for comparison during issues. 3. **Monitor all four resources**: CPU, memory, disk, and network. 4. **Set up alerts**: Alert on high utilization (CPU > 80%, memory > 85%, disk > 85%). 5. **Use perf for deep analysis**: Profile CPU when top doesn't show enough detail. 6. **Keep sysstat enabled**: Historical data is essential for trend analysis. 7. **Monitor swap**: High swap usage indicates memory pressure. 8. **Check disk queue length**: High queue indicates I/O bottleneck. 9. **Use htop for visual monitoring**: Better than top for real-time observation. 10. **Document performance incidents**: Record cause, resolution, and prevention.

Conclusion

Linux performance monitoring requires a systematic approach covering CPU, memory, disk, and network resources. Tools like top, vmstat, iostat, sar, and perf provide visibility into system behavior. Establishing baselines, monitoring continuously, and setting up alerts enable proactive performance management. When issues occur, a methodical troubleshooting approach identifies bottlenecks and guides resolution. Performance monitoring is an ongoing practice that ensures Linux systems remain responsive and reliable under varying workloads.

References

• sysstat Documentation (https://github.com/sysstat/sysstat)
• perf Documentation (https://perf.wiki.kernel.org/)
• Linux Performance Tools (https://www.brendangregg.com/linuxperf.html)
• top, htop Documentation
• vmstat, iostat man pages
• Brendan Gregg's USE Method (https://brendangregg.com/usemethod.html)