Linux Systemd Services and Boot Process

# Linux Systemd Services and Boot Process

Introduction

systemd is the system and service manager for most modern Linux distributions. It replaces the traditional SysV init system and provides parallelized service startup, dependency-based service control, on-demand daemon startup, and comprehensive logging through journald.

This tutorial covers the Linux boot process, systemd architecture, service management with systemctl, unit files, targets, timer units, logging with journalctl, and service troubleshooting.

By the end of this guide, you will be able to manage systemd services, understand the boot process, and troubleshoot service and boot issues.

Technical Background

Linux Boot Process (systemd)

BIOS/UEFI → Bootloader (GRUB) → Kernel → Initramfs → systemd (PID 1) → Target → Services

Boot Process Steps

| Step | Component | Description | |------|-----------|-------------| | 1 | BIOS/UEFI | Hardware initialization and POST | | 2 | Bootloader | GRUB loads kernel and initramfs | | 3 | Kernel | Kernel initializes hardware and mounts root filesystem | | 4 | Initramfs | Temporary filesystem with drivers; loads root filesystem | | 5 | systemd | PID 1; manages all processes and services | | 6 | Target | System target (multi-user, graphical) | | 7 | Services | System services started in dependency order |

systemd Architecture

systemd (PID 1)
├── Service Units (.service)
├── Target Units (.target)
├── Socket Units (.socket)
├── Timer Units (.timer)
├── Path Units (.path)
├── Device Units (.device)
├── Mount Units (.mount)
└── Slice Units (.slice)

systemd Unit Types

| Unit Type | Extension | Description | |-----------|-----------|-------------| | Service | .service | Manage daemons and processes | | Target | .target | Synchronization point (runlevels) | | Socket | .socket | Socket-based activation | | Timer | .timer | Scheduled jobs (replaces cron) | | Path | .path | File-based activation | | Mount | .mount | Filesystem mount points | | Device | .device | Kernel devices | | Slice | .slice | Process grouping for resource control |

systemd Targets (Runlevel Equivalents)

| Target | Alias | Description | |--------|-------|-------------| | poweroff.target | runlevel0 | System shutdown | | rescue.target | runlevel1 | Single-user rescue mode | | multi-user.target | runlevel3 | Multi-user, text mode | | graphical.target | runlevel5 | Multi-user, graphical mode | | reboot.target | runlevel6 | System reboot |

Prerequisites

• Linux system with systemd (most modern distributions)
• Root or sudo access
• Understanding of Linux services and processes

Architecture

Service Lifecycle

inactive → activating → active → deactivating → inactive
    ↓           ↓           ↓           ↓           ↓
  not running  starting   running    stopping    not running

systemd Unit Resolution Order

1. /etc/systemd/system/ (administrator overrides) 2. /run/systemd/system/ (runtime units) 3. /usr/lib/systemd/system/ (package-installed units) 4. /lib/systemd/system/ (legacy location)

Step-by-Step Implementation

Step 1: Understand the Boot Process

**Purpose**: Examine the boot process and systemd initialization.

**Commands**:

# Check systemd version
systemctl --version

# Check systemd is PID 1
ps -p 1 -o comm=

# View boot time and performance
systemd-analyze

# View detailed boot timeline
systemd-analyze blame
systemd-analyze critical-chain

# View boot log
journalctl -b

# View previous boot log
journalctl -b -1

**Expected output**: systemd information and boot analysis.

**Verification**:

ps -p 1 -o comm=
# Shows: systemd
systemctl --version
# Shows systemd version

Step 2: Manage System Services

**Purpose**: Start, stop, enable, and disable system services.

**Commands**:

# Check service status
sudo systemctl status nginx

# Start a service
sudo systemctl start nginx

# Stop a service
sudo systemctl stop nginx

# Restart a service
sudo systemctl restart nginx

# Reload a service (config reload without restart)
sudo systemctl reload nginx

# Enable service to start on boot
sudo systemctl enable nginx

# Disable service from starting on boot
sudo systemctl disable nginx

# Check if service is enabled
sudo systemctl is-enabled nginx

# Check if service is active
sudo systemctl is-active nginx

# List all services
systemctl list-units --type=service

# List failed services
systemctl --failed

**Expected output**: Service status and management operations completed.

**Verification**:

sudo systemctl status nginx
# Shows: Active: active (running) or inactive (dead)

Step 3: Create Custom Systemd Service Unit

**Purpose**: Create a custom service unit for an application.

**Commands**:

# Create service unit file
sudo nano /etc/systemd/system/myapp.service

**Example service unit file** (`/etc/systemd/system/myapp.service`):

[Unit]
Description=My Custom Application
After=network.target
Wants=network.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/opt/myapp
ExecStart=/opt/myapp/myapp-server
ExecReload=/bin/kill -HUP $MAINPID
Restart=always
RestartSec=5
StandardOutput=journal
StandardError=journal

[Install]
WantedBy=multi-user.target

**Apply and start service**:

# Reload systemd to recognize new unit
sudo systemctl daemon-reload

# Enable and start service
sudo systemctl enable myapp
sudo systemctl start myapp

# Check status
sudo systemctl status myapp

**Expected output**: Custom service created, enabled, and running.

**Verification**:

sudo systemctl status myapp
# Shows: Active: active (running)

Step 4: Manage systemd Targets

**Purpose**: Switch between system states (runlevel equivalents).

**Commands**:

# Check current target
systemctl get-default

# List all targets
systemctl list-units --type=target

# Set default target
sudo systemctl set-default multi-user.target

# Switch to target immediately
sudo systemctl isolate graphical.target

# Emergency/rescue mode
sudo systemctl rescue
sudo systemctl emergency

**Expected output**: System target changed.

**Verification**:

systemctl get-default
# Shows: multi-user.target or graphical.target

Step 5: View and Analyze Logs with journalctl

**Purpose**: Read systemd journal logs for troubleshooting.

**Commands**:

# View all logs
journalctl

# View logs for current boot
journalctl -b

# View logs for specific service
journalctl -u nginx

# Follow logs in real-time
journalctl -u nginx -f

# View logs since specific time
journalctl --since "2024-01-15 10:00:00"

# View kernel messages
journalctl -k

# View logs with priority
journalctl -p err

# Check disk usage of journal
journalctl --disk-usage

# Vacuum old logs
sudo journalctl --vacuum-size=500M

**Expected output**: Log entries for system and service troubleshooting.

**Verification**:

journalctl -u nginx --since "1 hour ago"
# Shows nginx logs from last hour

Step 6: Create Timer Units (Replacing cron)

**Purpose**: Schedule tasks using systemd timers.

**Commands**:

# Create timer unit
sudo nano /etc/systemd/system/backup.timer

**Example timer unit** (`/etc/systemd/system/backup.timer`):

[Unit]
Description=Daily Backup Timer

[Timer]
OnCalendar=daily
Persistent=true
Unit=backup.service

[Install]
WantedBy=timers.target

**Example service unit** (`/etc/systemd/system/backup.service`):

[Unit]
Description=Daily Backup Service

[Service]
Type=oneshot
ExecStart=/usr/local/bin/backup.sh

**Enable and start timer**:

sudo systemctl daemon-reload
sudo systemctl enable --now backup.timer

# List timers
systemctl list-timers --all

# Check next run
systemctl list-timers backup.timer

**Expected output**: Timer created and scheduled.

**Verification**:

systemctl list-timers --all
# Shows backup.timer with next run time

Step 7: Enable and Disable Services at Boot

**Purpose**: Control which services start automatically at boot.

**Commands**:

# List enabled services
systemctl list-unit-files --state=enabled

# List disabled services
systemctl list-unit-files --state=disabled

# Enable service
sudo systemctl enable ssh

# Disable service
sudo systemctl disable bluetooth

# Mask service (prevent manual start too)
sudo systemctl mask bluetooth

# Unmask service
sudo systemctl unmask bluetooth

**Expected output**: Services enabled or disabled for boot.

**Verification**:

systemctl is-enabled ssh
# Shows: enabled
systemctl is-enabled bluetooth
# Shows: disabled or masked

Configuration Examples

Custom Service Unit with Dependencies

[Unit]
Description=Database-backed Application
After=network.target postgresql.service
Wants=network.target
Requires=postgresql.service

[Service]
Type=forking
User=appuser
Group=appuser
Environment=DB_HOST=localhost
Environment=DB_PORT=5432
ExecStartPre=/usr/bin/mkdir -p /run/app
ExecStart=/opt/app/bin/app-server
ExecStop=/bin/kill -TERM $MAINPID
PIDFile=/run/app/app.pid
Restart=on-failure
RestartSec=10
TimeoutStartSec=30
TimeoutStopSec=30

[Install]
WantedBy=multi-user.target

Timer Unit Examples

# Daily at 2 AM
OnCalendar=*-*-* 02:00:00

# Every hour
OnCalendar=hourly

# Weekly on Sunday at 3 AM
OnCalendar=Sun *-*-* 03:00:00

# Every 15 minutes
OnUnitActiveSec=15m

systemd Slice Example (Resource Control)

[Slice]
Description=CPU-limited slice for non-critical services
CPUQuota=50%
MemoryMax=2G

Verification

systemd Verification Commands

systemctl status <service>    -- Service status
systemctl is-active <service> -- Check if active
systemctl is-enabled <service> -- Check if enabled
systemctl list-units --type=service --failed  -- Failed services
systemctl list-timers --all   -- All timers
systemctl get-default         -- Default target
journalctl -u <service>       -- Service logs
systemd-analyze blame         -- Boot time analysis
systemd-analyze critical-chain -- Boot dependency chain

Expected Results

• Services show Active: active (running) when started
• Services show enabled when configured for boot
• Custom service units load without errors
• Timer units show next run time
• journalctl shows service-specific logs

Troubleshooting

| Symptom | Possible Cause | Diagnostic Command | Corrective Action | Verification | |---------|---------------|-------------------|-------------------|--------------| | Service won't start | Configuration error | `sudo systemctl status <service>` | Fix unit file; `systemctl daemon-reload` | Service active | | Service fails repeatedly | Crash on startup | `journalctl -u <service>` | Fix application error; check logs | Service stable | | Service not enabled at boot | Not enabled | `systemctl is-enabled <service>` | `sudo systemctl enable <service>` | Service enabled | | Boot slow | Slow services | `systemd-analyze blame` | Optimize or disable slow services | Boot faster | | Cannot find unit file | Wrong path or typo | `systemctl list-unit-files` | Place unit in correct directory | Unit found | | Timer not running | Timer not enabled | `systemctl list-timers` | `sudo systemctl enable --now <timer>` | Timer active | | journalctl no permissions | Not in systemd-journal group | `sudo journalctl` | Add user to group or use sudo | Logs accessible | | Service depends on missing unit | Dependency not found | `systemctl list-dependencies <service>` | Install missing package or fix dependency | Dependencies resolved |

Best Practices

1. **Use Type=simple for most services**: Simplest and most common service type. 2. **Set Restart=always for critical services**: Automatic restart on failure. 3. **Use After= and Wants= for dependencies**: Ensure services start in correct order. 4. **Run services as non-root**: Use User= and Group= for security. 5. **Use journalctl for troubleshooting**: Centralized logging for all services. 6. **Test unit files with systemd-analyze verify**: Validate unit syntax before enabling. 7. **Use timers instead of cron**: Better integration with systemd and logging. 8. **Set resource limits**: CPUQuota, MemoryMax for service isolation. 9. **Document custom services**: Record purpose, dependencies, and restart behavior. 10. **Monitor failed services**: Set up alerts for systemctl --failed.

Conclusion

systemd is the central service manager for modern Linux systems, providing service management, boot process control, logging, and scheduling. Understanding systemd units, targets, service lifecycle, and journalctl logging enables administrators to manage services effectively, troubleshoot boot issues, and automate scheduled tasks. With systemd, Linux systems achieve faster boot times, better dependency management, and comprehensive service supervision.

References

• systemd Documentation (https://www.freedesktop.org/wiki/Software/systemd/)
• systemctl man page (systemctl(1))
• journalctl man page (journalctl(1))
• systemd.unit man page (systemd.unit(5))
• systemd.service man page (systemd.service(5))
• systemd.target man page (systemd.target(5))