Installing and Configuring Ollama on Linux

# Installing and Configuring Ollama on Linux

Introduction

Ollama is an open-source tool that simplifies running large language models locally on Linux, macOS, and Windows. It handles model downloading, quantization, and serving through a simple CLI and REST API. This tutorial covers Ollama installation, configuration, model management, and production deployment on Linux servers.

Learning Objectives

By the end of this tutorial, you will be able to:

• Install Ollama on Linux servers
• Download and manage LLMs through Ollama
• Configure Ollama for production use
• Use the Ollama REST API for inference
• Integrate Ollama with applications
• Optimize Ollama performance and resource usage

Technical Background

Ollama Architecture

flowchart TB
    Client[Client Application] -->|REST API| Ollama[Ollama Server]
    Ollama -->|Model Loading| Model[(Model Files)]
    Ollama -->|Inference| GPU[GPU/CPU]
    Ollama -->|Library| Registry[Ollama Model Registry]
    
    subgraph Ollama Components
    API[REST API]
    Library[Model Library]
    Engine[Inference Engine]
    end
    
    Ollama --> API & Library & Engine

Supported Models

| Model | Size | Parameters | Use Case | |-------|------|------------|----------| | llama3.2 | 2B | 2B | General tasks, fast inference | | llama3.1 | 8B/70B | 8B/70B | Complex reasoning, coding | | mistral | 7B | 7B | Efficient, multilingual | | codellama | 7B/13B/34B | 7B-34B | Code generation | | phi3 | 3.8B | 3.8B | Small, capable model | | gemma2 | 2B/9B/27B | 2B-27B | Google's open models |

Prerequisites

• Linux server (Ubuntu 22.04 LTS recommended)
• Root or sudo access
• NVIDIA GPU (recommended) or CPU-only
• 16+ GB RAM (32+ GB for larger models)
• 100+ GB storage for models

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | OS | Ubuntu 22.04 LTS, RHEL 9, or Debian 12 | | GPU | NVIDIA GPU with CUDA support (optional) | | RAM | 16 GB minimum, 32 GB recommended | | Storage | 100 GB+ SSD | | Network | Internet for model downloads |

Step-by-Step Implementation

Phase 1: Install Ollama

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Verify installation
ollama --version

# Check service status
systemctl status ollama

Phase 2: Download and Run Models

# List available models
ollama list

# Download a model
ollama pull llama3.2

# Run a model interactively
ollama run llama3.2

# Run with specific parameters
ollama run llama3.2 "Explain quantum computing"

Phase 3: Configure Ollama Service

#### Create systemd service override

# Edit Ollama service
sudo systemctl edit ollama.service

[Service]
Environment="OLLAMA_HOST=0.0.0.0"
Environment="OLLAMA_PORT=11434"
Environment="OLLAMA_MODELS=/opt/ollama/models"
Environment="OLLAMA_MAX_QUEUE=64"

# Apply changes
sudo systemctl daemon-reload
sudo systemctl restart ollama

#### Set custom model directory

# Create model directory
sudo mkdir -p /opt/ollama/models
sudo chown ollama:ollama /opt/ollama/models

# Update environment
sudo tee /etc/systemd/system/ollama.service.d/override.conf << 'EOF'
[Service]
Environment="OLLAMA_MODELS=/opt/ollama/models"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

Phase 4: Model Management

# List running models
ollama ps

# Show model details
ollama show llama3.2

# Copy a model
ollama cp llama3.2 my-custom-model

# Delete a model
ollama rm llama3.2

# Create a custom model from Modelfile
ollama create my-model -f Modelfile

#### Example Modelfile

# Modelfile
FROM llama3.2

# Set system prompt
SYSTEM """You are a helpful coding assistant specializing in Python and DevOps."""

# Set parameters
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER num_predict 512

# Set adapter (LoRA)
ADAPTER ./lora-adapter.bin

# Set template
TEMPLATE """{{ .System }} {{ .Prompt }}"""

Phase 5: Use Ollama REST API

# List models
curl http://localhost:11434/api/tags

# Generate text
curl http://localhost:11434/api/generate -d '{
  "model": "llama3.2",
  "prompt": "Explain Docker containers in one paragraph",
  "stream": false
}'

# Chat completion
curl http://localhost:11434/api/chat -d '{
  "model": "llama3.2",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "What is Kubernetes?"}
  ],
  "stream": false
}'

Phase 6: Production Deployment with Docker

# Run Ollama in Docker
docker run -d \
  --name ollama \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --gpus all \
  ollama/ollama

# Verify
curl http://localhost:11434/api/tags

# docker-compose.yml for production
version: '3.8'
services:
  ollama:
    image: ollama/ollama:latest
    ports:
      - "11434:11434"
    volumes:
      - ollama-data:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

volumes:
  ollama-data:

Phase 7: Optimize Performance

# Set GPU layers
OLLAMA_NUM_GPU=1 ollama run llama3.2

# Set context window
OLLAMA_NUM_CTX=4096 ollama run llama3.2

# Set parallel requests
OLLAMA_MAX_QUEUE=128 systemctl restart ollama

Expected Output

• Ollama server running and accessible
• Models downloaded and ready for inference
• REST API responding to requests
• Production configuration applied
• Performance optimized for hardware

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Ollama running | `systemctl status ollama` | Active and running | | API responding | `curl http://localhost:11434/api/tags` | JSON with model list | | Model generation | `ollama run llama3.2 "Hello"` | Coherent response | | GPU usage | `nvidia-smi` | GPU memory allocated |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Ollama not starting | Port already in use | Change OLLAMA_PORT or free port | | Model download fails | Network or storage issue | Check connection; verify disk space | | Slow inference | CPU instead of GPU | Install NVIDIA drivers; use --gpus all | | Out of memory | Model too large for RAM | Use smaller model or enable quantization | | API not accessible | OLLAMA_HOST not set | Configure OLLAMA_HOST=0.0.0.0 |

Best Practices

• **Use GPU acceleration:** Significantly faster inference than CPU
• **Quantize models:** Reduce memory usage with smaller quantizations
• **Set resource limits:** Prevent Ollama from consuming all system resources
• **Use systemd:** Manage Ollama as a system service
• **Backup models:** Store model files in persistent storage
• **Monitor logs:** Check journalctl for errors

Security Considerations

• **Network exposure:** Bind to localhost unless external access needed
• **Authentication:** Ollama has no built-in auth; use reverse proxy with auth
• **Model source:** Verify models come from trusted publishers
• **Input sanitization:** Validate prompts before sending to model
• **Rate limiting:** Implement API rate limiting at reverse proxy level

Conclusion

Ollama makes it easy to run large language models locally on Linux servers. By handling model management, quantization, and serving through a simple interface, Ollama enables teams to deploy self-hosted AI infrastructure without the complexity of managing model servers directly. Combined with Docker and systemd, Ollama provides a production-ready platform for local LLM inference.

References

• Ollama Documentation: https://github.com/ollama/ollama
• Ollama Models: https://ollama.com/library
• Ollama API: https://github.com/ollama/ollama/blob/main/docs/api.md
• Modelfile Reference: https://github.com/ollama/ollama/blob/main/docs/modelfile.md