Running Local AI Models on Windows and Ubuntu

# Running Local AI Models on Windows and Ubuntu

Introduction

Running AI models locally provides privacy, cost savings, and offline access compared to cloud-based APIs. This tutorial covers running local LLMs on Windows and Ubuntu using Ollama, LM Studio, Docker, and native installations with GPU acceleration.

Learning Objectives

By the end of this tutorial, you will be able to:

• Install and configure Ollama on Windows and Ubuntu
• Use LM Studio for GUI-based local AI
• Run models in Docker containers
• Configure GPU acceleration (NVIDIA CUDA)
• Optimize model performance for local hardware
• Compare local vs. cloud AI deployment

Technical Background

Local AI Architecture

flowchart TB
    User[User Interface] --> App[AI Application]
    App --> Runtime[Inference Runtime]
    Runtime --> Model[Model Files]
    Runtime --> GPU[GPU Acceleration]
    
    subgraph Windows
    WSL2[WSL2]
    Native[Native Windows]
    end
    
    subgraph Ubuntu
    NativeLinux[Native Linux]
    Docker[Docker Container]
    end
    
    App --> WSL2 & NativeLinux & Docker

Hardware Requirements by Model Size

| Model | RAM | VRAM | Disk | Use Case | |-------|-----|------|------|----------| | Phi-3 Mini (3.8B) | 8 GB | 4 GB | 5 GB | Simple tasks | | Llama-3.2 (3B) | 8 GB | 4 GB | 4 GB | Chat, summarization | | Llama-3.1 (8B) | 16 GB | 8 GB | 9 GB | General purpose | | Llama-3.1 (70B) | 64 GB | 40 GB | 80 GB | Complex reasoning |

Prerequisites

• Windows 10/11 or Ubuntu 22.04 LTS
• NVIDIA GPU (recommended for GPU acceleration)
• 16+ GB RAM
• 100+ GB free disk space

Hardware and Software Requirements

| Component | Windows | Ubuntu | |-----------|---------|--------| | OS | Windows 10/11 | Ubuntu 22.04 LTS | | GPU | NVIDIA RTX 3060+ | NVIDIA RTX 3060+ | | WSL2 | Required | N/A | | Docker | Docker Desktop | Docker Engine |

Step-by-Step Implementation

Phase 1: Ubuntu Installation

#### Install Ollama on Ubuntu

curl -fsSL https://ollama.com/install.sh | sh
ollama pull llama3.2
ollama run llama3.2

#### Install NVIDIA Drivers and CUDA on Ubuntu

# Add NVIDIA repository
sudo apt update
sudo apt install nvidia-driver-535 nvidia-cuda-toolkit

# Verify GPU
nvidia-smi

# Install Docker with NVIDIA support
sudo apt install docker.io nvidia-container-toolkit
sudo systemctl restart docker

#### Run Model with GPU in Docker

docker run -d \
  --gpus all \
  -p 11434:11434 \
  -v ollama:/root/.ollama \
  --name ollama \
  ollama/ollama

Phase 2: Windows Installation

#### Install WSL2

# Enable WSL2
dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart
dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

# Restart computer, then install Ubuntu from Microsoft Store
# Set WSL2 as default
wsl --set-default-version 2

#### Install Ollama on Windows

# Download and install Ollama
# https://ollama.com/download/windows

# Verify
ollama --version
ollama pull llama3.2
ollama run llama3.2

#### Use WSL2 with GPU

WSL2 supports GPU passthrough on Windows 11 with NVIDIA drivers installed on the host.

Phase 3: LM Studio (GUI Alternative)

#### Install LM Studio

1. Download from https://lmstudio.ai 2. Install on Windows or Ubuntu 3. Browse and download models from the built-in library 4. Load model and start local server

#### LM Studio API

# LM Studio provides OpenAI-compatible API
curl http://localhost:1234/v1/chat/completions -H "Content-Type: application/json" -d '{
  "model": "llama-3.2-3b-instruct",
  "messages": [{"role": "user", "content": "Hello!"}],
  "temperature": 0.7
}'

Phase 4: Model Quantization

| Quantization | Size Reduction | Quality Loss | VRAM Required | |-------------|---------------|--------------|---------------| | FP16 | 1x | None | Baseline | | Q8_0 | ~2x | Minimal | ~50% of FP16 | | Q5_K_M | ~4x | Small | ~30% of FP16 | | Q4_K_M | ~4x | Moderate | ~25% of FP16 | | Q2_K | ~8x | Significant | ~15% of FP16 |

# Ollama handles quantization automatically
ollama pull llama3.2:q4_K_M
ollama pull llama3.2:q8_0

Phase 5: Performance Optimization

#### Ubuntu Optimization

# Set huge pages for better performance
sudo sysctl -w vm.nr_hugepages=128

# Configure Ollama environment
sudo tee /etc/systemd/system/ollama.service.d/override.conf << 'EOF'
[Service]
Environment="OLLAMA_NUM_GPU=1"
Environment="OLLAMA_NUM_CTX=4096"
Environment="OLLAMA_MAX_QUEUE=64"
EOF

sudo systemctl daemon-reload
sudo systemctl restart ollama

#### Windows Optimization

# Enable WSL2 memory limit
# Create .wslconfig in user profile
[wsl2]
memory=32GB
processors=8
swap=8GB

Expected Output

• Local AI models running on Windows and Ubuntu
• GPU acceleration configured
• Multiple inference options (Ollama, LM Studio, Docker)
• Performance optimized for hardware

Verification Procedures

| Check | Command | Expected Result | |-------|---------|-----------------| | Ollama version | `ollama --version` | Version displayed | | GPU detected | `nvidia-smi` | GPU info shown | | Model runs | `ollama run llama3.2 "Hi"` | Response generated | | API works | `curl localhost:11434/api/tags` | Model list returned |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | GPU not detected | NVIDIA drivers missing | Install drivers; reboot | | Model too slow | Running on CPU | Enable GPU; check CUDA | | Out of memory | Model too large | Use quantized version | | WSL2 GPU not working | Windows 10 limitation | Upgrade to Windows 11 |

Best Practices

• **Use quantized models:** Reduce memory usage with Q4_K_M or Q8_0
• **Enable GPU acceleration:** Dramatically faster inference
• **Set context window:** Match context to your use case
• **Monitor resources:** Track RAM and VRAM usage
• **Use LM Studio for beginners:** GUI simplifies model management

Security Considerations

• **Local processing:** Data never leaves your machine
• **Model verification:** Download from trusted sources only
• **Network isolation:** Run offline for sensitive workloads
• **Access control:** Restrict API access on shared machines

Conclusion

Running AI models locally on Windows and Ubuntu provides privacy, cost control, and offline capability. With tools like Ollama and LM Studio, setting up local AI is accessible even without deep ML expertise. GPU acceleration and proper quantization make local inference practical for most use cases.

References

• Ollama: https://ollama.com/
• LM Studio: https://lmstudio.ai/
• WSL2 GPU: https://learn.microsoft.com/windows/ai/directml/gpu
• Hugging Face Models: https://huggingface.co/models