RAG Architecture and Document-Based AI Search

4 Nov 2026, 11:2828 min readby ITYantra Admin

Complete guide to Retrieval-Augmented Generation (RAG) architecture including vector databases, embedding models, document chunking, and production deployment patterns.

# RAG Architecture and Document-Based AI Search

Introduction

Large Language Models (LLMs) are powerful but have limitations: knowledge cutoff dates, hallucinations, and no access to private or recent data. Retrieval-Augmented Generation (RAG) addresses these limitations by combining LLMs with external knowledge retrieval, enabling accurate, grounded, and up-to-date responses based on organizational documents.

This tutorial provides a comprehensive guide to RAG architecture, covering vector databases, embedding models, document chunking strategies, retrieval mechanisms, and production deployment patterns.

Learning Objectives

By the end of this tutorial, you will be able to:

Design and implement a RAG architecture from components to production
Select and configure embedding models for semantic search
Choose and operate vector databases (pgvector, Milvus, Pinecone, Weaviate)
Implement document chunking strategies for optimal retrieval
Build hybrid search (semantic + keyword) pipelines
Deploy and monitor RAG applications in production
Evaluate RAG performance using RAGAS and other metrics

Technical Background

How RAG Works

RAG Pipeline:
1. Document Ingestion    → Load documents, chunk, embed, store in vector DB
2. User Query            → Embed the query using same embedding model
3. Retrieval             → Search vector DB for similar document chunks
4. Augmentation          → Prepend retrieved context to user query
5. Generation            → LLM generates answer using augmented context
6. Citation              → Return answer with source document references

RAG vs Fine-Tuning

| Aspect | RAG | Fine-Tuning | |--------|-----|-------------| | Knowledge source | External documents (dynamic) | Model weights (static) | | Updates | Add new documents instantly | Retrain model | | Privacy | Documents stay outside model | Training data embedded in model | | Hallucinations | Reduced (grounded in sources) | Can still hallucinate | | Cost | Lower (no retraining) | Higher (compute for training) | | Transparency | Citations available | Black box |

Core RAG Components

| Component | Purpose | Examples | |-----------|---------|----------| | Embedding Model | Convert text to vectors | text-embedding-3-small, BGE, E5, all-MiniLM-L6-v2 | | Vector Database | Store and search embeddings | pgvector, Milvus, Pinecone, Weaviate, Qdrant, Chroma | | Chunking Strategy | Split documents into pieces | Recursive, semantic, fixed-size, document-aware | | Retrieval Method | Find relevant chunks | Cosine similarity, dot product, hybrid search | | Reranker | Re-rank initial results | Cross-encoder, Cohere Rerank, BGE-Reranker | | LLM | Generate final answer | GPT-4, Claude, Llama, Mistral, DeepSeek |

Prerequisites

Understanding of LLMs and prompt engineering
Familiarity with Python and basic machine learning concepts
Knowledge of vector similarity (cosine, dot product, Euclidean)
Experience with databases and APIs

Hardware and Software Requirements

| Component | Requirement | |-----------|-------------| | Embedding Model | sentence-transformers, OpenAI embeddings, or local model | | Vector DB | pgvector (PostgreSQL), Milvus, Pinecone, Weaviate, Qdrant | | LLM API | OpenAI, Anthropic, or local (Ollama, vLLM) | | Framework | LangChain, LlamaIndex, Haystack, or custom | | Document Store | S3, Azure Blob, local filesystem | | Monitoring | LangSmith, Arize Phoenix, custom metrics |

Step-by-Step Implementation

Phase 1: Document Ingestion Pipeline

#### 1.1 Document Loading

# Load documents from various sources
from langchain.document_loaders import (
    PyPDFLoader, 
    UnstructuredMarkdownLoader,
    WebBaseLoader,
    TextLoader
)

pdf_loader = PyPDFLoader("documents/employee-handbook.pdf")
md_loader = UnstructuredMarkdownLoader("documents/api-docs.md")
web_loader = WebBaseLoader("https://docs.ityantra.co.in/")

documents = pdf_loader.load() + md_loader.load() + web_loader.load()

#### 1.2 Document Chunking Strategies

from langchain.text_splitter import (
    RecursiveCharacterTextSplitter,
    MarkdownHeaderTextSplitter,
    SentenceTransformersTokenTextSplitter
)

# Strategy 1: Recursive character splitting (general purpose)
recursive_splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
    length_function=len,
    separators=["\n\n", "\n", " ", ""]
)

# Strategy 2: Markdown-aware splitting (preserves structure)
markdown_splitter = MarkdownHeaderTextSplitter(
    headers_to_split_on=[
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
)

# Strategy 3: Token-based splitting (for LLM context limits)
token_splitter = SentenceTransformersTokenTextSplitter(
    chunk_overlap=0,
    model_name="sentence-transformers/all-MiniLM-L6-v2"
)

chunks = recursive_splitter.split_documents(documents)

**Chunking Best Practices:**

| Strategy | Best For | Chunk Size | Overlap | |----------|----------|------------|---------| | Recursive | General documents | 500-1000 chars | 100-200 chars | | Markdown headers | Technical docs | Section-based | Preserve headers | | Token-based | LLM context windows | 256-512 tokens | 50-100 tokens | | Semantic | Narrative documents | 3-5 sentences | 1 sentence |

Phase 2: Embedding Generation

#### 2.1 Select Embedding Model

| Model | Dimensions | Performance | Speed | Cost | |-------|-----------|-------------|-------|------| | text-embedding-3-small | 1536 | High | Fast | $0.02/1M tokens | | text-embedding-3-large | 3072 | Very High | Medium | $0.13/1M tokens | | all-MiniLM-L6-v2 | 384 | Medium | Very Fast | Free (local) | | BGE-base-en-v1.5 | 768 | High | Fast | Free (local) | | E5-large-v2 | 1024 | High | Medium | Free (local) |

#### 2.2 Generate Embeddings

from langchain.embeddings import OpenAIEmbeddings, HuggingFaceEmbeddings

# Option 1: OpenAI embeddings
embeddings = OpenAIEmbeddings(
    model="text-embedding-3-small",
    openai_api_key="sk-..."
)

# Option 2: Local HuggingFace embeddings
embeddings = HuggingFaceEmbeddings(
    model_name="sentence-transformers/all-MiniLM-L6-v2",
    model_kwargs={"device": "cpu"}
)

# Generate embeddings for chunks
texts = [chunk.page_content for chunk in chunks]
vectors = embeddings.embed_documents(texts)

Phase 3: Vector Database Setup

#### 3.1 pgvector (PostgreSQL Extension)

-- Enable pgvector extension
CREATE EXTENSION IF NOT EXISTS vector;

-- Create table for document chunks
CREATE TABLE document_chunks (
    id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    content TEXT NOT NULL,
    embedding vector(384),
    metadata JSONB DEFAULT '{}',
    document_id UUID,
    chunk_index INTEGER,
    created_at TIMESTAMP DEFAULT NOW()
);

-- Create HNSW index for fast similarity search
CREATE INDEX ON document_chunks 
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

-- Create IVFFlat index (alternative for large datasets)
CREATE INDEX ON document_chunks 
USING ivfflat (embedding vector_cosine_ops)
WITH (lists = 100);

Insert embeddings:

from langchain.vectorstores import PGVector

connection_string = "postgresql+psycopg2://user:pass@localhost:5432/rag_db"

vectorstore = PGVector.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection_string=connection_string,
    collection_name="ityantra_docs",
    pre_delete_collection=False
)

#### 3.2 Milvus (Standalone Vector Database)

from pymilvus import connections, FieldSchema, CollectionSchema, DataType, Collection

connections.connect(alias="default", host="localhost", port="19530")

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="content", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="embedding", dtype=DataType.FLOAT_VECTOR, dim=384),
    FieldSchema(name="metadata", dtype=DataType.JSON),
]

schema = CollectionSchema(fields=fields, description="RAG document chunks")
collection = Collection(name="document_chunks", schema=schema)

# Create index
index_params = {
    "metric_type": "COSINE",
    "index_type": "HNSW",
    "params": {"M": 16, "efConstruction": 64}
}
collection.create_index(field_name="embedding", index_params=index_params)

Phase 4: Retrieval Pipeline

#### 4.1 Semantic Search

# Query embedding
query = "What is the company's remote work policy?"
query_vector = embeddings.embed_query(query)

# Search vector database
results = vectorstore.similarity_search_with_score(
    query,
    k=5,
    filter={"department": "hr"}  # Optional metadata filter
)

for doc, score in results:
    print(f"Score: {score:.4f}")
    print(f"Content: {doc.page_content[:200]}...")
    print(f"Source: {doc.metadata.get('source', 'unknown')}")
    print("---")

#### 4.2 Hybrid Search (Semantic + Keyword)

from langchain.retrievers import EnsembleRetriever
from langchain.vectorstores import PGVector
from langchain.retrievers.bm25 import BM25Retriever

# Semantic retriever
semantic_retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# Keyword retriever (BM25)
bm25_retriever = BM25Retriever.from_documents(chunks)
bm25_retriever.k = 5

# Ensemble retriever (weighted combination)
ensemble_retriever = EnsembleRetriever(
    retrievers=[semantic_retriever, bm25_retriever],
    weights=[0.7, 0.3]
)

results = ensemble_retriever.get_relevant_documents(query)

#### 4.3 Reranking

from sentence_transformers import CrossEncoder

# Initial retrieval
initial_results = vectorstore.similarity_search(query, k=20)

# Rerank with cross-encoder
reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-12-v2")
pairs = [[query, doc.page_content] for doc in initial_results]
scores = reranker.predict(pairs)

# Sort by reranker score
ranked = sorted(zip(initial_results, scores), key=lambda x: x[1], reverse=True)
top_documents = [doc for doc, score in ranked[:5]]

Phase 5: Augmentation and Generation

#### 5.1 Build RAG Chain

from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI
from langchain.prompts import PromptTemplate

# Custom prompt with citation instructions
template = """Use the following pieces of context to answer the question at the end.
If you don't know the answer based on the context, say "I don't have enough information."
Always cite your sources using [Source: document_name] format.

Context: {context}

Question: {question}

Answer:"""

PROMPT = PromptTemplate(
    template=template,
    input_variables=["context", "question"]
)

qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(model="gpt-4", temperature=0),
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
    chain_type_kwargs={"prompt": PROMPT},
    return_source_documents=True
)

# Query
result = qa_chain({"query": "What is the remote work policy?"})
print(result["result"])
print("\nSources:")
for doc in result["source_documents"]:
    print(f"- {doc.metadata.get('source', 'unknown')}")

#### 5.2 Query Expansion

from langchain.retrievers.multi_query import MultiQueryRetriever

# Generate multiple query variations for better retrieval
multiquery_retriever = MultiQueryRetriever.from_llm(
    retriever=vectorstore.as_retriever(),
    llm=ChatOpenAI(model="gpt-3.5-turbo")
)

results = multiquery_retriever.get_relevant_documents(query)

Phase 6: Production Deployment

#### 6.1 Architecture Diagram

┌─────────────┐      ┌─────────────┐      ┌─────────────┐
│   User      │────▶│  API Gateway│────▶│  RAG Service│
│ Application │      │  (Auth,     │      │ (FastAPI)   │
│             │      │  RateLimit) │      │             │
└─────────────┘      └─────────────┘      └──────┬──────┘
                                                 │
                              ┌──────────────────┼──────────────────┐
                              ▼                  ▼                  ▼
                       ┌───────────┐      ┌───────────┐      ┌───────────┐
                       │  Embedding│      │  Vector   │      │   LLM     │
                       │  Service  │      │  Database │      │  Service  │
                       │  (Local)  │      │ (pgvector)│      │  (API)    │
                       └───────────┘      └───────────┘      └───────────┘
                              │                  │                  │
                              └──────────────────┴──────────────────┘
                                                 │
                                        ┌────────▼────────┐
                                        │   Observability │
                                        │ (Tracing, Logs, │
                                        │  Metrics)       │
                                        └─────────────────┘

#### 6.2 FastAPI RAG Service

from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel
from typing import List, Optional
import hashlib

app = FastAPI(title="ITYantra RAG Service")

class QueryRequest(BaseModel):
    question: str
    top_k: Optional[int] = 5
    filters: Optional[dict] = None

class SourceDocument(BaseModel):
    content: str
    metadata: dict
    score: float

class QueryResponse(BaseModel):
    answer: str
    sources: List[SourceDocument]
    query_id: str

@app.post("/query", response_model=QueryResponse)
async def query(request: QueryRequest):
    query_id = hashlib.md5(request.question.encode()).hexdigest()[:8]
    
    # Retrieve relevant documents
    docs = retriever.get_relevant_documents(request.question)
    
    # Generate answer
    result = qa_chain({"query": request.question})
    
    return QueryResponse(
        answer=result["result"],
        sources=[
            SourceDocument(
                content=doc.page_content[:500],
                metadata=doc.metadata,
                score=0.0  # Would include actual score
            )
            for doc in result["source_documents"]
        ],
        query_id=query_id
    )

@app.get("/health")
async def health():
    return {"status": "healthy", "service": "rag"}

#### 6.3 Docker Deployment

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8000

CMD ["uvicorn", "main:app", "--host", "0.0.0.0", "--port", "8000"]

# docker-compose.yml
version: "3.8"
services:
  rag-service:
    build: .
    ports:
      - "8000:8000"
    environment:
      - OPENAI_API_KEY=${OPENAI_API_KEY}
      - DATABASE_URL=postgresql+psycopg2://rag:rag@postgres:5432/rag_db
    depends_on:
      - postgres
  
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      - POSTGRES_PASSWORD=rag
      - POSTGRES_DB=rag_db
    volumes:
      - postgres_data:/var/lib/postgresql/data

volumes:
  postgres_data:

Phase 7: Evaluation and Monitoring

#### 7.1 RAG Evaluation Metrics

| Metric | Description | Target | |--------|-------------|--------| | Faithfulness | Answer grounded in retrieved context | > 0.8 | | Answer Relevancy | Answer addresses the question | > 0.8 | | Context Precision | Retrieved chunks are relevant | > 0.7 | | Context Recall | All relevant chunks retrieved | > 0.7 | | Latency | End-to-end response time | < 3 seconds |

#### 7.2 RAGAS Evaluation

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

eval_data = {
    "question": ["What is the remote work policy?"],
    "answer": ["Employees can work remotely 3 days per week..."],
    "contexts": [["Remote work policy: 3 days per week..."]],
    "ground_truth": ["Employees may work remotely up to 3 days per week with manager approval."]
}

result = evaluate(
    eval_data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall]
)
print(result)

#### 7.3 Production Monitoring

Key Metrics to Monitor:
  - Query latency (p50, p95, p99)
  - Token usage (input, output)
  - Error rate (LLM API failures, vector DB errors)
  - User feedback (thumbs up/down)
  - Retrieval quality (average similarity score)
  - Cost per query

Verification and Expected Outputs

| Check | Method | Expected Result | |-------|--------|-----------------| | Document ingestion | Load 100 PDFs, verify chunks | All documents chunked and embedded | | Semantic search | Query for known document | Relevant chunks returned with high scores | | RAG answer | Ask question from documents | Accurate answer with citations | | Hybrid search | Query with keyword + semantic | Better results than either alone | | Production deployment | Health endpoint responds | Service healthy, vector DB connected | | Evaluation | RAGAS metrics | Faithfulness > 0.8, Answer relevancy > 0.8 |

Troubleshooting Scenarios

| Issue | Diagnosis | Resolution | |-------|-----------|------------| | Poor retrieval quality | Wrong embedding model, poor chunking | Try different model, adjust chunk size, use reranker | | Hallucinated answers | Context doesn't contain answer | Improve retrieval, adjust prompt, increase top_k | | High latency | Large context, slow LLM | Reduce chunk size, use faster model, cache responses | | Duplicate results | Overlapping chunks | Reduce overlap, deduplicate by content hash | | Vector DB out of memory | Large index, high dimensions | Reduce dimensions, use quantization, scale horizontally | | Embedding drift | Model changed, old embeddings incompatible | Re-embed all documents with same model |

Security Best Practices

**Access control:** Filter results by user permissions (document-level security)
**PII detection:** Scan documents for PII before ingestion, redact or exclude
**Prompt injection:** Sanitize user queries, use system prompts to prevent manipulation
**Data encryption:** Encrypt vectors at rest, TLS for all API communications
**Audit logging:** Log all queries, responses, and source documents for compliance
**Rate limiting:** Prevent abuse of LLM API and embedding endpoints
**Model isolation:** Separate embedding and LLM models by security boundary

Enterprise Considerations

**Multi-tenancy:** Isolate documents by customer/department using metadata filters
**Document lifecycle:** Handle document updates, deletions, and re-indexing
**Version control:** Track document versions and their corresponding embeddings
**Cost optimization:** Cache frequent queries, use smaller models for simple questions
**Disaster recovery:** Backup vector database, document store, and configuration
**Compliance:** Meet GDPR, HIPAA, or other data protection requirements for document storage

Conclusion

RAG is the practical way to bring LLMs to enterprise knowledge. By combining document retrieval with generation, you create systems that are accurate, current, and grounded in your organization's data. Success depends on thoughtful chunking, quality embeddings, robust retrieval, and continuous evaluation.

Frequently Asked Questions

**Q: How many documents can RAG handle?** A: RAG scales to millions of documents. Vector databases are designed for large-scale similarity search.

**Q: Do I need to retrain when adding new documents?** A: No. Just ingest, chunk, embed, and index new documents. No model retraining needed.

**Q: What chunk size should I use?** A: Start with 500-1000 characters with 100-200 character overlap. Adjust based on document type and retrieval quality.

**Q: Can RAG handle images and tables?** A: Yes, but requires multimodal embedding models and specialized processing for images/tables.

Official Technical References

LangChain: https://python.langchain.com/
LlamaIndex: https://docs.llamaindex.ai/
pgvector: https://github.com/pgvector/pgvector
RAGAS: https://github.com/explodinggradients/ragas
Milvus: https://milvus.io/
Weaviate: https://weaviate.io/