AI Agent Development and Interview Guide: 03-Production-Grade RAG and Knowledge Engineering

1. RAG Objectives and Boundaries

RAG (Retrieval-Augmented Generation) retrieves external evidence prior to generation, decoupling knowledge updates from model parameter weights. It is ideally suited for:

• Internal enterprise document question answering;
• Frequently updated policies, product catalogs, and operating manuals;
• Tasks requiring verifiable citations and source text attribution;
• Scenarios where data cannot or should not enter model pretraining/fine-tuning.

RAG is not suited for solving in isolation:

• Scenarios requiring real-time transactional state without real-time data tools;
• Raw source data that is erroneous or internally contradictory;
• Tasks requiring executing actions rather than querying knowledge;
• Models lacking underlying reasoning or structural adherence capabilities;
• Strict arithmetic calculations, deterministic rules, or permission authorization gates.

🔥 P0 High-Frequency Essential: What is the difference between RAG and Fine-tuning? RAG supplies external knowledge at runtime during inference, offering rapid updates and verifiable citations; fine-tuning permanently alters behavioral styles or domain capabilities embedded in weights, incurring higher update costs and failing to guarantee real-time factual currency. Enterprise knowledge Q&A should always start with RAG, reserving fine-tuning for persistent formatting, style, or behavioral steering needs.

2. Complete End-to-End RAG Pipeline

Data Sources
  -> Parsing and Cleaning
  -> Document Structure Restoration
  -> Chunking and Metadata Tagging
  -> Dense Embedding / Sparse Indexing
  -> Candidate Recall
  -> Fusion and Deduplication
  -> Reranking
  -> Context Assembly
  -> Generation with Citations
  -> Evaluation and Feedback

A defect at any single stage will cause the final response to fail. In interviews, never focus solely on vector databases; demonstrate ability to isolate failures across the entire pipeline.

3. Data Ingestion & Parsing: Garbage In, Garbage Out

PDFs are not naturally flat sequential text. Common ingestion pitfalls include:

• Multi-column layout reading order confusion;
• Repeated headers and footers polluting context;
• Scanned documents lacking embedded OCR text layers;
• Tables fragmented into incoherent chunks;
• Lost hierarchical section heading paths;
• Displaced code blocks, mathematical equations, and footnotes;
• Duplicate multi-version ingestion of identical documents.

It is recommended to retain for each parsed unit:

from pydantic import BaseModel


class ParsedBlock(BaseModel):
    document_id: str
    document_version: str
    block_id: str
    block_type: str       # paragraph/table/code/title
    text: str
    page: int | None
    heading_path: list[str]
    source_uri: str
    updated_at: str
    access_tags: list[str]

source_uri, page numbers, and heading paths directly impact citation UX; access_tags enforce pre-retrieval authorization filters; document_version enables incremental updates and rollbacks.

Incremental Indexing

Never perform full rebuilds on every update. Compute content hashes:

import hashlib


def content_hash(text: str) -> str:
    normalized = " ".join(text.split())
    return hashlib.sha256(normalized.encode("utf-8")).hexdigest()

Only blocks with modified hashes require re-embedding. Deleted documents must have their indexed vectors synchronously purged, otherwise models will continue hallucinating on stale knowledge.

4. Chunking: Chunking Is Not Fixed-Character Slicing

Chunks too small: Incomplete semantic units, requiring answers to bridge across disjointed chunks. Chunks too large: Imprecise recall, bloated context budgets, and high Rerank latency.

Common strategies:

A simple structure-aware implementation:

from dataclasses import dataclass


@dataclass
class Chunk:
    chunk_id: str
    text: str
    heading_path: tuple[str, ...]
    source_uri: str


def chunk_sections(sections, max_chars: int = 1200) -> list[Chunk]:
    chunks = []
    for section in sections:
        paragraphs = section.paragraphs
        buffer = []
        length = 0

        for paragraph in paragraphs:
            if buffer and length + len(paragraph) > max_chars:
                index = len(chunks)
                chunks.append(Chunk(
                    chunk_id=f"{section.id}:{index}",
                    text="\n".join(buffer),
                    heading_path=tuple(section.heading_path),
                    source_uri=section.source_uri,
                ))
                buffer = []
                length = 0

            buffer.append(paragraph)
            length += len(paragraph)

        if buffer:
            index = len(chunks)
            chunks.append(Chunk(
                chunk_id=f"{section.id}:{index}",
                text="\n".join(buffer),
                heading_path=tuple(section.heading_path),
                source_uri=section.source_uri,
            ))
    return chunks

This example uses character approximation; production systems should tokenize using model tokenizers and strictly prevent splitting tables, code blocks, or heading hierarchies.

🔥 P0 High-Frequency Essential: How do you determine optimal chunk size? Never prescribe a fixed arbitrary number. Analyze document structure and query granularity, build a labeled evaluation dataset, and benchmark the empirical impact of chunk sizes and overlap ratios on Recall@k, answer quality, latency, and cost. Always start with hierarchical semantic chunking and tune parameters empirically.

5. Embeddings and Similarity Metrics

Embeddings map text into a vector space where semantically similar texts are placed closer. Common similarity metrics include Cosine Similarity, Dot Product, and Euclidean Distance. Always ensure that the indexing pipeline and query runtime use compatible models and identical normalization logic.

Engineering selection criteria:

• Performance on target language and multilingual domains;
• Whether asymmetric query vs. passage instruction prefixes are required;
• Maximum sequence token length;
• Vector dimensionality, storage footprint, and query latency;
• Support for specialized domain lexicons;
• Reindexing migration overhead upon embedding model upgrades.

Never select embedding models solely based on public leaderboards. Benchmark against your own labeled query-document datasets.

6. Dense, Sparse, and Hybrid Search

Dense retrieval excels at semantic nuances and paraphrases; Sparse retrieval like BM25 excels at product SKUs, proper nouns, error codes, and exact keyword matches. Hybrid Search harnesses both simultaneously.

Pinecone official documentation highlights: semantic search can miss exact keyword identifiers, while lexical search misses paraphrasing; Hybrid Search provides essential complementarity. Refer to Hybrid search.

RRF (Reciprocal Rank Fusion)

Reciprocal Rank Fusion does not require score normalization across heterogeneous retrieval engines:

RRF(d) = Σ 1 / (k + rank_i(d))

Implementation:

from collections import defaultdict


def reciprocal_rank_fusion(
    ranked_lists: list[list[str]],
    k: int = 60,
) -> list[tuple[str, float]]:
    scores = defaultdict(float)
    for ranked in ranked_lists:
        for rank, document_id in enumerate(ranked, start=1):
            scores[document_id] += 1.0 / (k + rank)

    return sorted(scores.items(), key=lambda item: item[1], reverse=True)

RRF is elegant and robust, though optimal hyperparameter k should still be calibrated via evaluation sets. Weighted linear combination is an alternative, provided dense and sparse score distributions are normalized.

🔥 P0 High-Frequency Essential: Why is BM25 still necessary alongside vector search? Vector search frequently misses exact alphanumeric identifiers, error codes, person names, and out-of-vocabulary terms; BM25 provides deterministic keyword precision. Hybrid search combines semantic and lexical signals, allowing downstream Rerankers to produce a unified ranking.

7. Query Transformation and Routing

Raw user questions are rarely optimal for direct vector queries:

• "When does it expire?" requires coreference resolution with conversational context;
• "Compare A and B" requires query decomposition into separate sub-queries;
• "How to reset XX-1042" contains exact error codes, requiring higher sparse retrieval weight;
• Casual chit-chat should bypass knowledge retrieval entirely;
• Real-time order queries should route to transactional APIs, not static knowledge bases.

Recommended structured plan output:

from typing import Literal
from pydantic import BaseModel


class RetrievalPlan(BaseModel):
    route: Literal["knowledge", "transaction_api", "no_retrieval"]
    queries: list[str]
    keyword_terms: list[str]
    filters: dict[str, str]

Query transformations can also introduce errors; always log both raw user inputs and rewritten queries in Traces to evaluate them independently.

8. Metadata Filtering: Filter Authorization First, Compute Similarity Second

The foundational rule of enterprise knowledge bases: documents a user has no privilege to access must never enter candidate sets, rather than retrieving everything and "hoping the model won't disclose it."

Common filter dimensions:

• tenant / organization;
• department;
• classification;
• document status;
• version;
• language;
• updated_at;
• product / region.

Permission filters must be generated server-side from authenticated user claims, never blindly trusting unverified model outputs like department="finance".

9. Rerank: Two-Stage Retrieval Architecture

The first stage recalls a broad candidate pool prioritizing high Recall; the second stage scores (query, document) pairs using Cross-Encoders or specialized Rerankers to maximize top-k Precision.

Hybrid Top 50 -> Deduplication -> Rerank -> Top 6 -> Context Assembly

Reranking adds latency and compute cost, requiring candidate pool capping, batching, and caching. Pinecone highlights Reranking as the most direct lever to elevate two-stage RAG quality; refer to Rerank results.

🔥 P0 High-Frequency Essential: How do the objectives of recall and reranking differ? The retrieval stage optimizes Recall, tolerating minor noise to avoid missing candidate evidence; the reranking stage optimizes Precision, reordering candidates so that truly relevant evidence surfaces at the top. Feeding raw Top 3 vector results directly into the LLM often fails due to retrieval omissions.

10. Context Assembly and Verifiable Attribution

Context should never be unstructured raw string concatenation. Structure evidence with persistent identifiers:

[S1]
source: employee_handbook.pdf
page: 12
section: Leave Policy > Annual Leave
text: ...

[S2]
source: hr_policy_2026.md
section: Special Leave
text: ...

Instruct models to cite factual assertions using [S1] notation. Post-validate generated answers:

• Whether cited citation IDs exist in retrieved context;
• Whether the cited snippet semantically supports the factual claim;
• Whether the model properly abstains when context lacks supporting evidence;
• Whether unauthorized sources were accidentally leaked or referenced.

The presence of citations does not equal factual truth. Enforce Citation Correctness evaluations and human spot-checks.

11. RAG Evaluation: Layer-by-Layer, Not Just End-to-End

11.1 Retrieval Metrics

• Recall@k: Whether ground-truth documents appear within the top k retrieved candidates; this is a critical execution standard and core baseline in production engineering architecture and system design.
• Precision@k: The proportion of retrieved candidates within top k that are genuinely relevant; this is a critical execution standard and core baseline in production engineering architecture and system design.
• MRR (Mean Reciprocal Rank): The reciprocal rank of the first relevant document; this is a critical execution standard and core baseline in production engineering architecture and system design.
• NDCG@k: Measuring graded relevance and rank position decay; this is a critical execution standard and core baseline in production engineering architecture and system design.

For example, if 82 out of 100 queries surface at least one relevant chunk in Top 5, Hit/Recall success rate is 82%. Ground-truth relevance standards must be established beforehand.

11.2 Generation Metrics

• Answer correctness and completeness;
• Faithfulness and factual grounding against retrieved evidence;
• Citation support accuracy;
• Proper abstention on unanswerable queries;
• Zero leakage of unauthorized data.

11.3 End-to-End Operational Metrics

• User task completion rate;
• First-contact resolution rate;
• Human escalation / deflection rate;
• P50 / P95 end-to-end latency;
• Token cost per request.

🔥 P0 High-Frequency Essential: If the final answer is wrong, how do you isolate retrieval vs. generation failure? First verify if annotated ground-truth documents entered candidate sets; if missing, it is a parsing, chunking, query rewrite, or recall defect. If present but ranked too low, it is a fusion or Rerank defect. If correct evidence was assembled into context but output remains wrong, the issue lies in generation, prompt constraints, or context distraction.

12. Common Failure Diagnostic Matrix

13. A Testable RAG Interface Architecture

from dataclasses import dataclass
from typing import Protocol


@dataclass
class Evidence:
    chunk_id: str
    text: str
    source_uri: str
    score: float


class Retriever(Protocol):
    async def search(self, query: str, filters: dict, top_k: int) -> list[Evidence]: ...


class Reranker(Protocol):
    async def rerank(
        self, query: str, candidates: list[Evidence], top_n: int
    ) -> list[Evidence]: ...


async def retrieve_context(
    query: str,
    filters: dict,
    retriever: Retriever,
    reranker: Reranker,
) -> list[Evidence]:
    candidates = await retriever.search(query, filters=filters, top_k=40)
    deduped = {item.chunk_id: item for item in candidates}
    ranked = await reranker.rerank(query, list(deduped.values()), top_n=6)
    return ranked

Adopting protocol-based dependency injection allows isolating and mocking Retriever and Reranker in unit tests, pinpointing component regressions deterministically.

14. Chapter Exercises

Exercise A: Building a Labeled Benchmark Dataset

Select 20–50 domain documents and curate 60 diverse questions:

• 30 standard informational Q&A pairs;
• 10 exact keyword/product code queries;
• 10 cross-section multi-hop queries;
• 5 unanswerable out-of-domain queries;
• 5 permission-gated access queries.

Annotate ground-truth relevant document chunk IDs and reference answers for each question.

Exercise B: Benchmarking Four Retrieval Pipelines

Compare: Dense, BM25, Hybrid, and Hybrid + Rerank. Measure and report Recall@5, MRR, P95 latency, and token cost.

Exercise C: Root-Cause Failure Classifier

Classify failures into 8 distinct buckets: parsing, chunking, query transformation, recall, rerank, generation, citation, and permissions. Log execution Traces for every failure.

15. High-Frequency Interview Q&A

🔥 P0: What is the complete lifecycle of an enterprise RAG pipeline?

Data parsing/cleaning, structural hierarchy restoration, chunking and metadata enrichment, dense/sparse indexing, query understanding and transformation, candidate recall, fusion deduplication, cross-encoder reranking, structured context assembly, grounded generation with citations, and multi-tier evaluation feedback loops. Candidates should explain observability across each layer.

🔥 P0: How do you eliminate hallucinations in RAG systems?

Enhance retrieval quality; enforce verifiable citations and explicit abstention on missing evidence; constrain generation strictly to context; run automated citation support verification; route deterministic queries to tools; and test continuously against unanswerable test suites. It cannot be absolutely eliminated purely via prompting.

🔥 P0: How do you select an enterprise vector database?

Evaluate across data volume, query latency, metadata filtering performance, native hybrid search, incremental index update latency, consistency model, multi-tenancy isolation, backup/restore, operational overhead, and total cost of ownership. Never rely solely on ANN algorithmic names; benchmark against your production query distribution.

⭐ P1: When is GraphRAG warranted?

When queries require deep entity relationship traversal, multi-hop reasoning across disjointed documents, and global structural synthesis (e.g., enterprise org charts, supply chain provenance, biomedical research). It introduces high extraction, graph maintenance, and evaluation complexity; standard document Q&A should always start with production Hybrid RAG.

⭐ P1: What must be considered during Embedding model upgrades?

Old and new vector spaces are mathematically incompatible, necessitating dual-writing or full re-indexing; execute offline A/B benchmarks; track model version metadata; architect zero-downtime traffic cutover; validate dimension sizes, normalization, multilingual capabilities, and domain transfer before switching live traffic.

16. Chapter Completion Criteria

• Able to diagram the end-to-end RAG architecture from source ingestion to verifiable citations;
• Able to implement and explain Hybrid Search + RRF + Rerank;
• Possess a labeled benchmark evaluation dataset with ground-truth chunk annotations;
• Able to independently measure and report retrieval, generation, and operational metrics;
• Able to diagnose failure stages directly from distributed Traces;
• Able to design pre-retrieval authorization filters, incremental indexing, and versioned migration pipelines.