AI Agent Development and Interview Guide: 09-Fine-Tuning, Post-Training and Inference Serving

1. Architectural Trade-Off Decision Matrix

🔥 P0 High-Frequency Essential: When should you NOT fine-tune? When knowledge facts update frequently, when high-quality training pairs are scarce, when problems can be solved deterministically with RAG/Tools/Schemas, when golden evaluation suites are absent, or when root-cause failure attribution is unverified. Fine-tuning bakes stale data into static model weights, and cannot provide real-time ground truth or IAM access controls.

2. Supervised Fine-Tuning (SFT) Fundamentals

Supervised Fine-Tuning (SFT) optimizes model parameters using pairs of inputs and target outputs, training the model to emulate desired behaviors. Best suited for:

• Consistently outputting specialized domain DSLs or schema formats;
• Adopting proprietary enterprise vocabulary, idioms, and stylistic tone;
• Improving instruction-following adherence on constrained tasks;
• Distilling complex intent classification or extraction tasks into lightweight Small Language Models (SLMs);
• Training robust adherence to complex tool invocation protocols.

Training Example Schema:

{
  "messages": [
    {"role": "system", "content": "You are a ticket classifier. Output strictly valid JSON matching the schema."},
    {"role": "user", "content": "I was charged twice for my membership fee last night."},
    {"role": "assistant", "content": "{\"category\":\"duplicate_charge\",\"priority\":\"high\"}"}
  ]
}

Specific data formatting adheres to your target framework and model chat template, but Train, Validation, and Test splits must remain strictly segregated.

3. Data Quality Over Data Quantity

Dataset Quality Audit Dimensions:

• Does the input distribution accurately reflect live production traffic?
• Are target completions verified by domain experts or deterministic rules?
• Are there conflicting or noisy ground-truth labels?
• Does the dataset contain unredacted PII, copyrighted secrets, or sensitive tokens?
• Does it comprehensively cover edge cases, unanswerable queries, and safety refusals?
• Does the formatting match the base model's exact tokenizer and chat template?
• Is there data contamination or leakage between train and test splits?
• Is the ratio between short and long sequence lengths balanced?

Deduplication Strategies

Exact duplicates artificially inflate evaluation scores, while near-duplicates cause data leakage. Recommended pipeline:

• Exact SHA-256 content hashing to purge exact duplicates;
• MinHash / SimHash locality-sensitive hashing for fuzzy near-duplicates;
• Group-based splitting by source document to prevent adjacent paragraphs from spanning Train/Test splits;
• Group-based splitting by business entity or template structure.

🔥 P0 High-Frequency Essential: Why must you avoid naive random dataset splitting? Near-duplicate samples from the same source document, user session, or prompt template will spill across both training and test splits, causing severe benchmark contamination and falsely inflated metrics. Datasets must be partitioned via Group Splitting (by document, customer, or time epoch), preserving genuinely unseen distributions for testing.

4. Parameter-Efficient Fine-Tuning with LoRA

Low-Rank Adaptation (LoRA) freezes the pre-trained model backbone and injects trainable rank decomposition matrices into targeted linear layers, dramatically reducing trainable parameters and GPU VRAM footprint while facilitating modular multi-adapter deployment.

Conceptual Formulation:

W' = W + ΔW
ΔW = B × A
rank(A, B) << dimension(W)

Core Hyperparameters:

• r: Low-rank dimension;
• alpha: Scaling factor;
• target_modules: Attention/MLP projection layers targeted for adaptation;
• dropout: LoRA layer dropout regularization;
• Learning rate, effective batch size, and epoch schedules;
• Maximum sequence context length.

Higher rank r is not inherently superior: it increases memory overhead and risks overfitting. Hyperparameter tuning must be guided by validation split performance.

QLoRA

QLoRA quantizes the frozen base model weights (typically to 4-bit NormalFloat) while backpropagating gradients through 16-bit LoRA adapter matrices, slashing GPU VRAM requirements even further. Quantization loss, throughput penalties, and framework compatibility must be empirically benchmarked.

5. Training Configuration Manifest

While framework APIs evolve, production workflows must maintain an immutable training manifest:

base_model: your-base-model
dataset_version: support-intent-v4
chat_template_version: model-default-v2

method: lora
lora:
  rank: 16
  alpha: 32
  dropout: 0.05
  target_modules: [q_proj, k_proj, v_proj, o_proj]

training:
  learning_rate: 0.0002
  epochs: 2
  max_sequence_length: 2048
  effective_batch_size: 64
  warmup_ratio: 0.03
  seed: 42

evaluation:
  dataset_version: support-hidden-test-v2
  metrics: [macro_f1, schema_pass_rate, safety_violation]

Persist the exact base model commit hash, tokenizer version, chat template definition, dataset snapshot ID, and training codebase commit to ensure deterministic reproducibility.

6. Direct Preference Optimization (DPO)

Direct Preference Optimization (DPO) optimizes policy models directly on pairwise preference tuples: given an input prompt, one completion is designated as chosen over a rejected alternative.

{
  "prompt": "Explain to a customer why their refund has not posted yet",
  "chosen": "The refund has been processed on our end. Banks typically take 3–5 business days to post credits to your account...",
  "rejected": "Please be patient, it will arrive eventually."
}

Ideal for calibrating: helpfulness, stylistic tone, refusal boundaries, conciseness, and tool selection preferences. Operational risks:

• Inter-annotator preference label inconsistency;
• Models learning superficial stylistic tropes rather than factual accuracy;
• Over-refusal on benign queries;
• Catastrophic performance degradation on out-of-distribution prompts;
• Confounded differences where chosen and rejected pairs diverge across multiple uncontrolled variables.

⭐ P1: What is the difference between SFT and DPO? SFT teaches a model "what ideal target completions look like" via imitation; DPO optimizes "which of two candidate completions is preferred" via pairwise contrast. Production pipelines typically run SFT first to establish foundational behavior, followed by DPO on curated pairs for alignment. Both require standalone evaluation suites verifying safety and general capability retention.

7. Role of RLHF and Agentic RL

Reinforcement Learning from Human Feedback (RLHF) integrates preference datasets, reward modeling, and policy gradient optimization (PPO). Agentic RL extends this by optimizing Agent trajectories based on task success metrics, tool invocation validity, or external environment rewards. Prerequisites:

• Fully deterministic, reproducible simulation environments;
• Scalable, objective, and high-fidelity reward functions;
• Vast quantities of interactive trajectory training data;
• Robust automated safety and trajectory evaluation harnesses;
• Substantial GPU compute clusters and research engineering expertise.

For most enterprise applications, behavioral cloning (SFT on expert traces), deterministic trajectory evaluators, and hard-coded state graph guards are far more cost-effective. Flawed reward functions inevitably incentivize reward hacking and loop exploitation.

8. Post-Fine-Tuning Evaluation Protocols

Never evaluate models on training loss curves alone. Benchmark across:

• Primary downstream task accuracy and completeness;
• General capability regression across foundational knowledge benchmarks;
• Structured schema and tool argument validation pass rates;
• Safety refusal accuracy and false positive refusal rates;
• Hallucination rates and factual groundedness against context;
• Multilingual consistency and long-context degradation;
• Serving latency, tokens-per-second throughput, VRAM footprint, and cost;
• Robustness against adversarial prompt perturbations.

Maintain three dedicated evaluation splits:

• Downstream Task Held-Out Test Split;
• General Capability Regression Split;
• Safety, Guardrail, and Red-Teaming Split.

🔥 P0 High-Frequency Essential: Does a dropping Training Loss indicate model improvement? No. Declining training loss often masks overfitting, benchmark contamination, or superficial memorization of prompt templates. Fine-tuned models must be validated on independent task benchmarks, broad regression suites, and adversarial safety splits, benchmarking quality, latency, and cost directly against original base models.

9. Core Inference Serving Telemetry

• Time to First Token (TTFT): Initial response latency;
• Time Per Output Token (TPOT): Inter-token generation latency;
• P50, P95, and P99 End-to-End Latency percentiles;
• System-Wide Output Tokens per Second (TPS);
• Concurrent Request Capacity and Queue Wait Duration;
• GPU Utilization and VRAM Allocation;
• Continuous Batch Size Dynamics;
• Request Success Rate and Out-Of-Memory (OOM) error frequency;
• Blended Monetary Cost per Million Tokens.

Streaming output drastically improves perceived user latency without reducing total GPU compute time. Continuous batching increases aggregate serving throughput, though under peak loads it can increase long-tail request queuing times.

10. KV Caching, Prefix Caching, and Quantization

KV Cache Management

Persists precomputed Key and Value attention tensors for historical prompt tokens in GPU VRAM, eliminating redundant recomputation during auto-regressive decoding. Long context windows and high concurrency create massive VRAM memory pressure.

Prompt Prefix Caching

Shares and reuses precomputed attention states across multiple requests that share identical static prefixes (e.g., system prompts, shared tool registries). Prompt template edits or tool schema updates immediately invalidate prefix cache entries.

Weight and Activation Quantization

Compresses model weights and activations to lower numerical precision (e.g., FP16/BF16 down to INT8/INT4/AWQ/GPTQ), reducing GPU memory footprints and accelerating throughput at the cost of slight precision loss and hardware kernel dependencies. Quantized models must be re-benchmarked across task accuracy and safety splits.

🔥 P0 High-Frequency Essential: What are the trade-offs of Model Quantization? Advantages: Drastically reduces GPU memory requirements, enables serving larger model backbones on consumer/edge hardware, and increases potential continuous batching throughput. Costs: Slight degradation in reasoning precision, kernel/hardware compatibility nuances, and potential latency regressions if quantization overhead is not hardware-accelerated. Never rely solely on perplexity metrics; always benchmark downstream task and tool calling accuracy.

11. Intelligent Model Routing

Route inbound requests dynamically based on:

• Reasoning complexity and task requirements;
• Operational risk and blast radius tier;
• Context window token volume;
• Target language or required multimodal capabilities;
• Latency SLA budgets;
• Real-time provider health, quotas, and error rates;
• Available financial token budgets.

Implementation Pattern:

def select_model(task) -> str:
    if task.risk == "high":
        return "high-accuracy-model"
    if task.kind in {"classify", "extract"} and task.context_tokens < 4000:
        return "small-fast-model"
    if task.requires_vision:
        return "vision-model"
    return "balanced-model"

Routing policies must be empirically validated against evaluation benchmarks; high-risk tasks cannot rely on "large models" alone, requiring human approval gates and deterministic safety guardrails.

12. Inference Deployment Version Tracking

Every inference trace must record the complete immutable runtime manifest:

base_model_version
adapter_version
quantization_version
tokenizer_version
chat_template_version
serving_engine_version
prompt_version
tool_schema_version

A change in any single variable alters model behavior and output quality. Canary deployments and A/B evaluations must stratify telemetry by exact configuration snapshots.

13. Chapter Exercises

Exercise A: Architecture Decision Assessment

Analyze three historical enterprise failure cases. For each, evaluate whether the root fix requires Prompt Engineering, RAG, Tool Calling, or SFT Fine-Tuning. Detail the data requirements, operational cost, update agility, and validation criteria.

Exercise B: SLM Classifier Fine-Tuning

Construct a curated dataset of 500+ classification examples using Group Splitting by source. Train an SLM adapter and compare Macro-F1 scores, schema pass rates, latency gains, and safety metrics before and after fine-tuning.

Exercise C: Inference Serving Stress Benchmark

Benchmark a base precision model against a quantized configuration (e.g., FP16 vs INT4-AWQ). Measure P95 latency, TTFT, aggregate throughput, VRAM consumption, task accuracy, and tool calling pass rates under synthetic concurrent load.

14. High-Frequency Interview Q&A

🔥 P0: Why does LoRA reduce GPU memory requirements during training?

LoRA freezes the vast majority of pre-trained model weights, training only low-rank adapter matrices. Consequently, optimizer states (such as Adam momentum and variance tensors) and backward gradients are computed and stored only for a tiny fraction of parameters. Active VRAM still houses layer activations, base weights, and KV caches; QLoRA further quantizes the base weights down to 4-bit precision.

🔥 P0: How do you prevent Catastrophic Forgetting during fine-tuning?

Train on high-quality curated datasets with conservative learning rates; limit training epochs; mix in diverse general domain and safety instruction samples; use parameter-efficient adaptation (LoRA/adapters) rather than full-parameter updates; maintain an automated general regression benchmark; apply early stopping; and evaluate downstream task performance simultaneously against general capability suites.

🔥 P0: How do you define success for an enterprise fine-tuning initiative?

Statistically significant accuracy gains on private held-out task benchmarks; general capability and safety degradation remaining within strict regression thresholds; serving latency, throughput, and costs meeting production SLAs; measured business outcome improvements in live canary traffic; and full end-to-end reproducibility backed by immutable dataset, code, and model versioning.

⭐ P1: How do you choose between Full Parameter Fine-Tuning and LoRA?

LoRA requires minimal GPU compute, iterates rapidly, and allows hot-swapping multiple task-specific adapters on a shared base model instance, making it optimal for the vast majority of domain adaptation tasks. Full parameter fine-tuning offers higher capacity for fundamental domain shifts but carries high compute costs and severe catastrophic forgetting risks. Always establish a LoRA baseline before attempting full fine-tuning.

⭐ P1: What core problems do modern LLM inference engines (e.g., vLLM, TGI, SGLang) solve?

Modern inference engines provide high-throughput continuous batching, dynamic PagedAttention KV cache memory management, concurrent request scheduling, tensor-parallel distributed execution, optimized INT8/INT4 quantization kernels, and standardized OpenAI-compatible APIs. Tool selection should be guided by empirical benchmarks on target model architectures, hardware platforms, throughput targets, and operational simplicity.

15. Chapter Completion Criteria

• Able to classify failure modes and recommend Prompt, RAG, Tool, or Fine-Tuning solutions;
• Explain the foundational distinctions between SFT, LoRA/QLoRA, DPO, and Agentic RL;
• Audit training datasets for provenance, group-split integrity, and deduplication;
• Implement tripartite evaluation protocols covering task accuracy, general capability retention, and safety;
• Explain KV Cache mechanics, prompt prefix caching, model quantization, and dynamic routing architectures;
• Maintain immutable end-to-end configuration manifests across inference and serving deployments.