Why Transformers replaced RNNs/LSTMs

Key Insight

RNNs process tokens sequentially — to understand the relationship between the first and last word in a sentence, information must pass through every intermediate hidden state. Transformers compute attention scores between all pairs of tokens simultaneously, making them vastly more efficient and effective at capturing long-range dependencies.

The QKV Framework

For each input token, three vectors are computed:

• Query (Q): “What am I looking for?”
• Key (K): “What do I contain?”
• Value (V): “What information do I provide?”

The attention score is computed as:

where is the dimension of the key vectors (the scaling factor prevents dot products from growing too large).

graph LR
    subgraph Self_Attention["Self-Attention Computation"]
        I["Input Embeddings"] --> Q["Q = X · W_Q"]
        I --> K["K = X · W_K"]
        I --> V["V = X · W_V"]
        Q --> DOT["Q · K^T"]
        K --> DOT
        DOT --> SCALE["÷ √d_k"]
        SCALE --> SOFT["Softmax"]
        SOFT --> MUL["× V"]
        V --> MUL
        MUL --> OUT["Output"]
    end

    style Self_Attention fill:#6cc3d5,stroke:#333,color:#fff

Multi-Head Attention

Instead of one attention function, Transformers run multiple attention heads in parallel (e.g., 8 or 16 heads). Each head learns different relationships:

• One head might learn syntactic relationships (subject-verb)
• Another might learn coreference (pronouns to their antecedents)
• Another might learn positional proximity

The outputs of all heads are concatenated and linearly projected.

Example

For the sentence: “The cat sat on the mat because it was tired”

The self-attention mechanism helps the model understand that “it” refers to “the cat” — the attention weight between “it” and “cat” will be high, while the weight between “it” and “mat” will be lower.

Main Tokenization Strategies

graph TD
    TEXT["Raw Text: 'The cats are playing'"]
    TEXT --> WL["Word-level: ['The', 'cats', 'are', 'playing']"]
    TEXT --> BPE["BPE: ['The', ' c', 'ats', ' are', ' play', 'ing']"]
    TEXT --> WP["WordPiece: ['The', 'cats', 'are', 'play', '##ing']"]

    style TEXT fill:#56cc9d,stroke:#333,color:#fff
    style BPE fill:#6cc3d5,stroke:#333,color:#fff
    style WP fill:#ffce67,stroke:#333

Why Subword Tokenization?

• Handles unknown words: Can represent any word by breaking it into known subwords
• Efficient vocabulary: Balances vocabulary size with sequence length
• Morphological awareness: Captures meaningful parts (prefixes, suffixes, roots)

Practical Considerations

• 1 token ≈ 4 characters (English) or ≈ 0.75 words
• Vocabulary sizes: GPT-4 uses ~100k tokens, LLaMA uses ~32k tokens
• Non-English languages and code often require more tokens per word

Detailed Comparison

Why Decoder-Only Dominates Today

Most modern LLMs (GPT-4, Claude, LLaMA, Gemini) are decoder-only because:

1. Simplicity: One unified architecture for all tasks
2. Scalability: Easier to scale with more parameters
3. Generality: Can handle classification, generation, and reasoning via prompting
4. Emergent abilities: Larger decoder-only models exhibit chain-of-thought reasoning

Fine-Tuning Approaches

LoRA (Low-Rank Adaptation) — Most Popular

LoRA freezes the pre-trained weights and injects trainable low-rank decomposition matrices:

where and , with rank .

When to Use What

• Prompt engineering first — no training needed, quick iteration
• LoRA/QLoRA — when you need task-specific behavior with limited compute
• Full fine-tuning — when you have large datasets and significant compute budget
• RLHF — when aligning model outputs with human preferences

The Three Steps

1. SFT (Supervised Fine-Tuning): Train the base model on high-quality human-written responses
2. Reward Model: Train a separate model to score responses based on human preference rankings
3. RL Optimization (PPO): Use the reward model as a signal to optimize the LLM’s outputs

Alternatives to RLHF

Why RLHF Matters

Without RLHF, base LLMs tend to:

• Continue text rather than answer questions
• Generate toxic, biased, or harmful content
• Hallucinate confidently
• Ignore user instructions

Types of Hallucinations

graph TD
    H["LLM Hallucinations"]
    H --> INT["Intrinsic Hallucination<br/>Contradicts the source input"]
    H --> EXT["Extrinsic Hallucination<br/>Cannot be verified from source"]

    INT --> INT_EX["Example: Summary says 'John went to Paris'<br/>when source says 'John went to London'"]
    EXT --> EXT_EX["Example: Model adds details<br/>not present in any source"]

    style H fill:#ff7851,stroke:#333,color:#fff
    style INT fill:#ffce67,stroke:#333
    style EXT fill:#6cc3d5,stroke:#333,color:#fff

Causes

Mitigation Strategies

Real-World Impact

Hallucinations are critical in high-stakes applications (legal, medical, financial). Production LLM systems almost always use RAG or other grounding techniques to minimize hallucinations.

RAG Pipeline Components

RAG vs. Fine-Tuning

When to Use RAG

• Knowledge changes frequently (news, documentation)
• Need verifiable, source-cited answers
• Domain-specific knowledge not in pre-training data
• Legal/compliance requirements for traceability

Key Prompting Techniques

graph TD
    PE["Prompt Engineering Techniques"]
    PE --> ZS["Zero-Shot<br/>'Classify this review as positive/negative'"]
    PE --> FS["Few-Shot<br/>'Here are 3 examples, now do this one'"]
    PE --> COT["Chain-of-Thought<br/>'Think step by step'"]
    PE --> SC["Self-Consistency<br/>Sample multiple CoT paths, majority vote"]
    PE --> TOT["Tree-of-Thought<br/>Explore multiple reasoning branches"]
    PE --> ROLE["Role Prompting<br/>'You are an expert data scientist...'"]

    style PE fill:#56cc9d,stroke:#333,color:#fff
    style COT fill:#6cc3d5,stroke:#333,color:#fff
    style SC fill:#ffce67,stroke:#333

Comparison of Techniques

System Prompt Best Practices

1. Be specific: “Extract the person’s name, company, and role” > “Extract information”
2. Define format: Specify JSON, markdown, or other output structures
3. Set constraints: “Answer only based on the provided context”
4. Provide examples: Show input-output pairs for complex tasks
5. Assign a role: “You are a senior Python developer reviewing code”

Temperature and Sampling Parameters

Key Production Patterns

Optimization Techniques

Evaluation in Production

• Automated: BLEU, ROUGE, BERTScore for generation quality
• LLM-as-Judge: Use a stronger model to evaluate outputs
• Human feedback: Thumbs up/down, preference ratings
• Task-specific: Accuracy, F1, faithfulness scores
• Safety: Toxicity rates, refusal rates, PII leakage

Security Considerations

• Prompt injection: Adversarial inputs that override system instructions
• Data leakage: Model revealing training data or system prompts
• PII exposure: Generating or storing personally identifiable information
• Jailbreaking: Users bypassing safety guardrails

The Chinchilla Scaling Law

The key finding (Hoffmann et al., 2022): for a given compute budget, model size and training tokens should be scaled equally.

where:

• = number of parameters
• = number of training tokens
• = loss (lower is better)
• = irreducible loss (entropy of natural language)

graph TD
    subgraph Scaling["Scaling Laws: Three Axes"]
        PARAMS["Model Parameters (N)<br/>More params → lower loss"]
        DATA["Training Data (D)<br/>More tokens → lower loss"]
        COMPUTE["Compute (C ≈ 6ND)<br/>More FLOPs → lower loss"]
    end

    subgraph Tradeoff["Chinchilla Optimal"]
        OPT["For budget C:<br/>Scale N and D equally<br/>D ≈ 20 × N tokens"]
    end

    Scaling --> Tradeoff

    style Scaling fill:#56cc9d,stroke:#333,color:#fff
    style Tradeoff fill:#6cc3d5,stroke:#333,color:#fff

Practical Implications

Emergent Abilities

Certain capabilities appear suddenly at scale rather than improving gradually:

• Chain-of-thought reasoning — emerges around 60-100B parameters
• In-context learning — improves dramatically with scale
• Multi-step math — requires large models to perform reliably

The Challenge: Quadratic Attention

Standard self-attention has complexity in both time and memory with respect to sequence length :

graph TD
    subgraph Problem["The Quadratic Problem"]
        S1["4K tokens → 16M attention entries"]
        S2["32K tokens → 1B attention entries"]
        S3["128K tokens → 16B attention entries"]
        S4["1M tokens → 1T attention entries"]
    end

    subgraph Solutions["Solutions"]
        SOL1["Efficient Attention<br/>Flash Attention, Ring Attention"]
        SOL2["Position Extrapolation<br/>RoPE, ALiBi, YaRN"]
        SOL3["Sparse Attention<br/>Sliding window, dilated"]
        SOL4["Memory/Retrieval<br/>Compress old context"]
    end

    Problem --> Solutions

    style Problem fill:#ff7851,stroke:#333,color:#fff
    style Solutions fill:#56cc9d,stroke:#333,color:#fff

Key Techniques for Long Context

The “Lost in the Middle” Problem

Research shows LLMs tend to:

• Remember information at the beginning and end of context well
• Struggle with information in the middle of long contexts
• Performance degrades as relevant information is placed further from query

Practical Considerations

• Longer context ≠ better retrieval (RAG often outperforms naive long context)
• KV cache memory grows linearly with sequence length
• Inference cost increases with context length even if most tokens are irrelevant

Memory Requirements Example (LLaMA-70B)

Quantization Approaches

Quality vs. Compression Tradeoff

Key Insight

4-bit quantization (GPTQ, AWQ) has become the standard for local LLM deployment because it offers near-lossless quality with 4x memory reduction — enabling 70B models to run on consumer hardware.

Core Agent Patterns

Tool Use

Tools transform LLMs from text generators into action-taking systems:

Agent Frameworks

Challenges

• Reliability: Agents can go off-track or loop infinitely
• Cost: Multiple LLM calls per task (tool reasoning is expensive)
• Safety: Autonomous actions need guardrails
• Evaluation: Hard to benchmark open-ended agent behavior

Metric Selection by Task

LLM-as-Judge

Using a stronger LLM to evaluate outputs has become standard:

System: You are an expert evaluator. Rate the following response on:
1. Helpfulness (1-5)
2. Accuracy (1-5)
3. Harmlessness (1-5)

Provide scores and brief justification.

Advantages: Scalable, consistent, correlates well with human judgment
Limitations: Biases (position bias, verbosity bias, self-preference)

Key Benchmarks (2024-2026)

Evaluation Anti-Patterns

• Benchmark contamination: Test data leaks into training
• Over-optimizing for benchmarks: Gaming metrics without real improvement
• Single metric reliance: Missing failure modes
• Static evaluation: Not tracking performance over time in production

Types of Embeddings

Similarity Metrics

Metric	Formula	Range	Best for
Cosine similarity		[-1, 1]	Most NLP tasks
Euclidean distance		[0, ∞)	Clustering
Dot product		(-∞, ∞)	When magnitude matters

Modern Embedding Models (2024-2026)

Practical Embedding Pipeline

1. Chunk documents into semantically meaningful segments
2. Embed chunks using a sentence embedding model
3. Store vectors in a vector database (Pinecone, Weaviate, pgvector)
4. Query-time: embed the user query with the same model
5. Retrieve: find top-k nearest neighbors via ANN (approximate nearest neighbor)

Common Pitfalls

• Using different embedding models for indexing vs. querying
• Chunks too large (loses specificity) or too small (loses context)
• Not normalizing vectors when using cosine similarity
• Ignoring embedding model’s max token limit

Dense vs. MoE Comparison

Notable MoE Models

Challenges with MoE

Why MoE Matters

MoE allows training models with trillions of parameters while keeping inference costs manageable — it’s likely the architecture behind the largest frontier models.

Why KV Cache is Necessary

During generation, the LLM produces one token at a time. Without caching, generating token requires recomputing attention over all previous tokens from scratch.

graph LR
    subgraph Without_Cache["Without KV Cache"]
        W1["Token 1: compute K,V for [1]"]
        W2["Token 2: compute K,V for [1,2]"]
        W3["Token 3: compute K,V for [1,2,3]"]
        W4["Token N: compute K,V for [1,...,N]"]
        W1 --> W2 --> W3 --> W4
    end

    subgraph With_Cache["With KV Cache"]
        C1["Token 1: compute & store K₁,V₁"]
        C2["Token 2: compute K₂,V₂, reuse K₁,V₁"]
        C3["Token 3: compute K₃,V₃, reuse K₁,V₁,K₂,V₂"]
        C1 --> C2 --> C3
    end

    style Without_Cache fill:#ff7851,stroke:#333,color:#fff
    style With_Cache fill:#56cc9d,stroke:#333,color:#fff

Complexity Comparison

Metric	Without KV Cache	With KV Cache
Compute per token	(recompute all)	(new token only)
Total generation
Memory		grows with sequence

KV Cache Memory Problem

For a model with layers, heads, head dimension, sequence length :

Example: LLaMA-70B with 128K context in FP16: GB — just for the cache!

KV Cache Optimization Techniques

Prefill vs. Decode Phases

The decode phase is typically memory-bandwidth-bound because each new token requires reading the entire KV cache from memory.

Comparison of Training Stages

What Makes Good Instruction Tuning Data?

Notable Instruction Datasets

Base Model vs. Instruction-Tuned Behavior

Prompt: “What is the capital of France?”

• Base model: “What is the capital of Germany? What is the capital of Spain?…” (continues the pattern)
• Instruction-tuned: “The capital of France is Paris.” (answers directly)

Speculative Decoding

Uses a small, fast draft model to predict multiple tokens, then the large model verifies them in parallel:

Continuous Batching vs. Static Batching

Parallelism Strategies

LLM Serving Frameworks

Key Metrics for Production Serving

Parameter Overview

Practical Configuration Examples

Mathematical Definition

Given logits for each token in the vocabulary:

where is the temperature.

graph LR
    subgraph T0["Temperature = 0 (Greedy)"]
        T0_1["Token A: 99.9%"]
        T0_2["Token B: 0.1%"]
        T0_3["Token C: ~0%"]
    end

    subgraph T07["Temperature = 0.7"]
        T07_1["Token A: 75%"]
        T07_2["Token B: 20%"]
        T07_3["Token C: 5%"]
    end

    subgraph T1["Temperature = 1.0 (Default)"]
        T1_1["Token A: 60%"]
        T1_2["Token B: 25%"]
        T1_3["Token C: 15%"]
    end

    subgraph T2["Temperature = 2.0"]
        T2_1["Token A: 40%"]
        T2_2["Token B: 32%"]
        T2_3["Token C: 28%"]
    end

    style T0 fill:#56cc9d,stroke:#333,color:#fff
    style T07 fill:#6cc3d5,stroke:#333,color:#fff
    style T1 fill:#ffce67,stroke:#333
    style T2 fill:#ff7851,stroke:#333,color:#fff

Effect of Temperature

Intuition

Think of temperature as a “creativity knob”:

• Low temperature (0–0.3): The model is confident and focused — it picks the most obvious next word. Great for factual tasks, code, structured extraction.
• Medium temperature (0.5–0.8): The model is balanced — it explores alternatives while staying coherent. Best for general chat and writing.
• High temperature (1.0+): The model is adventurous — it considers unlikely words, producing surprising or creative outputs.

Common Interview Follow-Up: “What does temperature=0 actually mean?”

Setting temperature=0 is a shortcut for greedy decoding — the model always selects the single highest-probability token. However:

• It’s still based on floating-point arithmetic, so minor non-determinism can occur across hardware
• Most APIs interpret temperature=0 as “return the argmax token” deterministically
• Some providers require setting a seed parameter for guaranteed reproducibility

Top-k Sampling

Select the k most probable tokens and redistribute probability among them:

Top-p (Nucleus) Sampling

Select the smallest set of tokens whose cumulative probability exceeds :

graph TD
    subgraph TopK["Top-k = 3 (Fixed Size)"]
        direction LR
        K1["'the' (0.40) ✓"]
        K2["'a' (0.25) ✓"]
        K3["'my' (0.15) ✓"]
        K4["'his' (0.10) ✗"]
        K5["'our' (0.05) ✗"]
        K6["'their' (0.03) ✗"]
    end

    subgraph TopP["Top-p = 0.9 (Dynamic Size)"]
        direction LR
        P1["'the' (0.40) ✓ → cumulative: 0.40"]
        P2["'a' (0.25) ✓ → cumulative: 0.65"]
        P3["'my' (0.15) ✓ → cumulative: 0.80"]
        P4["'his' (0.10) ✓ → cumulative: 0.90 ≥ p"]
        P5["'our' (0.05) ✗"]
        P6["'their' (0.03) ✗"]
    end

    style TopK fill:#6cc3d5,stroke:#333,color:#fff
    style TopP fill:#56cc9d,stroke:#333,color:#fff

Key Differences

Why Top-p is Generally Preferred

Consider two scenarios at different generation steps:

Step A (peaked distribution): Model is 95% sure the next word is “Paris”

• Top-k=50: Considers 50 tokens (49 are noise)
• Top-p=0.95: Considers only 1-2 tokens (adaptive!)

Step B (flat distribution): Model is uncertain, many tokens are equally likely

• Top-k=50: Might miss some reasonable candidates if vocabulary is large
• Top-p=0.95: Includes all tokens until 95% mass is covered (could be 100+ tokens)

Combining Top-k and Top-p

In practice, many systems use both simultaneously:

1. First apply Top-k to limit to k candidates
2. Then apply Top-p within those k candidates

This provides both an upper bound (Top-k) and adaptive filtering (Top-p).

Recommended Settings

Context Window Sizes (2024–2026)

What Happens When You Exceed the Context Window?

Context Window vs. Effective Context

Key insight for interviews: The advertised context window is not the same as effective context:

Strategies for Context Window Management

Cost Implications

Context window directly affects cost:

Longer contexts mean higher costs, higher latency, and more KV-cache memory usage.

Sources of Non-Determinism

How to Achieve Deterministic Output

When Determinism Matters

Important Caveat

Even with temperature=0 and a seed, exact determinism is not always guaranteed:

• GPU floating-point operations may vary across hardware versions
• API providers may route requests to different hardware
• Model quantization can introduce slight variations
• OpenAI states: “deterministic outputs are not guaranteed” even with seed (but are “mostly deterministic”)

Detailed Comparison

Greedy Search: The Simplest Strategy

At each step, pick the token with the highest probability:

Problem: Greedy search is locally optimal but not globally optimal. A low-probability token now might lead to a much better overall sequence.

Example: “The dog” (0.4) → “has” (0.9) gives sequence probability 0.36, while “The nice” (0.5) → “woman” (0.4) gives 0.20. Greedy picks “nice” first but misses the better path.

Beam Search: Exploring Multiple Paths

Maintains num_beams parallel hypotheses:

Beam 1: "The" → "dog" → "has" → "a"      (prob: 0.36 × ...)
Beam 2: "The" → "nice" → "woman" → "is"   (prob: 0.20 × ...)
Beam 3: "The" → "cat" → "sat" → "on"      (prob: 0.15 × ...)

When to use beam search:

• Translation (known output length)
• Summarization (structured output)
• NOT for open-ended generation (causes repetition)

When to Use Which Strategy

Mathematical Definitions

The logit for token is adjusted before sampling:

where is how many times token has appeared in the output so far.

graph TD
    subgraph FP["Frequency Penalty"]
        FP1["Penalizes proportionally to<br/>how MANY times token appeared"]
        FP2["Token appeared 5× → big penalty"]
        FP3["Token appeared 1× → small penalty"]
        FP4["Effect: Reduces repetitive words"]
    end

    subgraph PP["Presence Penalty"]
        PP1["Penalizes equally if token<br/>appeared AT ALL (binary)"]
        PP2["Token appeared 5× → same penalty as 1×"]
        PP3["Token never appeared → no penalty"]
        PP4["Effect: Encourages new topics"]
    end

    style FP fill:#6cc3d5,stroke:#333,color:#fff
    style PP fill:#ffce67,stroke:#333

Comparison

Practical Examples

Without penalties (both = 0): > “The weather is nice. The weather is really nice. The weather makes me happy. The weather…”

With frequency_penalty = 0.8: > “The weather is nice. It’s a beautiful day. The sunshine makes me happy. I think I’ll go outside…”

With presence_penalty = 1.0: > “The weather is nice. I’ve been reading a great book lately. My garden is blooming. Tomorrow I plan to cook…”

Repetition Penalty (Hugging Face)

Hugging Face uses a multiplicative repetition_penalty instead:

• repetition_penalty = 1.0: No effect
• repetition_penalty = 1.2: Moderate de-repetition (common default)
• repetition_penalty > 1.5: Strong — may cause incoherence

Key Relationships

If you set max_tokens higher than available space, the API will either:

• Silently cap it at the available space
• Return an error

max_tokens vs. max_new_tokens

Why Generation Stops

Generation terminates when any of these conditions is met:

Practical Implications

Cost Optimization

Since APIs charge per token:

• Set max_tokens appropriate to the task (not arbitrarily high)
• Use stop sequences to terminate early
• Monitor actual token usage vs. max_tokens budget

Common Stop Sequence Use Cases

Example: Controlling Multi-Turn Chat

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "List 3 fruits"}],
    stop=["\n\n", "4."],  # Stop after 3 items
    max_tokens=200
)

Without stop sequences: Model might continue listing dozens of fruits or add commentary.

With stop sequences: Generation stops cleanly after the third item.

Stop Sequences vs. Other Stopping Mechanisms

Best Practices

1. Include the delimiter that separates outputs (e.g., "\n\n" between paragraphs)
2. Test with variations — models might generate "\n " instead of "\n\n"
3. Combine with max_tokens as a safety net
4. Don’t over-specify — too many stop sequences can cause premature truncation

Decision Framework

Complete Configuration Recipes

temperature: 0.0
top_p: 1.0
max_tokens: 2048
stop: ["\n\n\n", "```"]
frequency_penalty: 0.0

temperature: 0.3
top_p: 0.9
max_tokens: 512
presence_penalty: 0.2
stop: ["Human:", "Customer:"]

temperature: 0.9
top_p: 0.95
max_tokens: 4096
frequency_penalty: 0.7
presence_penalty: 0.5

temperature: 0.0
top_p: 1.0
max_tokens: 256
response_format: {"type": "json_object"}
stop: ["}\n"]

temperature: 1.2
top_p: 0.95
max_tokens: 1024
frequency_penalty: 1.0
presence_penalty: 1.5
n: 5

Common Mistakes