Deploying and Serving LLM with Llama.cpp

End-to-end guide: deploy and serve LLMs locally and at scale with llama.cpp for efficient CPU/GPU inference

Keywords

llama.cpp, LLM serving, model deployment, GGUF, quantization, inference optimization, CPU inference, GPU inference, OpenAI API, production LLM

Deploying and serving LLMs locally with llama.cpp

Introduction

Not every LLM deployment requires a high-end GPU cluster. Many real-world use cases — edge devices, local development, cost-sensitive production — demand efficient inference on commodity hardware.

llama.cpp is an open-source C/C++ inference engine that makes LLM deployment:

• Hardware-flexible (runs on CPU, GPU, Apple Silicon, and hybrid CPU+GPU)
• Memory-efficient (via aggressive quantization — 2-bit to 8-bit GGUF formats)
• Production-ready (built-in OpenAI-compatible API server)
• Portable (no Python runtime or CUDA toolkit required at inference time)
• Easy to deploy (single binary, Docker, embedded systems)

In this tutorial, we will walk through a complete pipeline:

1. Install and build llama.cpp
2. Obtain and quantize models in GGUF format
3. Run offline inference from the CLI
4. Serve a model with the OpenAI-compatible API server
5. Query the model from Python
6. Optimize for production deployment

graph LR
    A["Install<br/>llama.cpp"] --> B["Obtain GGUF<br/>model"]
    B --> C["CLI inference<br/>or API server"]
    C --> D["Query from<br/>Python / curl"]
    D --> E["Deploy to<br/>production"]

    style A fill:#ffce67,stroke:#333
    style B fill:#ffce67,stroke:#333
    style C fill:#6cc3d5,stroke:#333,color:#fff
    style D fill:#6cc3d5,stroke:#333,color:#fff
    style E fill:#56cc9d,stroke:#333,color:#fff

What is llama.cpp?

llama.cpp is a high-performance C/C++ inference engine for LLMs, originally created by Georgi Gerganov. Key features include:

• GGUF format: Purpose-built model format with embedded metadata, supporting a wide range of quantization levels
• Quantization: Reduce model size and memory usage with minimal quality loss (Q2_K through Q8_0)
• CPU optimization: AVX, AVX2, AVX-512, ARM NEON for fast inference without a GPU
• GPU offloading: Offload layers to NVIDIA (CUDA), AMD (ROCm), Apple Metal, or Vulkan GPUs
• Built-in server: OpenAI-compatible HTTP API with continuous batching
• Support for many models: Llama, Mistral, Qwen, Phi, Gemma, DeepSeek, and more

graph TD
    A["llama.cpp"] --> B["GGUF Format<br/>Embedded metadata"]
    A --> C["Quantization<br/>Q2_K to Q8_0"]
    A --> D["CPU Optimized<br/>AVX / AVX2 / NEON"]
    A --> E["GPU Offloading<br/>CUDA / Metal / Vulkan"]
    A --> F["Built-in Server<br/>OpenAI-compatible"]

    style A fill:#56cc9d,stroke:#333,color:#fff
    style B fill:#6cc3d5,stroke:#333,color:#fff
    style C fill:#6cc3d5,stroke:#333,color:#fff
    style D fill:#6cc3d5,stroke:#333,color:#fff
    style E fill:#6cc3d5,stroke:#333,color:#fff
    style F fill:#6cc3d5,stroke:#333,color:#fff

Hardware Requirements

llama.cpp is designed to run on a wide range of hardware, from laptops to servers:

Model Size	Quantization	RAM/VRAM Needed	Recommended Hardware
0.5B–3B	Q4_K_M	2–3 GB	Any modern CPU / Raspberry Pi 5
7B–8B	Q4_K_M	5–6 GB	16 GB RAM laptop / RTX 3060
13B	Q4_K_M	9–10 GB	32 GB RAM / RTX 4090
70B	Q4_K_M	40–45 GB	64 GB RAM / A100 (multi-GPU)

Key advantage: llama.cpp can run entirely on CPU, making it ideal for environments without GPUs.

Installation

Option A: Install via pip (Recommended)

The easiest way to install llama.cpp’s Python bindings and server:

pip install llama-cpp-python

With GPU acceleration (CUDA):

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

With Apple Metal support:

CMAKE_ARGS="-DGGML_METAL=on" pip install llama-cpp-python

Option B: Build from Source

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

CPU-only build:

cmake -B build
cmake --build build --config Release

With CUDA support:

cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release

With Apple Metal support:

cmake -B build -DGGML_METAL=ON
cmake --build build --config Release

Verify installation

./build/bin/llama-cli --version

Obtaining GGUF Models

graph TD
    A["Need a GGUF model"] --> B{"Source?"}
    B -->|"Pre-quantized"| C["Download from<br/>HuggingFace"]
    B -->|"Own model"| D["Convert HF model<br/>to GGUF (F16)"]
    D --> E["Quantize to<br/>Q4_K_M / Q5_K_M"]
    C --> F["Ready for<br/>inference"]
    E --> F

    style A fill:#f8f9fa,stroke:#333
    style C fill:#56cc9d,stroke:#333,color:#fff
    style D fill:#ffce67,stroke:#333
    style E fill:#ffce67,stroke:#333
    style F fill:#56cc9d,stroke:#333,color:#fff

Download Pre-quantized Models from HuggingFace

Many models are available pre-quantized in GGUF format:

# Using huggingface-cli
pip install huggingface_hub
huggingface-cli download unsloth/Qwen3-0.6B-GGUF Q4_K_M.gguf --local-dir ./models

Quantize a Model Yourself

If you have a model in HuggingFace (safetensors) format, convert and quantize it:

# Step 1: Convert to GGUF (F16)
python convert_hf_to_gguf.py ./hf_model_dir --outfile model-f16.gguf --outtype f16

# Step 2: Quantize to Q4_K_M
./build/bin/llama-quantize model-f16.gguf model-Q4_K_M.gguf Q4_K_M

Common Quantization Levels

Quantization	Bits	Size (7B)	Quality	Speed
Q2_K	2	~2.7 GB	Low	Fastest
Q4_K_M	4	~4.1 GB	Good	Fast
Q5_K_M	5	~4.8 GB	Very Good	Medium
Q6_K	6	~5.5 GB	Excellent	Medium
Q8_0	8	~7.2 GB	Near-FP16	Slower
F16	16	~13.5 GB	Lossless	Slowest

Recommendation: Q4_K_M offers the best balance of quality, size, and speed for most use cases.

Offline Inference (CLI)

Use llama.cpp for fast inference directly from the command line.

Basic Text Generation

./build/bin/llama-cli \
    -m ./models/Q4_K_M.gguf \
    -p "Explain machine learning in simple terms." \
    -n 256 \
    --temp 0.7

Interactive Chat Mode

./build/bin/llama-cli \
    -m ./models/Q4_K_M.gguf \
    --chat-template chatml \
    -cnv \
    --temp 0.7

With GPU Offloading

Offload layers to GPU for faster inference (use -ngl to specify number of layers):

./build/bin/llama-cli \
    -m ./models/Q4_K_M.gguf \
    -p "What is the difference between AI and ML?" \
    -n 256 \
    -ngl 99 \
    --temp 0.7

-ngl 99 offloads all layers to GPU. Use a lower number for partial offloading when VRAM is limited.

Batch Inference with Python

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Q4_K_M.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,  # -1 = offload all layers to GPU
)

prompts = [
    "Explain machine learning in simple terms.",
    "What is the difference between AI and ML?",
    "Write a Python function to reverse a string.",
]

for prompt in prompts:
    output = llm(
        prompt,
        max_tokens=256,
        temperature=0.7,
    )
    print(output["choices"][0]["text"])
    print("---")

Chat-style Inference with Python

from llama_cpp import Llama

llm = Llama(
    model_path="./models/Q4_K_M.gguf",
    n_ctx=4096,
    n_gpu_layers=-1,
    chat_format="chatml",
)

messages = [
    {"role": "system", "content": "You are a helpful AI assistant."},
    {"role": "user", "content": "Explain llama.cpp in simple terms."},
]

output = llm.create_chat_completion(
    messages=messages,
    max_tokens=256,
    temperature=0.7,
)

print(output["choices"][0]["message"]["content"])

Serving with OpenAI-Compatible API

llama.cpp includes a built-in HTTP server that is fully compatible with the OpenAI API format.

graph TD
    A["llama-server"] --> B["Load GGUF model"]
    B --> C["OpenAI-compatible<br/>/v1/chat/completions"]
    C --> D["curl"]
    C --> E["Python requests"]
    C --> F["OpenAI Python client"]

    G["Options"] --> H["-ngl: GPU layers"]
    G --> I["-c: Context length"]
    G --> J["--slots: Parallelism"]

    style A fill:#56cc9d,stroke:#333,color:#fff
    style B fill:#6cc3d5,stroke:#333,color:#fff
    style C fill:#6cc3d5,stroke:#333,color:#fff
    style D fill:#f8f9fa,stroke:#333
    style E fill:#f8f9fa,stroke:#333
    style F fill:#f8f9fa,stroke:#333
    style G fill:#ffce67,stroke:#333

Start the Server (C++ binary)

./build/bin/llama-server \
    -m ./models/Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --api-key your-secret-key \
    --chat-template chatml \
    -ngl 99 \
    -c 4096 \
    --slots 4

Key options explained:

• -m: Path to the GGUF model file
• -ngl 99: Number of layers to offload to GPU (99 = all layers)
• -c 4096: Context length (max tokens for input + output)
• --slots 4: Number of concurrent request slots (controls parallelism)
• --chat-template: Chat template format (chatml, llama2, mistral, etc.)
• --api-key: API key for authentication

Start the Server (Python)

python -m llama_cpp.server \
    --model ./models/Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --n_gpu_layers -1 \
    --chat_format chatml \
    --n_ctx 4096

Verify the Server

curl http://localhost:8000/v1/models

Querying the API

Using curl

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer your-secret-key" \
    -d '{
        "model": "default",
        "messages": [
            {"role": "user", "content": "What is llama.cpp?"}
        ],
        "temperature": 0.7,
        "max_tokens": 256
    }'

Using Python (requests)

import requests

response = requests.post(
    "http://localhost:8000/v1/chat/completions",
    headers={"Authorization": "Bearer your-secret-key"},
    json={
        "model": "default",
        "messages": [
            {"role": "user", "content": "Explain GGUF quantization."}
        ],
        "temperature": 0.7,
        "max_tokens": 256,
    }
)

print(response.json()["choices"][0]["message"]["content"])

Using OpenAI Python Client (Recommended)

Since llama.cpp server is OpenAI-compatible, you can use the official OpenAI client:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key",
)

response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is quantization in LLMs?"},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)

Serving Custom / Fine-tuned Models

If you fine-tuned a small LLM with Unsloth and exported it to GGUF (e.g., gguf_model_small), here is how to serve it with llama.cpp.

graph TD
    A["Fine-tuned model"] --> B{"Format?"}
    B -->|"Already GGUF"| C["Serve directly<br/>with llama-server"]
    B -->|"HF safetensors"| D["Convert to GGUF<br/>(convert_hf_to_gguf.py)"]
    B -->|"LoRA adapter"| E["Serve with<br/>--lora flag"]
    D --> F["Quantize<br/>(llama-quantize)"]
    F --> C
    C --> G["OpenAI-compatible API"]
    E --> G

    style A fill:#f8f9fa,stroke:#333
    style C fill:#56cc9d,stroke:#333,color:#fff
    style D fill:#ffce67,stroke:#333
    style F fill:#ffce67,stroke:#333
    style E fill:#6cc3d5,stroke:#333,color:#fff
    style G fill:#56cc9d,stroke:#333,color:#fff

Option A: Serve a GGUF File Directly

Step 1: Prepare Your GGUF Model

After fine-tuning with Unsloth and exporting to GGUF, you should have a file like:

gguf_model_small/
├── added_tokens.json
├── chat_template.jinja
├── config.json
├── generation_config.json
├── merges.txt
├── model.safetensors
├── special_tokens_map.json
├── tokenizer.json
├── tokenizer_config.json
├── unsloth.BF16.gguf
├── unsloth.Q4_K_M.gguf
└── vocab.json

Step 2: Serve with llama.cpp

./build/bin/llama-server \
    -m ./gguf_model_small/unsloth.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --api-key your-secret-key \
    --chat-template chatml \
    -ngl 99 \
    -c 2048 \
    --slots 4

Step 3: Verify and Query

curl http://localhost:8000/v1/models \
    -H "Authorization: Bearer your-secret-key"

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="your-secret-key",
)

response = client.chat.completions.create(
    model="default",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is fine-tuning?"},
    ],
    temperature=0.7,
    max_tokens=256,
)

print(response.choices[0].message.content)

Option B: Convert from HuggingFace Format

If your model is in HuggingFace safetensors format, convert it first:

# Convert to GGUF
python convert_hf_to_gguf.py ./hf_model_small --outfile my-model-f16.gguf --outtype f16

# Quantize
./build/bin/llama-quantize my-model-f16.gguf my-model-Q4_K_M.gguf Q4_K_M

# Serve
./build/bin/llama-server \
    -m my-model-Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --api-key your-secret-key \
    --chat-template chatml \
    -ngl 99 \
    -c 2048

Serve with LoRA Adapters

llama.cpp supports applying LoRA adapters at inference time without merging:

./build/bin/llama-server \
    -m ./models/base-model-Q4_K_M.gguf \
    --lora ./lora-adapter.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    --api-key your-secret-key \
    -ngl 99

Then query normally:

response = client.chat.completions.create(
    model="default",
    messages=[{"role": "user", "content": "Hello!"}],
)

Docker Deployment

Deploy llama.cpp in a container for production environments.

graph LR
    subgraph cpu["CPU Deployment"]
        A1["llama.cpp:server"] --> B1["Mount models<br/>volume"]
        B1 --> C1["Expose port 8000"]
    end
    subgraph gpu["GPU Deployment"]
        A2["llama.cpp:server-cuda"] --> B2["--gpus all<br/>+ mount models"]
        B2 --> C2["Expose port 8000"]
    end

    style cpu fill:#6cc3d5,stroke:#333,color:#fff
    style gpu fill:#56cc9d,stroke:#333,color:#fff

Run with Docker (CPU)

docker run -p 8000:8000 \
    -v ./models:/models \
    ghcr.io/ggerganov/llama.cpp:server \
    -m /models/Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    -c 4096

Run with Docker (CUDA GPU)

docker run --gpus all -p 8000:8000 \
    -v ./models:/models \
    ghcr.io/ggerganov/llama.cpp:server-cuda \
    -m /models/Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8000 \
    -ngl 99 \
    -c 4096

Docker Compose

version: '3.8'
services:
  llama-server:
    image: ghcr.io/ggerganov/llama.cpp:server-cuda
    ports:
      - "8000:8000"
    volumes:
      - ./models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    command: >
      -m /models/Q4_K_M.gguf
      --host 0.0.0.0
      --port 8000
      -ngl 99
      -c 4096
      --slots 4

Performance Optimization Tips

• GPU offloading: Use -ngl 99 to offload all model layers to GPU; use a lower value for partial offloading when VRAM is limited
• Context length: Set -c to the minimum context length you need — larger context uses more memory
• Concurrent slots: Use --slots N to control how many requests can be processed in parallel
• Quantization choice: Q4_K_M is the sweet spot for most use cases; use Q5_K_M or Q6_K if quality matters more than speed
• Memory mapping: llama.cpp uses mmap by default, allowing models to be loaded without duplicating in RAM
• Batch size: Use -b to tune the prompt processing batch size (default 2048); larger values can speed up prompt processing
• Flash attention: Use --flash-attn to enable Flash Attention for faster inference (if supported)
• Streaming: Use stream=True for real-time token generation

llama.cpp vs Other Serving Solutions

Feature	llama.cpp	vLLM	Ollama	TGI
CPU Inference	Excellent	No	Good	No
GPU Inference	Good	Excellent	Good	Excellent
Throughput	Low-Medium	Very High	Medium	High
GPU Required	No	Yes	No	Yes
OpenAI API	Yes	Yes	Partial	Yes
Multi-GPU	Limited	Yes	No	Yes
Quantization	Extensive (GGUF)	AWQ/GPTQ	GGUF	AWQ/GPTQ
Ease of Use	Medium	Medium	Easy	Medium
Best For	Edge/CPU/Local	Production GPU	Local Dev	Production GPU

Conclusion

llama.cpp is the go-to solution for flexible, hardware-efficient LLM deployment:

• Runs on CPU, GPU, Apple Silicon, and hybrid configurations
• Supports extensive quantization for reduced memory and faster inference
• Provides an OpenAI-compatible API server for seamless integration
• Handles custom and fine-tuned models in GGUF format
• Deploys easily with Docker or as a single binary

This workflow is perfect for:

• Local development and prototyping
• Edge and embedded AI deployments
• Cost-sensitive production environments
• CPU-only server deployments
• Laptop and desktop AI applications

Read More

• Combine with a RAG pipeline (LangChain + llama.cpp)
• Add load balancing with Nginx or Traefik
• Deploy on Kubernetes with mixed CPU/GPU node pools
• Monitor with Prometheus + Grafana (llama.cpp exposes /metrics)
• Explore speculative decoding for faster inference
• Use grammar-constrained generation for structured output (JSON mode)

Explore LLMs Home