Building a self-hosted LLM system feels like choosing between hundreds of half-documented tools. You read marketing pages, find conflicting RAM claims, and still don't know if your hardware will run anything offline. This guide walks you through the actual workflow: verify specs, test locally, then deploy.
Start with what you have, not what sounds cutting-edge. Open a terminal and check RAM, GPU type, and storage.
On Linux/Mac: free -h and lspci | grep -i nvidia or system_profiler SPDisplaysDataType. On Windows: Task Manager → Performance. Note the total RAM, GPU model, and VRAM if you have a discrete GPU.
The baseline matters because tool specs in marketing copy often list "minimum" (won't work well) versus "recommended" (will work). A 7B-parameter LLM needs roughly 14 GB RAM to run comfortably on CPU alone. An 8 GB system can run it, but expect 3–5 second latency per token. If you have a GPU with 8 GB VRAM, you're in a different ballpark entirely—quantized models run at 50–100 tokens/second.
Document your setup: RAM, GPU yes/no, storage available, and your typical inference speed requirement. This becomes your filter.
Three categories exist: CPU-only runtimes (llama.cpp, Ollama), GPU-optimized runtimes (vLLM, text-generation-webui), and framework-based systems (LM Studio). Each trades speed, ease, and resource use differently.
If you have no GPU: llama.cpp runs any quantized GGUF model on CPU and outputs benchmark numbers upfront. Ollama wraps models and handles quantization for you—slower setup to specs, but friendlier. Both run fully offline once downloaded.
If you have an NVIDIA GPU: vLLM squeezes maximum throughput but requires CUDA 11.8+ and wins on batch inference. text-generation-webui (oobabooga) offers a web UI and works with multiple backends—slower development cycle, broader hardware support. LM Studio packages everything in one app but less control over quantization.
If you have a Mac with Metal GPU: Ollama handles it transparently. MLC LLM also works well. CPU fallback is automatic if quantization is set right.
Decision rule: Start with Ollama or llama.cpp. Both state hardware requirements clearly in their GitHub issues section. Test one model in 5 minutes. You will know if your hardware is undersized before you commit to a full stack.
A vector database stores document embeddings so your LLM can retrieve relevant context. For local deployment, options range from embedded (Chroma, Faiss) to standalone (Weaviate, Milvus).
Embedded vector databases live in your Python process. Chroma needs ~500 MB to start, scales to millions of vectors with minimal overhead, and requires no separate service. Use this if you have under 10 million vectors and want one command to deploy.
Standalone vector databases run as a service. Weaviate uses ~1–2 GB base memory, scales horizontally, and handles concurrent queries well. Use this if you have multiple applications querying the same vector store or need production-grade availability.
Simpler alternative: Faiss (Facebook AI Similarity Search) is a library, not a database. It's CPU-only, holds vectors in memory or on disk, and has zero network overhead. Good for prototypes under 100 million vectors. Not suitable for concurrent multi-user access.
Check the database's README for disk/memory claims and test with your document corpus size. A million 1536-dimensional vectors (OpenAI embedding size) uses roughly 6 GB disk and 3–4 GB RAM in a production system.
"Offline" means different things. One tool downloads weights once and never phones home. Another checks for updates every restart. A third silently falls back to cloud if local inference fails.
Test this yourself: Download your LLM and vector DB. Unplug the network. Run an inference. If it succeeds, that tool is offline. If it hangs or errors, it isn't—regardless of what the marketing page says.
Document: Can you download the model? What is the download size? Does the tool cache embeddings or recompute them? Can you run it without an internet connection for 24 hours? These facts belong in your architecture notes, not on a checklist.
Many tools claim offline but ship with telemetry or update checks. Read the source code or ask in the GitHub issues. If 100+ people use it and someone would have complained about cloud calls, you'll find the answer there.
Before scaling, test end-to-end: LLM runtime → vector DB → one inference query → answer returned. No distributed setup, no microservices. Just the three components talking.
Write a 20-line script: load a small document, embed it, query the vector DB, pass the results to the LLM, print the response. Time the latency. Measure memory use during inference. If this works, your stack is sound and scaling is just repeating it across more documents or instances.
Common failure point: Embedding model and vector DB dimensions don't match. Runtime uses 1536-dim embeddings but your DB is configured for 384-dim. This breaks silently in some frameworks. Catch it in the integration test.
Once one end-to-end test passes, you have a baseline. Deploy to production with the same tool versions, same Docker image, same quantization settings. Change one variable at a time afterward.