An Open Field Guide to Inference Engineering

Copy page Copy page Copy page as Markdown for LLMs View as Markdown View this page as plain text Copy MCP Server Copy MCP Server URL to clipboard Connect to Cursor Install MCP Server on Cursor Connect to VS Code Install MCP Server on VS Code An Open Field Guide to Inference Engineering On this page On this page Note Numbers, prices and tool versions are as of October 2026. This guide is for platform engineers new to inference. If you need to understand how models and GPUs work together and how to run inference in production, this guide will help. You’ll also learn about the tools to use and how to grow into an inference engineering role. This guide builds on the work of others in the inference community. If you’re interested in inference engineering, continue your learning with these: Viktor Farcic’s LLM Inference Explained Philip Kiely’s Inference Engineering The Pragmatic Engineer, What Is Inference Engineering? ByteByteGo’s Guide to AI Inference Engineering Data Gravity on The Rise of Inference Engineering Spheron’s Inference Engineering Guide Vishakha Sadhwani’s The AI Inference Engineering Roadmap Upbound, Project Champagne series 1. Why now? In an AI-forward organization, platform teams become inference teams. Coding assistants can’t send source code to third parties, and frontier model bills increase month to month. Support tools also need internal data that a hosted model can’t reach. Organizations want to run these models themselves, and open models now make that practical. Hugging Face hosts more than two million open models, and the best of them compete with closed models. DeepSeek released R1 in January 2025, and it matched OpenAI’s o1 on math and coding benchmarks about a month after o1 came out . When you run your own model, you control latency, uptime, and where your data goes. At high volume, you can also cut costs by about 80% . The work suits platform engineers. The price of a fixed level of AI capability falls roughly 10x a year, and most of that comes from how models are served, not from better GPUs . Upbound, the company behind Crossplane and Modelplane, saw this firsthand. In its Project Champagne blog series, the team ran its own models on Modelplane and found that vLLM and scheduling worked. The time went to TLS, identity, timeouts, storage, IAM and observability. Skills that carry over An inference engineer picks a model, gets it onto the right GPU, keeps it serving, makes it faster and cheaper, and gets paged when it breaks. That’s SRE for a new workload, and most of your skills carry over: What you already know What it becomes in inference Scheduling and node pools GPU placement with device selectors, taints and exclusive devices Capacity planning GPU memory math: weights, KV cache and runtime overhead, checked at runtime Load balancing Cache-aware routing and long-lived streaming connections Autoscaling on CPU Scaling on queue depth and KV cache use, with cold starts measured in minutes Rollouts Rollouts of exclusive GPUs, and a version matrix of model, hardware and engine build SLOs and latency percentiles Time to first token, time per output token and goodput Observability Engine metrics, real GPU activity and speculative decoding acceptance FinOps Cost per million tokens and cost per user session Identity and access Credentials that last as long as an agent session A hosted API is still the right call early in a product’s life . Self-hosting pays off when API spend becomes a real line item, when latency or reliability needs outgrow vendors, or when data can’t leave your network. In August 2026, Chris Zeoli counted about 58 open inference roles on public careers boards, with no settled job title. 2. Inference 101 This section covers the concepts in roughly the order you’ll meet them. Most are about what the engine keeps in GPU memory or what limits its speed. Viktor Farcic suggests asking when each concept starts to matter. Some matter as soon as you deploy, others only once many people use the model, and some you may never need. The table near the end of this section gives the signal for each. What an inference engine is An inference engine, like vLLM or SGLang , is the server that turns model weights into an API. For a platform engineer, the closest mental model is a stateful database server with a GPU attached. It loads a large dataset into memory at startup and keeps per-session state. It also schedules work internally and answers over HTTP. Ollama on a laptop is the same idea at small scale, and every hosted model API runs one behind the scenes. You know this from databases and services The engine’s counterpart Loading a dataset into the buffer pool at startup Loading model weights into GPU memory, which can take many minutes Per-session state in a session store The KV cache, one for each conversation in flight A query planner and connection pool The scheduler, which batches requests into each GPU pass A wire protocol clients already speak An OpenAI- or Anthropic-compatible HTTP API Warming caches after a restart Cold starts: pulling a large image and reloading the weights Unlike a database, the engine keeps its working set in GPU memory, not RAM. GPU memory is small and expensive, so most tuning is about deciding how much of it each request gets. The inference stack The engine is one layer of four. At the bottom, kernels do the arithmetic. You rarely choose them directly. Above them sits the engine . Above that, a control plane deploys and manages engines, because sooner or later you run more than one. On top, a gateway routes traffic across models and replicas. Inference stack: four layers and the clients they serve Requests pass through the gateway to an engine, which runs kernels on the GPU. The control plane deploys and manages the gateway and engines, but no request passes through it. Tokens A token is the unit a language model works in, roughly a word or word fragment. Context windows, prices and speeds are all counted in tokens. When someone quotes tokens per second, they mean this unit. Weights and the KV cache The weights are fixed, and the KV cache grows with load. Size for both, as you would a database’s footprint and working set. Weights are the fixed footprint. Most models are released at 2 bytes per parameter (BF16 or FP16), so an 8B model needs about 16 GB, and it has to live in GPU memory, not system RAM. On a 48 GB L40S, that leaves about two-thirds of the card for everything else. That number is the same for one user or a hundred. The KV cache is the working set. Each new token is computed against every token before it. Rather than recompute that history, the engine keeps the attention keys and values for every token in flight. Its size scales with context length multiplied by the number of concurrent conversations, much like a session store that grows with both session size and user count. You can estimate the KV cache per token from a model’s config: text KV bytes per token = 2 × layers × KV heads × head dimension × bytes per value For Llama 3 70B at FP16 (80 layers, 8 KV heads, head dimension 128), that is about 320 KB per token. One 128K-token conversation needs about 40 GB of cache. At that size, the KV cache usually limits how many users a GPU can serve. Watch for these sizing mistakes: Mixture-of-experts (MoE) models keep every weight resident. A model with 320B total parameters and 18B active still needs about 320 GB at FP8. Only compute scales with the active parameters. Read the checkpoint size, not the dtype table. While self-hosting its own models for Project Champagne , Upbound estimated one MoE model at about 315 GB from its dtypes (Part 2). The real checkpoint was 177.7 GB, because the experts were stored in a 4-bit format. Leave room for runtime overhead. CUDA graphs, activations a