Self-hosted LLM: a practical guide.

Should you self-host an LLM?

Start with the job, not the GPU. A self-hosted LLM can be useful when you need control of inference, have a task an available model can handle, and can maintain the deployment. Cost savings need a separate calculation. Use these four questions to decide what to validate next. They deliberately separate data control, workload, model quality and operational ownership. A low-volume deployment can still be justified for privacy; a busy one can still fail if the model is unsuitable.

What is a self-hosted LLM?

A self-hosted LLM runs model inference on infrastructure you operate: a workstation, a server or rented compute under your administration. You load the model weights and choose how applications reach the runtime. Hosting your own LLM does not require training a model from scratch. Separate three layers: the model, the serving runtime and the application. A chat UI is an application; it may use local inference or an external API. Local LLM hosting is only local end to end when all relevant processing follows the intended data boundary.

How to self-host an LLM: a small Docker pilot

Begin with one runtime, one small model and one chat interface. Keep the model service on the container network, persist the data and avoid public exposure while you establish that inference works. The example below connects Open WebUI to Ollama by service name. The tabs explain platform differences; they do not turn a CPU compose file into an accelerated deployment. Verify the basic path first, then introduce GPU settings one change at a time. Never use a pilot as proof of production capacity.

LinuxmacOSWindows / WSL2

Reference path: Docker Engine with Compose. This file runs on CPU. NVIDIA acceleration requires compatible drivers and the NVIDIA Container Toolkit plus a GPU reservation; do not assume that installing a container enables the GPU.

Two-container CPU reference: official documentation reviewed, not yet verified end to end. A separate macOS smoke test used native Ollama 0.34.3 and Docker Open WebUI 0.12.3 on 23 October 2026. It does not establish Linux GPU performance or dedicated VRAM requirements.

1. Create an isolated project directory

   Download the compose file below into this directory. The web interface uses port 3080 so it does not conflict with this website.

   mkdir llm-pilot
   cd llm-pilot
   docker version
   docker compose version

   Both Docker client and server respond, and Compose prints a version.

2. Set a persistent application secret

   Create the local environment file before starting. Keep it private and retain the same key across container updates.

   umask 077
   printf "WEBUI_SECRET_KEY=" > .env
   openssl rand -hex 32 >> .env
   docker compose config --quiet

   Compose validates without a missing-secret error. The secret stays in your local .env file.

3. Start the runtime and load a small test model

   The model download needs internet access and disk space. Start with a small evaluation model; this is a connectivity check, not a recommendation that it meets your quality needs.

   docker compose up -d
   docker compose exec ollama ollama pull llama3.2:1b
   docker compose exec ollama ollama run llama3.2:1b "Reply with: ready"

   The download completes and inference returns text. Exact wording and speed vary.

4. Check the UI and record the evidence

   Open http://localhost:3080, create your local account, and select the downloaded model. Test with synthetic prompts. Record digests, OS, context and concurrency; container RAM on a Mac is not GPU VRAM. For a model that does not support tool calling, turn off Builtin Tools in Workspace → Models → Edit → Capabilities before testing chat.

   docker compose ps
   docker compose images
   docker compose exec ollama ollama --version
   docker compose exec ollama ollama ps
   docker stats --no-stream

   The UI reaches the model, chat returns text, and you can save the runtime and memory observations.

compose.yaml

CopyDownload

# Reference CPU-only stack. No model port is exposed to the host.
# Replace image tags with validated digests before production use.
services:
  ollama:
    image: ${OLLAMA_IMAGE:-ollama/ollama:latest}
    volumes:
      - model-data:/root/.ollama
    restart: unless-stopped
  webui:
    image: ${WEBUI_IMAGE:-ghcr.io/open-webui/open-webui:main-slim}
    ports:
      - "127.0.0.1:3080:8080"
    environment:
      OLLAMA_BASE_URL: http://ollama:11434
      WEBUI_SECRET_KEY: ${WEBUI_SECRET_KEY:?Set WEBUI_SECRET_KEY in .env}
    volumes:
      - chat-data:/app/backend/data
    depends_on:
      - ollama
    restart: unless-stopped
volumes:
  model-data:
  chat-data:

Official references: Ollama Docker · Open WebUI Quick Start

Is hosting your own LLM actually cheaper?

Compare the same task at the same quality target. Input-heavy retrieval and output-heavy chat have different API bills, so a single blended token price can mislead. Add the infrastructure, maintenance and migration costs you will really pay. An idle reserved GPU still costs money. Useful-time utilization helps explain the cost per completed request; it does not reduce a fixed rental bill. For a stoppable hourly instance, enter actual billable hours, including idle time. The calculator holds a chosen deployment cost fixed and stops treating it as feasible beyond your capacity assumption.

Self-hosted LLM hardware: budget beyond the weights

Weight storage is a starting point, not a complete memory requirement. Quantization scales, runtime workspaces, attention implementation, context and concurrent sequences all affect the peak. For mixture-of-experts models, active parameters describe compute rather than the complete set of weights you may need to store. The table is arithmetic, not a hardware recommendation. Add a workload-specific KV-cache estimate and runtime headroom, then measure. Compare the same prompt length, generation length and concurrency when testing two configurations. Unified memory, host RAM and dedicated GPU VRAM are different budgets.

Weight GiB = parameters × bits per weight ÷ 8 ÷ 2³⁰. Quantization metadata, runtime buffers and KV cache are additional; a model file fitting in memory is not a performance guarantee.

Draw the data path before adding services

For a single user, a small runtime and UI may be enough. For a team, add explicit identity, access boundaries and observability before increasing the component count. A vector database, an agent framework and a cluster orchestrator each need a reason to exist. Sketch where the prompt, documents, embeddings, logs and model outputs go. Mark external connections and the identity on each hop. An architecture drawing becomes useful when it tells you where to apply a limit, an access check or a deletion policy.

Twelve checks before a deployment goes public

Self-hosting shifts responsibility; it does not create automatic privacy or compliance. Start with a loopback-only pilot. Review each check below before allowing remote use. The commands are inspection aids for this reference stack, not universal proof that a control is effective. Application permissions and recovery behavior require negative tests and a real restore. A green container status cannot establish either.

docker compose ps

curl -I http://127.0.0.1:3080/

test -s .env && echo "Environment file exists"

docker compose images

lsof -nP -iTCP -sTCP:LISTEN

docker volume ls --filter name=llm

docker compose logs --tail=50

# Manual negative test: two users, one restricted document

docker compose exec ollama ollama ps

docker stats --no-stream

# Record backup identifier, isolated restore result and recovery time

# Manual external check: only intended authenticated entry points respond

Measure a request, then measure a queue

Separate time to first token, generation rate and end-to-end latency. A fast single request does not show how the service behaves when multiple users arrive together. Prefill, decoding, batching and memory pressure can affect different parts of the request. Use a representative prompt set. Keep input and output lengths consistent, record warm and cold behavior, then increase concurrency while watching latency percentiles and error rates. Choose a serving runtime because the measured workload benefits, not because an unrelated benchmark reports a higher tokens-per-second number.

Owned hardware, hourly rental or monthly capacity?

Owned hardware brings purchase cost, power, cooling, replacement and administration. Amortize the purchase over an explicit useful life and include the rest of the machine. Do not treat a GPU you already own as free if the question is long-term total cost. Hourly rentals can suit intermittent work only when you can actually stop billing between jobs. Reserved capacity gives a predictable bill but can be expensive at low usage. Managed private deployments trade some operational work for service fees; include the support scope and minimum commitment in the comparison.

The best self-hosted LLM is task-dependent

Build a small evaluation set from the work you need done: extraction, grounded question answering, code edits or another clearly defined task. Define acceptable answers and failure conditions before comparing models. Test license suitability, tool-call behavior and response format as well as language quality. Change one variable at a time. Compare model revision, quantization and prompt format under the same memory and latency budget. Keep the useful baseline even when a newer model appears. A leaderboard can suggest candidates; it cannot replace this evaluation.

What to record after the first successful chat

Record the image digest, model identifier and revision, quantization, operating system, driver, runtime settings and test prompt. Keep context and concurrency beside every memory observation. A screenshot without these details is difficult to reproduce. Test persistence after a normal restart, export a backup and verify a restore in an isolated environment. Capture failed model loads, timeouts and queue saturation. Review upgrades against the same test set and keep a rollback path before changing the running version.

When you should not self-host an LLM

Pause when no available model meets your task quality, no one can own the service, or the required response time cannot be met within budget. A cheap failed request is not a saving. Small or highly variable workloads may be better served by an API, but there is no universal daily-token cutoff. A privacy requirement can outweigh pure cost. It can also rule out an external fallback. Write down the non-negotiable boundary before making a hybrid design; do not solve a capacity problem by silently sending sensitive requests somewhere else.

Self-hosted LLM vs API vs hybrid

Evaluate quality, data handling, latency, capacity, availability and total cost as separate axes. Self-hosting changes who operates the service; it does not remove resource limits. An API removes much of the infrastructure work, but still needs application monitoring and usage controls. Hybrid routing is useful only with clear rules. Decide which requests may leave your environment, how failures are handled and how costs are attributed. Keep the same evaluation tasks across all three options so the comparison answers your question rather than a vendor headline.

Run one pilot. Keep the evidence.

Choose one real task and an explicit data boundary. Complete the decision card, enter your costs, and run a small local pilot. Save the evidence before buying capacity or moving users. If your answer is still unclear, identify the missing fact: task quality, peak concurrency, response time or the actual hosting bill. Measure that next instead of adding more services.