Local Private LLM Serving with Ollama/vLLM behind Nginx
A private local LLM service has four layers: the model runtime, a loopback-only inference API, Nginx as the controlled entry point, and identity/network controls around Nginx. Ollama is ideal for simple single-host local models and exposes its API on localhost:11434 by default; vLLM is better suited to higher-throughput GPU serving and offers an OpenAI-compatible HTTP API. In both cases, do not expose the runtime port directly to a LAN or the internet.docs.ollama+1
Use Ollama for a private workstation, a small internal assistant, or a lightweight homelab endpoint. Use vLLM where you want several applications or users to share a GPU and prefer OpenAI-compatible clients. A vLLM –api-key is not enough by itself because some endpoints are outside its protected API path prefixes; place it behind an authenticated reverse proxy and restrict network exposure.
Run the inference runtime so that only Nginx can reach it locally. Ollama binds to 127.0.0.1:11434 by default, which is the preferred local-service posture.
For Ollama, verify the listening address:
ss -lntp | grep 11434
curl http://127.0.0.1:11434/api/tags
For vLLM, bind explicitly to loopback and set an application API key as a second layer:
vllm serve YOUR_MODEL_ID \
--host 127.0.0.1 \
--port 8000 \
--api-key "$VLLM_API_KEY"Store the API key in a root-readable environment file or secret manager, not in shell history, unit files readable by other users, Git repositories, or container-image layers. If using Ollama in an offline/private environment, disable cloud access with OLLAMA_NO_CLOUD=1 or the related server configuration, then restart the service.
Nginx terminates TLS, authenticates clients, limits abusive traffic, provides audit logs, and forwards only approved paths to the local runtime. Disable proxy buffering so token streaming reaches the client continuously rather than being held until the response completes.
# /etc/nginx/conf.d/private-llm.conf
limit_req_zone $binary_remote_addr zone=llm_per_ip:10m rate=5r/m;
limit_conn_zone $binary_remote_addr zone=llm_connections:10m;
upstream ollama_backend {
server 127.0.0.1:11434;
keepalive 16;
}
server {
listen 443 ssl http2;
server_name llm.internal.example;
ssl_certificate /etc/nginx/tls/llm.internal.example.crt;
ssl_certificate_key /etc/nginx/tls/llm.internal.example.key;
ssl_protocols TLSv1.2 TLSv1.3;
client_max_body_size 2m;
client_body_timeout 15s;
client_header_timeout 15s;
send_timeout 300s;
access_log /var/log/nginx/llm-access.log;
error_log /var/log/nginx/llm-error.log warn;
location / {
auth_basic “Private LLM”;
auth_basic_user_file /etc/nginx/auth/llm.htpasswd;
limit_req zone=llm_per_ip burst=10 nodelay;
limit_conn llm_connections 3;
proxy_pass http://ollama_backend;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto https;
proxy_buffering off;
proxy_request_buffering off;
proxy_connect_timeout 10s;
proxy_read_timeout 300s;
}
}
The same configuration works for vLLM if you change the upstream to 127.0.0.1:8000. vLLM itself supports OpenAI-style endpoints, while Nginx provides the surrounding access boundary. Nginx rate limiting tracks a request key such as client IP in shared memory and applies a configured request rate, helping protect an expensive GPU service from accidental loops or request floods.
HTTP Basic Auth is acceptable only for a tightly controlled internal lab over TLS; it is not ideal for a team or production environment. Prefer one of these patterns:
-
Put an identity-aware proxy in front of Nginx using OIDC/SAML and MFA.
-
Require VPN access first, then use OIDC at the proxy.
-
Restrict with a firewall so only approved management subnets, application servers, or VPN address pools can reach TCP 443.
-
Issue per-application API credentials through an authentication gateway, rather than sharing one static password.
-
Use mutual TLS for machine-to-machine callers that can manage client certificates.
Do not log prompt bodies, bearer tokens, authorization headers, or complete generated responses by default. Prompt data can contain internal code, tickets, credentials pasted by users, or customer information. Retain only the metadata necessary for operations: timestamp, caller identity or pseudonymous identifier, model, status code, latency, token usage if available, and request size.
Run Ollama/vLLM with a dedicated unprivileged account, keep model directories owned by that account, and prevent arbitrary users from writing model files. Use a host firewall to permit port 443 only from intended networks; block direct access to ports 11434 and 8000 from every external interface.
For a containerized vLLM deployment, do not run privileged, mount the GPU device only as needed, mount model-cache storage with controlled permissions, pin image versions/digests, scan images, and use a read-only root filesystem where the runtime supports it. Make outbound network access explicit: allow access only to model registries during controlled model pulls, then restrict egress during normal inference.
Test and Operate
Validate each boundary separately:
# Runtime reachable locally
curl http://127.0.0.1:11434/api/tags
# Runtime must not listen publicly
ss -lntp | egrep ‘11434|8000’
# Validate Nginx syntax and reload safely
sudo nginx -t
sudo systemctl reload nginx
# Test the protected external endpoint
curl -u USERNAME:PASSWORD \
https://llm.internal.example/api/tags
Monitor GPU memory/utilization, queue time, request latency, Nginx 401/403/429/5xx responses, disk capacity for model caches, and unexpected model downloads. Set strict resource limits and a request-size cap so one oversized prompt cannot exhaust RAM, VRAM, disk, or proxy workers.
For a secure first deployment, decide whether the clients are only local processes, devices on a VPN, or internal applications on separate servers. Then state which layer will authenticate them—Nginx Basic Auth, an OIDC proxy, mTLS, or a combination—and why.
[mai mult...]