Securing Self-Hosted AI APIs
A self-hosted AI API is both a conventional API and an AI-specific attack surface. Secure it in layers: isolate the model runtime, expose only a hardened gateway, authenticate every caller, authorize each action and data retrieval, constrain compute use, and log enough metadata to investigate abuse without storing sensitive prompts by default. Broken object-level authorization, broken authentication, resource exhaustion, misconfiguration, and unsafe downstream API use are all central API risks.
The key principle: do not treat an LLM endpoint as “just internal.” A compromised internal workstation, leaked API key, malicious prompt, or vulnerable integrated service can turn it into a path toward confidential data or privileged actions.
Reference Architecture
Caller
|
v
VPN / private network
|
v
API gateway or Nginx
- TLS / mTLS
- OIDC authentication
- Per-client rate and concurrency limits
- Request-size limits
- Audit metadata
|
v
AI API service
- Request schema validation
- Tenant and role authorization
- Prompt/input controls
- RAG permission filtering
- Tool policy enforcement
|
+--> Local LLM runtime (loopback/private subnet only)
+--> Vector database (ACL-filtered)
+--> Approved tools/services (least privilege)Bind Ollama, vLLM, or another inference runtime to 127.0.0.1 or a private container network. Block direct inbound access to runtime ports such as 11434 or 8000 using host firewalls, security groups, or Kubernetes NetworkPolicy; only the API layer should reach them. vLLM documentation warns that its API key option alone does not secure every endpoint, so defense must include a reverse proxy and network restrictions.
Use OIDC/OAuth2 access tokens for human and application clients, preferably validated at the gateway and again by the API service for high-value operations. Require MFA through the identity provider for administrator access, use short-lived tokens, rotate service credentials, and keep secrets in a vault or workload-identity mechanism rather than code, shell history, or container images. OWASP advises treating login, token, and account-recovery flows as high-risk endpoints that need brute-force protection, and it notes that API keys identify API clients rather than authenticating users.
Authentication answers who is calling. Authorization answers what that caller may do. Enforce authorization for every resource ID, model name, tenant, conversation, uploaded file, RAG collection, tool, and administrative operation—never rely on a route prefix or a client-supplied tenant field. For example, a caller authorized for team-a documents must not obtain team-b vector chunks merely by changing a collection ID; that is the AI equivalent of BOLA.
LLM inference is vulnerable to resource exhaustion because prompts, output length, concurrent requests, file uploads, embeddings, and tool loops consume CPU, GPU VRAM, memory, storage, and money. OWASP classifies this broader problem as unrestricted resource consumption for APIs and unbounded consumption for LLM applications.owasp+1
Apply limits at multiple points:
-
Request body and file-upload size.
-
Maximum prompt tokens and maximum generated tokens.
-
Per-user, per-API-key, and per-IP request rates.
-
Concurrent generation requests and queue depth.
-
Request duration and upstream read timeouts.
-
Embedding batch size and document-ingestion quotas.
-
Tool-call count, recursion depth, execution time, and spend budgets.
-
Tenant-specific monthly quotas and alerts.
Return a controlled 429 or 503 when overloaded; do not let queue growth exhaust host RAM or GPU memory. For longer jobs such as document ingestion, use an asynchronous queue with per-tenant limits rather than holding an API connection open.
Defend Prompts and RAG
Prompt injection happens when user-supplied or retrieved text tries to override the system’s intended behavior; OWASP identifies it as a leading LLM application risk. No prompt wording fully “solves” this, so place security boundaries outside the model’s judgment.
Treat prompts, files, web content, and retrieved RAG chunks as untrusted data. Never let the model decide whether a user is allowed to access a document, call a privileged API, read a secret, or execute a shell command. Instead:
-
Apply ACL filters before vector retrieval.
-
Pass only authorized chunks into the model context.
-
Clearly label retrieved content as untrusted reference text.
-
Require structured tool-call schemas and validate them server-side.
-
Map each tool to a narrowly scoped service account.
-
Require human approval for destructive or externally consequential actions.
-
Allowlist outbound domains and block link-local, loopback, private-address, and cloud-metadata targets for any URL-fetching tool.
Least-privilege tools, approval gates for consequential actions, tool-use auditing, and bounded agent execution are recommended safeguards against excessive agency.
Protect Data and Supply Chain
Classify data before it enters prompts, embeddings, logs, or model files. Redact secrets and sensitive identifiers where possible; encrypt data in transit and at rest; segregate tenants; and define retention/deletion rules for conversations, uploads, embeddings, and backups. Track document and dataset modifications with provenance so a suspicious RAG result can be traced to a source and rolled back.
Pin container images and model versions by digest, scan images and dependencies, restrict who may pull or register models, verify model provenance, and separate the model-download environment from normal inference egress. Do not automatically ingest arbitrary uploads, repositories, webpages, or tickets into a production knowledge base: scan, parse, classify, review, hash, and approve them first. Poisoned RAG documents can manipulate answers or introduce malicious operational guidance.
Monitoring and Testing
Log structured metadata: request ID, authenticated principal, tenant, selected model, endpoint, status, latency, token counts, rate-limit decision, retrieved document IDs, and tool invocations. Avoid recording raw prompts, generated text, authorization headers, secrets, or document contents unless a narrowly approved diagnostic policy requires it.
Test the service like both an API and an AI system:
-
Replay a valid request while swapping tenant, conversation, document, model, and tool IDs to test authorization.
-
Fuzz JSON fields, streaming endpoints, file upload parsers, and model parameters.
-
Test rate limits with parallel requests, long inputs, large max_tokens intentionally slow streaming clients.
-
Attempt prompt injection through direct prompts and indexed documents.
-
Verify RAG ACLs using users from different teams and classifications.
-
Test tools with malicious arguments, SSRF-style URLs, and requests for destructive actions.
-
Confirm that an API key cannot access operational or administrative endpoints outside its intended scope.
For your first hardening task, sketch your own inference service boundary and identify the one component that currently has the broadest privilege—for example, the API gateway, RAG service, tool runner, or model host—then describe the first privilege reduction you would apply.
[mai mult...]