Configurare program

Automating PostgreSQL/MongoDB Backups in Kubernetes

• Data 10/09/2026
• Autor Costin Dumitru
• 4
• 97

A Kubernetes backup system must protect three things: database data, database-level metadata, and the restore process itself. A scheduled backup that has never been restored is only an assumption, so design the workflow around a measurable recovery point objective (RPO) and recovery time objective (RTO).

A robust pattern is:

1. A Kubernetes CronJob starts a short-lived, non-root backup Pod.

2. The Pod reads credentials from a Secret mounted as environment variables or files.

3. It connects to PostgreSQL or MongoDB through an internal Service using least-privilege backup credentials.

4. It creates a compressed, encrypted backup artifact.

5. It uploads the artifact to object storage or a remote backup repository.

6. It writes logs, emits metrics, and removes temporary local files.

7. A separate scheduled restore test validates the latest backup in an isolated namespace.

Do not depend only on a PersistentVolume mounted beside the database. If the cluster, storage class, credentials, or namespace are compromised together, both the production data and “backup” may be lost.

PostgreSQL Backup Strategy

For logical backups, pg_dump backs up one PostgreSQL database, while pg_dumpall covers an entire cluster and preserves global objects such as roles and tablespaces. For individual application databases, use the custom archive format (-Fc): it is compressed, works with pg_restore, and supports selective restore operations.

A basic PostgreSQL backup command is:

The --no-owner option helps when restoring into a different environment where source role names do not exist. If your recovery requires roles and tablespaces too, run a separate global metadata export:

For large, high-change PostgreSQL systems, logical dumps may not meet your RPO/RTO. Use physical backups plus WAL archiving through a PostgreSQL-aware solution such as pgBackRest, WAL-G, or an operator-supported backup mechanism. The important distinction is that pg_dump captures a logical snapshot, while point-in-time recovery requires archived WAL files in addition to a suitable base backup.

For self-managed MongoDB, mongodump creates a binary export and mongorestore restores that export to a running MongoDB deployment. Use a single compressed archive for easier handling and avoid a large tree of loose BSON files:

mongodump can back up a full deployment, a database, a collection, or query-filtered content, depending on options and operational needs. For a replica set, ensure you understand consistency requirements and test restoration against the exact MongoDB topology you operate. For managed MongoDB Atlas, platform-native continuous backups and point-in-time recovery are often preferable to running mongodump as the sole protection mechanism.

This example uses PostgreSQL. The same container pattern works for MongoDB by replacing the image and backup command. It writes to /work; in production, add an upload command to S3-compatible storage, Azure Blob, GCS, or a hardened backup server.

concurrencyPolicy: Forbid prevents a slow backup from overlapping with the next scheduled run. Set startingDeadlineSeconds so an overdue backup does not start at an operationally unsafe time. CronJobs are appropriate for recurring work, but Kubernetes documents that scheduling can be approximate, so make the job idempotent and record success externally rather than trusting schedule timing alone.

Use a dedicated database account whose permissions are sufficient to read backup data but not to modify production records. Store connection strings, passwords, TLS certificates, and object-storage credentials in a secret-management system; restrict access with namespace RBAC, separate service accounts, and cloud workload identity where available.

Apply these controls:

• Use TLS for database connections and object-storage uploads.

• Encrypt backup files before leaving the Pod, or use a destination that provides strong server-side encryption with independently controlled keys.

• Keep backup storage in a separate account, project, or tenant from the production Kubernetes cluster.

• Enable versioning and retention locks/object immutability to resist accidental deletion and ransomware.

• Include a checksum or manifest containing backup timestamp, database/version, artifact size, checksum, and tool version.

• Never place database URIs, passwords, or cloud keys directly in CronJob YAML, Git repositories, or CI logs.

• Restrict egress from the backup Pod to only the database endpoint, DNS/time dependencies, and the backup destination.

Retention, Monitoring, and Restores

Define retention before automation. A typical scheme might keep daily backups for 14–30 days, weekly backups for several months, and monthly backups for a year, but the right policy follows business, legal, and storage requirements. Monitor at least: last-success timestamp, job duration, output size, upload success, checksum verification, and age of the newest restorable artifact.

Restore testing should be automated in a disposable namespace: provision an empty Postgres or MongoDB instance, download a recent backup, restore it, run integrity checks and representative application queries, then destroy the environment. PostgreSQL custom-format dumps restore with pg_restore; MongoDB archive dumps restore with mongorestore. A successful restore test is the most meaningful backup metric.mongodb+1

As a practical first step, define whether your most important database needs daily logical recovery or point-in-time recovery, then explain what maximum acceptable data loss in minutes or hours would be for that workload.

GitOps Deployment with Argo CD and Kustomize

• Data 10/09/2026
• Autor Costin Dumitru
• 3
• 129

GitOps makes Git the desired-state record for Kubernetes: instead of an engineer applying manifests manually with kubectl, a controller continuously compares what is declared in Git with what exists in the cluster and reconciles differences. Argo CD performs that reconciliation; Kustomize renders reusable Kubernetes configuration for different environments without copying the entire manifest set.argo-cd.readthedocs+1.

For a practical platform setup, keep application source code separate from deployment configuration. Your CI pipeline builds, scans, and publishes an immutable container image; it then updates an image tag or digest in the GitOps repository through a reviewed pull request. Argo CD sees the approved Git change and deploys it.

Kustomize uses a base for shared manifests and overlays for environment-specific changes. An overlay points to the base and applies only the patches or additions needed for an environment.

Keep environment differences narrowly scoped: image reference, replica count, resource requests/limits, ingress hostname, environment configuration, and possibly namespace. Avoid putting credentials in plain YAML; reference secrets managed through External Secrets, Sealed Secrets, SOPS, or another controlled mechanism.

The base defines stable application behavior shared by every environment:

Always render locally before committing:

The first command shows the generated manifests; the second helps you reason about the live-cluster impact before Argo CD applies it.

Define Environment Overlays

An overlay imports the base, changes its namespace, and patches only what differs. Here is a production overlay:

This separation is the main Kustomize benefit: the base contains universal configuration, while overlays supply only environment deltas. Prefer explicit patches over a growing chain of generators and transformations that make rendered output difficult to review.

An Argo CD Application tells Argo CD which repository path to render and the destination cluster and namespace to reconcile. The Application specification supports Git source details, Kustomize configuration, destination, and sync policy.

With automated, Argo CD applies desired-state changes found in Git. selfHeal: true tells it to correct out-of-band cluster changes, while prune: true allows it to remove resources that were deliberately removed from Git. These controls are powerful: in production, protect the Git branch, require review, restrict who can modify Application objects, and use Argo CD Projects to limit the repositories, clusters, namespaces, and resource kinds each team may deploy.

A secure GitOps delivery flow is:

1. A developer commits application code.

2. CI runs unit, integration, image, dependency, and policy checks.

3. CI builds an image, scans it, signs it if your supply chain supports signing, and pushes it using an immutable tag or digest.

4. CI opens a pull request that changes only the relevant overlay image tag or digest.

5. A reviewer verifies the rendered manifest and approves the GitOps change.

6. The merge becomes the desired state; Argo CD synchronizes it to the cluster.

7. Argo CD reports health and sync status; alerts fire on degraded, missing, or out-of-sync workloads.

Avoid allowing CI to run broad kubectl apply commands directly against production. That creates a second deployment authority and makes drift harder to explain. With self-healing enabled, a direct manual scale operation or kubectl edit change is eventually reverted because Git remains authoritative.

GitOps rollback means reverting the commit that introduced the bad desired state, then allowing Argo CD to reconcile the previous revision. Keep image references immutable: a tag like latest undermines auditability because the same Git commit could produce different runtime artifacts.

When an application is OutOfSync or Degraded, troubleshoot in order:

• Render locally: kubectl kustomize apps/inventory-api/overlays/prod.

• Inspect the Argo CD diff to find the desired-versus-live mismatch.

• Check Application events and sync history.

• Inspect the affected Kubernetes object with kubectl describe.

• Check Pod status, events, logs, readiness probes, resource quotas, and NetworkPolicies.

• Confirm the Argo CD service account has only the RBAC permissions it needs.

• Check whether a controller is legitimately mutating fields; configure ignore differences only for understood and documented cases.

For a production-quality next step, choose one application you would deploy and identify the three fields that should differ between dev and prod while the base stays identical.

Local Private LLM Serving with Ollama/vLLM behind Nginx

• Data 10/09/2026
• Autor Costin Dumitru
• 1
• 62

A private local LLM service has four layers: the model runtime, a loopback-only inference API, Nginx as the controlled entry point, and identity/network controls around Nginx. Ollama is ideal for simple single-host local models and exposes its API on localhost:11434 by default; vLLM is better suited to higher-throughput GPU serving and offers an OpenAI-compatible HTTP API. In both cases, do not expose the runtime port directly to a LAN or the internet.docs.ollama+1

Use Ollama for a private workstation, a small internal assistant, or a lightweight homelab endpoint. Use vLLM where you want several applications or users to share a GPU and prefer OpenAI-compatible clients. A vLLM –api-key is not enough by itself because some endpoints are outside its protected API path prefixes; place it behind an authenticated reverse proxy and restrict network exposure.

Run the inference runtime so that only Nginx can reach it locally. Ollama binds to 127.0.0.1:11434 by default, which is the preferred local-service posture.

For Ollama, verify the listening address:

For vLLM, bind explicitly to loopback and set an application API key as a second layer:

Store the API key in a root-readable environment file or secret manager, not in shell history, unit files readable by other users, Git repositories, or container-image layers. If using Ollama in an offline/private environment, disable cloud access with OLLAMA_NO_CLOUD=1 or the related server configuration, then restart the service.

Nginx terminates TLS, authenticates clients, limits abusive traffic, provides audit logs, and forwards only approved paths to the local runtime. Disable proxy buffering so token streaming reaches the client continuously rather than being held until the response completes.

The same configuration works for vLLM if you change the upstream to 127.0.0.1:8000. vLLM itself supports OpenAI-style endpoints, while Nginx provides the surrounding access boundary. Nginx rate limiting tracks a request key such as client IP in shared memory and applies a configured request rate, helping protect an expensive GPU service from accidental loops or request floods.

HTTP Basic Auth is acceptable only for a tightly controlled internal lab over TLS; it is not ideal for a team or production environment. Prefer one of these patterns:

• Put an identity-aware proxy in front of Nginx using OIDC/SAML and MFA.

• Require VPN access first, then use OIDC at the proxy.

• Restrict with a firewall so only approved management subnets, application servers, or VPN address pools can reach TCP 443.

• Issue per-application API credentials through an authentication gateway, rather than sharing one static password.

• Use mutual TLS for machine-to-machine callers that can manage client certificates.

Do not log prompt bodies, bearer tokens, authorization headers, or complete generated responses by default. Prompt data can contain internal code, tickets, credentials pasted by users, or customer information. Retain only the metadata necessary for operations: timestamp, caller identity or pseudonymous identifier, model, status code, latency, token usage if available, and request size.

Run Ollama/vLLM with a dedicated unprivileged account, keep model directories owned by that account, and prevent arbitrary users from writing model files. Use a host firewall to permit port 443 only from intended networks; block direct access to ports 11434 and 8000 from every external interface.

For a containerized vLLM deployment, do not run privileged, mount the GPU device only as needed, mount model-cache storage with controlled permissions, pin image versions/digests, scan images, and use a read-only root filesystem where the runtime supports it. Make outbound network access explicit: allow access only to model registries during controlled model pulls, then restrict egress during normal inference.

Test and Operate

Validate each boundary separately:

Monitor GPU memory/utilization, queue time, request latency, Nginx 401/403/429/5xx responses, disk capacity for model caches, and unexpected model downloads. Set strict resource limits and a request-size cap so one oversized prompt cannot exhaust RAM, VRAM, disk, or proxy workers.

For a secure first deployment, decide whether the clients are only local processes, devices on a VPN, or internal applications on separate servers. Then state which layer will authenticate them—Nginx Basic Auth, an OIDC proxy, mTLS, or a combination—and why.

Building a Local Enterprise Knowledge Base with RAG

• Data 10/09/2026
• Autor Costin Dumitru
• 2
• 108

A local enterprise RAG system lets an LLM answer questions using approved internal material at query time rather than relying only on its pretrained knowledge. It is not “training the model on company documents”: the system retrieves relevant passages from an index, places them into the model context, and asks the model to answer only from that evidence. This is usually preferable to fine-tuning for information that changes often, such as runbooks, policies, asset inventories, architecture diagrams, incident procedures, and internal APIs.

The main design principle is simple: the model generates; your governed documents supply the facts. If retrieval is weak, no model prompt can reliably compensate.

Reference Architecture

The RAG lifecycle has two distinct paths:

• Indexing path: Load documents, parse their content, split them into chunks, generate embeddings, and store chunks plus metadata in a vector store. This normally happens offline or when a source changes. langchain

• Question path: Authenticate the user, apply authorization filters, embed the user’s query, retrieve allowed chunks, optionally rerank them, give those chunks to the LLM, and return an answer with source references. langchain

A local stack could be: Ollama or vLLM for inference, bge/e5-class local embedding models, Qdrant/pgvector/Chroma for vector search, PostgreSQL for metadata/audit records, and a Python FastAPI service that performs retrieval and policy enforcement.

Ingestion and Document Quality

The most difficult part is rarely the LLM; it is turning messy enterprise content into trustworthy, searchable evidence. Build a controlled ingestion pipeline instead of allowing arbitrary users to upload anything directly into the vector database.

Each chunk should carry useful metadata:

Parse documents while preserving heading hierarchy, tables, code blocks, page numbers, repository revision, and source ownership. Deduplicate documents using a content hash, reject unsupported or encrypted file types unless there is an approved parser, and quarantine files that fail malware scanning or content validation. A typical RAG index converts document chunks to embeddings and stores them for later similarity search.

A chunk is the atomic unit the system retrieves. If it is too large, it includes irrelevant context and wastes the model’s context window; if it is too small, it may omit the condition, command, exception, or surrounding explanation needed for a correct answer. Large documents are split because whole documents are difficult to retrieve accurately and may exceed the finite context available to the model.

Start with structure-aware recursive chunking:

• Preserve headings and sections where possible.

• Use approximately 400–800 tokens per chunk.

• Use 50–120 tokens of overlap to avoid losing context at boundaries.

• Keep commands and their explanation together.

• Do not split tables row-by-row unless your parser preserves headers.

• Store parent-document and section metadata with every chunk.

At query time, combine semantic vector search with keyword search. Semantic search finds conceptually similar text; keyword search is essential for exact values such as hostnames, CVE IDs, command flags, error codes, IP addresses, and product versions. Retrieve an initial set, apply ACL filters first, rerank the remaining evidence, then send only the best few chunks to the LLM.

The largest enterprise mistake is treating RAG as ordinary search without enforcing permissions. A user must never retrieve a chunk they would not be allowed to open in the original system. Enforce access control as a hard retrieval filter, not as an instruction such as “do not disclose confidential information.”

A safe query pattern is:

For example, a database administrator can search classification <= confidential and allowed_groups contains database-admins, while a help-desk user may only search public and internal end-user documentation. Use isolated vector collections or namespaces for separate tenants and highly sensitive domains in addition to metadata filtering.

Retrieved text is data, not trusted instructions. A malicious or compromised document may contain content such as “ignore previous instructions,” secrets, misleading recovery commands, or instructions to exfiltrate data. OWASP identifies manipulated embedding or training data as a poisoning risk, and this includes knowledge-base sources used by RAG systems.owasp+1

Use a system instruction similar to:

Also apply source allowlists, document ownership, approval workflows, signatures or hashes for high-value runbooks, malware scanning, change review, re-indexing on approved updates, and anomaly alerts for sudden large-scale changes. Never let the RAG assistant execute retrieved shell commands, SQL, Kubernetes actions, or ticket updates without separate authorization, input validation, and human approval.

Build a test set before declaring the knowledge base successful. Include representative questions, expected sources, questions with no answer, outdated-document traps, confusing near-duplicates, access-control tests, and prompt-injection attempts embedded in sample documents. Measure:

• Retrieval recall: Did the correct source appear in the top results?

• Citation correctness: Does each cited source actually support the answer?

• Answer faithfulness: Does the response remain within the supplied evidence?

• Access-control accuracy: Can users retrieve only content allowed by their role?

• Freshness: Is the answer based on the latest approved version?

• Operational safety: Does it avoid inventing dangerous production steps?

Securing Self-Hosted AI APIs

• Data 10/09/2026
• Autor Costin Dumitru
• 3
• 97

A self-hosted AI API is both a conventional API and an AI-specific attack surface. Secure it in layers: isolate the model runtime, expose only a hardened gateway, authenticate every caller, authorize each action and data retrieval, constrain compute use, and log enough metadata to investigate abuse without storing sensitive prompts by default. Broken object-level authorization, broken authentication, resource exhaustion, misconfiguration, and unsafe downstream API use are all central API risks.

The key principle: do not treat an LLM endpoint as “just internal.” A compromised internal workstation, leaked API key, malicious prompt, or vulnerable integrated service can turn it into a path toward confidential data or privileged actions.

Reference Architecture

Bind Ollama, vLLM, or another inference runtime to 127.0.0.1 or a private container network. Block direct inbound access to runtime ports such as 11434 or 8000 using host firewalls, security groups, or Kubernetes NetworkPolicy; only the API layer should reach them. vLLM documentation warns that its API key option alone does not secure every endpoint, so defense must include a reverse proxy and network restrictions.

Use OIDC/OAuth2 access tokens for human and application clients, preferably validated at the gateway and again by the API service for high-value operations. Require MFA through the identity provider for administrator access, use short-lived tokens, rotate service credentials, and keep secrets in a vault or workload-identity mechanism rather than code, shell history, or container images. OWASP advises treating login, token, and account-recovery flows as high-risk endpoints that need brute-force protection, and it notes that API keys identify API clients rather than authenticating users.

Authentication answers who is calling. Authorization answers what that caller may do. Enforce authorization for every resource ID, model name, tenant, conversation, uploaded file, RAG collection, tool, and administrative operation—never rely on a route prefix or a client-supplied tenant field. For example, a caller authorized for team-a documents must not obtain team-b vector chunks merely by changing a collection ID; that is the AI equivalent of BOLA.

LLM inference is vulnerable to resource exhaustion because prompts, output length, concurrent requests, file uploads, embeddings, and tool loops consume CPU, GPU VRAM, memory, storage, and money. OWASP classifies this broader problem as unrestricted resource consumption for APIs and unbounded consumption for LLM applications.owasp+1

Apply limits at multiple points:

• Request body and file-upload size.

• Maximum prompt tokens and maximum generated tokens.

• Per-user, per-API-key, and per-IP request rates.

• Concurrent generation requests and queue depth.

• Request duration and upstream read timeouts.

• Embedding batch size and document-ingestion quotas.

• Tool-call count, recursion depth, execution time, and spend budgets.

• Tenant-specific monthly quotas and alerts.

Return a controlled 429 or 503 when overloaded; do not let queue growth exhaust host RAM or GPU memory. For longer jobs such as document ingestion, use an asynchronous queue with per-tenant limits rather than holding an API connection open.

Defend Prompts and RAG

Prompt injection happens when user-supplied or retrieved text tries to override the system’s intended behavior; OWASP identifies it as a leading LLM application risk. No prompt wording fully “solves” this, so place security boundaries outside the model’s judgment.

Treat prompts, files, web content, and retrieved RAG chunks as untrusted data. Never let the model decide whether a user is allowed to access a document, call a privileged API, read a secret, or execute a shell command. Instead:

• Apply ACL filters before vector retrieval.

• Pass only authorized chunks into the model context.

• Clearly label retrieved content as untrusted reference text.

• Require structured tool-call schemas and validate them server-side.

• Map each tool to a narrowly scoped service account.

• Require human approval for destructive or externally consequential actions.

• Allowlist outbound domains and block link-local, loopback, private-address, and cloud-metadata targets for any URL-fetching tool.

Least-privilege tools, approval gates for consequential actions, tool-use auditing, and bounded agent execution are recommended safeguards against excessive agency.

Protect Data and Supply Chain

Classify data before it enters prompts, embeddings, logs, or model files. Redact secrets and sensitive identifiers where possible; encrypt data in transit and at rest; segregate tenants; and define retention/deletion rules for conversations, uploads, embeddings, and backups. Track document and dataset modifications with provenance so a suspicious RAG result can be traced to a source and rolled back.

Pin container images and model versions by digest, scan images and dependencies, restrict who may pull or register models, verify model provenance, and separate the model-download environment from normal inference egress. Do not automatically ingest arbitrary uploads, repositories, webpages, or tickets into a production knowledge base: scan, parse, classify, review, hash, and approve them first. Poisoned RAG documents can manipulate answers or introduce malicious operational guidance.

Monitoring and Testing

Log structured metadata: request ID, authenticated principal, tenant, selected model, endpoint, status, latency, token counts, rate-limit decision, retrieved document IDs, and tool invocations. Avoid recording raw prompts, generated text, authorization headers, secrets, or document contents unless a narrowly approved diagnostic policy requires it.

Test the service like both an API and an AI system:

• Replay a valid request while swapping tenant, conversation, document, model, and tool IDs to test authorization.

• Fuzz JSON fields, streaming endpoints, file upload parsers, and model parameters.

• Test rate limits with parallel requests, long inputs, large max_tokens intentionally slow streaming clients.

• Attempt prompt injection through direct prompts and indexed documents.

• Verify RAG ACLs using users from different teams and classifications.

• Test tools with malicious arguments, SSRF-style URLs, and requests for destructive actions.

• Confirm that an API key cannot access operational or administrative endpoints outside its intended scope.

For your first hardening task, sketch your own inference service boundary and identify the one component that currently has the broadest privilege—for example, the API gateway, RAG service, tool runner, or model host—then describe the first privilege reduction you would apply.