Configurare program

GitOps Deployment with Argo CD and Kustomize

• Data 10/09/2026
• Autor Costin Dumitru
• 2
• 31

GitOps makes Git the desired-state record for Kubernetes: instead of an engineer applying manifests manually with kubectl, a controller continuously compares what is declared in Git with what exists in the cluster and reconciles differences. Argo CD performs that reconciliation; Kustomize renders reusable Kubernetes configuration for different environments without copying the entire manifest set.argo-cd.readthedocs+1.

For a practical platform setup, keep application source code separate from deployment configuration. Your CI pipeline builds, scans, and publishes an immutable container image; it then updates an image tag or digest in the GitOps repository through a reviewed pull request. Argo CD sees the approved Git change and deploys it.

Kustomize uses a base for shared manifests and overlays for environment-specific changes. An overlay points to the base and applies only the patches or additions needed for an environment.

Keep environment differences narrowly scoped: image reference, replica count, resource requests/limits, ingress hostname, environment configuration, and possibly namespace. Avoid putting credentials in plain YAML; reference secrets managed through External Secrets, Sealed Secrets, SOPS, or another controlled mechanism.

The base defines stable application behavior shared by every environment:

Always render locally before committing:

The first command shows the generated manifests; the second helps you reason about the live-cluster impact before Argo CD applies it.

Define Environment Overlays

An overlay imports the base, changes its namespace, and patches only what differs. Here is a production overlay:

This separation is the main Kustomize benefit: the base contains universal configuration, while overlays supply only environment deltas. Prefer explicit patches over a growing chain of generators and transformations that make rendered output difficult to review.

An Argo CD Application tells Argo CD which repository path to render and the destination cluster and namespace to reconcile. The Application specification supports Git source details, Kustomize configuration, destination, and sync policy.

With automated, Argo CD applies desired-state changes found in Git. selfHeal: true tells it to correct out-of-band cluster changes, while prune: true allows it to remove resources that were deliberately removed from Git. These controls are powerful: in production, protect the Git branch, require review, restrict who can modify Application objects, and use Argo CD Projects to limit the repositories, clusters, namespaces, and resource kinds each team may deploy.

A secure GitOps delivery flow is:

1. A developer commits application code.

2. CI runs unit, integration, image, dependency, and policy checks.

3. CI builds an image, scans it, signs it if your supply chain supports signing, and pushes it using an immutable tag or digest.

4. CI opens a pull request that changes only the relevant overlay image tag or digest.

5. A reviewer verifies the rendered manifest and approves the GitOps change.

6. The merge becomes the desired state; Argo CD synchronizes it to the cluster.

7. Argo CD reports health and sync status; alerts fire on degraded, missing, or out-of-sync workloads.

Avoid allowing CI to run broad kubectl apply commands directly against production. That creates a second deployment authority and makes drift harder to explain. With self-healing enabled, a direct manual scale operation or kubectl edit change is eventually reverted because Git remains authoritative.

GitOps rollback means reverting the commit that introduced the bad desired state, then allowing Argo CD to reconcile the previous revision. Keep image references immutable: a tag like latest undermines auditability because the same Git commit could produce different runtime artifacts.

When an application is OutOfSync or Degraded, troubleshoot in order:

• Render locally: kubectl kustomize apps/inventory-api/overlays/prod.

• Inspect the Argo CD diff to find the desired-versus-live mismatch.

• Check Application events and sync history.

• Inspect the affected Kubernetes object with kubectl describe.

• Check Pod status, events, logs, readiness probes, resource quotas, and NetworkPolicies.

• Confirm the Argo CD service account has only the RBAC permissions it needs.

• Check whether a controller is legitimately mutating fields; configure ignore differences only for understood and documented cases.

For a production-quality next step, choose one application you would deploy and identify the three fields that should differ between dev and prod while the base stays identical.

Local Private LLM Serving with Ollama/vLLM behind Nginx

• Data 10/09/2026
• Autor Costin Dumitru
• 1
• 29

A private local LLM service has four layers: the model runtime, a loopback-only inference API, Nginx as the controlled entry point, and identity/network controls around Nginx. Ollama is ideal for simple single-host local models and exposes its API on localhost:11434 by default; vLLM is better suited to higher-throughput GPU serving and offers an OpenAI-compatible HTTP API. In both cases, do not expose the runtime port directly to a LAN or the internet.docs.ollama+1

Use Ollama for a private workstation, a small internal assistant, or a lightweight homelab endpoint. Use vLLM where you want several applications or users to share a GPU and prefer OpenAI-compatible clients. A vLLM –api-key is not enough by itself because some endpoints are outside its protected API path prefixes; place it behind an authenticated reverse proxy and restrict network exposure.

Run the inference runtime so that only Nginx can reach it locally. Ollama binds to 127.0.0.1:11434 by default, which is the preferred local-service posture.

For Ollama, verify the listening address:

For vLLM, bind explicitly to loopback and set an application API key as a second layer:

Store the API key in a root-readable environment file or secret manager, not in shell history, unit files readable by other users, Git repositories, or container-image layers. If using Ollama in an offline/private environment, disable cloud access with OLLAMA_NO_CLOUD=1 or the related server configuration, then restart the service.

Nginx terminates TLS, authenticates clients, limits abusive traffic, provides audit logs, and forwards only approved paths to the local runtime. Disable proxy buffering so token streaming reaches the client continuously rather than being held until the response completes.

The same configuration works for vLLM if you change the upstream to 127.0.0.1:8000. vLLM itself supports OpenAI-style endpoints, while Nginx provides the surrounding access boundary. Nginx rate limiting tracks a request key such as client IP in shared memory and applies a configured request rate, helping protect an expensive GPU service from accidental loops or request floods.

HTTP Basic Auth is acceptable only for a tightly controlled internal lab over TLS; it is not ideal for a team or production environment. Prefer one of these patterns:

• Put an identity-aware proxy in front of Nginx using OIDC/SAML and MFA.

• Require VPN access first, then use OIDC at the proxy.

• Restrict with a firewall so only approved management subnets, application servers, or VPN address pools can reach TCP 443.

• Issue per-application API credentials through an authentication gateway, rather than sharing one static password.

• Use mutual TLS for machine-to-machine callers that can manage client certificates.

Do not log prompt bodies, bearer tokens, authorization headers, or complete generated responses by default. Prompt data can contain internal code, tickets, credentials pasted by users, or customer information. Retain only the metadata necessary for operations: timestamp, caller identity or pseudonymous identifier, model, status code, latency, token usage if available, and request size.

Run Ollama/vLLM with a dedicated unprivileged account, keep model directories owned by that account, and prevent arbitrary users from writing model files. Use a host firewall to permit port 443 only from intended networks; block direct access to ports 11434 and 8000 from every external interface.

For a containerized vLLM deployment, do not run privileged, mount the GPU device only as needed, mount model-cache storage with controlled permissions, pin image versions/digests, scan images, and use a read-only root filesystem where the runtime supports it. Make outbound network access explicit: allow access only to model registries during controlled model pulls, then restrict egress during normal inference.

Test and Operate

Validate each boundary separately:

Monitor GPU memory/utilization, queue time, request latency, Nginx 401/403/429/5xx responses, disk capacity for model caches, and unexpected model downloads. Set strict resource limits and a request-size cap so one oversized prompt cannot exhaust RAM, VRAM, disk, or proxy workers.

For a secure first deployment, decide whether the clients are only local processes, devices on a VPN, or internal applications on separate servers. Then state which layer will authenticate them—Nginx Basic Auth, an OIDC proxy, mTLS, or a combination—and why.

Building a Local Enterprise Knowledge Base with RAG

• Data 10/09/2026
• Autor Costin Dumitru
• -2
• 22

A local enterprise RAG system lets an LLM answer questions using approved internal material at query time rather than relying only on its pretrained knowledge. It is not “training the model on company documents”: the system retrieves relevant passages from an index, places them into the model context, and asks the model to answer only from that evidence. This is usually preferable to fine-tuning for information that changes often, such as runbooks, policies, asset inventories, architecture diagrams, incident procedures, and internal APIs.

The main design principle is simple: the model generates; your governed documents supply the facts. If retrieval is weak, no model prompt can reliably compensate.

Reference Architecture

The RAG lifecycle has two distinct paths:

• Indexing path: Load documents, parse their content, split them into chunks, generate embeddings, and store chunks plus metadata in a vector store. This normally happens offline or when a source changes. langchain

• Question path: Authenticate the user, apply authorization filters, embed the user’s query, retrieve allowed chunks, optionally rerank them, give those chunks to the LLM, and return an answer with source references. langchain

A local stack could be: Ollama or vLLM for inference, bge/e5-class local embedding models, Qdrant/pgvector/Chroma for vector search, PostgreSQL for metadata/audit records, and a Python FastAPI service that performs retrieval and policy enforcement.

Ingestion and Document Quality

The most difficult part is rarely the LLM; it is turning messy enterprise content into trustworthy, searchable evidence. Build a controlled ingestion pipeline instead of allowing arbitrary users to upload anything directly into the vector database.

Each chunk should carry useful metadata:

Parse documents while preserving heading hierarchy, tables, code blocks, page numbers, repository revision, and source ownership. Deduplicate documents using a content hash, reject unsupported or encrypted file types unless there is an approved parser, and quarantine files that fail malware scanning or content validation. A typical RAG index converts document chunks to embeddings and stores them for later similarity search.

A chunk is the atomic unit the system retrieves. If it is too large, it includes irrelevant context and wastes the model’s context window; if it is too small, it may omit the condition, command, exception, or surrounding explanation needed for a correct answer. Large documents are split because whole documents are difficult to retrieve accurately and may exceed the finite context available to the model.

Start with structure-aware recursive chunking:

• Preserve headings and sections where possible.

• Use approximately 400–800 tokens per chunk.

• Use 50–120 tokens of overlap to avoid losing context at boundaries.

• Keep commands and their explanation together.

• Do not split tables row-by-row unless your parser preserves headers.

• Store parent-document and section metadata with every chunk.

At query time, combine semantic vector search with keyword search. Semantic search finds conceptually similar text; keyword search is essential for exact values such as hostnames, CVE IDs, command flags, error codes, IP addresses, and product versions. Retrieve an initial set, apply ACL filters first, rerank the remaining evidence, then send only the best few chunks to the LLM.

The largest enterprise mistake is treating RAG as ordinary search without enforcing permissions. A user must never retrieve a chunk they would not be allowed to open in the original system. Enforce access control as a hard retrieval filter, not as an instruction such as “do not disclose confidential information.”

A safe query pattern is:

For example, a database administrator can search classification <= confidential and allowed_groups contains database-admins, while a help-desk user may only search public and internal end-user documentation. Use isolated vector collections or namespaces for separate tenants and highly sensitive domains in addition to metadata filtering.

Retrieved text is data, not trusted instructions. A malicious or compromised document may contain content such as “ignore previous instructions,” secrets, misleading recovery commands, or instructions to exfiltrate data. OWASP identifies manipulated embedding or training data as a poisoning risk, and this includes knowledge-base sources used by RAG systems.owasp+1

Use a system instruction similar to:

Also apply source allowlists, document ownership, approval workflows, signatures or hashes for high-value runbooks, malware scanning, change review, re-indexing on approved updates, and anomaly alerts for sudden large-scale changes. Never let the RAG assistant execute retrieved shell commands, SQL, Kubernetes actions, or ticket updates without separate authorization, input validation, and human approval.

Build a test set before declaring the knowledge base successful. Include representative questions, expected sources, questions with no answer, outdated-document traps, confusing near-duplicates, access-control tests, and prompt-injection attempts embedded in sample documents. Measure:

• Retrieval recall: Did the correct source appear in the top results?

• Citation correctness: Does each cited source actually support the answer?

• Answer faithfulness: Does the response remain within the supplied evidence?

• Access-control accuracy: Can users retrieve only content allowed by their role?

• Freshness: Is the answer based on the latest approved version?

• Operational safety: Does it avoid inventing dangerous production steps?

Securing Self-Hosted AI APIs

• Data 10/09/2026
• Autor Costin Dumitru
• 1
• 33

A self-hosted AI API is both a conventional API and an AI-specific attack surface. Secure it in layers: isolate the model runtime, expose only a hardened gateway, authenticate every caller, authorize each action and data retrieval, constrain compute use, and log enough metadata to investigate abuse without storing sensitive prompts by default. Broken object-level authorization, broken authentication, resource exhaustion, misconfiguration, and unsafe downstream API use are all central API risks.

The key principle: do not treat an LLM endpoint as “just internal.” A compromised internal workstation, leaked API key, malicious prompt, or vulnerable integrated service can turn it into a path toward confidential data or privileged actions.

Reference Architecture

Bind Ollama, vLLM, or another inference runtime to 127.0.0.1 or a private container network. Block direct inbound access to runtime ports such as 11434 or 8000 using host firewalls, security groups, or Kubernetes NetworkPolicy; only the API layer should reach them. vLLM documentation warns that its API key option alone does not secure every endpoint, so defense must include a reverse proxy and network restrictions.

Use OIDC/OAuth2 access tokens for human and application clients, preferably validated at the gateway and again by the API service for high-value operations. Require MFA through the identity provider for administrator access, use short-lived tokens, rotate service credentials, and keep secrets in a vault or workload-identity mechanism rather than code, shell history, or container images. OWASP advises treating login, token, and account-recovery flows as high-risk endpoints that need brute-force protection, and it notes that API keys identify API clients rather than authenticating users.

Authentication answers who is calling. Authorization answers what that caller may do. Enforce authorization for every resource ID, model name, tenant, conversation, uploaded file, RAG collection, tool, and administrative operation—never rely on a route prefix or a client-supplied tenant field. For example, a caller authorized for team-a documents must not obtain team-b vector chunks merely by changing a collection ID; that is the AI equivalent of BOLA.

LLM inference is vulnerable to resource exhaustion because prompts, output length, concurrent requests, file uploads, embeddings, and tool loops consume CPU, GPU VRAM, memory, storage, and money. OWASP classifies this broader problem as unrestricted resource consumption for APIs and unbounded consumption for LLM applications.owasp+1

Apply limits at multiple points:

• Request body and file-upload size.

• Maximum prompt tokens and maximum generated tokens.

• Per-user, per-API-key, and per-IP request rates.

• Concurrent generation requests and queue depth.

• Request duration and upstream read timeouts.

• Embedding batch size and document-ingestion quotas.

• Tool-call count, recursion depth, execution time, and spend budgets.

• Tenant-specific monthly quotas and alerts.

Return a controlled 429 or 503 when overloaded; do not let queue growth exhaust host RAM or GPU memory. For longer jobs such as document ingestion, use an asynchronous queue with per-tenant limits rather than holding an API connection open.

Defend Prompts and RAG

Prompt injection happens when user-supplied or retrieved text tries to override the system’s intended behavior; OWASP identifies it as a leading LLM application risk. No prompt wording fully “solves” this, so place security boundaries outside the model’s judgment.

Treat prompts, files, web content, and retrieved RAG chunks as untrusted data. Never let the model decide whether a user is allowed to access a document, call a privileged API, read a secret, or execute a shell command. Instead:

• Apply ACL filters before vector retrieval.

• Pass only authorized chunks into the model context.

• Clearly label retrieved content as untrusted reference text.

• Require structured tool-call schemas and validate them server-side.

• Map each tool to a narrowly scoped service account.

• Require human approval for destructive or externally consequential actions.

• Allowlist outbound domains and block link-local, loopback, private-address, and cloud-metadata targets for any URL-fetching tool.

Least-privilege tools, approval gates for consequential actions, tool-use auditing, and bounded agent execution are recommended safeguards against excessive agency.

Protect Data and Supply Chain

Classify data before it enters prompts, embeddings, logs, or model files. Redact secrets and sensitive identifiers where possible; encrypt data in transit and at rest; segregate tenants; and define retention/deletion rules for conversations, uploads, embeddings, and backups. Track document and dataset modifications with provenance so a suspicious RAG result can be traced to a source and rolled back.

Pin container images and model versions by digest, scan images and dependencies, restrict who may pull or register models, verify model provenance, and separate the model-download environment from normal inference egress. Do not automatically ingest arbitrary uploads, repositories, webpages, or tickets into a production knowledge base: scan, parse, classify, review, hash, and approve them first. Poisoned RAG documents can manipulate answers or introduce malicious operational guidance.

Monitoring and Testing

Log structured metadata: request ID, authenticated principal, tenant, selected model, endpoint, status, latency, token counts, rate-limit decision, retrieved document IDs, and tool invocations. Avoid recording raw prompts, generated text, authorization headers, secrets, or document contents unless a narrowly approved diagnostic policy requires it.

Test the service like both an API and an AI system:

• Replay a valid request while swapping tenant, conversation, document, model, and tool IDs to test authorization.

• Fuzz JSON fields, streaming endpoints, file upload parsers, and model parameters.

• Test rate limits with parallel requests, long inputs, large max_tokens intentionally slow streaming clients.

• Attempt prompt injection through direct prompts and indexed documents.

• Verify RAG ACLs using users from different teams and classifications.

• Test tools with malicious arguments, SSRF-style URLs, and requests for destructive actions.

• Confirm that an API key cannot access operational or administrative endpoints outside its intended scope.

For your first hardening task, sketch your own inference service boundary and identify the one component that currently has the broadest privilege—for example, the API gateway, RAG service, tool runner, or model host—then describe the first privilege reduction you would apply.

Multi-Agent Workflows with Persistent Memory and Human Gatekeeping

• Data 10/09/2026
• Autor Costin Dumitru
• 1
• 26

A multi-agent workflow divides a complex task among specialized AI components instead of asking one model to do everything. For example, one agent can collect evidence, another can analyze it, a third can draft an answer, and a final agent can check policy or quality. Persistent memory gives the workflow continuity across sessions, while human gatekeeping ensures that high-impact actions remain under human control.

The central rule is simple: agents may recommend, prepare, classify, and summarize, but they should not independently make irreversible decisions, access sensitive data outside their scope, or execute consequential actions.

Core Architecture

A practical design separates orchestration, agents, memory, tools, and approval controls.

The orchestrator owns the workflow state. It decides which agent runs next, passes only the required context to that agent, records results, detects failures, and stops the workflow when an approval requirement is reached. Do not let agents communicate freely without control. Unbounded agent-to-agent conversation can create loops, hidden assumptions, duplicated work, and unclear accountability.

An agent should have a narrow responsibility, limited data access, and minimal tools.

• Research agent: Retrieves approved internal documents, tickets, runbooks, or external evidence.

• Analysis agent: Compares evidence, identifies gaps, calculates risk, or proposes options.

• Drafting agent: Produces a report, email, change request, or technical plan.

• Validation agent: Checks required fields, policy constraints, source references, formatting, and safety rules.

• Execution agent: Performs an approved action through a narrowly scoped API or automation account.

For example, in an infrastructure incident workflow, the research agent gathers alerts and related runbook sections. The analysis agent determines likely impact. The drafting agent prepares a change proposal. A human approves the proposal. Only then can an execution agent create a ticket, scale a deployment, restart a service, or apply an approved remediation.

This separation makes failures easier to investigate. If a bad recommendation appears, you can determine whether it came from retrieval, reasoning, drafting, validation, or an incorrect approval decision. Persistent memory should not mean storing every conversation forever. Store specific, governed facts that improve future work.

Use three memory types:

• Working memory: Temporary state for one workflow run, such as task ID, intermediate findings, selected documents, and pending approvals. Delete or expire it when the task finishes.

• Episodic memory: Records of previous interactions, decisions, approvals, incidents, and outcomes. This supports auditability and allows the system to learn from completed work.

• Semantic memory: Durable facts such as service ownership, approved procedures, business rules, known environment constraints, and validated technical preferences.

Each stored memory item should include metadata similar to this:

The most important fields are source, owner, classification, allowed roles, freshness, and approval status. An agent should not treat an old chat message as equivalent to an approved runbook or a signed operational decision.

Retrieve Memory Safely

Before an agent receives memory, apply authorization and relevance filtering.

Never rely on the model to decide whether the caller may access a memory record. The memory service must enforce that decision before information reaches the prompt.

For a team knowledge base, separate records by tenant, department, project, environment, and classification. A production incident agent should not automatically receive HR records, customer data, credentials, private keys, unrelated security reports, or another team’s confidential project history.

Human gatekeeping is a policy control, not a decorative confirmation button. Define what requires approval in advance.

Require explicit approval for actions such as:

• Changing production infrastructure or Kubernetes resources.

• Restarting, scaling, deleting, or reconfiguring services.

• Sending emails, tickets, messages, or reports outside the system.

• Accessing sensitive datasets or exporting internal information.

• Creating credentials, rotating keys, changing IAM roles, or modifying firewall rules.

• Executing SQL write operations or running potentially destructive commands.

• Publishing recommendations that affect legal, financial, HR, compliance, or customer decisions.

The approval request should include the proposed action, intended target, expected impact, evidence used, uncertainty level, rollback method, risk level, and exact command or API payload where applicable.

A human should approve a specific action, not a vague goal. “Fix the deployment” is too broad; “roll back deployment X to image digest Y” is reviewable and auditable.

Control Agent Tools

An LLM should never receive unrestricted shell, Kubernetes, database, cloud, or network access. Give each action agent a dedicated service account with permissions limited to one environment, namespace, service, or operation type.

A safe execution pattern is:

Validate tool arguments server-side. If an agent proposes a Kubernetes operation, verify the allowed namespace, resource type, name pattern, operation, image source, and change window before execution. Do not allow free-form shell commands when a structured API operation can be used instead.

Persistent memory introduces risks beyond normal prompt handling.

• Memory poisoning: A malicious document or user message adds misleading instructions or false facts to long-term memory.

• Stale memory: An old procedure is retrieved after infrastructure or policy has changed.

• Cross-tenant disclosure: One user’s information is returned to another team or customer.

• Over-retention: Sensitive personal, operational, or credential-related data remains longer than necessary.

• False confidence: The model treats a low-confidence note as an established fact.

Mitigate these risks with approval workflows for durable memory, source provenance, expiration dates, versioning, review queues, audit trails, role-based retrieval filters, and a deletion process. Treat user-provided instructions as untrusted unless they are verified and promoted through a controlled process.

Monitor and Test

Track each workflow through a correlation ID. Log the workflow version, agent invoked, input classification, memory records retrieved, tools requested, policy decisions, human approver, execution result, and final outcome. Avoid logging raw sensitive prompts or secrets unless a specific approved investigation requires it.

Test workflows with realistic failure cases:

• A malicious memory entry that tells an agent to ignore policy.

• An expired runbook that conflicts with a newer approved procedure.

• A user requesting data outside their role.

• An agent attempting to call an unauthorized tool.

• A proposed production action with incomplete rollback information.

• An approval request where the target resource differs from the original task.

• A failed action that must stop rather than retry indefinitely.