Configurare program

API Security Testing in CI/CD

Since you already work with pentesting tools like Burp Suite, Metasploit, and Nikto, think of this as extending that manual mindset into an automated, repeatable pipeline stage. The goal of CI/CD-integrated API security testing is to catch vulnerabilities like broken authorization or injection flaws on every code change, not just during periodic audits.

Testing an API before it reaches production is far cheaper than fixing it after deployment, especially for issues like Broken Object Level Authorization (BOLA) or Broken Function Level Authorization (BFLA), which are the top risks in the OWASP API Security Top 10 2023. The core idea of “shift-left” is simple: generate and lint your OpenAPI specification, run authorization and fuzzing tests automatically, scan infrastructure-as-code for misconfigurations, and fail the build when high-severity issues appear.

Given your background in penetration testing, here’s a question to get you thinking before we go further: when you manually test an API for BOLA today, what steps do you take to confirm one user can’t access another user’s data? Hold that thought, because we’re about to automate exactly that.

CI/CD API security testing generally falls into three buckets, each targeting a different class of OWASP API risks:

  • Authentication and authorization tests — verifying BOLA (swap object IDs between two valid users), BFLA (replay admin-only requests with a lower-privilege token), mass assignment (attempt to modify read-only fields via POST/PATCH), and token lifecycle behavior (expired or revoked tokens must return 401, not stale cached data).

  • Input and request validation tests — submitting oversized payloads, SQL/NoSQL/command injection strings, path traversal attempts, malformed JSON/XML, and checking that GraphQL endpoints enforce query complexity limits and disable introspection in production.

  • Transport and configuration tests — enforcing HTTPS everywhere, validating TLS versions and cipher suites, checking security headers, testing CORS against unauthorized origins, and scanning for verbose debug output that leaks internal paths.

A practical automation pattern for BOLA/BFLA, drawn from real testing workflows, looks like this: extract every route from your OpenAPI spec to build your attack surface list, capture three token sets (admin, standard user, unauthenticated), then programmatically replay the same request while swapping tokens or object IDs between the two user accounts — if a low-privilege token or wrong user’s ID returns a 200 instead of 403/404, you’ve found a flaw.

Building the Pipeline Stage

Rather than treating a security scan as an afterthought, it should run as a pipeline job that blocks deployment on critical findings, similar to how you might already gate deployments on failed unit tests. A minimal GitHub Actions-style stage would download a scanner such as the OWASP API Security Testing Framework, point it at your API with an auth token, output results in a machine-readable format like SARIF, and upload that report to your code scanning dashboard so findings show up alongside pull requests. Tools like OWASP ZAP, Burp Suite (which you already use manually), and 42Crunch can run these scans directly against your OpenAPI/Swagger spec, meaning your existing spec becomes the test target instead of something you write test cases for by hand.

Beyond deployment-triggered scans, it’s worth scheduling separate recurring scans (weekly or monthly) against your live production API, since configuration drift — like a firewall rule change or an expired cert — can introduce vulnerabilities between deployments even without a code change. NIST’s guidance on API protection for cloud-native systems frames this as two complementary layers: static definition-time protections (defining schemas, valid value ranges, and maintaining an API inventory) and runtime protections applied to every request and response, including encryption in transit, authentication/authorization checks, and rate limiting or circuit breaking.

Prioritizing What to Automate First

If you’re setting this up from scratch, the recommended order is: start with authorization testing (BOLA, then BFLA), then authentication, then input validation and rate limiting, then configuration and API inventory checks. A condensed priority view looks like this:

Risk Priority CI/CD Action
BOLA High Add server-side ownership checks; enforce in automated tests
Broken authentication High Reject alg:none JWTs; enforce RS256; enable key rotation
Broken object/property-level auth High Apply field allowlists and strict schema validation
Unrestricted resource consumption High Enforce rate limits and payload size caps in pipeline tests
Broken function-level auth High Map admin-only endpoints; enforce via policy engine

Given your Rocky Linux and Kubernetes experience, you’re well positioned to containerize a scanner like ZAP or the OWASP ASTF as a pipeline job that spins up, hits a staging deployment, and tears down automatically.

To check your understanding: if a CI/CD pipeline scan finds that a standard-user token can successfully call an admin-only /users/delete endpoint, which OWASP risk category does that fall under, and what specific pipeline check would you add to catch it on every future pull request?

[mai mult...]

Hardening POS/Payment Terminal Integration Networks

A payment-terminal network is not just “a VLAN for card machines.” It is a tightly controlled path between the POS/terminal, payment gateway or acquirer, terminal-management system (TMS), and a small set of operational services. The main security objective is to prevent a compromise of a workstation, Wi-Fi network, or back-office server from reaching payment devices or affecting payment traffic. PCI guidance strongly recommends segmentation: without effective isolation, a much broader network can fall into PCI scope.

Before implementing controls, map each payment-data flow: terminal to processor, terminal to TMS, POS to payment middleware, DNS/NTP/DHCP dependencies, logging, monitoring, and vendor-support paths. An asset inventory should identify every point-of-interaction device, its model, serial number, owner, location, firmware, network identity, and support status.

Use separate zones with default-deny controls between them:

  • Terminal zone: Dedicated wired switch ports or a dedicated SSID/VLAN for payment terminals only. No corporate laptops, printers, guest devices, or general POS browsing systems.

  • Payment services zone: The payment gateway, payment middleware, and/or approved connector. Permit only required terminal-to-service flows.

  • Management zone: A hardened TMS, monitoring system, patch repository, and a jump host. It should not be reachable directly from normal administrator workstations.

  • Corporate zone: Office users, back-office systems, and standard endpoints. Treat this zone as untrusted relative to the payment environment.

  • Internet/processor edge: Allow outbound connections only to documented acquirer, processor, DNS, time, certificate-status, and update destinations.

Segmentation can be physical, logical, or both, but it must stop out-of-scope systems from communicating with—or influencing—the cardholder-data environment. A useful rule is: the terminal may initiate only the precise connections needed to process a transaction and receive authorized management updates; everything else is blocked. 

Avoid broad rules such as “terminal VLAN to Internet: allow.” Replace them with allowlisted flows that have a documented business purpose:

Source Destination Permit Deny
Terminal VLAN Approved processor endpoints Required encrypted payment traffic only All other internet traffic
Terminal VLAN Internal DNS/NTP/DHCP Only approved infrastructure services General internal access
TMS / jump host Terminal VLAN Vendor-approved management ports only Direct admin access from corporate PCs
Corporate LAN Terminal VLAN Usually none All inbound traffic by default
Guest Wi-Fi / IoT Payment zones None All traffic
Payment zones Corporate databases Only explicitly required tokenized/payment integrations Broad database and SMB access

Build policy around both network and identity. For example, a TMS server should be the only system that can manage terminals, and a named, MFA-protected administrator should be the only identity allowed to access that TMS. PCI guidance specifically calls for MFA for access into the cardholder-data environment, including remote access to terminals or terminal-management systems.

Treat the terminal as an embedded security appliance, not as a generic network device.

  • Change vendor-default credentials and disable unused local, web, SSH, Telnet, Bluetooth, Wi-Fi, USB, debug, and remote-management interfaces.

  • Keep vendor-supported firmware and payment applications current through a controlled, authenticated update path; unsupported terminal software is a serious operational and security risk.

  • Enforce certificate validation and modern TLS for terminal-to-processor and terminal-to-TMS communications. Do not accept invalid certificates “to make connectivity work.”

  • Use device certificates, mutual TLS, or equivalent strong device identity when supported; IP allowlisting alone is not enough because IP addresses can be spoofed or reassigned.

  • Remove sensitive card data from POS logs, error messages, packet captures, databases, and support tickets. Prefer processor tokenization so business systems handle tokens rather than primary account numbers.

  • Protect terminals from substitution and tampering: maintain a device inventory, inspect devices regularly, train staff to spot altered equipment, and investigate unexpected reboots, serial-number changes, or new network identities.

Remote vendor support is a high-risk integration path. Never expose a terminal, POS controller, or TMS directly to the public internet. Instead, require an approved remote-access gateway or VPN, MFA, a hardened jump host, time-limited approval, session recording where feasible, and centralized logs. Give vendors a narrowly scoped role: a support account should access only the specified terminal group or TMS function, not your full internal network.

If a payment processor requires a static inbound path, terminate it in a dedicated edge zone and forward only the required protocol to a purpose-built integration component. Do not make a firewall exception from the internet to the terminal VLAN.

Hardening is incomplete until it is tested. Monitor firewall-deny events, new MAC addresses, terminal certificate failures, firmware changes, repeated transaction errors, unusual outbound destinations, TMS admin logins, and configuration drift. Review whether each terminal is communicating only with its allowlisted destinations; a terminal making DNS requests to arbitrary resolvers or HTTPS connections to unfamiliar hosts deserves immediate investigation.

Run segmentation validation from a compromised-user perspective: place a test device in the corporate or guest network and verify it cannot discover, route to, scan, administer, or reach terminal-management interfaces. PCI guidance emphasizes that segmentation should be documented, tested, and maintained—not merely shown as a diagram. Re-test after switch changes, firewall changes, new terminals, processor migrations, TMS upgrades, or vendor remote-access changes.

[mai mult...]

Fine-Tuning Small language models on Linux

Fine-tuning adapts a pretrained model to a narrow task, vocabulary, tone, or response format using your own examples. For a Linux/sysadmin use case, a small language model (SLM) could be adapted to turn Rocky Linux alerts into triage steps, answer internal runbook questions, or classify security-ticket descriptions. The usual efficient approach is supervised fine-tuning (SFT): train on input-output examples so the model learns to predict the desired answer conditioned on the prompt.

Start with a modest model that fits your GPU and a narrow goal. A small, clean dataset usually beats a large, inconsistent one: avoid secrets, access tokens, customer data, passwords, production logs containing identifiers, and unlicensed text. Split the dataset into train, validation, and test sets before training, so you can measure whether the model generalizes rather than merely memorizing examples.

A practical Linux stack is Python in a virtual environment, NVIDIA drivers plus CUDA-compatible PyTorch for GPU training, and the Hugging Face ecosystem: transformers for models, datasets for data, peft for efficient adapters, trl for supervised training, and optionally bitsandbytes for low-bit loading. PEFT methods update a small set of adapter parameters rather than the full model, reducing both compute and storage needs.

bash
python3 -m venv slm-ft
source slm-ft/bin/activate

python -m pip install –upgrade pip
pip install torch transformers datasets accelerate peft trl bitsandbytes

Before training, confirm the machine sees the GPU and that PyTorch can use it:

bash
nvidia-smi

python -c “import torch; print(torch.cuda.is_available()); print(torch.cuda.get_device_name(0) if torch.cuda.is_available() else ‘CPU only’)”

CPU fine-tuning is possible for very small models, but it is generally slow. On a single consumer NVIDIA GPU, 4-bit quantization plus LoRA is often the practical route: loading weights in 4-bit or 8-bit precision saves memory, while the adapter remains trainable.

Prepare the Dataset

For chat-oriented tuning, use JSON Lines, with one valid JSON object per line. Keep the prompt explicit and make answer style consistent:

json
{"messages":[{"role":"system","content":"You are a cautious Linux operations assistant. Never invent commands or system state."},{"role":"user","content":"Rocky Linux: how do I see failed systemd services?"},{"role":"assistant","content":"Run systemctl --failed. Then inspect a unit with systemctl status <unit> and journalctl -u <unit>."}]}

Create three files: train.jsonl, validation.jsonl, and test.jsonl. Validation data guides training decisions; test data stays untouched until the end. Include both normal and adversarial cases relevant to the deployment: ambiguous requests, requests for unavailable information, commands with unsafe consequences, and examples where the correct answer is to ask for context instead of guessing.

LoRA (Low-Rank Adaptation) freezes the base model and trains small adapter matrices attached to selected model layers. The result is a compact adapter rather than a full duplicate of the model; PEFT’s documented workflow is to load the base model, define LoraConfig, wrap it as a trainable PEFT model, and train normally.

For an SLM, target the attention projection layers commonly named q_proj and v_proj; verify your particular architecture first because module names vary. Key settings are:

  • r — adapter rank. Start around 8 or 16; a larger rank can learn more but costs more VRAM.

  • lora_alpha — scaling factor. A common starting relationship is approximately two times r.

  • lora_dropout — regularization. Start around 0.05 for smaller datasets.

  • learning_rate — start conservatively, such as 1e-4 to 2e-4 for LoRA SFT.

  • max_seq_length — cap it to the real task need; longer sequences consume much more memory.

Minimal Training Script

Save the following as train.py. Replace YOUR_BASE_MODEL with a model you are licensed and authorized to use, and inspect its model card for its expected prompt format and intended use.

python
from datasets import load_dataset
from peft import LoraConfig
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

model_id = “YOUR_BASE_MODEL”

quant_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type=“nf4”,
bnb_4bit_compute_dtype=“float16”,
)

tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
model_id,
quantization_config=quant_config,
device_map=“auto”,
)

dataset = load_dataset(
“json”,
data_files={
“train”: “train.jsonl”,
“validation”: “validation.jsonl”,
},
)

peft_config = LoraConfig(
r=16,
lora_alpha=32,
lora_dropout=0.05,
target_modules=[“q_proj”, “v_proj”],
task_type=“CAUSAL_LM”,
)

training_args = SFTConfig(
output_dir=“./output-slm-adapter”,
num_train_epochs=3,
per_device_train_batch_size=1,
gradient_accumulation_steps=8,
learning_rate=2e-4,
logging_steps=10,
eval_strategy=“steps”,
eval_steps=50,
save_steps=50,
bf16=True,
report_to=“none”,
)

trainer = SFTTrainer(
model=model,
args=training_args,
train_dataset=dataset[“train”],
eval_dataset=dataset[“validation”],
processing_class=tokenizer,
peft_config=peft_config,
)

trainer.train()
trainer.save_model(“./output-slm-adapter”)
tokenizer.save_pretrained(“./output-slm-adapter”)

TRL’s SFTTrainer is built specifically to simplify supervised fine-tuning, and when given a PEFT configuration it initializes the adapter training path for you. If bf16=True fails on your GPU, use fp16=True instead; do not enable both.

Do not judge success only by a declining training loss. Compare the base model and tuned model against the held-out test set, then manually probe realistic requests. For an internal Linux assistant, assess command correctness, whether prerequisites are stated, whether destructive commands are guarded, whether outputs are invented, and whether the model declines to fabricate environment-specific facts.

Watch for these common failures:

  • Overfitting: Training loss drops while validation loss rises. Reduce epochs, lower rank, improve dataset variety, or add examples.

  • Bad formatting: The model’s response template differs from your training format. Match the base model’s chat template during data preparation and inference.

  • Catastrophic behavior change: A narrow dataset makes the model repetitive or overly confident. Mix high-quality general instruction examples with your specialist examples.

  • Out-of-memory errors: Lower max_seq_length, batch size, LoRA rank, or use gradient accumulation and 4-bit loading.

  • Unsafe operational output: Add counterexamples that require warnings, confirmation, backup advice, or requests for logs before recommending a risky command.

A LoRA output normally contains adapter weights and configuration, not a standalone base model. At inference time, load the original base model and then attach the adapter; PEFT supports saving adapters and loading them on the correct base model through the adapter configuration. This makes versioning practical: store a model manifest containing the base-model revision, adapter version, dataset version/hash, training parameters, evaluation result, owner, and approval status.

For your Linux-first workflow, package inference in a container, run it with a non-root user, mount model data read-only, avoid embedding access tokens in images, restrict egress if the model does not need external access, and expose the API only behind authentication and rate limits. Keep model artifacts separate from training data, especially if the data originated from internal tickets or infrastructure documentation.

[mai mult...]

Automating PostgreSQL/MongoDB Backups in Kubernetes

A Kubernetes backup system must protect three things: database data, database-level metadata, and the restore process itself. A scheduled backup that has never been restored is only an assumption, so design the workflow around a measurable recovery point objective (RPO) and recovery time objective (RTO).

A robust pattern is:

  1. A Kubernetes CronJob starts a short-lived, non-root backup Pod.

  2. The Pod reads credentials from a Secret mounted as environment variables or files.

  3. It connects to PostgreSQL or MongoDB through an internal Service using least-privilege backup credentials.

  4. It creates a compressed, encrypted backup artifact.

  5. It uploads the artifact to object storage or a remote backup repository.

  6. It writes logs, emits metrics, and removes temporary local files.

  7. A separate scheduled restore test validates the latest backup in an isolated namespace.

Do not depend only on a PersistentVolume mounted beside the database. If the cluster, storage class, credentials, or namespace are compromised together, both the production data and “backup” may be lost.

PostgreSQL Backup Strategy

For logical backups, pg_dump backs up one PostgreSQL database, while pg_dumpall covers an entire cluster and preserves global objects such as roles and tablespaces. For individual application databases, use the custom archive format (-Fc): it is compressed, works with pg_restore, and supports selective restore operations.

A basic PostgreSQL backup command is:

bash
pg_dump \
--host="$PGHOST" \
--port="$PGPORT" \
--username="$PGUSER" \
--format=custom \
--no-owner \
--file="/work/${PGDATABASE}-${STAMP}.dump" \
"$PGDATABASE"

The --no-owner option helps when restoring into a different environment where source role names do not exist. If your recovery requires roles and tablespaces too, run a separate global metadata export:

bash
pg_dumpall \
--host="$PGHOST" \
--username="$PGUSER" \
--globals-only \
--file="/work/${PGDATABASE}-${STAMP}-globals.sql"

For large, high-change PostgreSQL systems, logical dumps may not meet your RPO/RTO. Use physical backups plus WAL archiving through a PostgreSQL-aware solution such as pgBackRest, WAL-G, or an operator-supported backup mechanism. The important distinction is that pg_dump captures a logical snapshot, while point-in-time recovery requires archived WAL files in addition to a suitable base backup.

For self-managed MongoDB, mongodump creates a binary export and mongorestore restores that export to a running MongoDB deployment. Use a single compressed archive for easier handling and avoid a large tree of loose BSON files:

bash
mongodump \
--uri="$MONGODB_URI" \
--archive="/work/mongodb-${STAMP}.archive.gz" \
--gzip

mongodump can back up a full deployment, a database, a collection, or query-filtered content, depending on options and operational needs. For a replica set, ensure you understand consistency requirements and test restoration against the exact MongoDB topology you operate. For managed MongoDB Atlas, platform-native continuous backups and point-in-time recovery are often preferable to running mongodump as the sole protection mechanism.

This example uses PostgreSQL. The same container pattern works for MongoDB by replacing the image and backup command. It writes to /work; in production, add an upload command to S3-compatible storage, Azure Blob, GCS, or a hardened backup server.

text
apiVersion: batch/v1
kind: CronJob
metadata:
name: postgresql-backup
namespace: data
spec:
schedule: "15 2 * * *"
timeZone: "Europe/Bucharest"
concurrencyPolicy: Forbid
startingDeadlineSeconds: 600
successfulJobsHistoryLimit: 3
failedJobsHistoryLimit: 5
jobTemplate:
spec:
backoffLimit: 2
ttlSecondsAfterFinished: 86400
template:
spec:
restartPolicy: Never
serviceAccountName: database-backup
securityContext:
runAsNonRoot: true
runAsUser: 10001
fsGroup: 10001
containers:
- name: backup
image: postgres:16
imagePullPolicy: IfNotPresent
envFrom:
- secretRef:
name: postgres-backup-credentials
command:
- /bin/sh
- -ec
- |
umask 077
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
OUT="/work/${PGDATABASE}-${STAMP}.dump"

pg_dump \
–host=”$PGHOST” \
–port=”${PGPORT:-5432}” \
–username=”$PGUSER” \
–format=custom \
–no-owner \
–file=”$OUT” \
“$PGDATABASE”

test -s “$OUT”
sha256sum “$OUT” > “${OUT}.sha256”

# Upload OUT and OUT.sha256 to remote immutable storage here.
# Delete local files only after upload verification succeeds.
volumeMounts:
– name: backup-work
mountPath: /work
volumes:
– name: backup-work
emptyDir:
sizeLimit: 20Gi

concurrencyPolicy: Forbid prevents a slow backup from overlapping with the next scheduled run. Set startingDeadlineSeconds so an overdue backup does not start at an operationally unsafe time. CronJobs are appropriate for recurring work, but Kubernetes documents that scheduling can be approximate, so make the job idempotent and record success externally rather than trusting schedule timing alone.

Use a dedicated database account whose permissions are sufficient to read backup data but not to modify production records. Store connection strings, passwords, TLS certificates, and object-storage credentials in a secret-management system; restrict access with namespace RBAC, separate service accounts, and cloud workload identity where available.

Apply these controls:

  • Use TLS for database connections and object-storage uploads.

  • Encrypt backup files before leaving the Pod, or use a destination that provides strong server-side encryption with independently controlled keys.

  • Keep backup storage in a separate account, project, or tenant from the production Kubernetes cluster.

  • Enable versioning and retention locks/object immutability to resist accidental deletion and ransomware.

  • Include a checksum or manifest containing backup timestamp, database/version, artifact size, checksum, and tool version.

  • Never place database URIs, passwords, or cloud keys directly in CronJob YAML, Git repositories, or CI logs.

  • Restrict egress from the backup Pod to only the database endpoint, DNS/time dependencies, and the backup destination.

Retention, Monitoring, and Restores

Define retention before automation. A typical scheme might keep daily backups for 14–30 days, weekly backups for several months, and monthly backups for a year, but the right policy follows business, legal, and storage requirements. Monitor at least: last-success timestamp, job duration, output size, upload success, checksum verification, and age of the newest restorable artifact.

Restore testing should be automated in a disposable namespace: provision an empty Postgres or MongoDB instance, download a recent backup, restore it, run integrity checks and representative application queries, then destroy the environment. PostgreSQL custom-format dumps restore with pg_restore; MongoDB archive dumps restore with mongorestore. A successful restore test is the most meaningful backup metric.mongodb+1

As a practical first step, define whether your most important database needs daily logical recovery or point-in-time recovery, then explain what maximum acceptable data loss in minutes or hours would be for that workload.

[mai mult...]

Bare-Metal Kubernetes Networking with Cilium

On bare metal, Kubernetes has no cloud load balancer, no VPC-native pod routing, and often no automatic way to advertise LoadBalancer IPs. Cilium fills those gaps with an eBPF-based datapath, Kubernetes networking, service handling, optional kube-proxy replacement, load-balancer IP allocation, BGP or Layer-2 service advertisement, and identity-aware network policy. The key design decision is how external clients will reach Service IPs: Layer 2 announcements for a flat local subnet, or BGP for routed and larger environments.docs.cilium+1

Start by separating the address spaces:

  • Node subnet: Real IPs assigned to physical servers, such as 10.10.0.0/24.

  • Pod CIDR: Addresses allocated to pods, such as 10.244.0.0/16.

  • Service CIDR: Virtual Kubernetes ClusterIP range, such as 10.96.0.0/12.

  • LoadBalancer IP pool: Addresses reserved for externally reachable Kubernetes Services, such as 10.10.0.200-10.10.0.220.

Traffic usually follows one of these paths:

  1. A pod contacts another pod directly through Cilium’s datapath.

  2. A pod or external client contacts a Kubernetes Service IP; Cilium selects a backend pod.

  3. An external client contacts a LoadBalancer IP; a Cilium node advertises reachability through ARP/NDP (L2) or BGP routing, then forwards traffic to the service backend.

Do not overlap Pod, Service, LoadBalancer, node, VPN, or corporate network CIDRs. CIDR overlap causes routing ambiguity that can resemble random packet loss or failed service access.

Choose L2 or BGP

Method Best fit How it works Main limitation
L2 announcements Homelabs, offices, small clusters, one LAN/VLAN A selected Cilium node responds to ARP/NDP for a LoadBalancer IP so clients on that subnet send it traffic docs.cilium Clients must share the relevant Layer-2 network; it is not a routing protocol for multiple subnets notes.kodekloud
BGP control plane Datacenters, enterprise networks, routed environments Cilium peers with routers/switches and advertises service routes Requires BGP-capable network infrastructure and coordination with network operations
NodePort Simple bootstrap/exposure Client uses a node IP plus a high port Less elegant; exposes node addresses and complicates firewalling

For a typical small on-premises cluster with nodes and clients on the same VLAN, L2 announcements are the simplest option. For your infrastructure-oriented use cases—multiple VLANs, edge firewalls, and routed corporate networks—BGP is usually the better long-term architecture because upstream routers learn where service IPs are reachable.

Install Cilium Safely

When using kube-proxy replacement, Cilium must know how to reach the Kubernetes API server without depending on the kube-proxy service path. Define the control-plane endpoint explicitly and roll out in a maintenance window.

Example Helm values:

text
kubeProxyReplacement: true

k8sServiceHost: “10.10.0.10”
k8sServicePort: 6443

ipam:
mode: kubernetes

l2announcements:
enabled: true

externalIPs:
enabled: true

devices: “eno1”

L2 announcements require kube-proxy replacement and require the announcing interface to be included in Cilium’s managed-device configuration if you set devices explicitly. Before removing or disabling kube-proxy, validate service connectivity, DNS resolution, NodePort behavior, and control-plane API reachability from every node. docs.cilium

Useful verification commands:

bash
cilium status --wait

cilium connectivity test

kubectl -n kube-system get pods -l k8s-app=cilium -o wide

kubectl get ciliumnodes

kubectl get svc -A

Provide LoadBalancer IPs

Cilium LoadBalancer IPAM assigns an address from a defined pool to a Service of type LoadBalancer. Keep this pool outside DHCP ranges and do not use IPs assigned to nodes, printers, VPN clients, or other infrastructure.

text
apiVersion: cilium.io/v2alpha1
kind: CiliumLoadBalancerIPPool
metadata:
name: lan-services
spec:
blocks:
- start: 10.10.0.200
stop: 10.10.0.220

Next, tell Cilium which nodes/interfaces may announce those IPs on the LAN:

text
apiVersion: cilium.io/v2alpha1
kind: CiliumL2AnnouncementPolicy
metadata:
name: lan-lb-announcement
spec:
interfaces:
- eno1
loadBalancerIPs: true

Then expose an application:

text
apiVersion: v1
kind: Service
metadata:
name: demo-web
namespace: default
spec:
type: LoadBalancer
selector:
app: demo-web
ports:
- name: http
port: 80
targetPort: 8080

Cilium’s L2-aware load-balancer capability makes services reachable on a local network without BGP routing, while Cilium’s IPAM feature supplies the address assigned to the Service. Verify the assigned IP with kubectl get svc demo-web, then test it from a host on the same VLAN using curl http://<external-ip> and inspect ARP with ip neigh or arp -an.

Cilium enforces standard Kubernetes NetworkPolicy resources for L3/L4 rules and extends policy options to Layer 7, allowing restrictions based on HTTP methods, paths, hosts, gRPC methods, Kafka topics, and DNS names. Begin with an explicit default-deny policy in one noncritical namespace, then add only required ingress and egress paths; applying cluster-wide default deny before allowing DNS, ingress, observability, and required dependencies can break workloads.

Example namespace default deny:

text
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny
namespace: apps
spec:
podSelector: {}
policyTypes:
- Ingress
- Egress

After applying this, explicitly allow application-to-database traffic, ingress-controller-to-application traffic, and DNS egress to CoreDNS. Validate with a test pod rather than assuming the policy does what its YAML suggests.

When a bare-metal Service is unreachable, troubleshoot in this order:

  1. Confirm the Service has endpoints: kubectl get endpointslice -l kubernetes.io/service-name=demo-web.

  2. Test the Pod directly from inside the cluster.

  3. Test the ClusterIP from a different Pod.

  4. Test the LoadBalancer IP from a node, then from an external host on the same VLAN.

  5. Check that the address lies in the Cilium IP pool and is not used elsewhere.

  6. For L2, confirm the chosen interface, VLAN membership, ARP response, and switch port configuration.

  7. For BGP, verify peer sessions, advertised prefixes, route acceptance, and return routing.

  8. Inspect Cilium health and BPF/service state using cilium status, Hubble, and Cilium agent logs.

A strong first architecture exercise is to decide whether your external clients are on the same VLAN as the service IP range or reach it through a router. Describe that path—from a client IP to a Kubernetes Service—and identify whether L2 announcements or BGP would carry the service route.

[mai mult...]

GitOps Deployment with Argo CD and Kustomize

GitOps makes Git the desired-state record for Kubernetes: instead of an engineer applying manifests manually with kubectl, a controller continuously compares what is declared in Git with what exists in the cluster and reconciles differences. Argo CD performs that reconciliation; Kustomize renders reusable Kubernetes configuration for different environments without copying the entire manifest set.argo-cd.readthedocs+1.

For a practical platform setup, keep application source code separate from deployment configuration. Your CI pipeline builds, scans, and publishes an immutable container image; it then updates an image tag or digest in the GitOps repository through a reviewed pull request. Argo CD sees the approved Git change and deploys it.

Kustomize uses a base for shared manifests and overlays for environment-specific changes. An overlay points to the base and applies only the patches or additions needed for an environment.

text
platform-gitops/
├── apps/
│ └── inventory-api/
│ ├── base/
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ ├── kustomization.yaml
│ │ └── networkpolicy.yaml
│ └── overlays/
│ ├── dev/
│ │ ├── kustomization.yaml
│ │ └── deployment-patch.yaml
│ ├── staging/
│ │ ├── kustomization.yaml
│ │ └── deployment-patch.yaml
│ └── prod/
│ ├── kustomization.yaml
│ └── deployment-patch.yaml
└── argocd/
├── projects/
└── applications/

Keep environment differences narrowly scoped: image reference, replica count, resource requests/limits, ingress hostname, environment configuration, and possibly namespace. Avoid putting credentials in plain YAML; reference secrets managed through External Secrets, Sealed Secrets, SOPS, or another controlled mechanism.

The base defines stable application behavior shared by every environment:

text
# apps/inventory-api/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
– deployment.yaml
– service.yaml
– networkpolicy.yaml

commonLabels:
app.kubernetes.io/name: inventory-api
app.kubernetes.io/managed-by: argocd

text
# apps/inventory-api/base/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-api
spec:
replicas: 2
selector:
matchLabels:
app: inventory-api
template:
metadata:
labels:
app: inventory-api
spec:
securityContext:
runAsNonRoot: true
containers:
- name: api
image: registry.example.internal/inventory-api:0.0.0
ports:
- containerPort: 8080
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ["ALL"]
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
cpu: 500m
memory: 256Mi

Always render locally before committing:

bash
kubectl kustomize apps/inventory-api/overlays/dev

kubectl diff -k apps/inventory-api/overlays/dev

The first command shows the generated manifests; the second helps you reason about the live-cluster impact before Argo CD applies it.

Define Environment Overlays

An overlay imports the base, changes its namespace, and patches only what differs. Here is a production overlay:

text
# apps/inventory-api/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

namespace: inventory-prod

resources:
– ../../base

images:
– name: registry.example.internal/inventory-api
newName: registry.example.internal/inventory-api
newTag: “1.7.3”

patches:
– path: deployment-patch.yaml

text
# apps/inventory-api/overlays/prod/deployment-patch.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: inventory-api
spec:
replicas: 4
template:
spec:
containers:
- name: api
env:
- name: LOG_LEVEL
value: "info"

This separation is the main Kustomize benefit: the base contains universal configuration, while overlays supply only environment deltas. Prefer explicit patches over a growing chain of generators and transformations that make rendered output difficult to review.

An Argo CD Application tells Argo CD which repository path to render and the destination cluster and namespace to reconcile. The Application specification supports Git source details, Kustomize configuration, destination, and sync policy.

text
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: inventory-api-prod
namespace: argocd
spec:
project: production

source:
repoURL: https://git.example.internal/platform-gitops.git
targetRevision: main
path: apps/inventory-api/overlays/prod

destination:
server: https://kubernetes.default.svc
namespace: inventory-prod

syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
– CreateNamespace=true
– PruneLast=true
retry:
limit: 3
backoff:
duration: 10s
factor: 2
maxDuration: 3m

With automated, Argo CD applies desired-state changes found in Git. selfHeal: true tells it to correct out-of-band cluster changes, while prune: true allows it to remove resources that were deliberately removed from Git. These controls are powerful: in production, protect the Git branch, require review, restrict who can modify Application objects, and use Argo CD Projects to limit the repositories, clusters, namespaces, and resource kinds each team may deploy.

A secure GitOps delivery flow is:

  1. A developer commits application code.

  2. CI runs unit, integration, image, dependency, and policy checks.

  3. CI builds an image, scans it, signs it if your supply chain supports signing, and pushes it using an immutable tag or digest.

  4. CI opens a pull request that changes only the relevant overlay image tag or digest.

  5. A reviewer verifies the rendered manifest and approves the GitOps change.

  6. The merge becomes the desired state; Argo CD synchronizes it to the cluster.

  7. Argo CD reports health and sync status; alerts fire on degraded, missing, or out-of-sync workloads.

Avoid allowing CI to run broad kubectl apply commands directly against production. That creates a second deployment authority and makes drift harder to explain. With self-healing enabled, a direct manual scale operation or kubectl edit change is eventually reverted because Git remains authoritative.

GitOps rollback means reverting the commit that introduced the bad desired state, then allowing Argo CD to reconcile the previous revision. Keep image references immutable: a tag like latest undermines auditability because the same Git commit could produce different runtime artifacts.

When an application is OutOfSync or Degraded, troubleshoot in order:

  • Render locally: kubectl kustomize apps/inventory-api/overlays/prod.

  • Inspect the Argo CD diff to find the desired-versus-live mismatch.

  • Check Application events and sync history.

  • Inspect the affected Kubernetes object with kubectl describe.

  • Check Pod status, events, logs, readiness probes, resource quotas, and NetworkPolicies.

  • Confirm the Argo CD service account has only the RBAC permissions it needs.

  • Check whether a controller is legitimately mutating fields; configure ignore differences only for understood and documented cases.

For a production-quality next step, choose one application you would deploy and identify the three fields that should differ between dev and prod while the base stays identical.

[mai mult...]