Alert enrichment: send Prometheus Alertmanager alerts to Slack with contextual runbooks (webhook + small webhook enricher)

Situatie

Summary

Alertmanager will POST alerts to a small webhook service you host (/alert-enrich) that:

parses incoming alerts,
appends a runbook URL or short remediation steps based on alert labels (e.g., alertname, instance),
forwards the enriched message to Slack via an Incoming Webhook.

Components

Alertmanager config change to call webhook.
A tiny Python Flask service that enriches alerts and forwards to Slack.
Slack Incoming Webhook URL stored securely.

Solutie

Alertmanager snippet

Add to receivers:

Then route important alerts to webhook-enrich.

Python enricher (alert_enricher.py)

from flask import Flask, request, jsonify

import requests

import os

app = Flask(__name__)
SLACK_WEBHOOK = os.environ[‘SLACK_WEBHOOK’]

# simple runbook map; in real life load from DB or files
RUNBOOKS = {
“HighCpu”: “https://kb.example.com/runbooks/high_cpu”,
“PGDown”: “https://kb.example.com/runbooks/postgres_down”,
}

def build_slack_message(alert):
labels = alert.get(‘labels’, {})
name = labels.get(‘alertname’, ‘Unknown’)
runbook = RUNBOOKS.get(name, “https://kb.example.com/runbooks/general”)
summary = alert.get(‘annotations’, {}).get(‘summary’, ”)
desc = alert.get(‘annotations’, {}).get(‘description’, ”)
instance = labels.get(‘instance’, ‘unknown’)
msg = {
“text”: f”*Alert:* {name}\n*Instance:* {instance}\n*Summary:* {summary}\n{desc}\n*Runbook:* {runbook}”
}
return msg

@app.route(‘/alert-enrich’, methods=[‘POST’])
def enrich():
data = request.get_json()
alerts = data.get(‘alerts’, [])
for a in alerts:
msg = build_slack_message(a)
requests.post(SLACK_WEBHOOK, json=msg, timeout=10)
return jsonify({“status”: “ok”}), 200

if __name__ == ‘__main__’:
app.run(host=‘0.0.0.0’, port=5000)

Steps

Create Slack Incoming Webhook and save URL.
Deploy alert_enricher.py on a small container/VPS, set SLACK_WEBHOOK env var.
Update Alertmanager receivers to point to the public URL of the enricher.
Add runbook mapping (either static like above, or store runbooks in a repo / DB and fetch dynamically).
Test by sending a synthetic alert through Alertmanager API.

Enhancements

Add deduplication & rate-limiting in the enricher.
Include escalation links and PagerDuty integration.
Support richer Slack blocks with buttons (e.g., I am investigating).
Attach recent logs by querying Loki or ELK (if available) and embedding a short excerpt.

Troubleshooting & notes

Ensure Alertmanager can reach the webhook (network, firewall)
Protect the webhook endpoint (IP allowlist or a shared secret)
For high volume, use a queue (RabbitMQ/SQS) to avoid blocking Alertmanager retries.

Tip solutie

Permanent

Follow Us