Send Alertmanager notifications to Redis

Use this optional example when another application should consume alerts from a Redis Stream. The path is Prometheus → Alertmanager → HTTP webhook → Redis. It complements email and Slack; it does not change simple mode, cron, or the dependencies of the monit-docker agent.

The small receiver uses HTTPdis and its sonicprobe server, then calls DWhoNotifierRedis.send() synchronously. It acknowledges a notification only after Redis returns the ID of the appended entry.

Start the example

Docker Compose with build support and access to Python package downloads is required. From examples/monitoring, prepare a private directory. For a new Redis-only setup, run:

umask 077
mkdir -p notifications.local
chmod 700 notifications.local
cp alertmanager.redis.yml notifications.local/alertmanager.yml
od -An -N32 -tx1 /dev/urandom | tr -d ' \n' > notifications.local/redis_webhook_token
printf '%s\n' 'redis://redis:6379/0' > notifications.local/redis_url
# The shared notification overlay requires both files, even when unused.
touch notifications.local/smtp_password notifications.local/slack_webhook_url
chmod 644 notifications.local/alertmanager.yml notifications.local/redis_webhook_token \
  notifications.local/redis_url notifications.local/smtp_password notifications.local/slack_webhook_url
sh start.sh --redis

The parent directory is private on the host; individual files are readable by the unprivileged container user through their mounts. Keep the directory out of git, images and distribution archives. The repository already excludes it.

If email or Slack is already configured, keep your existing config and secrets. Create only the two Redis secret files, and copy the webhook_configs block from alertmanager.redis.yml into the receiver that should also write to Redis. Start with --redis (which also enables the notifications overlay). Each receiver integration has its own delivery outcome; sending to several destinations is not an atomic transaction.

The receiver is built locally. Its requirements.txt pins HTTPdis, sonicprobe, redis-py, and dwho 0.3.61 from PyPI. This release supplies the strict send() and Streams APIs. To update an existing receiver image after pulling these examples, run from examples/monitoring:

docker compose -f compose.yaml -f compose.notifications.yaml -f compose.redis.yaml \
  build redis-webhook
sh start.sh --redis

Read notifications

Run these commands from examples/monitoring:

docker compose -f compose.yaml -f compose.notifications.yaml -f compose.redis.yaml \
  exec redis redis-cli XRANGE monit-docker:alerts - + COUNT 5

Each stream entry contains one field named payload: a JSON string containing the complete Alertmanager webhook group, including version, groupKey, status, alerts, labels and annotations. One entry may contain several alerts. Both firing and resolved messages are sent. Repeat notifications also produce entries. Interpret resolution together with collection health, since missing container measurements can clear a memory alert.

Consumers may use XREAD, or XREADGROUP and XACK for consumer groups. Consumer implementation and acknowledgment are separate from this example’s HTTP receipt. Choose an idempotency policy for your consumer: retries after a lost response can produce duplicates, and a fingerprint alone cannot distinguish firing, resolved and repeat events.

Retention and persistence

The stream defaults to monit-docker:alerts and the latest 10,000 entries. Set REDIS_STREAM and REDIS_STREAM_MAXLEN in the monitoring .env to change these values, then recreate redis-webhook. The limit is exact and must be a positive integer. Trimming also removes entries that a slow consumer has not processed. This is an entry count, not a byte or time limit; budget Redis memory for your payload sizes.

Redis stores AOF data in the redis-notifications-data named volume with appendfsync everysec. Normal restarts preserve history; a crash can lose about one second of writes. A successful HTTP response confirms Redis accepted the entry, not that it was flushed to disk or processed by a consumer. See Redis persistence. docker compose down --volumes deletes this history.

Delivery and operation

  • POST /alerts requires the Bearer token mounted in both Alertmanager and the receiver. Keep max_alerts: 0 to avoid truncation. Requests are limited to 1 MiB.

  • The receiver returns 202 with the stream and entry ID after a successful append. Redis errors return 503; Alertmanager retries failed server requests. Invalid payloads return a client error and require fixing the source/config.

  • /healthz checks the HTTP process; /readyz checks Redis connectivity. Redis failure does not stop the process. Delivery resumes when connectivity returns.

  • Redis socket connect/read timeouts default to three seconds each; URL query parameters can change them to positive values up to five seconds. Redis client timeout retries are disabled; Alertmanager owns notification retries.

  • Redis and the receiver have no published host ports or Docker socket access. The bundled Redis is unauthenticated on an internal Docker network shared only with the receiver. Do not expose it publicly. A different Redis service needs its own networking, authentication and persistence configuration; redis_url accepts redis:// or rediss:// URLs, including credentials.

After changing files, recreate the services to refresh individually mounted secrets (both services read the webhook token at startup):

docker compose -f compose.yaml -f compose.notifications.yaml -f compose.redis.yaml \
  up -d --force-recreate --no-deps redis-webhook alertmanager

Use the same three -f arguments with ps, logs --tail=100, or down. Alertmanager remains available on http://127.0.0.1:9093 by default. Check its notification failures as well as the receiver’s readiness; an outage of the notification stack cannot reliably report itself through that same stack.

Regression coverage

CI builds the actual receiver image and runs the three Compose overlays. It checks Prometheus discovers Alertmanager, grouped firing/resolved delivery, authentication, malformed/oversized requests, a Redis outage with Alertmanager retry after recovery, AOF history across a Redis restart, and exact stream retention. Tests use disposable local services and dummy credentials.

Event journal

See the audit journal guide for persistent events, UTC timestamps, actors, delivery-result boundaries, JSONL/CSV export and external forwarding.