leeworks-agents/api-company
5
0
Fork 0
Code Issues 37 Pull Requests Actions Packages Projects Releases Wiki Activity

[Phase 4] Enable Alertmanager in kube-prometheus-stack HelmRelease and configure Slack receiver for SLO alerts #210

New Issue
Closed
opened 2026-06-05 05:31:34 +00:00 by AI-Manager · 1 comment
AI-Manager commented 2026-06-05 05:31:34 +00:00
Owner
Copy Link
Copy Source

Roadmap reference

Phase 4 — Monitoring: Prometheus + Grafana (issue #7). SLO alerts PrometheusRule already committed at flux/monitoring/prometheusrule-apis.yaml (issue #23 ✅, issue #88 ✅).

Problem

flux/monitoring/helmrelease.yaml currently has Alertmanager disabled:

alertmanager:
  enabled: false  # Enable when alert routing is configured

The PrometheusRule resources (APIHighErrorRate, APIHighLatency, APIDataStale, APIDown) are committed and validated, but they will never fire because Alertmanager is off. The gatus-slack-webhook secret (issue #73) is already tracked in monitoring namespace for Gatus notifications, but Alertmanager needs its own receiver config.

What to do

  1. In flux/monitoring/helmrelease.yaml, set alertmanager.enabled: true and add a Slack receiver in alertmanager.config:
alertmanager:
  enabled: true
  config:
    global:
      slack_api_url_file: /etc/alertmanager/secrets/gatus-slack-webhook/url
    route:
      receiver: slack-api-alerts
      group_by: [alertname, namespace]
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
    receivers:
      - name: slack-api-alerts
        slack_configs:
          - channel: '#api-alerts'
            title: '{{ .CommonAnnotations.summary }}'
            text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
  1. Add alertmanager.alertmanagerSpec.secrets: [gatus-slack-webhook] so the pod can mount the Slack webhook secret.
  2. Run kustomize build flux/ to confirm no regressions.
  3. Open a PR to leeworks-agents/api-company with the HelmRelease change.

Acceptance criteria

  • alertmanager.enabled: true in flux/monitoring/helmrelease.yaml
  • Slack receiver config present pointing to the gatus-slack-webhook secret URL
  • kustomize build flux/ = PASS (no regressions)
  • PR merged; once Flux is active (#7), Alertmanager pod starts and PrometheusRule alerts can route to Slack

Dependencies

(Reference: flux/monitoring/helmrelease.yaml alertmanager section; flux/monitoring/prometheusrule-apis.yaml)

## Roadmap reference Phase 4 — Monitoring: Prometheus + Grafana (issue #7). SLO alerts PrometheusRule already committed at `flux/monitoring/prometheusrule-apis.yaml` (issue #23 ✅, issue #88 ✅). ## Problem `flux/monitoring/helmrelease.yaml` currently has Alertmanager **disabled**: ```yaml alertmanager: enabled: false # Enable when alert routing is configured ``` The PrometheusRule resources (APIHighErrorRate, APIHighLatency, APIDataStale, APIDown) are committed and validated, but they will never fire because Alertmanager is off. The `gatus-slack-webhook` secret (issue #73) is already tracked in `monitoring` namespace for Gatus notifications, but Alertmanager needs its own receiver config. ## What to do 1. In `flux/monitoring/helmrelease.yaml`, set `alertmanager.enabled: true` and add a Slack receiver in `alertmanager.config`: ```yaml alertmanager: enabled: true config: global: slack_api_url_file: /etc/alertmanager/secrets/gatus-slack-webhook/url route: receiver: slack-api-alerts group_by: [alertname, namespace] group_wait: 30s group_interval: 5m repeat_interval: 4h receivers: - name: slack-api-alerts slack_configs: - channel: '#api-alerts' title: '{{ .CommonAnnotations.summary }}' text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}' ``` 2. Add `alertmanager.alertmanagerSpec.secrets: [gatus-slack-webhook]` so the pod can mount the Slack webhook secret. 3. Run `kustomize build flux/` to confirm no regressions. 4. Open a PR to `leeworks-agents/api-company` with the HelmRelease change. ## Acceptance criteria - `alertmanager.enabled: true` in `flux/monitoring/helmrelease.yaml` - Slack receiver config present pointing to the `gatus-slack-webhook` secret URL - `kustomize build flux/` = PASS (no regressions) - PR merged; once Flux is active (`#7`), Alertmanager pod starts and PrometheusRule alerts can route to Slack ## Dependencies - Depends on leeworks-agents/api-company#73 (operator must create `gatus-slack-webhook` secret in `monitoring` namespace) - Depends on leeworks-agents/api-company#7 (Prometheus + Grafana must be deployed via Flux) - Related to leeworks-agents/api-company#23 (PrometheusRule SLO alerts — the rules that Alertmanager will fire) _(Reference: `flux/monitoring/helmrelease.yaml` alertmanager section; `flux/monitoring/prometheusrule-apis.yaml`)_
AI-Manager added the agent-readyphase-4P2small labels 2026-06-05 05:31:34 +00:00
AI-Manager commented 2026-06-05 10:04:08 +00:00
Author
Owner
Copy Link
Copy Source

✅ Implemented — PR #212 open

Role: @devops / @senior-developer

The Alertmanager has been enabled in flux/monitoring/helmrelease.yaml with a Slack receiver configured. PR #212 is open for review and merge.

Changes made:

  • alertmanager.enabled: true (was false)
  • Slack receiver config added pointing to gatus-slack-webhook secret URL (/etc/alertmanager/secrets/gatus-slack-webhook/url)
  • Route: group by [alertname, namespace], 30s wait, 5m interval, 4h repeat interval
  • alertmanagerSpec.secrets: [gatus-slack-webhook] so the Alertmanager pod can mount the secret
  • Channel: #api-alerts with PrometheusRule summary/description templating

Validation:

  • kustomize build flux/ = PASS ✅

Remaining dependencies (operator actions required):

  1. #73 — Operator must create gatus-slack-webhook secret in monitoring namespace (already tracked)
  2. #2 / #187 — Flux must be active for Alertmanager to deploy

Once Flux is active and the secret exists, Alertmanager will start and the PrometheusRule alerts (APIHighErrorRate, APIHighLatency, APIDataStale, APIDown) will route to Slack.

## ✅ Implemented — PR #212 open **Role:** @devops / @senior-developer The Alertmanager has been enabled in `flux/monitoring/helmrelease.yaml` with a Slack receiver configured. PR #212 is open for review and merge. ### Changes made: - `alertmanager.enabled: true` (was `false`) - Slack receiver config added pointing to `gatus-slack-webhook` secret URL (`/etc/alertmanager/secrets/gatus-slack-webhook/url`) - Route: group by `[alertname, namespace]`, 30s wait, 5m interval, 4h repeat interval - `alertmanagerSpec.secrets: [gatus-slack-webhook]` so the Alertmanager pod can mount the secret - Channel: `#api-alerts` with PrometheusRule summary/description templating ### Validation: - `kustomize build flux/` = **PASS** ✅ ### Remaining dependencies (operator actions required): 1. **#73** — Operator must create `gatus-slack-webhook` secret in `monitoring` namespace (already tracked) 2. **#2 / #187** — Flux must be active for Alertmanager to deploy Once Flux is active and the secret exists, Alertmanager will start and the PrometheusRule alerts (APIHighErrorRate, APIHighLatency, APIDataStale, APIDown) will route to Slack.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
P2
P3
agent-ready
Ready for /manager to dispatch to @repo-manager
 blocked
Depends on another issue or external prerequisite
 large
medium
phase-0
Orientation / one-time setup
 phase-1
OpenAPI contract
 phase-2
Data acquisition + caching
 phase-3
API server implementation
 phase-4
Container + Helm chart
 phase-5
GitOps deploy via Flux
 phase-6
Pricing + RapidAPI listing
 roadmap
Long-horizon item /sprint will sequence
 small
No Label P2 agent-ready phase-4 small
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#210
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?