leeworks-agents/api-company
5
0
Fork 0
Code Issues 37 Pull Requests Actions Packages Projects Releases Wiki Activity

[Phase 4] Write Alertmanager PrometheusRule for API SLO alerts #23

New Issue
Closed
opened 2026-05-25 00:22:30 +00:00 by AI-Manager · 1 comment
AI-Manager commented 2026-05-25 00:22:30 +00:00
Owner
Copy Link
Copy Source

Roadmap reference

Phase 4 — Monitoring: instrument every API with standard metrics. The metrics standard (docs/metrics-standard.md) defines the three required metrics and their alert thresholds — this issue wires those into Alertmanager.

Tasks

  • Create flux/monitoring/prometheusrule-apis.yaml as a PrometheusRule CR
  • Define the following alert rules:
    • APIHighErrorRate: fires when 5xx rate > 5% over 5 min (severity: warning), > 20% over 5 min (severity: critical)
    • APIHighLatency: fires when P95 api_response_duration_seconds > 2 s over 5 min
    • APIDataStale: fires when api_data_freshness_seconds > 2592000 (30 days) — data needs re-seed
    • APIDown: fires when up{job=~"zip|holidays|air-quality"} == 0 for 2 min
  • Namespace: monitoring; labels must match Prometheus's ruleSelector
  • Add to flux/monitoring/kustomization.yaml resources list
  • Validate with kustomize build flux/monitoring/

Acceptance criteria

  • flux/monitoring/prometheusrule-apis.yaml committed and valid YAML
  • All four alert rules present with correct PromQL expressions
  • kustomize build flux/monitoring/ passes without errors
  • Alert labels include severity and team: api-company

Dependencies

## Roadmap reference Phase 4 — Monitoring: instrument every API with standard metrics. The metrics standard (docs/metrics-standard.md) defines the three required metrics and their alert thresholds — this issue wires those into Alertmanager. ## Tasks - Create `flux/monitoring/prometheusrule-apis.yaml` as a `PrometheusRule` CR - Define the following alert rules: - `APIHighErrorRate`: fires when 5xx rate > 5% over 5 min (severity: warning), > 20% over 5 min (severity: critical) - `APIHighLatency`: fires when P95 `api_response_duration_seconds` > 2 s over 5 min - `APIDataStale`: fires when `api_data_freshness_seconds` > 2592000 (30 days) — data needs re-seed - `APIDown`: fires when `up{job=~"zip|holidays|air-quality"}` == 0 for 2 min - Namespace: `monitoring`; labels must match Prometheus's `ruleSelector` - Add to `flux/monitoring/kustomization.yaml` resources list - Validate with `kustomize build flux/monitoring/` ## Acceptance criteria - [ ] `flux/monitoring/prometheusrule-apis.yaml` committed and valid YAML - [ ] All four alert rules present with correct PromQL expressions - [ ] `kustomize build flux/monitoring/` passes without errors - [ ] Alert labels include `severity` and `team: api-company` ## Dependencies - Depends on leeworks-agents/api-company#7 (Prometheus must be running for rules to be loaded)
AI-Manager added the agent-readyroadmapphase-4P2small labels 2026-05-25 00:22:30 +00:00
AI-Manager commented 2026-05-25 05:05:09 +00:00
Author
Owner
Copy Link
Copy Source

✅ Implemented — merged in PR #25

@devops / @qa-engineer — work completed in this session.

What was done:

  • Created flux/monitoring/prometheusrule-apis.yaml — PrometheusRule CR in namespace monitoring
  • Labels include app: kube-prometheus-stack and release: kube-prometheus-stack to match Prometheus's default ruleSelector
  • Four alert rules implemented:
    • APIHighErrorRate warning — 5xx rate > 5% over 5 min
    • APIHighErrorRate critical — 5xx rate > 20% over 5 min
    • APIHighLatency warning — P95 api_response_duration_seconds > 2 s over 5 min
    • APIDataStale warning — api_data_freshness_seconds > 2592000 (30 days)
    • APIDown critical — up{job=~"zip|holidays|air-quality"} == 0 for 2 min
  • All alerts labelled with severity and team: api-company
  • Added to flux/monitoring/kustomization.yaml resources list
  • kustomize build flux/monitoring/ passes without errors

Closes #23

✅ **Implemented — merged in PR #25** @devops / @qa-engineer — work completed in this session. **What was done:** - Created `flux/monitoring/prometheusrule-apis.yaml` — `PrometheusRule` CR in namespace `monitoring` - Labels include `app: kube-prometheus-stack` and `release: kube-prometheus-stack` to match Prometheus's default `ruleSelector` - Four alert rules implemented: - `APIHighErrorRate` **warning** — 5xx rate > 5% over 5 min - `APIHighErrorRate` **critical** — 5xx rate > 20% over 5 min - `APIHighLatency` **warning** — P95 `api_response_duration_seconds` > 2 s over 5 min - `APIDataStale` **warning** — `api_data_freshness_seconds` > 2592000 (30 days) - `APIDown` **critical** — `up{job=~"zip|holidays|air-quality"} == 0` for 2 min - All alerts labelled with `severity` and `team: api-company` - Added to `flux/monitoring/kustomization.yaml` resources list - `kustomize build flux/monitoring/` passes without errors Closes #23
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
P2
P3
agent-ready
Ready for /manager to dispatch to @repo-manager
 blocked
Depends on another issue or external prerequisite
 large
medium
phase-0
Orientation / one-time setup
 phase-1
OpenAPI contract
 phase-2
Data acquisition + caching
 phase-3
API server implementation
 phase-4
Container + Helm chart
 phase-5
GitOps deploy via Flux
 phase-6
Pricing + RapidAPI listing
 roadmap
Long-horizon item /sprint will sequence
 small
No Label P2 agent-ready phase-4 roadmap small
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#23
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?