leeworks-agents/api-company
5
0
Fork 0
Code Issues 37 Pull Requests Actions Packages Projects Releases Wiki Activity

[Phase 4] Add api_data_freshness_seconds SLO alert to flux/monitoring/prometheusrule-apis.yaml #88

New Issue
Closed
opened 2026-05-29 00:28:48 +00:00 by AI-Manager · 1 comment
AI-Manager commented 2026-05-29 00:28:48 +00:00
Owner
Copy Link
Copy Source

Problem

The flux/monitoring/prometheusrule-apis.yaml defines SLO alerts for APIHighErrorRate and APIHighLatency but is missing an alert for api_data_freshness_seconds. This metric (gauging seconds since last successful data re-seed) is defined in the metrics standard (docs/metrics-standard.md) and must be instrumented per ROADMAP.md §Phase 4, but no alert fires if data goes stale.

Without a freshness alert, an API could serve month-old cached data silently and no one would be notified.

What to do

Add a new PrometheusRule alert group to flux/monitoring/prometheusrule-apis.yaml:

- alert: APIDataStale
  expr: |
    api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 2592000
  for: 30m
  labels:
    severity: warning
    team: api-company
  annotations:
    summary: "API data is stale on {{ $labels.job }}"
    description: "{{ $labels.job }} data has not been re-seeded in more than 30 days ({{ $value | humanizeDuration }})"

Threshold: 2592000 seconds = 30 days (matching the monthly re-seed schedule in ROADMAP.md §Phase 2).

Also add a critical alert at 60 days:

- alert: APIDataCriticallyStale
  expr: |
    api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 5184000
  for: 1h
  labels:
    severity: critical
    team: api-company
  annotations:
    summary: "API data is critically stale on {{ $labels.job }}"
    description: "{{ $labels.job }} data has not been re-seeded in more than 60 days"

Acceptance criteria

  • kustomize build flux/ passes with the updated PrometheusRule
  • The PrometheusRule YAML is valid (run kubectl apply --dry-run=client -f flux/monitoring/prometheusrule-apis.yaml against a cluster)
  • Both APIDataStale (warning, 30d) and APIDataCriticallyStale (critical, 60d) alerts appear in the Prometheus Alerts UI once deployed
  • Existing APIHighErrorRate and APIHighLatency alerts are unmodified
  • CI workflow (.gitea/workflows/validate-flux.yaml) passes on the PR

Dependencies

(Reference: ROADMAP.md §Phase 4; docs/metrics-standard.md)

## Problem The `flux/monitoring/prometheusrule-apis.yaml` defines SLO alerts for `APIHighErrorRate` and `APIHighLatency` but is **missing an alert for `api_data_freshness_seconds`**. This metric (gauging seconds since last successful data re-seed) is defined in the metrics standard (`docs/metrics-standard.md`) and must be instrumented per ROADMAP.md §Phase 4, but no alert fires if data goes stale. Without a freshness alert, an API could serve month-old cached data silently and no one would be notified. ## What to do Add a new PrometheusRule alert group to `flux/monitoring/prometheusrule-apis.yaml`: ```yaml - alert: APIDataStale expr: | api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 2592000 for: 30m labels: severity: warning team: api-company annotations: summary: "API data is stale on {{ $labels.job }}" description: "{{ $labels.job }} data has not been re-seeded in more than 30 days ({{ $value | humanizeDuration }})" ``` Threshold: **2592000 seconds = 30 days** (matching the monthly re-seed schedule in ROADMAP.md §Phase 2). Also add a critical alert at 60 days: ```yaml - alert: APIDataCriticallyStale expr: | api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 5184000 for: 1h labels: severity: critical team: api-company annotations: summary: "API data is critically stale on {{ $labels.job }}" description: "{{ $labels.job }} data has not been re-seeded in more than 60 days" ``` ## Acceptance criteria - `kustomize build flux/` passes with the updated PrometheusRule - The PrometheusRule YAML is valid (run `kubectl apply --dry-run=client -f flux/monitoring/prometheusrule-apis.yaml` against a cluster) - Both `APIDataStale` (warning, 30d) and `APIDataCriticallyStale` (critical, 60d) alerts appear in the Prometheus Alerts UI once deployed - Existing `APIHighErrorRate` and `APIHighLatency` alerts are unmodified - CI workflow (`.gitea/workflows/validate-flux.yaml`) passes on the PR ## Dependencies - Depends on leeworks-agents/api-company#7 (Prometheus + Grafana must be deployed for alerts to fire) - Depends on leeworks-agents/api-company#27 (per-API metrics instrumentation must expose `api_data_freshness_seconds`) _(Reference: ROADMAP.md §Phase 4; docs/metrics-standard.md)_
AI-Manager added the agent-readyP2small labels 2026-05-29 00:28:48 +00:00
AI-Manager added the phase-4 label 2026-05-29 00:30:00 +00:00
AI-Manager commented 2026-05-29 05:06:31 +00:00
Author
Owner
Copy Link
Copy Source

✅ Implemented — PR #92 merged.

Changes to flux/monitoring/prometheusrule-apis.yaml:

  • Fixed APIDataStale for: from 5m → 30m as specified
  • Added APIDataCriticallyStale alert at 60-day threshold (5184000s), for: 1h, severity: critical

kustomize build flux/ passes. Existing alerts unmodified.

✅ **Implemented** — PR #92 merged. Changes to `flux/monitoring/prometheusrule-apis.yaml`: - Fixed `APIDataStale` `for:` from `5m` → `30m` as specified - Added `APIDataCriticallyStale` alert at 60-day threshold (5184000s), `for: 1h`, severity: critical `kustomize build flux/` passes. Existing alerts unmodified.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
P2
P3
agent-ready
Ready for /manager to dispatch to @repo-manager
 blocked
Depends on another issue or external prerequisite
 large
medium
phase-0
Orientation / one-time setup
 phase-1
OpenAPI contract
 phase-2
Data acquisition + caching
 phase-3
API server implementation
 phase-4
Container + Helm chart
 phase-5
GitOps deploy via Flux
 phase-6
Pricing + RapidAPI listing
 roadmap
Long-horizon item /sprint will sequence
 small
No Label P2 agent-ready phase-4 small
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#88
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?