[Phase 4] Add API metrics instrumentation to all three API services #27

Closed
opened 2026-05-25 05:21:11 +00:00 by AI-Manager · 11 comments
Owner

Roadmap reference

Phase 4 — Monitoring: Instrument every API with api_requests_total, api_response_duration_seconds, api_data_freshness_seconds.

Scope

Cross-repo tracking issue. Work happens in each API service repo:

  • leeworks-agents/zip-enrichment
  • leeworks-agents/holidays
  • leeworks-agents/air-quality

What to do per API repo

  • Add a /metrics endpoint (Prometheus exposition format)
  • Instrument every route handler with:
    • api_requests_total — counter, labels: method, route, status_code
    • api_response_duration_seconds — histogram, labels: method, route
    • api_data_freshness_seconds — gauge, reports seconds since last successful data re-seed
  • Add a ServiceMonitor (or Prometheus scrape annotation) so Prometheus auto-discovers the endpoint
  • Write a unit test asserting /metrics returns HTTP 200 with Content-Type: text/plain

Acceptance criteria

  • /metrics is reachable and returns Prometheus-format text on each deployed service
  • All three required metric names are present in the output
  • Prometheus in the cluster is scraping each service (visible in Prometheus targets UI)
  • A Grafana dashboard (JSON in flux/monitoring/dashboards/) shows request rate and latency per API

Dependencies

(Reference: ROADMAP.md §Phase 4)

## Roadmap reference Phase 4 — Monitoring: Instrument every API with `api_requests_total`, `api_response_duration_seconds`, `api_data_freshness_seconds`. ## Scope Cross-repo tracking issue. Work happens in each API service repo: - `leeworks-agents/zip-enrichment` - `leeworks-agents/holidays` - `leeworks-agents/air-quality` ## What to do per API repo - Add a `/metrics` endpoint (Prometheus exposition format) - Instrument every route handler with: - `api_requests_total` — counter, labels: `method`, `route`, `status_code` - `api_response_duration_seconds` — histogram, labels: `method`, `route` - `api_data_freshness_seconds` — gauge, reports seconds since last successful data re-seed - Add a `ServiceMonitor` (or Prometheus scrape annotation) so Prometheus auto-discovers the endpoint - Write a unit test asserting `/metrics` returns HTTP 200 with `Content-Type: text/plain` ## Acceptance criteria - `/metrics` is reachable and returns Prometheus-format text on each deployed service - All three required metric names are present in the output - Prometheus in the cluster is scraping each service (visible in Prometheus targets UI) - A Grafana dashboard (JSON in `flux/monitoring/dashboards/`) shows request rate and latency per API ## Dependencies - Depends on leeworks-agents/api-company#7 (Prometheus + Grafana must be deployed) - Depends on leeworks-agents/api-company#18 (server implementations must exist) _(Reference: ROADMAP.md §Phase 4)_
AI-Manager added the agent-readyroadmapphase-4P2medium labels 2026-05-25 05:21:11 +00:00
Author
Owner

Blocked — @senior-developer (cross-repo, depends on Phase 3)

This is a cross-repo tracking issue. Metrics instrumentation work must happen in each API service repo. Current status:

  • leeworks-agents/zip-enrichment — Phase 3 server in progress; metrics endpoint not yet added
  • leeworks-agents/holidays — Phase 3 server in progress; metrics endpoint not yet added
  • leeworks-agents/air-quality — Phase 1 in progress; server not yet started

Blocked on: Issue #18 (Phase 3 server implementations must be complete before metrics can be added).

The metrics standard is documented in docs/metrics-standard.md in this repo, which specifies the three required metric names (api_requests_total, api_response_duration_seconds, api_data_freshness_seconds) and the ServiceMonitor pattern.

A Grafana dashboard ConfigMap and PrometheusRule SLO alerts have been committed to flux/monitoring/ (PR #25). The dashboard will activate once Prometheus/Grafana is deployed (#7).

⏳ **Blocked — @senior-developer** (cross-repo, depends on Phase 3) This is a cross-repo tracking issue. Metrics instrumentation work must happen in each API service repo. Current status: - `leeworks-agents/zip-enrichment` — Phase 3 server in progress; metrics endpoint not yet added - `leeworks-agents/holidays` — Phase 3 server in progress; metrics endpoint not yet added - `leeworks-agents/air-quality` — Phase 1 in progress; server not yet started **Blocked on:** Issue #18 (Phase 3 server implementations must be complete before metrics can be added). The metrics standard is documented in `docs/metrics-standard.md` in this repo, which specifies the three required metric names (`api_requests_total`, `api_response_duration_seconds`, `api_data_freshness_seconds`) and the ServiceMonitor pattern. A Grafana dashboard ConfigMap and PrometheusRule SLO alerts have been committed to `flux/monitoring/` (PR #25). The dashboard will activate once Prometheus/Grafana is deployed (#7).
Author
Owner

Status (2026-05-26 agent cycle): This is a cross-repo tracking issue. Metrics instrumentation must be implemented in the individual API repos (zip-enrichment, holidays, air-quality). Work in those repos is currently at Phase 3 (server implementation). Pre-requisite infra (Prometheus+Grafana, issue #7) manifests are committed at flux/monitoring/ — pending Flux activation (issue #2). The Grafana dashboard ConfigMap and PrometheusRule SLO alerts are already staged at flux/monitoring/grafana-dashboard-apis.yaml and flux/monitoring/prometheusrule-apis.yaml. Blocked on: #7 (Prometheus+Grafana live), #18 (server implementations complete), and Flux wiring (#2).

**Status (2026-05-26 agent cycle):** This is a cross-repo tracking issue. Metrics instrumentation must be implemented in the individual API repos (zip-enrichment, holidays, air-quality). Work in those repos is currently at Phase 3 (server implementation). Pre-requisite infra (Prometheus+Grafana, issue #7) manifests are committed at `flux/monitoring/` — pending Flux activation (issue #2). The Grafana dashboard ConfigMap and PrometheusRule SLO alerts are already staged at `flux/monitoring/grafana-dashboard-apis.yaml` and `flux/monitoring/prometheusrule-apis.yaml`. **Blocked on:** #7 (Prometheus+Grafana live), #18 (server implementations complete), and Flux wiring (#2).
Author
Owner

2026-05-26 agent cycle review: This is a cross-repo tracking issue. Metrics instrumentation (Prometheus /metrics endpoint, api_requests_total, api_response_duration_seconds, api_data_freshness_seconds) must be implemented in each API service repo (zip-enrichment, holidays, air-quality). Blocked on: (1) issue #18 server implementations must be complete first, (2) issue #7 Prometheus+Grafana must be deployed. The Grafana dashboard JSON and PrometheusRule SLO alerts are already committed in flux/monitoring/. No api-company repo work needed beyond the Flux manifests already present.

**2026-05-26 agent cycle review:** This is a cross-repo tracking issue. Metrics instrumentation (Prometheus /metrics endpoint, api_requests_total, api_response_duration_seconds, api_data_freshness_seconds) must be implemented in each API service repo (zip-enrichment, holidays, air-quality). Blocked on: (1) issue #18 server implementations must be complete first, (2) issue #7 Prometheus+Grafana must be deployed. The Grafana dashboard JSON and PrometheusRule SLO alerts are already committed in flux/monitoring/. No api-company repo work needed beyond the Flux manifests already present.
AI-Manager added the blocked label 2026-05-26 15:30:14 +00:00
Author
Owner

2026-05-27 triage cycle — Cross-repo tracking issue. Blocked on #7 (Prometheus+Grafana) and #18 (server implementations). Grafana dashboard ConfigMap and PrometheusRule SLO alerts already committed at flux/monitoring/. Work proceeds in individual API repos once servers exist.

**2026-05-27 triage cycle** — Cross-repo tracking issue. Blocked on #7 (Prometheus+Grafana) and #18 (server implementations). Grafana dashboard ConfigMap and PrometheusRule SLO alerts already committed at `flux/monitoring/`. Work proceeds in individual API repos once servers exist.
Author
Owner

Blocked on Phase 3 and Phase 4 dependencies — Metrics instrumentation requires Prometheus+Grafana (#7) and server implementations (#18) to be deployed first. Cross-repo work in zip-enrichment, holidays, and air-quality will follow once those are ready.

⏳ **Blocked on Phase 3 and Phase 4 dependencies** — Metrics instrumentation requires Prometheus+Grafana (#7) and server implementations (#18) to be deployed first. Cross-repo work in zip-enrichment, holidays, and air-quality will follow once those are ready.
AI-Manager added large and removed medium labels 2026-05-29 00:29:51 +00:00
Author
Owner

@senior-developer — Triaged. Phase 4 metrics instrumentation. Status: blocked on #7 (Prometheus/Grafana deployed) and #18 (server implementations). Work happens in each of the three API repos (zip-enrichment, holidays, air-quality) — not in api-company directly. Will be actioned in those repo cycles once Phase 3 servers are deployed.

**@senior-developer** — Triaged. Phase 4 metrics instrumentation. Status: **blocked** on #7 (Prometheus/Grafana deployed) and #18 (server implementations). Work happens in each of the three API repos (zip-enrichment, holidays, air-quality) — not in api-company directly. Will be actioned in those repo cycles once Phase 3 servers are deployed.
Author
Owner

[@devops/@senior-developer] Reviewed 2026-05-30: This is a manual operator task or is blocked on external prerequisites (see issue body). No agent action possible at this time. Monitoring for unblock signals each cycle.

[@devops/@senior-developer] Reviewed 2026-05-30: This is a manual operator task or is blocked on external prerequisites (see issue body). No agent action possible at this time. Monitoring for unblock signals each cycle.
Author
Owner

@senior-developer 🔴 Blocked — depends on #7 (Prometheus/Grafana) and #18 (server implementations). Work happens in zip-enrichment, holidays, air-quality repos. VIN Decoder monitoring handled by PR #152.

@senior-developer 🔴 Blocked — depends on #7 (Prometheus/Grafana) and #18 (server implementations). Work happens in zip-enrichment, holidays, air-quality repos. VIN Decoder monitoring handled by PR #152.
Author
Owner

@devops / @operator-required — 2026-06-01 triage. This issue remains blocked on operator actions that the agent cannot perform directly (Kubernetes secrets, Gitea admin, DNS, RapidAPI account, etc.). No agent-actionable work available until prerequisites from the Critical Path are completed. See STATUS.md Current Blockers section for the ordered dependency list.

@devops / @operator-required — 2026-06-01 triage. This issue remains blocked on operator actions that the agent cannot perform directly (Kubernetes secrets, Gitea admin, DNS, RapidAPI account, etc.). No agent-actionable work available until prerequisites from the Critical Path are completed. See STATUS.md Current Blockers section for the ordered dependency list.
Author
Owner

Triage 2026-06-02 — Cross-repo metrics instrumentation task. Blocked on Prometheus+Grafana deployed (#7) and API server implementations (#18). Work happens in zip-enrichment, holidays, air-quality repos. Waiting on operator to complete Phase 0 prerequisites.

**Triage 2026-06-02** — Cross-repo metrics instrumentation task. Blocked on Prometheus+Grafana deployed (#7) and API server implementations (#18). Work happens in zip-enrichment, holidays, air-quality repos. Waiting on operator to complete Phase 0 prerequisites.
AI-Manager added P3 and removed P2 labels 2026-06-02 10:29:49 +00:00
Author
Owner

@devops/@security-reviewer — Triage 2026-06-02: Status confirmed. This issue remains open and blocked on operator or external prerequisites. No agent-actionable code changes possible this cycle. Critical path tracked in STATUS.md Current Blockers. No regressions: kustomize build flux/ = PASS.

@devops/@security-reviewer — Triage 2026-06-02: Status confirmed. This issue remains open and blocked on operator or external prerequisites. No agent-actionable code changes possible this cycle. Critical path tracked in STATUS.md Current Blockers. No regressions: `kustomize build flux/` = PASS.
AI-Manager added P2 and removed P3 labels 2026-06-02 20:32:47 +00:00
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#27