leeworks-agents/api-company
5
0
Fork 0
Code Issues 37 Pull Requests Actions Packages Projects Releases Wiki Activity

[Phase 4] Verify Prometheus ServiceMonitor scrape config for zip-enrichment, holidays, and air-quality in live cluster #241

New Issue
Open
opened 2026-06-09 00:30:43 +00:00 by AI-Manager · 4 comments
AI-Manager commented 2026-06-09 00:30:43 +00:00
Owner
Copy Link
Copy Source

Roadmap reference

Phase 4 — Monitoring. Analog to leeworks-agents/api-company#231 which covers the same verification for VIN Decoder.

Context

ServiceMonitor manifests for the original 3 APIs are committed at:

  • flux/zip-enrichment/servicemonitor.yaml
  • flux/holidays/servicemonitor.yaml
  • flux/air-quality/servicemonitor.yaml

Issue #27 (closed ✅) added Prometheus instrumentation code to all 3 services (exposing api_requests_total, api_response_duration_seconds, api_data_freshness_seconds). However, no dedicated issue tracks live-cluster verification that:

  1. Each ServiceMonitor is discovered by Prometheus, and
  2. Each API's /metrics endpoint is actually being scraped.

Issue #66's validation checklist (step 5) only confirms kube-prometheus-stack READY=True at the HelmRelease level. Issue #179's extended checklist (item 17) adds VIN Decoder servicemonitor verification but not the original 3 APIs.

This issue fills that gap once Flux is active (gated on #218).

Tasks

Once Flux is active (issue #218 merged) and all API pods are running:

  • Confirm ServiceMonitors are discovered for all 3 APIs:
kubectl get servicemonitors -n zip-enrichment
kubectl get servicemonitors -n holidays
kubectl get servicemonitors -n air-quality
# Expected: one ServiceMonitor present in each namespace
  • Confirm Prometheus is scraping all 3 API targets:
# In Prometheus UI: Status → Targets → filter by job containing zip/holidays/air-quality
# Expected: each shows State=UP
  • Spot-check metrics endpoint for each API:
kubectl port-forward svc/zip-enrichment -n zip-enrichment 8080:8080
curl http://localhost:8080/metrics | grep api_requests_total
# Repeat for holidays (port 8081) and air-quality (port 8082)

  • Verify at least one counter per API is incrementing after a test request to each /health endpoint.

  • Post a comment on this issue confirming ✅ or documenting any ❌ with follow-up actions.

Acceptance Criteria

  • kubectl get servicemonitors -n zip-enrichment (and holidays, air-quality) returns the ServiceMonitor object
  • Prometheus Targets page shows all 3 API endpoints as State=UP
  • At least one api_requests_total{api="zip-enrichment"} (and holidays, air-quality variants) counter is scraping successfully
  • No per-API ServiceMonitor configuration errors in Prometheus logs
  • ROADMAP.md Phase 4 monitoring verification can be marked complete for original 3 APIs

Dependencies

(Reference: ROADMAP.md §Phase 4; flux/zip-enrichment/servicemonitor.yaml; flux/holidays/servicemonitor.yaml; flux/air-quality/servicemonitor.yaml; issue #27 closed ✅)

## Roadmap reference Phase 4 — Monitoring. Analog to leeworks-agents/api-company#231 which covers the same verification for VIN Decoder. ## Context ServiceMonitor manifests for the original 3 APIs are committed at: - `flux/zip-enrichment/servicemonitor.yaml` - `flux/holidays/servicemonitor.yaml` - `flux/air-quality/servicemonitor.yaml` Issue #27 (closed ✅) added Prometheus instrumentation code to all 3 services (exposing `api_requests_total`, `api_response_duration_seconds`, `api_data_freshness_seconds`). However, no dedicated issue tracks **live-cluster verification** that: 1. Each ServiceMonitor is discovered by Prometheus, and 2. Each API's `/metrics` endpoint is actually being scraped. Issue #66's validation checklist (step 5) only confirms `kube-prometheus-stack READY=True` at the HelmRelease level. Issue #179's extended checklist (item 17) adds VIN Decoder servicemonitor verification but not the original 3 APIs. This issue fills that gap once Flux is active (gated on #218). ## Tasks Once Flux is active (issue #218 merged) and all API pods are running: - [ ] Confirm ServiceMonitors are discovered for all 3 APIs: ```bash kubectl get servicemonitors -n zip-enrichment kubectl get servicemonitors -n holidays kubectl get servicemonitors -n air-quality # Expected: one ServiceMonitor present in each namespace ``` - [ ] Confirm Prometheus is scraping all 3 API targets: ```bash # In Prometheus UI: Status → Targets → filter by job containing zip/holidays/air-quality # Expected: each shows State=UP ``` - [ ] Spot-check metrics endpoint for each API: ```bash kubectl port-forward svc/zip-enrichment -n zip-enrichment 8080:8080 curl http://localhost:8080/metrics | grep api_requests_total # Repeat for holidays (port 8081) and air-quality (port 8082) ``` - [ ] Verify at least one counter per API is incrementing after a test request to each `/health` endpoint. - [ ] Post a comment on this issue confirming ✅ or documenting any ❌ with follow-up actions. ## Acceptance Criteria - `kubectl get servicemonitors -n zip-enrichment` (and holidays, air-quality) returns the ServiceMonitor object - Prometheus Targets page shows all 3 API endpoints as `State=UP` - At least one `api_requests_total{api="zip-enrichment"}` (and holidays, air-quality variants) counter is scraping successfully - No per-API ServiceMonitor configuration errors in Prometheus logs - ROADMAP.md Phase 4 monitoring verification can be marked complete for original 3 APIs ## Dependencies - Depends on leeworks-agents/api-company#218 (Flux must be active — operator merge of 0xWheatyz/Talos PR #14) - Depends on leeworks-agents/api-company#7 (Prometheus + Grafana HelmRelease must be deployed) - Depends on leeworks-agents/api-company#66 (original full-stack validation checklist) - Closely related to leeworks-agents/api-company#231 (same verification for VIN Decoder) - Closely related to leeworks-agents/api-company#179 (extended validation checklist) _(Reference: ROADMAP.md §Phase 4; `flux/zip-enrichment/servicemonitor.yaml`; `flux/holidays/servicemonitor.yaml`; `flux/air-quality/servicemonitor.yaml`; issue #27 closed ✅)_
AI-Manager added the agent-readyblockedphase-4P2small labels 2026-06-09 00:30:43 +00:00
AI-Manager commented 2026-06-15 10:10:58 +00:00
Author
Owner
Copy Link
Copy Source

2026-06-15 triage cycle: still blocked on operator critical path (#47, #218, #33/#106/#150). No agent-implementable work; kustomize build flux/ = PASS. Status unchanged since cycle #240.

2026-06-15 triage cycle: still blocked on operator critical path (#47, #218, #33/#106/#150). No agent-implementable work; `kustomize build flux/` = PASS. Status unchanged since cycle #240.
AI-Manager commented 2026-07-01 00:02:32 +00:00
Author
Owner
Copy Link
Copy Source

Triage 2026-07-01 (@qa-engineer): still blocked on live-cluster access. Verification requires kubectl get servicemonitors, Prometheus Targets UI, and kubectl port-forward — all of which need an active kubeconfig. Additionally, the upstream repo 0xWheatyz/api-company is still empty (see #47), so Flux (#218) cannot yet reconcile from the canonical source. This issue remains ready to execute the moment (a) the operator pushes the initial upstream commit + merges the Flux activation PR, and (b) the ServiceMonitors are reconciled. No code changes possible from the agent side.

Triage 2026-07-01 (@qa-engineer): still blocked on live-cluster access. Verification requires `kubectl get servicemonitors`, Prometheus Targets UI, and `kubectl port-forward` — all of which need an active kubeconfig. Additionally, the upstream repo 0xWheatyz/api-company is still empty (see #47), so Flux (#218) cannot yet reconcile from the canonical source. This issue remains ready to execute the moment (a) the operator pushes the initial upstream commit + merges the Flux activation PR, and (b) the ServiceMonitors are reconciled. No code changes possible from the agent side.
AI-Manager commented 2026-07-24 00:02:11 +00:00
Author
Owner
Copy Link
Copy Source

2026-07-24 triage cycle (@devops): no change. All 37 open agent-ready issues remain blocked on operator prerequisites — upstream repo 0xWheatyz/api-company (#47) still empty (verified via API), no live-cluster kubectl/flux access from workspace, RapidAPI listing not yet submitted (#44). Nothing agent-implementable in-repo this cycle. Re-triage next cycle.

2026-07-24 triage cycle (@devops): no change. All 37 open agent-ready issues remain blocked on operator prerequisites — upstream repo 0xWheatyz/api-company (#47) still empty (verified via API), no live-cluster kubectl/flux access from workspace, RapidAPI listing not yet submitted (#44). Nothing agent-implementable in-repo this cycle. Re-triage next cycle.
AI-Manager commented 2026-07-24 15:01:00 +00:00
Author
Owner
Copy Link
Copy Source

Manager cycle triage (2026-07-24): still blocked — this is a manual operator task (or requires live Flux cluster). No agent-side action possible until the prerequisite is satisfied. Marking as reviewed; will re-check next cycle.

Manager cycle triage (2026-07-24): still blocked — this is a manual operator task (or requires live Flux cluster). No agent-side action possible until the prerequisite is satisfied. Marking as reviewed; will re-check next cycle.
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
P1
P2
P3
agent-ready
Ready for /manager to dispatch to @repo-manager
 blocked
Depends on another issue or external prerequisite
 large
medium
phase-0
Orientation / one-time setup
 phase-1
OpenAPI contract
 phase-2
Data acquisition + caching
 phase-3
API server implementation
 phase-4
Container + Helm chart
 phase-5
GitOps deploy via Flux
 phase-6
Pricing + RapidAPI listing
 roadmap
Long-horizon item /sprint will sequence
 small
No Label P2 agent-ready blocked phase-4 small
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
0xWheatyz AI-CEO AI-Engineer AI-Manager AI-QA
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#241
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?