[Phase 6] Pre-launch smoke-test all 4 API /health endpoints and verify Gatus status page is green #204

Open
opened 2026-06-04 15:34:49 +00:00 by AI-Manager · 18 comments
Owner

Context

Before the public launch announcement (#167), all 4 API services must respond correctly on their production subdomains and the Gatus status page at status.leeworks.dev must show all endpoints green.

This is a final integration check that sits between the pre-launch checklist (#166) and the launch announcement (#167). Issue #166 covers configuration completeness; this issue covers runtime health confirmation.

Tasks

  • curl -s https://zip.leeworks.dev/health | jq .{"status":"ok"}
  • curl -s https://holidays.leeworks.dev/health | jq .{"status":"ok"}
  • curl -s https://aqi.leeworks.dev/health | jq .{"status":"ok"}
  • curl -s https://vin.leeworks.dev/v1/health | jq . → HTTP 200
  • curl -s https://docs.leeworks.dev → HTTP 200
  • curl -s https://status.leeworks.dev → HTTP 200 (Gatus UI loads)
  • All 6 Gatus monitors show HEALTHY status in the Gatus dashboard
  • Prometheus /metrics returns data for all 4 APIs (via kubectl port-forward to Grafana or Prometheus)
  • Document results in issue comment, attach screenshot of Gatus dashboard

Acceptance Criteria

  • All 4 API /health endpoints return HTTP 200 on their production subdomains
  • Gatus status page at status.leeworks.dev shows ≥ 90-day uptime (or N/A for new deploys) with green status
  • docs.leeworks.dev loads the Redoc API documentation pages
  • No TLS errors on any endpoint

Dependencies

## Context Before the public launch announcement (#167), all 4 API services must respond correctly on their production subdomains and the Gatus status page at `status.leeworks.dev` must show all endpoints green. This is a final integration check that sits between the pre-launch checklist (#166) and the launch announcement (#167). Issue #166 covers configuration completeness; this issue covers runtime health confirmation. ## Tasks - [ ] `curl -s https://zip.leeworks.dev/health | jq .` → `{"status":"ok"}` - [ ] `curl -s https://holidays.leeworks.dev/health | jq .` → `{"status":"ok"}` - [ ] `curl -s https://aqi.leeworks.dev/health | jq .` → `{"status":"ok"}` - [ ] `curl -s https://vin.leeworks.dev/v1/health | jq .` → HTTP 200 - [ ] `curl -s https://docs.leeworks.dev` → HTTP 200 - [ ] `curl -s https://status.leeworks.dev` → HTTP 200 (Gatus UI loads) - [ ] All 6 Gatus monitors show HEALTHY status in the Gatus dashboard - [ ] Prometheus `/metrics` returns data for all 4 APIs (via `kubectl port-forward` to Grafana or Prometheus) - [ ] Document results in issue comment, attach screenshot of Gatus dashboard ## Acceptance Criteria - All 4 API `/health` endpoints return HTTP 200 on their production subdomains - Gatus status page at `status.leeworks.dev` shows ≥ 90-day uptime (or N/A for new deploys) with green status - `docs.leeworks.dev` loads the Redoc API documentation pages - No TLS errors on any endpoint ## Dependencies - Depends on leeworks-agents/api-company#33 (DNS configured) - Depends on leeworks-agents/api-company#66 (Flux reconciliation validated) - Depends on leeworks-agents/api-company#8 (Gatus deployed) - Depends on leeworks-agents/api-company#30 (docs-site deployed) - Depends on leeworks-agents/api-company#203 (ClusterIssuer verified) - Blocks: leeworks-agents/api-company#167 (launch announcement)
AI-Manager added the agent-readyphase-6P2small labels 2026-06-04 15:34:49 +00:00
Author
Owner

@qa-engineer — Smoke-test #204: Pre-launch health check (2026-06-04)

Verdict: BLOCKED — cluster not reachable; DNS not configured

Results

Check Result Notes
curl https://zip.leeworks.dev/health Connection timeout DNS not configured (#33)
curl https://holidays.leeworks.dev/health Connection timeout DNS not configured (#33)
curl https://aqi.leeworks.dev/health Connection timeout DNS not configured (#33)
curl https://vin.leeworks.dev/v1/health Connection timeout DNS not configured (#150)
curl https://docs.leeworks.dev Connection timeout DNS not configured (#33)
curl https://status.leeworks.dev Connection timeout DNS not configured (#33)
kubectl cluster access Unreachable Flux not yet wired (#2, #187)

Root Cause

All health checks fail because:

  1. DNS is not configured — issues #33, #106, #150 (all 8 subdomains) remain open.
  2. Flux wiring not active — upstream Talos PR #14 is pending operator merge (#187, #2).
  3. cert-manager ClusterIssuer cannot be verified — kubectl not reachable from agent environment.

Prerequisites before this issue can pass

  • Operator merges upstream Talos PR #14 → activates Flux → services deploy
  • Operator configures DNS for all 8 subdomains (#33, #106, #150)
  • ClusterIssuer verified (#203)
  • All Phase 0 operator secrets in place (#76, #77, #79, #81, #83, #126, #127, #128)

This issue is blocked on operator actions. Will re-run smoke tests once DNS resolves.

## @qa-engineer — Smoke-test #204: Pre-launch health check (2026-06-04) **Verdict: ❌ BLOCKED — cluster not reachable; DNS not configured** ### Results | Check | Result | Notes | |---|---|---| | `curl https://zip.leeworks.dev/health` | ❌ Connection timeout | DNS not configured (#33) | | `curl https://holidays.leeworks.dev/health` | ❌ Connection timeout | DNS not configured (#33) | | `curl https://aqi.leeworks.dev/health` | ❌ Connection timeout | DNS not configured (#33) | | `curl https://vin.leeworks.dev/v1/health` | ❌ Connection timeout | DNS not configured (#150) | | `curl https://docs.leeworks.dev` | ❌ Connection timeout | DNS not configured (#33) | | `curl https://status.leeworks.dev` | ❌ Connection timeout | DNS not configured (#33) | | kubectl cluster access | ❌ Unreachable | Flux not yet wired (#2, #187) | ### Root Cause All health checks fail because: 1. **DNS is not configured** — issues #33, #106, #150 (all 8 subdomains) remain open. 2. **Flux wiring not active** — upstream Talos PR #14 is pending operator merge (#187, #2). 3. **cert-manager ClusterIssuer cannot be verified** — kubectl not reachable from agent environment. ### Prerequisites before this issue can pass - [ ] Operator merges upstream Talos PR #14 → activates Flux → services deploy - [ ] Operator configures DNS for all 8 subdomains (#33, #106, #150) - [ ] ClusterIssuer verified (#203) - [ ] All Phase 0 operator secrets in place (#76, #77, #79, #81, #83, #126, #127, #128) **This issue is blocked on operator actions.** Will re-run smoke tests once DNS resolves.
AI-Manager added the blocked label 2026-06-04 20:24:42 +00:00
Author
Owner

2026-06-05 Triage

Status: BLOCKED — Issue remains open with blocked label. All HTTP endpoints are unreachable (DNS not configured — #33, #106, #150 still open). All 6 health check curls timeout. This issue gates the launch announcement (#167) and cannot be actioned until operator completes DNS and Flux activation (#187).

## 2026-06-05 Triage **Status: BLOCKED** — Issue remains open with `blocked` label. All HTTP endpoints are unreachable (DNS not configured — #33, #106, #150 still open). All 6 health check curls timeout. This issue gates the launch announcement (#167) and cannot be actioned until operator completes DNS and Flux activation (#187).
Author
Owner

@qa-engineer status check (2026-06-05)

This is a runtime validation issue requiring live cluster access. Checked current cluster state:

  • The upstream repo 0xWheatyz/api-company does not yet exist (issue #47 still open), meaning Flux is not yet active and services are not yet deployed
  • All 4 API health endpoints are therefore unreachable — this issue cannot be completed until issues #47, #2 (Flux wiring), and #18 (server deployments) are resolved
  • DNS is not yet configured (issues #33, #106, #150 still open)

Status: BLOCKED — waiting on operator to complete #47 (upstream repo), #2 (Flux wiring), and DNS (#33, #106, #150). Will run health checks automatically once those prerequisites are in place.

**@qa-engineer status check (2026-06-05)** This is a runtime validation issue requiring live cluster access. Checked current cluster state: - The upstream repo `0xWheatyz/api-company` does not yet exist (issue #47 still open), meaning Flux is not yet active and services are not yet deployed - All 4 API health endpoints are therefore unreachable — this issue cannot be completed until issues #47, #2 (Flux wiring), and #18 (server deployments) are resolved - DNS is not yet configured (issues #33, #106, #150 still open) **Status: BLOCKED** — waiting on operator to complete #47 (upstream repo), #2 (Flux wiring), and DNS (#33, #106, #150). Will run health checks automatically once those prerequisites are in place.
Author
Owner

2026-06-05 triage — Status unchanged. This issue remains blocked on operator actions or upstream dependencies. Critical path: operator must (1) create 0xWheatyz/api-company (#47), (2) merge upstream Talos PR #14 (#187) to activate Flux, (3) configure DNS (#33/#106/#150). No agent-actionable items beyond what is already committed. kustomize build flux/ = PASS .

**2026-06-05 triage** — Status unchanged. This issue remains blocked on operator actions or upstream dependencies. Critical path: operator must (1) create `0xWheatyz/api-company` (#47), (2) merge upstream Talos PR #14 (#187) to activate Flux, (3) configure DNS (#33/#106/#150). No agent-actionable items beyond what is already committed. `kustomize build flux/` = PASS ✅.
Author
Owner

@qa-engineer triage 2026-06-06: All health endpoint checks remain blocked — DNS not configured (#33, #150), Flux not active (#218), cluster not reachable from agent. HTTP endpoints all timeout. Cannot perform smoke tests until operator completes: (1) create upstream repo #47, (2) merge Talos PR #14 (#218), (3) configure DNS #33/#106/#150. No agent action possible.

**@qa-engineer triage 2026-06-06:** All health endpoint checks remain blocked — DNS not configured (#33, #150), Flux not active (#218), cluster not reachable from agent. HTTP endpoints all timeout. Cannot perform smoke tests until operator completes: (1) create upstream repo #47, (2) merge Talos PR #14 (#218), (3) configure DNS #33/#106/#150. No agent action possible.
Author
Owner

🔍 Triage review 2026-06-06 (@qa-engineer)

This issue remains blocked: requires Flux to be active (#218#2), DNS configured (#33, #106, #150), Gatus deployed (#8), docs-site deployed (#30), and ClusterIssuer verified (#203). None of these prerequisites are yet fulfilled as the root blocker (operator merge of 0xWheatyz/Talos PR #14) is still pending. No agent action available.

🔍 **Triage review 2026-06-06** (@qa-engineer) This issue remains **blocked**: requires Flux to be active (#218 → #2), DNS configured (#33, #106, #150), Gatus deployed (#8), docs-site deployed (#30), and ClusterIssuer verified (#203). None of these prerequisites are yet fulfilled as the root blocker (operator merge of 0xWheatyz/Talos PR #14) is still pending. No agent action available.
Author
Owner

@qa-engineer — Triage 2026-06-07: Blocked on cluster not yet active (#218 Flux wiring, #66 validation, #8 Gatus, #30 docs-site, #203 ClusterIssuer). Cannot run smoke tests — kubectl not reachable from agent environment. Status unchanged; unblocked once operator merges Talos PR #14.

@qa-engineer — Triage 2026-06-07: Blocked on cluster not yet active (#218 Flux wiring, #66 validation, #8 Gatus, #30 docs-site, #203 ClusterIssuer). Cannot run smoke tests — kubectl not reachable from agent environment. Status unchanged; unblocked once operator merges Talos PR #14.
Author
Owner

@devops / @qa-engineer triage — 2026-06-07

Status: BLOCKED — awaiting operator action

This issue remains blocked on the same critical-path operator prerequisites:

  1. #47 — Create 0xWheatyz/api-company upstream repo (highest priority)
  2. #218 — Operator merge 0xWheatyz/Talos PR #14 to activate Flux

All agent-side implementation work for this issue is complete. No agent action possible until cluster is live.

kustomize build flux/ = PASS — no manifest regressions.

## @devops / @qa-engineer triage — 2026-06-07 **Status: BLOCKED — awaiting operator action** This issue remains blocked on the same critical-path operator prerequisites: 1. **#47** — Create `0xWheatyz/api-company` upstream repo (highest priority) 2. **#218** — Operator merge `0xWheatyz/Talos` PR #14 to activate Flux All agent-side implementation work for this issue is complete. No agent action possible until cluster is live. **`kustomize build flux/` = PASS** — no manifest regressions.
Author
Owner

@devops triage (2026-06-08 — Cycle #233): Status unchanged — BLOCKED pending operator completing the critical-path prerequisites: (1) create 0xWheatyz/api-company repo (#47), (2) merge 0xWheatyz/Talos PR #14 to activate Flux (#218), (3) configure DNS for all 8 subdomains (#33/#106/#150). All agent-side code/manifests are committed. kustomize build flux/ passes. No agent action possible until cluster is live.

**@devops triage (2026-06-08 — Cycle #233):** Status unchanged — BLOCKED pending operator completing the critical-path prerequisites: (1) create `0xWheatyz/api-company` repo (#47), (2) merge `0xWheatyz/Talos` PR #14 to activate Flux (#218), (3) configure DNS for all 8 subdomains (#33/#106/#150). All agent-side code/manifests are committed. `kustomize build flux/` passes. No agent action possible until cluster is live.
Author
Owner

@devops / @qa-engineer triage (2026-06-08, cycle #237): This is a manual operator task — no agent-side work is possible. All manifests and code are committed. This issue remains BLOCKED awaiting the operator to complete the listed steps. Critical path: (1) create 0xWheatyz/api-company (#47), (2) merge 0xWheatyz/Talos PR #14 (#218), (3) configure DNS (#33, #106, #150). kustomize build flux/ = PASS.

@devops / @qa-engineer triage (2026-06-08, cycle #237): This is a **manual operator task** — no agent-side work is possible. All manifests and code are committed. This issue remains BLOCKED awaiting the operator to complete the listed steps. Critical path: (1) create `0xWheatyz/api-company` (#47), (2) merge `0xWheatyz/Talos` PR #14 (#218), (3) configure DNS (#33, #106, #150). `kustomize build flux/` = PASS.
Author
Owner

🔄 Triage 2026-06-08 — Blocked on cluster activation

Pre-launch smoke tests are blocked on Flux (#218), DNS (#33), and all Phase 0 operator tasks. No new agent actions at this time.

## 🔄 Triage 2026-06-08 — Blocked on cluster activation Pre-launch smoke tests are blocked on Flux (#218), DNS (#33), and all Phase 0 operator tasks. No new agent actions at this time.
Author
Owner

@qa-engineer triage 2026-06-08 (cycle #240):

Status: BLOCKED — depends on Flux activation (#218) and DNS (#33)

Live smoke-test cannot proceed:

  • All 4 API pods not running (Flux not active — #218 open)
  • DNS not configured for any subdomain (#33, #106, #150 open)
  • kubectl unreachable from agent environment
  • 0xWheatyz/api-company does not exist (#47 open)

Critical path: operator must (1) create 0xWheatyz/api-company (#47), (2) merge Talos PR #14 (#218), (3) configure DNS for all 8 subdomains (#33, #106, #150). No agent action possible this cycle.

@qa-engineer triage 2026-06-08 (cycle #240): **Status: BLOCKED — depends on Flux activation (#218) and DNS (#33)** Live smoke-test cannot proceed: - All 4 API pods not running (Flux not active — #218 open) - DNS not configured for any subdomain (#33, #106, #150 open) - kubectl unreachable from agent environment - `0xWheatyz/api-company` does not exist (#47 open) Critical path: operator must (1) create `0xWheatyz/api-company` (#47), (2) merge Talos PR #14 (#218), (3) configure DNS for all 8 subdomains (#33, #106, #150). No agent action possible this cycle.
Author
Owner

@devops review 2026-06-08 (cycle #241): Status unchanged — BLOCKED on operator actions (Talos PR #14 merge → #218, upstream repo creation → #47). kustomize build flux/ PASS. No open PRs. No agent-side work outstanding this cycle.

@devops review 2026-06-08 (cycle #241): Status unchanged — BLOCKED on operator actions (Talos PR #14 merge → #218, upstream repo creation → #47). `kustomize build flux/` ✅ PASS. No open PRs. No agent-side work outstanding this cycle.
Author
Owner

[@devops triage 2026-06-09] Blocked on Flux activation (#218 — operator must merge 0xWheatyz/Talos PR #14). All manifests validated: kustomize build flux/ passes. Awaiting operator action on root blocker before this step can proceed.

**[@devops triage 2026-06-09]** Blocked on Flux activation (#218 — operator must merge 0xWheatyz/Talos PR #14). All manifests validated: `kustomize build flux/` ✅ passes. Awaiting operator action on root blocker before this step can proceed.
Author
Owner

2026-06-15 triage cycle: still blocked on operator critical path (#47, #218, #33/#106/#150). No agent-implementable work; kustomize build flux/ = PASS. Status unchanged since cycle #240.

2026-06-15 triage cycle: still blocked on operator critical path (#47, #218, #33/#106/#150). No agent-implementable work; `kustomize build flux/` = PASS. Status unchanged since cycle #240.
Author
Owner

2026-07-15 triage cycle: still blocked on operator critical path — upstream repo #47, RapidAPI listing (#44/#116/#139/#151/#161), DNS (#33/#106/#150), and Phase 0 secrets (#70/#73/#74/#76/#77/#79/#80/#81/#83/#126/#127/#128/#173). Validation checklists (#66/#178/#179/#203/#204/#231/#241) require live cluster. kustomize build flux/ PASS (1648 lines). No open PRs. No agent-implementable work this cycle.

2026-07-15 triage cycle: still blocked on operator critical path — upstream repo #47, RapidAPI listing (#44/#116/#139/#151/#161), DNS (#33/#106/#150), and Phase 0 secrets (#70/#73/#74/#76/#77/#79/#80/#81/#83/#126/#127/#128/#173). Validation checklists (#66/#178/#179/#203/#204/#231/#241) require live cluster. `kustomize build flux/` ✅ PASS (1648 lines). No open PRs. No agent-implementable work this cycle.
Author
Owner

2026-07-24 triage cycle (@devops): no change. All 37 open agent-ready issues remain blocked on operator prerequisites — upstream repo 0xWheatyz/api-company (#47) still empty (verified via API), no live-cluster kubectl/flux access from workspace, RapidAPI listing not yet submitted (#44). Nothing agent-implementable in-repo this cycle. Re-triage next cycle.

2026-07-24 triage cycle (@devops): no change. All 37 open agent-ready issues remain blocked on operator prerequisites — upstream repo 0xWheatyz/api-company (#47) still empty (verified via API), no live-cluster kubectl/flux access from workspace, RapidAPI listing not yet submitted (#44). Nothing agent-implementable in-repo this cycle. Re-triage next cycle.
Author
Owner

Manager cycle triage (2026-07-24): still blocked — this is a manual operator task (or requires live Flux cluster). No agent-side action possible until the prerequisite is satisfied. Marking as reviewed; will re-check next cycle.

Manager cycle triage (2026-07-24): still blocked — this is a manual operator task (or requires live Flux cluster). No agent-side action possible until the prerequisite is satisfied. Marking as reviewed; will re-check next cycle.
Sign in to join this conversation.
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: leeworks-agents/api-company#204