diff --git a/docs/operator-runbook.md b/docs/operator-runbook.md new file mode 100644 index 0000000..727dc08 --- /dev/null +++ b/docs/operator-runbook.md @@ -0,0 +1,299 @@ +# Operator Runbook + +**Audience:** Human operator (0xWheatyz) +**Purpose:** Ordered, copy-paste-ready guide to bring the full `api-company` stack live. +**Last updated:** 2026-05-26 +**Closes:** leeworks-agents/api-company#50 + +--- + +## Overview + +The agent has committed all Flux manifests and documentation. The only remaining +work is a set of manual steps that require Gitea admin access, `kubectl` access to +the `testing1` cluster, and external service accounts (RapidAPI, Slack, PayPal). + +Work through these phases **in order** — each phase unblocks the next. + +--- + +## Phase 0 — Create upstream repo (unblocks all ship cycles) + +> **Why first?** Every agent deployment cycle fails to open a PR to upstream +> because `0xWheatyz/api-company` does not yet exist. This one step unblocks +> all automated deployments. See issues #41, #47. + +### Step 0-A — Create `0xWheatyz/api-company` on Gitea + +1. Log into Gitea as `0xWheatyz`. +2. **+** → **New Repository**. +3. Owner: `0xWheatyz`, Name: `api-company`. +4. Visibility: Public (or Private — your choice). +5. **Do not** initialise with a README. +6. Click **Create Repository**. + +Once created, the agent's next ship cycle will open a deployment PR automatically. + +--- + +## Phase 1 — Wire Flux to this repo (unblocks all GitOps reconciliation) + +> **Why second?** Until Flux watches `leeworks-agents/api-company`, none of the +> manifests in `flux/` are applied to the cluster. See issue #2. + +### Step 1-A — Create `gitea-leeworks-agents-token` secret in `flux-system` + +```bash +# In Gitea: User Settings → Applications → Generate Token +# Scopes: read:repository (read-only is sufficient for Flux) +# Copy the token, then: + +kubectl create secret generic gitea-leeworks-agents-token \ + -n flux-system \ + --from-literal=username=leeworks-agents \ + --from-literal=password= +``` + +### Step 1-B — Copy Flux source + kustomization into 0xWheatyz/Talos + +Reference manifests are at `flux/api-company-source/` in this repo. +Copy them verbatim to: + +``` +0xWheatyz/Talos:testing1/first-cluster/cluster/flux/api-company-source/ +├── gitrepository.yaml +└── kustomization.yaml +``` + +You can do this via the Gitea web editor or locally: + +```bash +cd /path/to/Talos-checkout +mkdir -p testing1/first-cluster/cluster/flux/api-company-source +# copy the two files from api-company/flux/api-company-source/ +git add . +git commit -m "feat: wire Flux GitRepository + Kustomization for api-company" +git push origin main +``` + +**Verify reconciliation (after ~5 minutes):** +```bash +flux get sources git -n flux-system +flux get kustomizations -n flux-system +``` + +Both `api-company` entries should show `Ready = True`. + +--- + +## Phase 2 — Secrets for already-staged services + +Once Flux is watching the repo, it will attempt to reconcile all `flux/` +sub-directories. The HelmReleases will stall on missing secrets. Create them: + +### Step 2-A — `gitea-runner-token` (unblocks Gitea Actions runner, issue #3) + +```bash +# In Gitea: Admin Panel → Site Administration → Actions → Runners +# → Create new Runner → copy registration token + +kubectl create secret generic gitea-runner-token \ + -n gitea-runner \ + --from-literal=token= +``` + +**Verify:** +```bash +kubectl get pods -n gitea-runner +# Then check Gitea Admin → Actions → Runners — runner should appear Online +``` + +### Step 2-B — `grafana-admin` (unblocks Grafana, issue #7) + +```bash +kubectl create secret generic grafana-admin \ + -n monitoring \ + --from-literal=admin-password= +``` + +Grafana URL: `https://grafana.leeworks.dev` (login: `admin` / ``) + +### Step 2-C — `gatus-slack-webhook` (unblocks Gatus alerts, issue #8) + +```bash +# Create an incoming webhook at: https://api.slack.com/messaging/webhooks + +kubectl create secret generic gatus-slack-webhook \ + -n monitoring \ + --from-literal=url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL +``` + +Gatus URL: `https://status.leeworks.dev` + +### Step 2-D — Enable Gitea packages + registry DNS (issue #4) + +**4a — Enable packages in Gitea `app.ini`:** +```ini +[packages] +ENABLED = true +``` +Restart Gitea after editing `app.ini`. + +**4b — Add DNS A record:** +``` +registry.leeworks.dev → +``` + +Find the ingress IP: +```bash +kubectl get svc -n ingress-nginx +``` + +See `docs/registry.md` for additional context. + +--- + +## Phase 3 — Enable CI image push (unblocks API service deployments) + +Once the runner is online and the registry is reachable, CI pipelines can build +and push container images. + +### Step 3-A — Add `GITEA_TOKEN` Actions Secret to each repo + +Repos to configure: +- `leeworks-agents/api-company` +- `leeworks-agents/zip-enrichment` +- `leeworks-agents/holidays` +- `leeworks-agents/air-quality` + +**For each repo:** Repo → Settings → Actions → Secrets → Add Secret +- **Name:** `GITEA_TOKEN` +- **Value:** Gitea personal access token with `write:packages` scope + +### Step 3-B — Create image-automation token secret (issue #51) + +The agent has added `ImageRepository` + `ImagePolicy` + `ImageUpdateAutomation` +manifests to `flux/image-automation/`. Flux will automatically update image tags +in HelmReleases when CI pushes new images — but it needs write access to commit +back: + +```bash +kubectl create secret generic gitea-image-automation-token \ + -n flux-system \ + --from-literal=username=leeworks-agents \ + --from-literal=password= +``` + +--- + +## Phase 4 — DNS for API services (issue #33) + +Add DNS A records for all six leeworks.dev subdomains (all point to the same +cluster ingress IP): + +| Hostname | Target | +|-------------------------|------------------------| +| `zip.leeworks.dev` | `` | +| `holidays.leeworks.dev` | `` | +| `aqi.leeworks.dev` | `` | +| `docs.leeworks.dev` | `` | +| `grafana.leeworks.dev` | `` | +| `status.leeworks.dev` | `` | + +cert-manager will obtain Let's Encrypt certificates automatically once DNS +propagates (typically minutes, up to 48 h). + +--- + +## Phase 5 — RapidAPI + PayPal (issue #44, #19) + +> **Blocked on operator being 18+ for PayPal.** Complete when eligible. + +1. Create accounts on [rapidapi.com](https://rapidapi.com) and [paypal.com](https://www.paypal.com). +2. Link PayPal to RapidAPI as the payout method. +3. Submit each API to the RapidAPI marketplace using `docs/rapidapi-listings.md`. +4. Configure paid tiers per `ROADMAP.md`. + +After submission, RapidAPI generates a `X-RapidAPI-Proxy-Secret` per API. Create: + +```bash +# zip-enrichment +kubectl create secret generic rapidapi-proxy-secret \ + -n zip-enrichment \ + --from-literal=X-RapidAPI-Proxy-Secret= + +# holidays +kubectl create secret generic rapidapi-proxy-secret \ + -n holidays \ + --from-literal=X-RapidAPI-Proxy-Secret= + +# air-quality +kubectl create secret generic rapidapi-proxy-secret \ + -n air-quality \ + --from-literal=X-RapidAPI-Proxy-Secret= +``` + +--- + +## Quick Verification Checklist + +```bash +# Flux overall health +flux get all -A + +# API service pods +kubectl get pods -n zip-enrichment +kubectl get pods -n holidays +kubectl get pods -n air-quality + +# Ingress + TLS +kubectl get ingress -A +kubectl get certificates -A + +# Gitea runner +kubectl get pods -n gitea-runner + +# Monitoring stack +kubectl get pods -n monitoring + +# Image automation +flux get imagepolicies -A +flux get imagerepositories -A +``` + +--- + +## Dependency Summary + +``` +Phase 0: Create 0xWheatyz/api-company repo + └─► unblocks agent deployment PRs to upstream + +Phase 1: Wire Flux (gitea-token secret + Talos manifests) + └─► all flux/ manifests reconcile + +Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry) + └─► runner online, monitoring live, registry reachable + +Phase 3: CI secrets + image-automation token + └─► images build, push, and auto-update → API services deploy + +Phase 4: DNS records + └─► HTTPS certs issued → public URLs go live + +Phase 5: RapidAPI + PayPal + └─► revenue enabled +``` + +--- + +## Related Documents + +| Document | Purpose | +|----------|---------| +| `docs/secrets-checklist.md` | Full checklist of all required secrets | +| `docs/registry.md` | Container registry architecture decision | +| `docs/cluster-audit.md` | Node/namespace/ingress inventory | +| `docs/rapidapi-listings.md` | RapidAPI marketplace submission details | +| `ROADMAP.md` | Full project roadmap and milestones | +| `STATUS.md` | Current cycle status and blockers |