api-company/docs/operator-runbook.md

# Operator Runbook

**Audience:** Human operator (0xWheatyz)  
**Purpose:** Ordered, copy-paste-ready guide to bring the full `api-company` stack live.  
**Last updated:** 2026-05-26  
**Closes:** leeworks-agents/api-company#50

---

## Overview

The agent has committed all Flux manifests and documentation. The only remaining
work is a set of manual steps that require Gitea admin access, `kubectl` access to
the `testing1` cluster, and external service accounts (RapidAPI, Slack, PayPal).

Work through these phases **in order** — each phase unblocks the next.

---

## Phase 0 — Create upstream repo (unblocks all ship cycles)

> **Why first?** Every agent deployment cycle fails to open a PR to upstream
> because `0xWheatyz/api-company` does not yet exist. This one step unblocks
> all automated deployments. See issues #41, #47.

### Step 0-A — Create `0xWheatyz/api-company` on Gitea

1. Log into Gitea as `0xWheatyz`.
2. **+** → **New Repository**.
3. Owner: `0xWheatyz`, Name: `api-company`.
4. Visibility: Public (or Private — your choice).
5. **Do not** initialise with a README.
6. Click **Create Repository**.

Once created, the agent's next ship cycle will open a deployment PR automatically.

---

## Phase 1 — Wire Flux to this repo (unblocks all GitOps reconciliation)

> **Why second?** Until Flux watches `leeworks-agents/api-company`, none of the
> manifests in `flux/` are applied to the cluster. See issue #2.

### Step 1-A — Create `gitea-leeworks-agents-token` secret in `flux-system`

```bash
# In Gitea: User Settings → Applications → Generate Token
# Scopes: read:repository  (read-only is sufficient for Flux)
# Copy the token, then:

kubectl create secret generic gitea-leeworks-agents-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN>
```

### Step 1-B — Copy Flux source + kustomization into 0xWheatyz/Talos

Reference manifests are at `flux/api-company-source/` in this repo.  
Copy them verbatim to:

```
0xWheatyz/Talos:testing1/first-cluster/cluster/flux/api-company-source/
├── gitrepository.yaml
└── kustomization.yaml
```

You can do this via the Gitea web editor or locally:

```bash
cd /path/to/Talos-checkout
mkdir -p testing1/first-cluster/cluster/flux/api-company-source
# copy the two files from api-company/flux/api-company-source/
git add .
git commit -m "feat: wire Flux GitRepository + Kustomization for api-company"
git push origin main
```

**Verify reconciliation (after ~5 minutes):**
```bash
flux get sources git -n flux-system
flux get kustomizations -n flux-system
```

Both `api-company` entries should show `Ready = True`.

---

## Phase 2 — Secrets for already-staged services

Once Flux is watching the repo, it will attempt to reconcile all `flux/`
sub-directories. The HelmReleases will stall on missing secrets. Create them:

### Step 2-A — `gitea-runner-token` (unblocks Gitea Actions runner, issue #3)

```bash
# In Gitea: Admin Panel → Site Administration → Actions → Runners
# → Create new Runner → copy registration token

kubectl create secret generic gitea-runner-token \
  -n gitea-runner \
  --from-literal=token=<RUNNER_REGISTRATION_TOKEN>
```

**Verify:**
```bash
kubectl get pods -n gitea-runner
# Then check Gitea Admin → Actions → Runners — runner should appear Online
```

### Step 2-B — `grafana-admin` (unblocks Grafana, issue #7)

```bash
kubectl create secret generic grafana-admin \
  -n monitoring \
  --from-literal=admin-password=<CHOOSE_STRONG_PASSWORD>
```

Grafana URL: `https://grafana.leeworks.dev` (login: `admin` / `<PASSWORD>`)

### Step 2-C — `gatus-slack-webhook` (unblocks Gatus alerts, issue #8)

```bash
# Create an incoming webhook at: https://api.slack.com/messaging/webhooks

kubectl create secret generic gatus-slack-webhook \
  -n monitoring \
  --from-literal=url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```

Gatus URL: `https://status.leeworks.dev`

### Step 2-D — Enable Gitea packages + registry DNS (issue #4)

**4a — Enable packages in Gitea `app.ini`:**
```ini
[packages]
ENABLED = true
```
Restart Gitea after editing `app.ini`.

**4b — Add DNS A record:**
```
registry.leeworks.dev  →  <cluster ingress IP>
```

Find the ingress IP:
```bash
kubectl get svc -n ingress-nginx
```

See `docs/registry.md` for additional context.

---

## Phase 3 — Enable CI image push (unblocks API service deployments)

Once the runner is online and the registry is reachable, CI pipelines can build
and push container images.

### Step 3-A — Add `GITEA_TOKEN` Actions Secret to each repo

Repos to configure:
- `leeworks-agents/api-company`
- `leeworks-agents/zip-enrichment`
- `leeworks-agents/holidays`
- `leeworks-agents/air-quality`

**For each repo:** Repo → Settings → Actions → Secrets → Add Secret
- **Name:** `GITEA_TOKEN`
- **Value:** Gitea personal access token with `write:packages` scope

### Step 3-B — Create image-automation token secret (issue #51)

The agent has added `ImageRepository` + `ImagePolicy` + `ImageUpdateAutomation`
manifests to `flux/image-automation/`. Flux will automatically update image tags
in HelmReleases when CI pushes new images — but it needs write access to commit
back:

```bash
kubectl create secret generic gitea-image-automation-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN_WITH_WRITE_REPO>
```

---

## Phase 4 — DNS for API services (issue #33)

Add DNS A records for all six leeworks.dev subdomains (all point to the same
cluster ingress IP):

| Hostname                | Target                 |
|-------------------------|------------------------|
| `zip.leeworks.dev`      | `<cluster ingress IP>` |
| `holidays.leeworks.dev` | `<cluster ingress IP>` |
| `aqi.leeworks.dev`      | `<cluster ingress IP>` |
| `docs.leeworks.dev`     | `<cluster ingress IP>` |
| `grafana.leeworks.dev`  | `<cluster ingress IP>` |
| `status.leeworks.dev`   | `<cluster ingress IP>` |

cert-manager will obtain Let's Encrypt certificates automatically once DNS
propagates (typically minutes, up to 48 h).

---

## Phase 5 — RapidAPI + PayPal (issue #44, #19)

> **Blocked on operator being 18+ for PayPal.** Complete when eligible.

1. Create accounts on [rapidapi.com](https://rapidapi.com) and [paypal.com](https://www.paypal.com).
2. Link PayPal to RapidAPI as the payout method.
3. Submit each API to the RapidAPI marketplace using `docs/rapidapi-listings.md`.
4. Configure paid tiers per `ROADMAP.md`.

After submission, RapidAPI generates a `X-RapidAPI-Proxy-Secret` per API. Create:

```bash
# zip-enrichment
kubectl create secret generic rapidapi-proxy-secret \
  -n zip-enrichment \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# holidays
kubectl create secret generic rapidapi-proxy-secret \
  -n holidays \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# air-quality
kubectl create secret generic rapidapi-proxy-secret \
  -n air-quality \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>
```

---

## Quick Verification Checklist

```bash
# Flux overall health
flux get all -A

# API service pods
kubectl get pods -n zip-enrichment
kubectl get pods -n holidays
kubectl get pods -n air-quality

# Ingress + TLS
kubectl get ingress -A
kubectl get certificates -A

# Gitea runner
kubectl get pods -n gitea-runner

# Monitoring stack
kubectl get pods -n monitoring

# Image automation
flux get imagepolicies -A
flux get imagerepositories -A
```

---

## Dependency Summary

```
Phase 0: Create 0xWheatyz/api-company repo
    └─► unblocks agent deployment PRs to upstream

Phase 1: Wire Flux (gitea-token secret + Talos manifests)
    └─► all flux/ manifests reconcile

Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry)
    └─► runner online, monitoring live, registry reachable

Phase 3: CI secrets + image-automation token
    └─► images build, push, and auto-update → API services deploy

Phase 4: DNS records
    └─► HTTPS certs issued → public URLs go live

Phase 5: RapidAPI + PayPal
    └─► revenue enabled
```

---

## Related Documents

| Document | Purpose |
|----------|---------|
| `docs/secrets-checklist.md` | Full checklist of all required secrets |
| `docs/registry.md` | Container registry architecture decision |
| `docs/cluster-audit.md` | Node/namespace/ingress inventory |
| `docs/rapidapi-listings.md` | RapidAPI marketplace submission details |
| `ROADMAP.md` | Full project roadmap and milestones |
| `STATUS.md` | Current cycle status and blockers |