api-company/docs/operator-runbook.md

# Operator Runbook

**Audience:** Human operator (0xWheatyz)  
**Purpose:** Ordered, copy-paste-ready guide to bring the full `api-company` stack live.  
**Last updated:** 2026-06-01  
**Closes:** leeworks-agents/api-company#50

---

## Overview

The agent has committed all Flux manifests and documentation. The only remaining
work is a set of manual steps that require Gitea admin access, `kubectl` access to
the `testing1` cluster, and external service accounts (RapidAPI, Slack, PayPal).

Work through these phases **in order** — each phase unblocks the next.

---

## Phase 0 — Create upstream repo (unblocks all ship cycles)

> **Why first?** Every agent deployment cycle fails to open a PR to upstream
> because `0xWheatyz/api-company` does not yet exist. This one step unblocks
> all automated deployments. See issues #41, #47.

### Step 0-A — Create `0xWheatyz/api-company` on Gitea

1. Log into Gitea as `0xWheatyz`.
2. **+** → **New Repository**.
3. Owner: `0xWheatyz`, Name: `api-company`.
4. Visibility: Public (or Private — your choice).
5. **Do not** initialise with a README.
6. Click **Create Repository**.

Once created, the agent's next ship cycle will open a deployment PR automatically.

---

## Phase 1 — Wire Flux to this repo (unblocks all GitOps reconciliation)

> **Why second?** Until Flux watches `leeworks-agents/api-company`, none of the
> manifests in `flux/` are applied to the cluster. See issue #2.

### Step 1-A — Create `gitea-leeworks-agents-token` secret in `flux-system`

```bash
# In Gitea: User Settings → Applications → Generate Token
# Scopes: read:repository  (read-only is sufficient for Flux)
# Copy the token, then:

kubectl create secret generic gitea-leeworks-agents-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN>
```

### Step 1-B — Copy Flux source + kustomization into 0xWheatyz/Talos

Reference manifests are at `flux/api-company-source/` in this repo.  
Copy them verbatim to:

```
0xWheatyz/Talos:testing1/first-cluster/cluster/flux/api-company-source/
├── gitrepository.yaml
└── kustomization.yaml
```

You can do this via the Gitea web editor or locally:

```bash
cd /path/to/Talos-checkout
mkdir -p testing1/first-cluster/cluster/flux/api-company-source
# copy the two files from api-company/flux/api-company-source/
git add .
git commit -m "feat: wire Flux GitRepository + Kustomization for api-company"
git push origin main
```

**Verify reconciliation (after ~5 minutes):**
```bash
flux get sources git -n flux-system
flux get kustomizations -n flux-system
```

Both `api-company` entries should show `Ready = True`.

---

## Phase 2 — Secrets for already-staged services

Once Flux is watching the repo, it will attempt to reconcile all `flux/`
sub-directories. The HelmReleases will stall on missing secrets. Create them:

### Step 2-A — `gitea-runner-token` (unblocks Gitea Actions runner, issue #3)

```bash
# In Gitea: Admin Panel → Site Administration → Actions → Runners
# → Create new Runner → copy registration token

kubectl create secret generic gitea-runner-token \
  -n gitea-runner \
  --from-literal=token=<RUNNER_REGISTRATION_TOKEN>
```

**Verify:**
```bash
kubectl get pods -n gitea-runner
# Then check Gitea Admin → Actions → Runners — runner should appear Online
```

### Step 2-B — `grafana-admin` (unblocks Grafana, issue #7)

```bash
kubectl create secret generic grafana-admin \
  -n monitoring \
  --from-literal=admin-password=<CHOOSE_STRONG_PASSWORD>
```

Grafana URL: `https://grafana.leeworks.dev` (login: `admin` / `<PASSWORD>`)

### Step 2-C — `gatus-slack-webhook` (unblocks Gatus alerts, issue #8)

```bash
# Create an incoming webhook at: https://api.slack.com/messaging/webhooks

kubectl create secret generic gatus-slack-webhook \
  -n monitoring \
  --from-literal=url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL
```

Gatus URL: `https://status.leeworks.dev`

### Step 2-D — Enable Gitea packages + registry DNS (issue #4)

**4a — Enable packages in Gitea `app.ini`:**
```ini
[packages]
ENABLED = true
```
Restart Gitea after editing `app.ini`.

**4b — Add DNS A record:**
```
registry.leeworks.dev  →  <cluster ingress IP>
```

Find the ingress IP:
```bash
kubectl get svc -n ingress-nginx
```

See `docs/registry.md` for additional context.

---

## Phase 3 — Enable CI image push (unblocks API service deployments)

Once the runner is online and the registry is reachable, CI pipelines can build
and push container images.

### Step 3-A — Add `GITEA_TOKEN` Actions Secret to each repo

Repos to configure:
- `leeworks-agents/api-company`
- `leeworks-agents/zip-enrichment`
- `leeworks-agents/holidays`
- `leeworks-agents/air-quality`
- `leeworks-agents/vin-decoder` (see also Phase 4-A Step 4)

**For each repo:** Repo → Settings → Actions → Secrets → Add Secret
- **Name:** `GITEA_TOKEN`
- **Value:** Gitea personal access token with `write:packages` scope

### Step 3-B — Create image-automation token secret (issue #51)

The agent has added `ImageRepository` + `ImagePolicy` + `ImageUpdateAutomation`
manifests to `flux/image-automation/`. Flux will automatically update image tags
in HelmReleases when CI pushes new images — but it needs write access to commit
back:

```bash
kubectl create secret generic gitea-image-automation-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN_WITH_WRITE_REPO>
```

---

## Phase 4 — DNS for API services (issues #33, #106, #150)

Add DNS A records for **all eight** leeworks.dev subdomains (all point to the same
cluster ingress IP):

| Hostname                | Target                 |
|-------------------------|------------------------|
| `zip.leeworks.dev`      | `<cluster ingress IP>` |
| `holidays.leeworks.dev` | `<cluster ingress IP>` |
| `aqi.leeworks.dev`      | `<cluster ingress IP>` |
| `vin.leeworks.dev`      | `<cluster ingress IP>` |
| `docs.leeworks.dev`     | `<cluster ingress IP>` |
| `grafana.leeworks.dev`  | `<cluster ingress IP>` |
| `status.leeworks.dev`   | `<cluster ingress IP>` |
| `registry.leeworks.dev` | `<cluster ingress IP>` |

Verify DNS propagation:

```bash
for host in zip holidays aqi vin docs grafana status registry; do
  echo -n "${host}.leeworks.dev: "
  dig ${host}.leeworks.dev +short
done
```

cert-manager will obtain Let's Encrypt certificates automatically once DNS
propagates (typically minutes, up to 48 h).

---

---

## Phase 4-A — VIN Decoder namespace setup (issues #126, #127, #128, #139)

Before VIN Decoder pods can start, the following manual steps are required.
Do these alongside Phase 3 (they are independent of the DNS batch):

### Step 4A-1 — Create `vin-decoder` namespace

```bash
kubectl create namespace vin-decoder --dry-run=client -o yaml | kubectl apply -f -
```

### Step 4A-2 — Create `gitea-registry` imagePullSecret in `vin-decoder` namespace (issue #127)

Reuse the same Gitea token with `read:packages` scope from secrets checklist item #8.

```bash
kubectl create secret docker-registry gitea-registry \
  --namespace=vin-decoder \
  --docker-server=registry.leeworks.dev \
  --docker-username=leeworks-agents \
  --docker-password=<GITEA_TOKEN_WITH_READ_PACKAGES> \
  --docker-email=agent@leeworks.dev
```

Verify:
```bash
kubectl get secret gitea-registry -n vin-decoder -o jsonpath='{.type}'
# Expected: kubernetes.io/dockerconfigjson
```

### Step 4A-3 — Create `rapidapi-proxy-secret` in `vin-decoder` namespace (issue #128)

**Now (placeholder — unblocks deploy testing):**
```bash
kubectl create secret generic rapidapi-proxy-secret \
  --namespace=vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=PLACEHOLDER_REPLACE_AFTER_RAPIDAPI_LISTING
```

**After VIN Decoder is listed on RapidAPI (Phase 5), update with real secret:**
```bash
kubectl create secret generic rapidapi-proxy-secret \
  -n vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_RAPIDAPI_PROXY_SECRET> \
  --save-config --dry-run=client -o yaml | kubectl apply -f -
```

### Step 4A-4 — Add `GITEA_TOKEN` Actions secret to `leeworks-agents/vin-decoder` repo (issue #126)

Gitea → `leeworks-agents/vin-decoder` → Settings → Secrets and Variables → Actions
- **Name:** `GITEA_TOKEN`
- **Value:** Gitea personal access token with `write:packages` scope

Verify it appears in the repo's Actions Secrets list before the next push to `main`.

### Step 4A-5 — Add RapidAPI VIN secrets to `leeworks-agents/api-company` Actions (issue #139)

> **Blocked** — requires VIN Decoder to be listed on RapidAPI first (issue #131 tracker; operator action required).

Once the VIN Decoder listing is live:

Gitea → `leeworks-agents/api-company` → Settings → Secrets and Variables → Actions

| Secret name               | Where to find it |
|---------------------------|------------------|
| `RAPIDAPI_VIN_API_ID`     | RapidAPI dashboard → VIN Decoder listing → Overview |
| `RAPIDAPI_VIN_VERSION_ID` | RapidAPI dashboard → VIN Decoder listing → Versions tab |

## Phase 5 — RapidAPI + PayPal (issue #44, #19)

> **Blocked on operator being 18+ for PayPal.** Complete when eligible.

1. Create accounts on [rapidapi.com](https://rapidapi.com) and [paypal.com](https://www.paypal.com).
2. Link PayPal to RapidAPI as the payout method.
3. Submit each API to the RapidAPI marketplace using `docs/rapidapi-listings.md`.
4. Configure paid tiers per `ROADMAP.md`.

After submission, RapidAPI generates a `X-RapidAPI-Proxy-Secret` per API. Create:

```bash
# zip-enrichment
kubectl create secret generic rapidapi-proxy-secret \
  -n zip-enrichment \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# holidays
kubectl create secret generic rapidapi-proxy-secret \
  -n holidays \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# air-quality
kubectl create secret generic rapidapi-proxy-secret \
  -n air-quality \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# vin-decoder (updates the placeholder secret from Phase 4-A Step 3)
kubectl create secret generic rapidapi-proxy-secret \
  -n vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_VALUE> \
  --save-config --dry-run=client -o yaml | kubectl apply -f -
```

Also configure VIN Decoder pricing tiers on RapidAPI (issue #151):

| Tier  | Monthly Price | Request Limit  | Rate Limit  |
|-------|---------------|----------------|-------------|
| Free  | $0            | 100 req/mo     | 5 req/min   |
| Basic | $9            | 5,000 req/mo   | 60 req/min  |
| Pro   | $19           | 20,000 req/mo  | 200 req/min |
| Ultra | $49           | 100,000 req/mo | 500 req/min |

Confirm VIN Decoder revenue flows through the same PayPal account as the other 3 APIs.

---

## Quick Verification Checklist

```bash
# Flux overall health
flux get all -A

# API service pods
kubectl get pods -n zip-enrichment
kubectl get pods -n holidays
kubectl get pods -n air-quality

# Ingress + TLS
kubectl get ingress -A
kubectl get certificates -A

# Gitea runner
kubectl get pods -n gitea-runner

# Monitoring stack
kubectl get pods -n monitoring

# Image automation
flux get imagepolicies -A
flux get imagerepositories -A
```

---

## Dependency Summary

```
Phase 0: Create 0xWheatyz/api-company repo
    └─► unblocks agent deployment PRs to upstream

Phase 1: Wire Flux (gitea-token secret + Talos manifests)
    └─► all flux/ manifests reconcile

Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry)
    └─► runner online, monitoring live, registry reachable

Phase 3: CI secrets + image-automation token
    └─► images build, push, and auto-update → API services deploy
    (includes leeworks-agents/vin-decoder GITEA_TOKEN — Step 3-A)

Phase 4: DNS records (8 subdomains including vin.leeworks.dev)
    └─► HTTPS certs issued → public URLs go live

Phase 4-A: VIN Decoder namespace setup (#126, #127, #128)
    4A-1: vin-decoder namespace created
    4A-2: gitea-registry imagePullSecret in vin-decoder (#127)
    4A-3: rapidapi-proxy-secret placeholder in vin-decoder (#128)
    4A-4: GITEA_TOKEN Actions secret in vin-decoder repo (#126)
    4A-5: RAPIDAPI_VIN_API_ID + RAPIDAPI_VIN_VERSION_ID Actions secrets (#139)
          [blocked on VIN Decoder RapidAPI listing]
    └─► VIN Decoder pods start; CI can push images

Phase 5: RapidAPI + PayPal (all 4 APIs including VIN Decoder; issue #151)
    └─► revenue enabled for all 4 APIs
```

---

## Related Documents

| Document | Purpose |
|----------|---------|
| `docs/secrets-checklist.md` | Full checklist of all required secrets |
| `docs/registry.md` | Container registry architecture decision |
| `docs/cluster-audit.md` | Node/namespace/ingress inventory |
| `docs/rapidapi-listings.md` | RapidAPI marketplace submission details |
| `ROADMAP.md` | Full project roadmap and milestones |
| `STATUS.md` | Current cycle status and blockers |