api-company/docs/operator-runbook.md

Operator Runbook

Audience: Human operator (0xWheatyz)
Purpose: Ordered, copy-paste-ready guide to bring the full api-company stack live.
Last updated: 2026-05-26
Closes: leeworks-agents/api-company#50

---

Overview

The agent has committed all Flux manifests and documentation. The only remaining work is a set of manual steps that require Gitea admin access, kubectl access to the testing1 cluster, and external service accounts (RapidAPI, Slack, PayPal).

Work through these phases in order — each phase unblocks the next.

---

Phase 0 — Create upstream repo (unblocks all ship cycles)

Why first? Every agent deployment cycle fails to open a PR to upstream because 0xWheatyz/api-company does not yet exist. This one step unblocks all automated deployments. See issues #41, #47.

Step 0-A — Create 0xWheatyz/api-company on Gitea

1. Log into Gitea as 0xWheatyz.
2. + → New Repository.
3. Owner: 0xWheatyz, Name: api-company.
4. Visibility: Public (or Private — your choice).
5. Do not initialise with a README.
6. Click Create Repository.

Once created, the agent's next ship cycle will open a deployment PR automatically.

---

Phase 1 — Wire Flux to this repo (unblocks all GitOps reconciliation)

Why second? Until Flux watches leeworks-agents/api-company, none of the manifests in flux/ are applied to the cluster. See issue #2.

Step 1-A — Create gitea-leeworks-agents-token secret in flux-system

# In Gitea: User Settings → Applications → Generate Token
# Scopes: read:repository  (read-only is sufficient for Flux)
# Copy the token, then:

kubectl create secret generic gitea-leeworks-agents-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN>

Step 1-B — Copy Flux source + kustomization into 0xWheatyz/Talos

Reference manifests are at flux/api-company-source/ in this repo.
Copy them verbatim to:

0xWheatyz/Talos:testing1/first-cluster/cluster/flux/api-company-source/
├── gitrepository.yaml
└── kustomization.yaml

You can do this via the Gitea web editor or locally:

cd /path/to/Talos-checkout
mkdir -p testing1/first-cluster/cluster/flux/api-company-source
# copy the two files from api-company/flux/api-company-source/
git add .
git commit -m "feat: wire Flux GitRepository + Kustomization for api-company"
git push origin main

Verify reconciliation (after ~5 minutes):

flux get sources git -n flux-system
flux get kustomizations -n flux-system

Both api-company entries should show Ready = True.

---

Phase 2 — Secrets for already-staged services

Once Flux is watching the repo, it will attempt to reconcile all flux/ sub-directories. The HelmReleases will stall on missing secrets. Create them:

Step 2-A — gitea-runner-token (unblocks Gitea Actions runner, issue #3)

# In Gitea: Admin Panel → Site Administration → Actions → Runners
# → Create new Runner → copy registration token

kubectl create secret generic gitea-runner-token \
  -n gitea-runner \
  --from-literal=token=<RUNNER_REGISTRATION_TOKEN>

Verify:

kubectl get pods -n gitea-runner
# Then check Gitea Admin → Actions → Runners — runner should appear Online

Step 2-B — grafana-admin (unblocks Grafana, issue #7)

kubectl create secret generic grafana-admin \
  -n monitoring \
  --from-literal=admin-password=<CHOOSE_STRONG_PASSWORD>

Grafana URL: https://grafana.leeworks.dev (login: admin / <PASSWORD>)

Step 2-C — gatus-slack-webhook (unblocks Gatus alerts, issue #8)

# Create an incoming webhook at: https://api.slack.com/messaging/webhooks

kubectl create secret generic gatus-slack-webhook \
  -n monitoring \
  --from-literal=url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Gatus URL: https://status.leeworks.dev

Step 2-D — Enable Gitea packages + registry DNS (issue #4)

4a — Enable packages in Gitea app.ini:

[packages]
ENABLED = true

Restart Gitea after editing app.ini.

4b — Add DNS A record:

registry.leeworks.dev  →  <cluster ingress IP>

Find the ingress IP:

kubectl get svc -n ingress-nginx

See docs/registry.md for additional context.

---

Phase 3 — Enable CI image push (unblocks API service deployments)

Once the runner is online and the registry is reachable, CI pipelines can build and push container images.

Step 3-A — Add GITEA_TOKEN Actions Secret to each repo

Repos to configure:

• leeworks-agents/api-company
• leeworks-agents/zip-enrichment
• leeworks-agents/holidays
• leeworks-agents/air-quality

For each repo: Repo → Settings → Actions → Secrets → Add Secret

• Name: GITEA_TOKEN
• Value: Gitea personal access token with write:packages scope

Step 3-B — Create image-automation token secret (issue #51)

The agent has added ImageRepository + ImagePolicy + ImageUpdateAutomation manifests to flux/image-automation/. Flux will automatically update image tags in HelmReleases when CI pushes new images — but it needs write access to commit back:

kubectl create secret generic gitea-image-automation-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN_WITH_WRITE_REPO>

---

Phase 4 — DNS for API services (issue #33)

Add DNS A records for all seven leeworks.dev subdomains (all point to the same cluster ingress IP):

Hostname	Target
zip.leeworks.dev	<cluster ingress IP>
holidays.leeworks.dev	<cluster ingress IP>
aqi.leeworks.dev	<cluster ingress IP>
docs.leeworks.dev	<cluster ingress IP>
grafana.leeworks.dev	<cluster ingress IP>
status.leeworks.dev	<cluster ingress IP>
registry.leeworks.dev	<cluster ingress IP>

Verify DNS propagation:

for host in zip holidays aqi docs grafana status registry; do
  echo -n "${host}.leeworks.dev: "
  dig ${host}.leeworks.dev +short
done

cert-manager will obtain Let's Encrypt certificates automatically once DNS propagates (typically minutes, up to 48 h).

---

Phase 5 — RapidAPI + PayPal (issue #44, #19)

Blocked on operator being 18+ for PayPal. Complete when eligible.

1. Create accounts on rapidapi.com and paypal.com.
2. Link PayPal to RapidAPI as the payout method.
3. Submit each API to the RapidAPI marketplace using docs/rapidapi-listings.md.
4. Configure paid tiers per ROADMAP.md.

After submission, RapidAPI generates a X-RapidAPI-Proxy-Secret per API. Create:

# zip-enrichment
kubectl create secret generic rapidapi-proxy-secret \
  -n zip-enrichment \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# holidays
kubectl create secret generic rapidapi-proxy-secret \
  -n holidays \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# air-quality
kubectl create secret generic rapidapi-proxy-secret \
  -n air-quality \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

---

Quick Verification Checklist

# Flux overall health
flux get all -A

# API service pods
kubectl get pods -n zip-enrichment
kubectl get pods -n holidays
kubectl get pods -n air-quality

# Ingress + TLS
kubectl get ingress -A
kubectl get certificates -A

# Gitea runner
kubectl get pods -n gitea-runner

# Monitoring stack
kubectl get pods -n monitoring

# Image automation
flux get imagepolicies -A
flux get imagerepositories -A

---

Dependency Summary

Phase 0: Create 0xWheatyz/api-company repo
    └─► unblocks agent deployment PRs to upstream

Phase 1: Wire Flux (gitea-token secret + Talos manifests)
    └─► all flux/ manifests reconcile

Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry)
    └─► runner online, monitoring live, registry reachable

Phase 3: CI secrets + image-automation token
    └─► images build, push, and auto-update → API services deploy

Phase 4: DNS records
    └─► HTTPS certs issued → public URLs go live

Phase 5: RapidAPI + PayPal
    └─► revenue enabled

---

Related Documents

Document	Purpose
docs/secrets-checklist.md	Full checklist of all required secrets
docs/registry.md	Container registry architecture decision
docs/cluster-audit.md	Node/namespace/ingress inventory
docs/rapidapi-listings.md	RapidAPI marketplace submission details
ROADMAP.md	Full project roadmap and milestones
STATUS.md	Current cycle status and blockers