api-company/docs/operator-runbook.md

Operator Runbook

Audience: Human operator (0xWheatyz)
Purpose: Ordered, copy-paste-ready guide to bring the full api-company stack live.
Last updated: 2026-06-01
Closes: leeworks-agents/api-company#50

---

Overview

The agent has committed all Flux manifests and documentation. The only remaining work is a set of manual steps that require Gitea admin access, kubectl access to the testing1 cluster, and external service accounts (RapidAPI, Slack, PayPal).

Work through these phases in order — each phase unblocks the next.

---

Phase 0 — Create upstream repo (unblocks all ship cycles)

Why first? Every agent deployment cycle fails to open a PR to upstream because 0xWheatyz/api-company does not yet exist. This one step unblocks all automated deployments. See issues #41, #47.

Step 0-A — Create 0xWheatyz/api-company on Gitea

1. Log into Gitea as 0xWheatyz.
2. + → New Repository.
3. Owner: 0xWheatyz, Name: api-company.
4. Visibility: Public (or Private — your choice).
5. Do not initialise with a README.
6. Click Create Repository.

Once created, the agent's next ship cycle will open a deployment PR automatically.

---

Phase 1 — Wire Flux to this repo (unblocks all GitOps reconciliation)

Why second? Until Flux watches leeworks-agents/api-company, none of the manifests in flux/ are applied to the cluster. See issue #2.

Step 1-A — Create gitea-leeworks-agents-token secret in flux-system

# In Gitea: User Settings → Applications → Generate Token
# Scopes: read:repository  (read-only is sufficient for Flux)
# Copy the token, then:

kubectl create secret generic gitea-leeworks-agents-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN>

Step 1-B — Copy Flux source + kustomization into 0xWheatyz/Talos

Reference manifests are at flux/api-company-source/ in this repo.
Copy them verbatim to:

0xWheatyz/Talos:testing1/first-cluster/cluster/flux/api-company-source/
├── gitrepository.yaml
└── kustomization.yaml

You can do this via the Gitea web editor or locally:

cd /path/to/Talos-checkout
mkdir -p testing1/first-cluster/cluster/flux/api-company-source
# copy the two files from api-company/flux/api-company-source/
git add .
git commit -m "feat: wire Flux GitRepository + Kustomization for api-company"
git push origin main

Verify reconciliation (after ~5 minutes):

flux get sources git -n flux-system
flux get kustomizations -n flux-system

Both api-company entries should show Ready = True.

---

Phase 2 — Secrets for already-staged services

Once Flux is watching the repo, it will attempt to reconcile all flux/ sub-directories. The HelmReleases will stall on missing secrets. Create them:

Step 2-A — gitea-runner-token (unblocks Gitea Actions runner, issue #3)

# In Gitea: Admin Panel → Site Administration → Actions → Runners
# → Create new Runner → copy registration token

kubectl create secret generic gitea-runner-token \
  -n gitea-runner \
  --from-literal=token=<RUNNER_REGISTRATION_TOKEN>

Verify:

kubectl get pods -n gitea-runner
# Then check Gitea Admin → Actions → Runners — runner should appear Online

Step 2-B — grafana-admin (unblocks Grafana, issue #7)

kubectl create secret generic grafana-admin \
  -n monitoring \
  --from-literal=admin-password=<CHOOSE_STRONG_PASSWORD>

Grafana URL: https://grafana.leeworks.dev (login: admin / <PASSWORD>)

Step 2-C — gatus-slack-webhook (unblocks Gatus alerts, issue #8)

# Create an incoming webhook at: https://api.slack.com/messaging/webhooks

kubectl create secret generic gatus-slack-webhook \
  -n monitoring \
  --from-literal=url=https://hooks.slack.com/services/YOUR/WEBHOOK/URL

Gatus URL: https://status.leeworks.dev

Step 2-D — Enable Gitea packages + registry DNS (issue #4)

4a — Enable packages in Gitea app.ini:

[packages]
ENABLED = true

Restart Gitea after editing app.ini.

4b — Add DNS A record:

registry.leeworks.dev  →  <cluster ingress IP>

Find the ingress IP:

kubectl get svc -n ingress-nginx

See docs/registry.md for additional context.

---

Phase 3 — Enable CI image push (unblocks API service deployments)

Once the runner is online and the registry is reachable, CI pipelines can build and push container images.

Step 3-A — Add GITEA_TOKEN Actions Secret to each repo

Repos to configure:

• leeworks-agents/api-company
• leeworks-agents/zip-enrichment
• leeworks-agents/holidays
• leeworks-agents/air-quality
• leeworks-agents/vin-decoder (see also Phase 4-A Step 4)

For each repo: Repo → Settings → Actions → Secrets → Add Secret

• Name: GITEA_TOKEN
• Value: Gitea personal access token with write:packages scope

Step 3-B — Create image-automation token secret (issue #51)

The agent has added ImageRepository + ImagePolicy + ImageUpdateAutomation manifests to flux/image-automation/. Flux will automatically update image tags in HelmReleases when CI pushes new images — but it needs write access to commit back:

kubectl create secret generic gitea-image-automation-token \
  -n flux-system \
  --from-literal=username=leeworks-agents \
  --from-literal=password=<GITEA_TOKEN_WITH_WRITE_REPO>

---

Phase 4 — DNS for API services (issues #33, #106, #150)

Add DNS A records for all eight leeworks.dev subdomains (all point to the same cluster ingress IP):

Hostname	Target
zip.leeworks.dev	<cluster ingress IP>
holidays.leeworks.dev	<cluster ingress IP>
aqi.leeworks.dev	<cluster ingress IP>
vin.leeworks.dev	<cluster ingress IP>
docs.leeworks.dev	<cluster ingress IP>
grafana.leeworks.dev	<cluster ingress IP>
status.leeworks.dev	<cluster ingress IP>
registry.leeworks.dev	<cluster ingress IP>

Verify DNS propagation:

for host in zip holidays aqi vin docs grafana status registry; do
  echo -n "${host}.leeworks.dev: "
  dig ${host}.leeworks.dev +short
done

cert-manager will obtain Let's Encrypt certificates automatically once DNS propagates (typically minutes, up to 48 h).

---

---

Phase 4-A — VIN Decoder namespace setup (issues #126, #127, #128, #139)

Before VIN Decoder pods can start, the following manual steps are required. Do these alongside Phase 3 (they are independent of the DNS batch):

Step 4A-1 — Create vin-decoder namespace

kubectl create namespace vin-decoder --dry-run=client -o yaml | kubectl apply -f -

Step 4A-2 — Create gitea-registry imagePullSecret in vin-decoder namespace (issue #127)

Reuse the same Gitea token with read:packages scope from secrets checklist item #8.

kubectl create secret docker-registry gitea-registry \
  --namespace=vin-decoder \
  --docker-server=registry.leeworks.dev \
  --docker-username=leeworks-agents \
  --docker-password=<GITEA_TOKEN_WITH_READ_PACKAGES> \
  --docker-email=agent@leeworks.dev

Verify:

kubectl get secret gitea-registry -n vin-decoder -o jsonpath='{.type}'
# Expected: kubernetes.io/dockerconfigjson

Step 4A-3 — Create rapidapi-proxy-secret in vin-decoder namespace (issue #128)

Now (placeholder — unblocks deploy testing):

kubectl create secret generic rapidapi-proxy-secret \
  --namespace=vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=PLACEHOLDER_REPLACE_AFTER_RAPIDAPI_LISTING

After VIN Decoder is listed on RapidAPI (Phase 5), update with real secret:

kubectl create secret generic rapidapi-proxy-secret \
  -n vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_RAPIDAPI_PROXY_SECRET> \
  --save-config --dry-run=client -o yaml | kubectl apply -f -

Step 4A-4 — Add GITEA_TOKEN Actions secret to leeworks-agents/vin-decoder repo (issue #126)

Gitea → leeworks-agents/vin-decoder → Settings → Secrets and Variables → Actions

• Name: GITEA_TOKEN
• Value: Gitea personal access token with write:packages scope

Verify it appears in the repo's Actions Secrets list before the next push to main.

Step 4A-5 — Add RapidAPI VIN secrets to leeworks-agents/api-company Actions (issue #139)

Blocked — requires VIN Decoder to be listed on RapidAPI first (issue #131 tracker; operator action required).

Once the VIN Decoder listing is live:

Gitea → leeworks-agents/api-company → Settings → Secrets and Variables → Actions

Secret name	Where to find it
RAPIDAPI_VIN_API_ID	RapidAPI dashboard → VIN Decoder listing → Overview
RAPIDAPI_VIN_VERSION_ID	RapidAPI dashboard → VIN Decoder listing → Versions tab

Phase 5 — RapidAPI + PayPal (issue #44, #19)

Blocked on operator being 18+ for PayPal. Complete when eligible.

1. Create accounts on rapidapi.com and paypal.com.
2. Link PayPal to RapidAPI as the payout method.
3. Submit each API to the RapidAPI marketplace using docs/rapidapi-listings.md.
4. Configure paid tiers per ROADMAP.md.

After submission, RapidAPI generates a X-RapidAPI-Proxy-Secret per API. Create:

# zip-enrichment
kubectl create secret generic rapidapi-proxy-secret \
  -n zip-enrichment \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# holidays
kubectl create secret generic rapidapi-proxy-secret \
  -n holidays \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# air-quality
kubectl create secret generic rapidapi-proxy-secret \
  -n air-quality \
  --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>

# vin-decoder (updates the placeholder secret from Phase 4-A Step 3)
kubectl create secret generic rapidapi-proxy-secret \
  -n vin-decoder \
  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_VALUE> \
  --save-config --dry-run=client -o yaml | kubectl apply -f -

Also configure VIN Decoder pricing tiers on RapidAPI (issue #151):

Tier	Monthly Price	Request Limit	Rate Limit
Free	$0	100 req/mo	5 req/min
Basic	$9	5,000 req/mo	60 req/min
Pro	$19	20,000 req/mo	200 req/min
Ultra	$49	100,000 req/mo	500 req/min

Confirm VIN Decoder revenue flows through the same PayPal account as the other 3 APIs.

---

Quick Verification Checklist

# Flux overall health
flux get all -A

# API service pods
kubectl get pods -n zip-enrichment
kubectl get pods -n holidays
kubectl get pods -n air-quality

# Ingress + TLS
kubectl get ingress -A
kubectl get certificates -A

# Gitea runner
kubectl get pods -n gitea-runner

# Monitoring stack
kubectl get pods -n monitoring

# Image automation
flux get imagepolicies -A
flux get imagerepositories -A

---

Dependency Summary

Phase 0: Create 0xWheatyz/api-company repo
    └─► unblocks agent deployment PRs to upstream

Phase 1: Wire Flux (gitea-token secret + Talos manifests)
    └─► all flux/ manifests reconcile

Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry)
    └─► runner online, monitoring live, registry reachable

Phase 3: CI secrets + image-automation token
    └─► images build, push, and auto-update → API services deploy
    (includes leeworks-agents/vin-decoder GITEA_TOKEN — Step 3-A)

Phase 4: DNS records (8 subdomains including vin.leeworks.dev)
    └─► HTTPS certs issued → public URLs go live

Phase 4-A: VIN Decoder namespace setup (#126, #127, #128)
    4A-1: vin-decoder namespace created
    4A-2: gitea-registry imagePullSecret in vin-decoder (#127)
    4A-3: rapidapi-proxy-secret placeholder in vin-decoder (#128)
    4A-4: GITEA_TOKEN Actions secret in vin-decoder repo (#126)
    4A-5: RAPIDAPI_VIN_API_ID + RAPIDAPI_VIN_VERSION_ID Actions secrets (#139)
          [blocked on VIN Decoder RapidAPI listing]
    └─► VIN Decoder pods start; CI can push images

Phase 5: RapidAPI + PayPal (all 4 APIs including VIN Decoder; issue #151)
    └─► revenue enabled for all 4 APIs

---

Related Documents

Document	Purpose
docs/secrets-checklist.md	Full checklist of all required secrets
docs/registry.md	Container registry architecture decision
docs/cluster-audit.md	Node/namespace/ingress inventory
docs/rapidapi-listings.md	RapidAPI marketplace submission details
ROADMAP.md	Full project roadmap and milestones
STATUS.md	Current cycle status and blockers