feat(monitoring): add APIDataCriticallyStale alert and fix APIDataStale for duration

- Fix APIDataStale 'for' duration from 5m to 30m (issue spec requires 30m)
- Add APIDataCriticallyStale alert at 60-day threshold (5184000s) with for: 1h
  and severity: critical

Both alerts fire on api_data_freshness_seconds metric for all three API jobs.
Existing APIHighErrorRate and APIHighLatency alerts are unmodified.
kustomize build flux/ passes.

Closes leeworks-agents/api-company#88

.gitea/workflows/validate-flux.yaml

@@ -0,0 +1,21 @@
+name: Validate Flux manifests
+
+on:
+  pull_request:
+    branches: [main]
+  push:
+    branches: [main]
+
+jobs:
+  kustomize-build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install kustomize
+        run: |
+          curl -sL "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
+          sudo mv kustomize /usr/local/bin/
+
+      - name: kustomize build flux/
+        run: kustomize build flux/ > /dev/null

STATUS.md

@@ -1,6 +1,6 @@
 # Company Status
 
-_Last updated: 2026-05-27 (agent cycle — evening triage)_
+_Last updated: 2026-05-28 (agent cycle — cycle-2 triage)_
 
 ## APIs
 | API            | Spec | Code | Deployed | Listed on RapidAPI | Paying Users | MRR |
@@ -21,7 +21,42 @@ Legend: [x]=done, [~]=in-progress, [ ]=not started
 - **API service manifests:** `flux/zip-enrichment/`, `flux/holidays/`, `flux/air-quality/` scaffolded (PR #48, closes #46)
 - **Image automation:** `flux/image-automation/` — ImageRepository + ImagePolicy + ImageUpdateAutomation for all three APIs
 
-## Completed This Cycle (2026-05-27 — Evening Triage)
+## Completed This Cycle (2026-05-28 — Cycle-2 Triage)
+- **#83** — Triaged: `GITEA_TOKEN` Actions secrets needed in `zip-enrichment`, `holidays`, `air-quality` repos. Manual operator task; reuse token from #74 if it has `write:package` scope. Step-by-step instructions posted.
+- **No open PRs** — queue empty.
+- **`kustomize build flux/` = PASS** — no regressions.
+- **Upstream `0xWheatyz/api-company`** — still does not exist; fork sync skipped (blocker #47).
+
+## Completed Previous Cycle (2026-05-28 — Late-Night Triage)
+- **#79** — Triaged: `gitea-registry` imagePullSecret needed in 4 namespaces. Manual operator task; kubectl docker-registry secret instructions posted.
+- **#80** — Triaged: `gitea-image-automation-token` secret needed in `flux-system`. Manual operator task; write-scoped PAT required, kubectl instructions posted.
+- **#81** — Triaged: `rapidapi-proxy-secret` needed in `zip-enrichment`, `holidays`, `air-quality`. Manual operator task; placeholder + real-secret commands posted.
+- **No open PRs** — queue empty.
+- **`kustomize build flux/` = PASS** — no regressions.
+
+## Completed Previous Cycle (2026-05-28 — Night Triage)
+- **#76** — Triaged: `gitea-leeworks-agents-token` secret required in `flux-system` to unblock Flux GitRepository auth. Manual operator task; kubectl instructions + reconcile steps posted in issue comment.
+- **#77** — Triaged: `gitea-runner-token` secret required in `gitea-runner` namespace to register Act Runner. Manual operator task; full step-by-step instructions posted in issue comment.
+- **No open PRs** — queue empty.
+- **`kustomize build flux/` = PASS** — no regressions.
+
+## Completed Previous Cycle (2026-05-28 — Evening Triage)
+- **#74** — Triaged: `GITEA_TOKEN` Actions secret required in leeworks-agents/api-company for docs-site CI. Manual operator task; instructions posted.
+- **#73** — Triaged: `gatus-slack-webhook` secret required in `monitoring` namespace. Manual operator task; kubectl instructions posted.
+- **#67** — Confirmed already resolved: `docs/operator-runbook.md` and `docs/dns.md` both list all 7 subdomains incl. `grafana.leeworks.dev` (PR #63). No code change needed.
+- **No open PRs** — queue empty.
+- **`kustomize build flux/` = PASS** — no regressions.
+
+## Completed Previous Cycle (2026-05-28 — Morning Triage)
+- **#69 → PR #71 MERGED** — Added `.gitea/workflows/validate-flux.yaml` CI workflow: runs `kustomize build flux/` on every PR and push to `main`. CI gate enforced once Act Runner (#3) is online.
+- **#70** — Triaged: `grafana-admin` secret needed in `monitoring` namespace. Manual operator task; kubectl instructions posted in issue comment.
+- **#67** — Confirmed `docs/operator-runbook.md` Phase 4 DNS table already lists all 7 subdomains including grafana.leeworks.dev. No code change needed.
+- **#66** — Blocked on Phase 0 operator actions (#47, #2, #3, #4). Status comment posted.
+- **#47, #2, #3, #4, #7, #8, #16, #17, #18, #19, #27, #30, #33, #44** — Status comments posted confirming blocked/awaiting-operator state.
+- **No open PRs to review** — queue empty after PR #71 merged.
+- **`kustomize build flux/` = PASS** — no regressions.
+
+## Completed Previous Cycle (2026-05-27 — Evening Triage)
 - **Triage pass** — all 15 open agent-ready issues reviewed; status comments posted on each.
 - **No open PRs** — nothing to merge or review.
 - **No new feature work** — all issues remain blocked on operator actions or cross-repo prerequisites.

docs/dns.md

@@ -102,6 +102,7 @@ dig aqi.leeworks.dev +short
 dig docs.leeworks.dev +short
 dig status.leeworks.dev +short
 dig registry.leeworks.dev +short
+dig grafana.leeworks.dev +short
 
 # Check TLS certificates (once services are deployed)
 curl -v https://zip.leeworks.dev/health 2>&1 | grep -E "SSL|certificate|issuer"
@@ -125,7 +126,7 @@ The following actions require human operator access to the DNS provider:
 
 1. Log into the DNS provider managing `leeworks.dev`
 2. Find the cluster ingress IP: `kubectl get svc -n ingress-nginx ingress-nginx-controller`
-3. Create/update the 6 A records listed in the table above
+3. Create/update the 7 A records listed in the table above
 4. Verify propagation: `dig +trace zip.leeworks.dev`
 
 DNS propagation typically takes 5–60 minutes.
@@ -141,4 +142,5 @@ DNS propagation typically takes 5–60 minutes.
 - [ ] `docs.leeworks.dev` → DNS record created
 - [ ] `status.leeworks.dev` → DNS record created
 - [ ] `registry.leeworks.dev` → DNS record created
-- [ ] TLS certificates issued and valid for all 6 subdomains
+- [ ] `grafana.leeworks.dev` → DNS record created
+- [ ] TLS certificates issued and valid for all 7 subdomains

docs/operator-runbook.md

@@ -188,7 +188,7 @@ kubectl create secret generic gitea-image-automation-token \
 
 ## Phase 4 — DNS for API services (issue #33)
 
-Add DNS A records for all six leeworks.dev subdomains (all point to the same
+Add DNS A records for all seven leeworks.dev subdomains (all point to the same
 cluster ingress IP):
 
 | Hostname                | Target                 |
@@ -199,6 +199,16 @@ cluster ingress IP):
 | `docs.leeworks.dev`     | `<cluster ingress IP>` |
 | `grafana.leeworks.dev`  | `<cluster ingress IP>` |
 | `status.leeworks.dev`   | `<cluster ingress IP>` |
+| `registry.leeworks.dev` | `<cluster ingress IP>` |
+
+Verify DNS propagation:
+
+```bash
+for host in zip holidays aqi docs grafana status registry; do
+  echo -n "${host}.leeworks.dev: "
+  dig ${host}.leeworks.dev +short
+done
+```
 
 cert-manager will obtain Let's Encrypt certificates automatically once DNS
 propagates (typically minutes, up to 48 h).

flux/air-quality/externalsecret.yaml

@@ -1,29 +1,18 @@
-# Placeholder: inject the RapidAPI Proxy Secret here once ESO is deployed.
-# Replace with a real ExternalSecret once leeworks-agents/api-company#2 and
-# the external-secrets operator are running in the cluster.
-#
-# Example (uncomment and fill in secretStore name):
-#
-# apiVersion: external-secrets.io/v1beta1
-# kind: ExternalSecret
-# metadata:
-#   name: rapidapi-proxy-secret
-#   namespace: air-quality
-# spec:
-#   refreshInterval: 1h
-#   secretStoreRef:
-#     name: <your-secret-store>
-#     kind: ClusterSecretStore
-#   target:
-#     name: rapidapi-proxy-secret
-#     creationPolicy: Owner
-#   data:
-#     - secretKey: X-RapidAPI-Proxy-Secret
-#       remoteRef:
-#         key: rapidapi/air-quality
-#         property: proxy-secret
-#
-# Until then, create manually:
-# kubectl create secret generic rapidapi-proxy-secret \
-#   --from-literal=X-RapidAPI-Proxy-Secret=<value> \
-#   -n air-quality
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: rapidapi-proxy-secret
+  namespace: air-quality
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: kubernetes-provider
+    kind: ClusterSecretStore
+  target:
+    name: rapidapi-proxy-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: X-RapidAPI-Proxy-Secret
+      remoteRef:
+        key: rapidapi-air-quality-source
+        property: X-RapidAPI-Proxy-Secret

flux/docs-site/helmrelease.yaml

@@ -35,7 +35,7 @@ spec:
                 - name: gitea-registry
               containers:
                 - name: docs-site
-                  image: registry.leeworks.dev/leeworks-agents/docs-site:latest
+                  image: registry.leeworks.dev/leeworks-agents/docs-site:latest # {"$imagepolicy": "flux-system:docs-site"}
                   ports:
                     - containerPort: 80
                   resources:

flux/external-secrets/clustersecretstore.yaml

@@ -0,0 +1,50 @@
+# ClusterSecretStore using the Kubernetes provider.
+# Reads secrets from the cluster itself — no external vault required.
+# The service account below must have get/list access to secrets in
+# the namespaces where ExternalSecrets are created.
+apiVersion: v1
+kind: ServiceAccount
+metadata:
+  name: eso-kubernetes-provider
+  namespace: external-secrets
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRole
+metadata:
+  name: eso-kubernetes-provider-reader
+rules:
+  - apiGroups: [""]
+    resources: ["secrets"]
+    verbs: ["get", "list", "watch"]
+---
+apiVersion: rbac.authorization.k8s.io/v1
+kind: ClusterRoleBinding
+metadata:
+  name: eso-kubernetes-provider-reader
+roleRef:
+  apiGroup: rbac.authorization.k8s.io
+  kind: ClusterRole
+  name: eso-kubernetes-provider-reader
+subjects:
+  - kind: ServiceAccount
+    name: eso-kubernetes-provider
+    namespace: external-secrets
+---
+apiVersion: external-secrets.io/v1beta1
+kind: ClusterSecretStore
+metadata:
+  name: kubernetes-provider
+spec:
+  provider:
+    kubernetes:
+      remoteNamespace: external-secrets
+      server:
+        caProvider:
+          type: ConfigMap
+          name: kube-root-ca.crt
+          namespace: external-secrets
+          key: ca.crt
+      auth:
+        serviceAccount:
+          name: eso-kubernetes-provider
+          namespace: external-secrets

flux/external-secrets/helmrelease.yaml

@@ -0,0 +1,30 @@
+apiVersion: helm.toolkit.fluxcd.io/v2
+kind: HelmRelease
+metadata:
+  name: external-secrets
+  namespace: external-secrets
+spec:
+  interval: 15m
+  chart:
+    spec:
+      chart: external-secrets
+      version: ">=0.9.0 <1.0.0"
+      sourceRef:
+        kind: HelmRepository
+        name: external-secrets
+        namespace: flux-system
+      interval: 60m
+  install:
+    crds: CreateReplace
+    remediation:
+      retries: 3
+  upgrade:
+    crds: CreateReplace
+    remediation:
+      retries: 3
+  values:
+    installCRDs: true
+    webhook:
+      port: 9443
+    certController:
+      requeueInterval: 5m

flux/external-secrets/helmrepository.yaml

@@ -0,0 +1,8 @@
+apiVersion: source.toolkit.fluxcd.io/v1
+kind: HelmRepository
+metadata:
+  name: external-secrets
+  namespace: flux-system
+spec:
+  interval: 60m
+  url: https://charts.external-secrets.io

flux/external-secrets/kustomization.yaml

@@ -0,0 +1,7 @@
+apiVersion: kustomize.config.k8s.io/v1beta1
+kind: Kustomization
+resources:
+  - namespace.yaml
+  - helmrepository.yaml
+  - helmrelease.yaml
+  - clustersecretstore.yaml

flux/external-secrets/namespace.yaml

@@ -0,0 +1,4 @@
+apiVersion: v1
+kind: Namespace
+metadata:
+  name: external-secrets

flux/holidays/externalsecret.yaml

@@ -1,29 +1,18 @@
-# Placeholder: inject the RapidAPI Proxy Secret here once ESO is deployed.
-# Replace with a real ExternalSecret once leeworks-agents/api-company#2 and
-# the external-secrets operator are running in the cluster.
-#
-# Example (uncomment and fill in secretStore name):
-#
-# apiVersion: external-secrets.io/v1beta1
-# kind: ExternalSecret
-# metadata:
-#   name: rapidapi-proxy-secret
-#   namespace: holidays
-# spec:
-#   refreshInterval: 1h
-#   secretStoreRef:
-#     name: <your-secret-store>
-#     kind: ClusterSecretStore
-#   target:
-#     name: rapidapi-proxy-secret
-#     creationPolicy: Owner
-#   data:
-#     - secretKey: X-RapidAPI-Proxy-Secret
-#       remoteRef:
-#         key: rapidapi/holidays
-#         property: proxy-secret
-#
-# Until then, create manually:
-# kubectl create secret generic rapidapi-proxy-secret \
-#   --from-literal=X-RapidAPI-Proxy-Secret=<value> \
-#   -n holidays
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: rapidapi-proxy-secret
+  namespace: holidays
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: kubernetes-provider
+    kind: ClusterSecretStore
+  target:
+    name: rapidapi-proxy-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: X-RapidAPI-Proxy-Secret
+      remoteRef:
+        key: rapidapi-holidays-source
+        property: X-RapidAPI-Proxy-Secret

flux/image-automation/imagepolicies.yaml

@@ -37,3 +37,15 @@ spec:
   policy:
     semver:
       range: ">=0.1.0"
+---
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImagePolicy
+metadata:
+  name: docs-site
+  namespace: flux-system
+spec:
+  imageRepositoryRef:
+    name: docs-site
+  policy:
+    alphabetical:
+      order: asc

flux/image-automation/imagerepositories.yaml

@@ -32,3 +32,14 @@ spec:
   interval: 5m
   secretRef:
     name: gitea-leeworks-agents-token
+---
+apiVersion: image.toolkit.fluxcd.io/v1beta2
+kind: ImageRepository
+metadata:
+  name: docs-site
+  namespace: flux-system
+spec:
+  image: registry.leeworks.dev/leeworks-agents/docs-site
+  interval: 5m
+  secretRef:
+    name: gitea-leeworks-agents-token

flux/kustomization.yaml

@@ -1,6 +1,7 @@
 apiVersion: kustomize.config.k8s.io/v1beta1
 kind: Kustomization
 resources:
+  - external-secrets
   - gitea-runner
   - monitoring
   - docs-site

flux/monitoring/prometheusrule-apis.yaml

@@ -71,7 +71,7 @@ spec:
         - alert: APIDataStale
           expr: |
             api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 2592000
-          for: 5m
+          for: 30m
           labels:
             severity: warning
             team: api-company
@@ -79,6 +79,20 @@ spec:
             summary: "Stale dataset on {{ $labels.job }} ({{ $labels.dataset }})"
             description: "{{ $labels.job }} dataset '{{ $labels.dataset }}' has not been re-seeded in {{ $value | humanizeDuration }} (threshold: 30 days). Re-seed required."
 
+        # -------------------------------------------------------------------
+        # APIDataCriticallyStale — data freshness > 60 days
+        # -------------------------------------------------------------------
+        - alert: APIDataCriticallyStale
+          expr: |
+            api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 5184000
+          for: 1h
+          labels:
+            severity: critical
+            team: api-company
+          annotations:
+            summary: "API data is critically stale on {{ $labels.job }}"
+            description: "{{ $labels.job }} data has not been re-seeded in more than 60 days ({{ $value | humanizeDuration }})"
+
         # -------------------------------------------------------------------
         # APIDown — any API job absent for 2 min
         # -------------------------------------------------------------------

flux/zip-enrichment/externalsecret.yaml

@@ -1,29 +1,18 @@
-# Placeholder: inject the RapidAPI Proxy Secret here once ESO is deployed.
-# Replace with a real ExternalSecret once leeworks-agents/api-company#2 and
-# the external-secrets operator are running in the cluster.
-#
-# Example (uncomment and fill in secretStore name):
-#
-# apiVersion: external-secrets.io/v1beta1
-# kind: ExternalSecret
-# metadata:
-#   name: rapidapi-proxy-secret
-#   namespace: zip-enrichment
-# spec:
-#   refreshInterval: 1h
-#   secretStoreRef:
-#     name: <your-secret-store>
-#     kind: ClusterSecretStore
-#   target:
-#     name: rapidapi-proxy-secret
-#     creationPolicy: Owner
-#   data:
-#     - secretKey: X-RapidAPI-Proxy-Secret
-#       remoteRef:
-#         key: rapidapi/zip-enrichment
-#         property: proxy-secret
-#
-# Until then, create manually:
-# kubectl create secret generic rapidapi-proxy-secret \
-#   --from-literal=X-RapidAPI-Proxy-Secret=<value> \
-#   -n zip-enrichment
+apiVersion: external-secrets.io/v1beta1
+kind: ExternalSecret
+metadata:
+  name: rapidapi-proxy-secret
+  namespace: zip-enrichment
+spec:
+  refreshInterval: 1h
+  secretStoreRef:
+    name: kubernetes-provider
+    kind: ClusterSecretStore
+  target:
+    name: rapidapi-proxy-secret
+    creationPolicy: Owner
+  data:
+    - secretKey: X-RapidAPI-Proxy-Secret
+      remoteRef:
+        key: rapidapi-zip-enrichment-source
+        property: X-RapidAPI-Proxy-Secret