feat: extend monitoring to include VIN Decoder API

- flux/monitoring/prometheusrule-apis.yaml: extend all 5 SLO alert
  rules to include vin-decoder in the job regex filter
- flux/monitoring/grafana-dashboard-apis.yaml: update all PromQL
  expressions to include vin-decoder in the api regex filter;
  update dashboard description
- flux/monitoring/vin-decoder-dashboard.yaml: new dedicated Grafana
  dashboard for VIN Decoder showing request rate by route, 5xx error
  rate, P50/P95/P99 latency by route, and data freshness gauge
- flux/monitoring/kustomization.yaml: add vin-decoder-dashboard.yaml
- flux/vin-decoder/servicemonitor.yaml: Prometheus ServiceMonitor
  for auto-discovery of /metrics on the vin-decoder service
- flux/vin-decoder/kustomization.yaml: add servicemonitor.yaml

Closes leeworks-agents/api-company#129

docs/operator-runbook.md

@@ -2,7 +2,7 @@
 
 **Audience:** Human operator (0xWheatyz)  
 **Purpose:** Ordered, copy-paste-ready guide to bring the full `api-company` stack live.  
-**Last updated:** 2026-06-01  
+**Last updated:** 2026-05-26  
 **Closes:** leeworks-agents/api-company#50
 
 ---
@@ -165,7 +165,6 @@ Repos to configure:
 - `leeworks-agents/zip-enrichment`
 - `leeworks-agents/holidays`
 - `leeworks-agents/air-quality`
-- `leeworks-agents/vin-decoder` (see also Phase 4-A Step 4)
 
 **For each repo:** Repo → Settings → Actions → Secrets → Add Secret
 - **Name:** `GITEA_TOKEN`
@@ -187,9 +186,9 @@ kubectl create secret generic gitea-image-automation-token \
 
 ---
 
-## Phase 4 — DNS for API services (issues #33, #106, #150)
+## Phase 4 — DNS for API services (issue #33)
 
-Add DNS A records for **all eight** leeworks.dev subdomains (all point to the same
+Add DNS A records for all seven leeworks.dev subdomains (all point to the same
 cluster ingress IP):
 
 | Hostname                | Target                 |
@@ -197,7 +196,6 @@ cluster ingress IP):
 | `zip.leeworks.dev`      | `<cluster ingress IP>` |
 | `holidays.leeworks.dev` | `<cluster ingress IP>` |
 | `aqi.leeworks.dev`      | `<cluster ingress IP>` |
-| `vin.leeworks.dev`      | `<cluster ingress IP>` |
 | `docs.leeworks.dev`     | `<cluster ingress IP>` |
 | `grafana.leeworks.dev`  | `<cluster ingress IP>` |
 | `status.leeworks.dev`   | `<cluster ingress IP>` |
@@ -206,7 +204,7 @@ cluster ingress IP):
 Verify DNS propagation:
 
 ```bash
-for host in zip holidays aqi vin docs grafana status registry; do
+for host in zip holidays aqi docs grafana status registry; do
   echo -n "${host}.leeworks.dev: "
   dig ${host}.leeworks.dev +short
 done
@@ -217,76 +215,6 @@ propagates (typically minutes, up to 48 h).
 
 ---
 
----
-
-## Phase 4-A — VIN Decoder namespace setup (issues #126, #127, #128, #139)
-
-Before VIN Decoder pods can start, the following manual steps are required.
-Do these alongside Phase 3 (they are independent of the DNS batch):
-
-### Step 4A-1 — Create `vin-decoder` namespace
-
-```bash
-kubectl create namespace vin-decoder --dry-run=client -o yaml | kubectl apply -f -
-```
-
-### Step 4A-2 — Create `gitea-registry` imagePullSecret in `vin-decoder` namespace (issue #127)
-
-Reuse the same Gitea token with `read:packages` scope from secrets checklist item #8.
-
-```bash
-kubectl create secret docker-registry gitea-registry \
-  --namespace=vin-decoder \
-  --docker-server=registry.leeworks.dev \
-  --docker-username=leeworks-agents \
-  --docker-password=<GITEA_TOKEN_WITH_READ_PACKAGES> \
-  --docker-email=agent@leeworks.dev
-```
-
-Verify:
-```bash
-kubectl get secret gitea-registry -n vin-decoder -o jsonpath='{.type}'
-# Expected: kubernetes.io/dockerconfigjson
-```
-
-### Step 4A-3 — Create `rapidapi-proxy-secret` in `vin-decoder` namespace (issue #128)
-
-**Now (placeholder — unblocks deploy testing):**
-```bash
-kubectl create secret generic rapidapi-proxy-secret \
-  --namespace=vin-decoder \
-  --from-literal=X-RapidAPI-Proxy-Secret=PLACEHOLDER_REPLACE_AFTER_RAPIDAPI_LISTING
-```
-
-**After VIN Decoder is listed on RapidAPI (Phase 5), update with real secret:**
-```bash
-kubectl create secret generic rapidapi-proxy-secret \
-  -n vin-decoder \
-  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_RAPIDAPI_PROXY_SECRET> \
-  --save-config --dry-run=client -o yaml | kubectl apply -f -
-```
-
-### Step 4A-4 — Add `GITEA_TOKEN` Actions secret to `leeworks-agents/vin-decoder` repo (issue #126)
-
-Gitea → `leeworks-agents/vin-decoder` → Settings → Secrets and Variables → Actions
-- **Name:** `GITEA_TOKEN`
-- **Value:** Gitea personal access token with `write:packages` scope
-
-Verify it appears in the repo's Actions Secrets list before the next push to `main`.
-
-### Step 4A-5 — Add RapidAPI VIN secrets to `leeworks-agents/api-company` Actions (issue #139)
-
-> **Blocked** — requires VIN Decoder to be listed on RapidAPI first (issue #131 tracker; operator action required).
-
-Once the VIN Decoder listing is live:
-
-Gitea → `leeworks-agents/api-company` → Settings → Secrets and Variables → Actions
-
-| Secret name               | Where to find it |
-|---------------------------|------------------|
-| `RAPIDAPI_VIN_API_ID`     | RapidAPI dashboard → VIN Decoder listing → Overview |
-| `RAPIDAPI_VIN_VERSION_ID` | RapidAPI dashboard → VIN Decoder listing → Versions tab |
-
 ## Phase 5 — RapidAPI + PayPal (issue #44, #19)
 
 > **Blocked on operator being 18+ for PayPal.** Complete when eligible.
@@ -313,25 +241,8 @@ kubectl create secret generic rapidapi-proxy-secret \
 kubectl create secret generic rapidapi-proxy-secret \
   -n air-quality \
   --from-literal=X-RapidAPI-Proxy-Secret=<VALUE>
-
-# vin-decoder (updates the placeholder secret from Phase 4-A Step 3)
-kubectl create secret generic rapidapi-proxy-secret \
-  -n vin-decoder \
-  --from-literal=X-RapidAPI-Proxy-Secret=<VIN_VALUE> \
-  --save-config --dry-run=client -o yaml | kubectl apply -f -
 ```
 
-Also configure VIN Decoder pricing tiers on RapidAPI (issue #151):
-
-| Tier  | Monthly Price | Request Limit  | Rate Limit  |
-|-------|---------------|----------------|-------------|
-| Free  | $0            | 100 req/mo     | 5 req/min   |
-| Basic | $9            | 5,000 req/mo   | 60 req/min  |
-| Pro   | $19           | 20,000 req/mo  | 200 req/min |
-| Ultra | $49           | 100,000 req/mo | 500 req/min |
-
-Confirm VIN Decoder revenue flows through the same PayPal account as the other 3 APIs.
-
 ---
 
 ## Quick Verification Checklist
@@ -376,22 +287,12 @@ Phase 2: Service secrets (runner-token, grafana-admin, gatus-webhook, registry)
 
 Phase 3: CI secrets + image-automation token
     └─► images build, push, and auto-update → API services deploy
-    (includes leeworks-agents/vin-decoder GITEA_TOKEN — Step 3-A)
 
-Phase 4: DNS records (8 subdomains including vin.leeworks.dev)
+Phase 4: DNS records
     └─► HTTPS certs issued → public URLs go live
 
-Phase 4-A: VIN Decoder namespace setup (#126, #127, #128)
-    4A-1: vin-decoder namespace created
-    4A-2: gitea-registry imagePullSecret in vin-decoder (#127)
-    4A-3: rapidapi-proxy-secret placeholder in vin-decoder (#128)
-    4A-4: GITEA_TOKEN Actions secret in vin-decoder repo (#126)
-    4A-5: RAPIDAPI_VIN_API_ID + RAPIDAPI_VIN_VERSION_ID Actions secrets (#139)
-          [blocked on VIN Decoder RapidAPI listing]
-    └─► VIN Decoder pods start; CI can push images
-
-Phase 5: RapidAPI + PayPal (all 4 APIs including VIN Decoder; issue #151)
-    └─► revenue enabled for all 4 APIs
+Phase 5: RapidAPI + PayPal
+    └─► revenue enabled
 ```
 
 ---

flux/monitoring/grafana-dashboard-apis.yaml

@@ -9,7 +9,7 @@ data:
   api-dashboard.json: |
     {
       "annotations": { "list": [] },
-      "description": "Request rate, latency, error rate, and data freshness for zip-enrichment, holidays, and air-quality APIs",
+      "description": "Request rate, latency, error rate, and data freshness for zip-enrichment, holidays, air-quality, and vin-decoder APIs",
       "editable": true,
       "graphTooltip": 1,
       "panels": [

flux/monitoring/kustomization.yaml

@@ -8,3 +8,4 @@ resources:
   - gatus-helmrelease.yaml
   - grafana-dashboard-apis.yaml
   - prometheusrule-apis.yaml
+  - vin-decoder-dashboard.yaml

flux/monitoring/prometheusrule-apis.yaml

@@ -18,9 +18,9 @@ spec:
         - alert: APIHighErrorRate
           expr: |
             (
-              sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality"}[5m]))
+              sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality|vin-decoder"}[5m]))
               /
-              sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality"}[5m]))
+              sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality|vin-decoder"}[5m]))
             ) > 0.05
           for: 5m
           labels:
@@ -36,9 +36,9 @@ spec:
         - alert: APIHighErrorRate
           expr: |
             (
-              sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality"}[5m]))
+              sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality|vin-decoder"}[5m]))
               /
-              sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality"}[5m]))
+              sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality|vin-decoder"}[5m]))
             ) > 0.20
           for: 5m
           labels:
@@ -55,7 +55,7 @@ spec:
           expr: |
             histogram_quantile(
               0.95,
-              sum by (job, le) (rate(api_response_duration_seconds_bucket{job=~"zip|holidays|air-quality"}[5m]))
+              sum by (job, le) (rate(api_response_duration_seconds_bucket{job=~"zip|holidays|air-quality|vin-decoder"}[5m]))
             ) > 2
           for: 5m
           labels:
@@ -70,7 +70,7 @@ spec:
         # -------------------------------------------------------------------
         - alert: APIDataStale
           expr: |
-            api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 2592000
+            api_data_freshness_seconds{job=~"zip|holidays|air-quality|vin-decoder"} > 2592000
           for: 30m
           labels:
             severity: warning
@@ -84,7 +84,7 @@ spec:
         # -------------------------------------------------------------------
         - alert: APIDataCriticallyStale
           expr: |
-            api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 5184000
+            api_data_freshness_seconds{job=~"zip|holidays|air-quality|vin-decoder"} > 5184000
           for: 1h
           labels:
             severity: critical
@@ -98,9 +98,9 @@ spec:
         # -------------------------------------------------------------------
         - alert: APIDown
           expr: |
-            absent(up{job=~"zip|holidays|air-quality"} == 1)
+            absent(up{job=~"zip|holidays|air-quality|vin-decoder"} == 1)
             or
-            up{job=~"zip|holidays|air-quality"} == 0
+            up{job=~"zip|holidays|air-quality|vin-decoder"} == 0
           for: 2m
           labels:
             severity: critical

flux/monitoring/vin-decoder-dashboard.yaml

@@ -0,0 +1,161 @@
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: grafana-dashboard-vin-decoder
+  namespace: monitoring
+  labels:
+    grafana_dashboard: "1"
+data:
+  vin-decoder-dashboard.json: |
+    {
+      "annotations": { "list": [] },
+      "description": "Request rate, latency, error rate, and cache hit ratio for the VIN Decoder API",
+      "editable": true,
+      "graphTooltip": 1,
+      "panels": [
+        {
+          "collapsed": false,
+          "gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
+          "id": 1,
+          "title": "VIN Decoder — Request Rate",
+          "type": "row"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": { "axisLabel": "requests/sec", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
+              "unit": "reqps"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },
+          "id": 2,
+          "options": {
+            "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
+            "tooltip": { "mode": "multi" }
+          },
+          "targets": [
+            {
+              "expr": "sum by (route) (rate(api_requests_total{api=\"vin-decoder\"}[5m]))",
+              "legendFormat": "{{route}}",
+              "refId": "A"
+            }
+          ],
+          "title": "Request Rate by Route",
+          "type": "timeseries"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": { "axisLabel": "", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
+              "unit": "percentunit"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },
+          "id": 3,
+          "options": {
+            "legend": { "calcs": ["mean", "last"], "displayMode": "table", "placement": "bottom" },
+            "tooltip": { "mode": "multi" }
+          },
+          "targets": [
+            {
+              "expr": "sum(rate(api_requests_total{api=\"vin-decoder\",status_code=~\"5..\"}[5m])) / sum(rate(api_requests_total{api=\"vin-decoder\"}[5m]))",
+              "legendFormat": "5xx error rate",
+              "refId": "A"
+            }
+          ],
+          "title": "5xx Error Rate",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
+          "id": 10,
+          "title": "VIN Decoder — Latency",
+          "type": "row"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "palette-classic" },
+              "custom": { "axisLabel": "seconds", "drawStyle": "line", "fillOpacity": 5, "lineWidth": 1, "showPoints": "never" },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 24, "x": 0, "y": 10 },
+          "id": 4,
+          "options": {
+            "legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
+            "tooltip": { "mode": "multi" }
+          },
+          "targets": [
+            {
+              "expr": "histogram_quantile(0.50, sum by (route, le) (rate(api_response_duration_seconds_bucket{api=\"vin-decoder\"}[5m])))",
+              "legendFormat": "P50 {{route}}",
+              "refId": "A"
+            },
+            {
+              "expr": "histogram_quantile(0.95, sum by (route, le) (rate(api_response_duration_seconds_bucket{api=\"vin-decoder\"}[5m])))",
+              "legendFormat": "P95 {{route}}",
+              "refId": "B"
+            },
+            {
+              "expr": "histogram_quantile(0.99, sum by (route, le) (rate(api_response_duration_seconds_bucket{api=\"vin-decoder\"}[5m])))",
+              "legendFormat": "P99 {{route}}",
+              "refId": "C"
+            }
+          ],
+          "title": "Response Latency P50 / P95 / P99 by Route",
+          "type": "timeseries"
+        },
+        {
+          "collapsed": false,
+          "gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 },
+          "id": 20,
+          "title": "VIN Decoder — Cache & Data Freshness",
+          "type": "row"
+        },
+        {
+          "datasource": { "type": "prometheus", "uid": "prometheus" },
+          "fieldConfig": {
+            "defaults": {
+              "color": { "mode": "thresholds" },
+              "thresholds": {
+                "steps": [
+                  { "color": "green", "value": null },
+                  { "color": "yellow", "value": 2592000 },
+                  { "color": "red", "value": 5184000 }
+                ]
+              },
+              "unit": "s"
+            },
+            "overrides": []
+          },
+          "gridPos": { "h": 8, "w": 12, "x": 0, "y": 19 },
+          "id": 5,
+          "options": { "reduceOptions": { "calcs": ["lastNotNull"] }, "orientation": "auto", "textMode": "auto", "colorMode": "background" },
+          "targets": [
+            {
+              "expr": "api_data_freshness_seconds{api=\"vin-decoder\"}",
+              "legendFormat": "data freshness",
+              "refId": "A"
+            }
+          ],
+          "title": "Data Freshness (seconds since last NHTSA fetch)",
+          "type": "stat"
+        }
+      ],
+      "refresh": "1m",
+      "schemaVersion": 38,
+      "tags": ["vin-decoder", "api-company"],
+      "title": "VIN Decoder API",
+      "uid": "vin-decoder-api",
+      "version": 1
+    }

flux/vin-decoder/kustomization.yaml

@@ -4,3 +4,4 @@ resources:
   - namespace.yaml
   - externalsecret.yaml
   - helmrelease.yaml
+  - servicemonitor.yaml

flux/vin-decoder/servicemonitor.yaml

@@ -0,0 +1,15 @@
+apiVersion: monitoring.coreos.com/v1
+kind: ServiceMonitor
+metadata:
+  name: vin-decoder
+  namespace: vin-decoder
+  labels:
+    release: kube-prometheus-stack
+spec:
+  selector:
+    matchLabels:
+      app: vin-decoder
+  endpoints:
+    - port: http
+      path: /metrics
+      interval: 30s