api-company/docs/metrics-standard.md

# API Metrics Instrumentation Standard

**Version:** 1.0  
**Date:** 2026-05-24  
**Applies to:** All leeworks.dev API services (zip-enrichment, holidays, air-quality)

---

## Overview

Every API service MUST expose Prometheus-compatible metrics at `GET /metrics`. This document defines the required metrics, label conventions, and provides reference middleware implementations for both Fastify (Node.js) and FastAPI (Python).

---

## Required Metrics

### 1. `api_requests_total`

| Field | Value |
|-------|-------|
| **Type** | Counter |
| **Description** | Total number of HTTP requests received |
| **Labels** | `api`, `route`, `method`, `status` |

**Label values:**
- `api`: one of `zip-enrichment`, `holidays`, `air-quality`
- `route`: the matched route pattern, e.g. `/v1/lookup`, `/v1/holidays`
- `method`: HTTP method, e.g. `GET`, `POST`
- `status`: HTTP status code as string, e.g. `200`, `404`, `429`, `403`

**Example:**
```
api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="200"} 1234
api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="429"} 12
api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="403"} 3
```

---

### 2. `api_response_duration_seconds`

| Field | Value |
|-------|-------|
| **Type** | Histogram |
| **Description** | HTTP response latency in seconds |
| **Labels** | `api`, `route` |
| **Buckets** | `0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5` |

**Example:**
```
api_response_duration_seconds_bucket{api="holidays",route="/v1/holidays",le="0.05"} 800
api_response_duration_seconds_bucket{api="holidays",route="/v1/holidays",le="0.1"} 990
api_response_duration_seconds_sum{api="holidays",route="/v1/holidays"} 45.2
api_response_duration_seconds_count{api="holidays",route="/v1/holidays"} 1000
```

---

### 3. `api_data_freshness_seconds`

| Field | Value |
|-------|-------|
| **Type** | Gauge |
| **Description** | Seconds since the local dataset was last seeded/refreshed |
| **Labels** | `api`, `dataset` |
| **Unit** | Seconds (Unix timestamp diff: `now - last_seed_time`) |

**Label values:**
- `dataset`: a descriptive name for the dataset, e.g. `zip_codes`, `us_holidays`, `aqi_readings`

**Example:**
```
api_data_freshness_seconds{api="air-quality",dataset="aqi_readings"} 86400
api_data_freshness_seconds{api="zip-enrichment",dataset="zip_codes"} 2592000
```

A value of `0` means freshly seeded; values growing toward `2592000` (30 days) are expected for monthly re-seed schedules.

---

## Reference Implementations

### Fastify (Node.js/TypeScript)

Install dependencies:
```bash
npm install prom-client
```

**`src/metrics.ts`:**
```typescript
import { Registry, Counter, Histogram, Gauge } from 'prom-client';

export const register = new Registry();

export const requestsTotal = new Counter({
  name: 'api_requests_total',
  help: 'Total number of HTTP requests received',
  labelNames: ['api', 'route', 'method', 'status'],
  registers: [register],
});

export const responseDuration = new Histogram({
  name: 'api_response_duration_seconds',
  help: 'HTTP response latency in seconds',
  labelNames: ['api', 'route'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
  registers: [register],
});

export const dataFreshness = new Gauge({
  name: 'api_data_freshness_seconds',
  help: 'Seconds since the local dataset was last seeded',
  labelNames: ['api', 'dataset'],
  registers: [register],
});
```

**`src/metricsMiddleware.ts`:**
```typescript
import { FastifyPluginAsync } from 'fastify';
import { register, requestsTotal, responseDuration } from './metrics';

const API_NAME = process.env.API_NAME ?? 'unknown'; // set per service

export const metricsPlugin: FastifyPluginAsync = async (fastify) => {
  // Expose /metrics endpoint
  fastify.get('/metrics', async (_req, reply) => {
    reply.header('Content-Type', register.contentType);
    return register.metrics();
  });

  // Instrument all routes
  fastify.addHook('onRequest', async (request, _reply) => {
    (request as any)._startTime = process.hrtime.bigint();
  });

  fastify.addHook('onResponse', async (request, reply) => {
    const startTime = (request as any)._startTime as bigint;
    const durationMs = Number(process.hrtime.bigint() - startTime) / 1e6;
    const route = request.routerPath ?? request.url;

    requestsTotal.labels(API_NAME, route, request.method, String(reply.statusCode)).inc();
    responseDuration.labels(API_NAME, route).observe(durationMs / 1000);
  });
};
```

**Register in main:**
```typescript
import { metricsPlugin } from './metricsMiddleware';
await fastify.register(metricsPlugin);
```

**Update data freshness gauge (call after each seed):**
```typescript
import { dataFreshness } from './metrics';
// Call this after each DB seed completes:
dataFreshness.labels('zip-enrichment', 'zip_codes').set(0);
// Or set it to seconds since last seed on startup:
dataFreshness.labels('zip-enrichment', 'zip_codes').set(secondsSinceLastSeed);
```

---

### FastAPI (Python)

Install dependencies:
```bash
pip install prometheus-client starlette
```

**`metrics.py`:**
```python
from prometheus_client import Counter, Histogram, Gauge, REGISTRY, CollectorRegistry

registry = CollectorRegistry()

requests_total = Counter(
    'api_requests_total',
    'Total number of HTTP requests received',
    ['api', 'route', 'method', 'status'],
    registry=registry,
)

response_duration = Histogram(
    'api_response_duration_seconds',
    'HTTP response latency in seconds',
    ['api', 'route'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
    registry=registry,
)

data_freshness = Gauge(
    'api_data_freshness_seconds',
    'Seconds since the local dataset was last seeded',
    ['api', 'dataset'],
    registry=registry,
)
```

**`metrics_middleware.py`:**
```python
import time
import os
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
from metrics import requests_total, response_duration, registry

API_NAME = os.getenv("API_NAME", "unknown")


class MetricsMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        start = time.time()
        response = await call_next(request)
        duration = time.time() - start

        route = request.url.path
        requests_total.labels(
            api=API_NAME,
            route=route,
            method=request.method,
            status=str(response.status_code),
        ).inc()
        response_duration.labels(api=API_NAME, route=route).observe(duration)

        return response


async def metrics_endpoint(request: Request):
    return Response(
        generate_latest(registry),
        media_type=CONTENT_TYPE_LATEST,
    )
```

**Register in FastAPI app:**
```python
from fastapi import FastAPI
from starlette.routing import Route
from metrics_middleware import MetricsMiddleware, metrics_endpoint

app = FastAPI()
app.add_middleware(MetricsMiddleware)
app.add_route("/metrics", metrics_endpoint)
```

---

## Prometheus Scrape Configuration

Add to your Prometheus `scrape_configs` (or ServiceMonitor for kube-prometheus-stack):

```yaml
# prometheus-additional-scrapes.yaml
- job_name: 'leeworks-apis'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: "true"
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__
```

Add annotations to each API pod:
```yaml
annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "3000"   # or 8000 for FastAPI
  prometheus.io/path: "/metrics"
```

---

## Grafana Dashboard

Grafana dashboards are committed as Flux ConfigMaps and are auto-loaded by Grafana via the `grafana_dashboard: "1"` label:
- `flux/monitoring/grafana-dashboard-apis.yaml` — consolidated 4-API dashboard (all APIs)
- `flux/monitoring/vin-decoder-dashboard.yaml` — VIN Decoder-specific dashboard

Dashboards will load automatically once Grafana is deployed (issue #7) and Flux is active (issue #218).

Key panels to include:
1. Request rate by API and status (`rate(api_requests_total[5m])`)
2. P50/P95/P99 latency (`histogram_quantile(0.99, rate(api_response_duration_seconds_bucket[5m]))`)
3. Error rate = non-2xx / total requests
4. Data freshness gauge per API
5. Request volume heatmap

---

## Compliance Checklist

Before marking an API server PR as ready:

- [ ] `GET /metrics` returns `text/plain; version=0.0.4; charset=utf-8`
- [ ] `api_requests_total` increments on every request with correct labels
- [ ] `api_response_duration_seconds` has observations on every request
- [ ] `api_data_freshness_seconds` is set on startup and after each seed
- [ ] Pod annotations for Prometheus scraping are present in the Helm chart values
- [ ] `API_NAME` env var is set correctly per deployment