api-company/docs/metrics-standard.md

API Metrics Instrumentation Standard

Version: 1.0
Date: 2026-05-24
Applies to: All leeworks.dev API services (zip-enrichment, holidays, air-quality)

---

Overview

Every API service MUST expose Prometheus-compatible metrics at GET /metrics. This document defines the required metrics, label conventions, and provides reference middleware implementations for both Fastify (Node.js) and FastAPI (Python).

---

Required Metrics

1. api_requests_total

Field	Value
Type	Counter
Description	Total number of HTTP requests received
Labels	api, route, method, status

Label values:

• api: one of zip-enrichment, holidays, air-quality
• route: the matched route pattern, e.g. /v1/lookup, /v1/holidays
• method: HTTP method, e.g. GET, POST
• status: HTTP status code as string, e.g. 200, 404, 429, 403

Example:

api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="200"} 1234
api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="429"} 12
api_requests_total{api="zip-enrichment",route="/v1/lookup",method="GET",status="403"} 3

---

2. api_response_duration_seconds

Field	Value
Type	Histogram
Description	HTTP response latency in seconds
Labels	api, route
Buckets	0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5

Example:

api_response_duration_seconds_bucket{api="holidays",route="/v1/holidays",le="0.05"} 800
api_response_duration_seconds_bucket{api="holidays",route="/v1/holidays",le="0.1"} 990
api_response_duration_seconds_sum{api="holidays",route="/v1/holidays"} 45.2
api_response_duration_seconds_count{api="holidays",route="/v1/holidays"} 1000

---

3. api_data_freshness_seconds

Field	Value
Type	Gauge
Description	Seconds since the local dataset was last seeded/refreshed
Labels	api, dataset
Unit	Seconds (Unix timestamp diff: now - last_seed_time)

Label values:

• dataset: a descriptive name for the dataset, e.g. zip_codes, us_holidays, aqi_readings

Example:

api_data_freshness_seconds{api="air-quality",dataset="aqi_readings"} 86400
api_data_freshness_seconds{api="zip-enrichment",dataset="zip_codes"} 2592000

A value of 0 means freshly seeded; values growing toward 2592000 (30 days) are expected for monthly re-seed schedules.

---

Reference Implementations

Fastify (Node.js/TypeScript)

Install dependencies:

npm install prom-client

src/metrics.ts:

import { Registry, Counter, Histogram, Gauge } from 'prom-client';

export const register = new Registry();

export const requestsTotal = new Counter({
  name: 'api_requests_total',
  help: 'Total number of HTTP requests received',
  labelNames: ['api', 'route', 'method', 'status'],
  registers: [register],
});

export const responseDuration = new Histogram({
  name: 'api_response_duration_seconds',
  help: 'HTTP response latency in seconds',
  labelNames: ['api', 'route'],
  buckets: [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
  registers: [register],
});

export const dataFreshness = new Gauge({
  name: 'api_data_freshness_seconds',
  help: 'Seconds since the local dataset was last seeded',
  labelNames: ['api', 'dataset'],
  registers: [register],
});

src/metricsMiddleware.ts:

import { FastifyPluginAsync } from 'fastify';
import { register, requestsTotal, responseDuration } from './metrics';

const API_NAME = process.env.API_NAME ?? 'unknown'; // set per service

export const metricsPlugin: FastifyPluginAsync = async (fastify) => {
  // Expose /metrics endpoint
  fastify.get('/metrics', async (_req, reply) => {
    reply.header('Content-Type', register.contentType);
    return register.metrics();
  });

  // Instrument all routes
  fastify.addHook('onRequest', async (request, _reply) => {
    (request as any)._startTime = process.hrtime.bigint();
  });

  fastify.addHook('onResponse', async (request, reply) => {
    const startTime = (request as any)._startTime as bigint;
    const durationMs = Number(process.hrtime.bigint() - startTime) / 1e6;
    const route = request.routerPath ?? request.url;

    requestsTotal.labels(API_NAME, route, request.method, String(reply.statusCode)).inc();
    responseDuration.labels(API_NAME, route).observe(durationMs / 1000);
  });
};

Register in main:

import { metricsPlugin } from './metricsMiddleware';
await fastify.register(metricsPlugin);

Update data freshness gauge (call after each seed):

import { dataFreshness } from './metrics';
// Call this after each DB seed completes:
dataFreshness.labels('zip-enrichment', 'zip_codes').set(0);
// Or set it to seconds since last seed on startup:
dataFreshness.labels('zip-enrichment', 'zip_codes').set(secondsSinceLastSeed);

---

FastAPI (Python)

Install dependencies:

pip install prometheus-client starlette

metrics.py:

from prometheus_client import Counter, Histogram, Gauge, REGISTRY, CollectorRegistry

registry = CollectorRegistry()

requests_total = Counter(
    'api_requests_total',
    'Total number of HTTP requests received',
    ['api', 'route', 'method', 'status'],
    registry=registry,
)

response_duration = Histogram(
    'api_response_duration_seconds',
    'HTTP response latency in seconds',
    ['api', 'route'],
    buckets=[0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5],
    registry=registry,
)

data_freshness = Gauge(
    'api_data_freshness_seconds',
    'Seconds since the local dataset was last seeded',
    ['api', 'dataset'],
    registry=registry,
)

metrics_middleware.py:

import time
import os
from starlette.middleware.base import BaseHTTPMiddleware
from starlette.requests import Request
from starlette.responses import Response
from prometheus_client import generate_latest, CONTENT_TYPE_LATEST
from metrics import requests_total, response_duration, registry

API_NAME = os.getenv("API_NAME", "unknown")


class MetricsMiddleware(BaseHTTPMiddleware):
    async def dispatch(self, request: Request, call_next):
        start = time.time()
        response = await call_next(request)
        duration = time.time() - start

        route = request.url.path
        requests_total.labels(
            api=API_NAME,
            route=route,
            method=request.method,
            status=str(response.status_code),
        ).inc()
        response_duration.labels(api=API_NAME, route=route).observe(duration)

        return response


async def metrics_endpoint(request: Request):
    return Response(
        generate_latest(registry),
        media_type=CONTENT_TYPE_LATEST,
    )

Register in FastAPI app:

from fastapi import FastAPI
from starlette.routing import Route
from metrics_middleware import MetricsMiddleware, metrics_endpoint

app = FastAPI()
app.add_middleware(MetricsMiddleware)
app.add_route("/metrics", metrics_endpoint)

---

Prometheus Scrape Configuration

Add to your Prometheus scrape_configs (or ServiceMonitor for kube-prometheus-stack):

# prometheus-additional-scrapes.yaml
- job_name: 'leeworks-apis'
  kubernetes_sd_configs:
    - role: pod
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
      action: keep
      regex: "true"
    - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
      action: replace
      target_label: __metrics_path__
      regex: (.+)
    - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
      action: replace
      regex: ([^:]+)(?::\d+)?;(\d+)
      replacement: $1:$2
      target_label: __address__

Add annotations to each API pod:

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/port: "3000"   # or 8000 for FastAPI
  prometheus.io/path: "/metrics"

---

Grafana Dashboard

Grafana dashboards are committed as Flux ConfigMaps and are auto-loaded by Grafana via the grafana_dashboard: "1" label:

• flux/monitoring/grafana-dashboard-apis.yaml — consolidated 4-API dashboard (all APIs)
• flux/monitoring/vin-decoder-dashboard.yaml — VIN Decoder-specific dashboard

Dashboards will load automatically once Grafana is deployed (issue #7) and Flux is active (issue #218).

Key panels to include:

1. Request rate by API and status (rate(api_requests_total[5m]))
2. P50/P95/P99 latency (histogram_quantile(0.99, rate(api_response_duration_seconds_bucket[5m])))
3. Error rate = non-2xx / total requests
4. Data freshness gauge per API
5. Request volume heatmap

---

Compliance Checklist

Before marking an API server PR as ready:

• GET /metrics returns text/plain; version=0.0.4; charset=utf-8
• api_requests_total increments on every request with correct labels
• api_response_duration_seconds has observations on every request
• api_data_freshness_seconds is set on startup and after each seed
• Pod annotations for Prometheus scraping are present in the Helm chart values
• API_NAME env var is set correctly per deployment