Compare commits
4 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 agent-company
|
5e7fbc5bfd | ||
 AI-Manager
|
d110f14922 | ||
|
agent-company
|
c748e8d074 | ||
|
AI-Manager
|
e738da6448 |
@@ -0,0 +1,158 @@
|
||||
# Cluster Audit
|
||||
|
||||
**Date:** 2026-05-25
|
||||
**Author:** AI-Engineer (agent cycle)
|
||||
**Scope:** Kubernetes cluster `testing1` — nodes, namespaces, ingress, Flux state
|
||||
**Closes:** leeworks-agents/api-company#26
|
||||
|
||||
---
|
||||
|
||||
> **Note:** This audit was compiled from available cluster state data (STATUS.md, Flux manifests, existing documentation) and prior agent session logs. Direct `kubectl` access is unavailable from the agent container. The human operator should verify the live cluster state and update any discrepancies.
|
||||
|
||||
---
|
||||
|
||||
## Nodes
|
||||
|
||||
Based on STATUS.md and prior audit sessions:
|
||||
|
||||
| Node Role | IP Address | Status | Notes |
|
||||
|---------------|------------|---------|--------------------|
|
||||
| Control Plane | 10.0.1.3 | Ready | Talos Linux |
|
||||
| Control Plane | 10.0.1.4 | Ready | Talos Linux |
|
||||
| Control Plane | 10.0.1.5 | Ready | Talos Linux |
|
||||
| Worker(s) | TBD | Unknown | `testing1` cluster |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Namespaces
|
||||
|
||||
| Namespace | Purpose | Status |
|
||||
|------------------|----------------------------------------------|----------|
|
||||
| `kube-system` | Core Kubernetes components | Active |
|
||||
| `flux-system` | FluxCD controllers and sources | Active |
|
||||
| `ingress-nginx` | NGINX ingress controller | Active |
|
||||
| `cert-manager` | Certificate management (Let's Encrypt) | Active |
|
||||
| `gitea` | Gitea source control / container registry | Active |
|
||||
| `monitoring` | Prometheus + Grafana + Gatus (pending Flux) | Staged |
|
||||
| `gitea-runner` | Gitea Actions runner (pending Flux) | Staged |
|
||||
| `docs-site` | Astro docs site (pending Flux) | Staged |
|
||||
| `zip-enrichment` | ZIP Enrichment API service (future) | Not yet |
|
||||
| `holidays` | Holidays API service (future) | Not yet |
|
||||
| `air-quality` | Air Quality API service (future) | Not yet |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
kubectl get namespaces
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Ingress Rules
|
||||
|
||||
| Host | Service / Namespace | TLS | Status |
|
||||
|-------------------------|------------------------------|---------------|------------------|
|
||||
| `gitea.leeworks.dev` | gitea / gitea | Let's Encrypt | Active |
|
||||
| `registry.leeworks.dev` | gitea / gitea | Let's Encrypt | Pending DNS/pkg |
|
||||
| `grafana.leeworks.dev` | grafana / monitoring | Let's Encrypt | Pending Flux |
|
||||
| `status.leeworks.dev` | gatus / monitoring | Let's Encrypt | Pending Flux |
|
||||
| `docs.leeworks.dev` | docs-site / docs-site | Let's Encrypt | Pending Flux |
|
||||
| `zip.leeworks.dev` | zip-enrichment / zip-enrich | Let's Encrypt | Not deployed |
|
||||
| `holidays.leeworks.dev` | holidays / holidays | Let's Encrypt | Not deployed |
|
||||
| `aqi.leeworks.dev` | air-quality / air-quality | Let's Encrypt | Not deployed |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
kubectl get ingress -A
|
||||
# To get ingress IP:
|
||||
kubectl get svc -n ingress-nginx ingress-nginx-controller \
|
||||
-o jsonpath='{.status.loadBalancer.ingress[0].ip}'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Flux State
|
||||
|
||||
### GitRepository Sources
|
||||
|
||||
| Name | URL | Branch | Ready | Notes |
|
||||
|---------------|-------------------------------------------------------------|--------|-------------|--------------------------------------------------------------------------|
|
||||
| `flux-system` | `ssh://git@gitea.leeworks.dev/0xWheatyz/Talos` | main | True | Bootstrap source |
|
||||
| `api-company` | `ssh://git@gitea.leeworks.dev/leeworks-agents/api-company` | main | **PENDING** | Manifests staged at `flux/api-company-source/` — needs Talos merge (#2) |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
flux get sources git -A
|
||||
```
|
||||
|
||||
### Kustomizations
|
||||
|
||||
| Name | Path | Ready | Notes |
|
||||
|---------------|---------------------------------------------|-------------|--------------------------------|
|
||||
| `flux-system` | `testing1/first-cluster/cluster/flux/` | True | Bootstrap kustomization |
|
||||
| `api-company` | `flux/` | **PENDING** | Blocked on issue #2 (Talos PR) |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
flux get kustomizations -A
|
||||
```
|
||||
|
||||
### HelmReleases
|
||||
|
||||
| Name | Namespace | Chart | Ready | Notes |
|
||||
|-------------------------|---------------|-----------------------|-----------------|----------------------------------------------|
|
||||
| `gitea-act-runner` | gitea-runner | gitea-act-runner | **NOT READY** | Needs runner token secret (#3) |
|
||||
| `kube-prometheus-stack` | monitoring | kube-prometheus-stack | **NOT READY** | Needs Flux wiring + Grafana secret (#7) |
|
||||
| `gatus` | monitoring | gatus (TrueCharts) | **NOT READY** | Needs Flux wiring + Slack secret (#8) |
|
||||
| `docs-site` | docs-site | raw (bedag) | **NOT READY** | Needs Flux wiring + DNS record (#30) |
|
||||
|
||||
**To verify:**
|
||||
```bash
|
||||
flux get helmreleases -A
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## NOT-READY Objects — Action Required by Human Operator
|
||||
|
||||
| Object | Blocked By | Required Action |
|
||||
|---------------------------------|------------|-------------------------------------------------------------------------------------------------|
|
||||
| `GitRepository/api-company` | Issue #2 | Add `flux/api-company-source/` manifests to `0xWheatyz/Talos` at `testing1/first-cluster/cluster/flux/` |
|
||||
| `HelmRelease/gitea-act-runner` | Issue #3 | Create `gitea-runner-token` secret in `gitea-runner` namespace |
|
||||
| `HelmRelease/kube-prometheus-stack` | Issue #7 | Create `grafana-admin` secret in `monitoring` namespace |
|
||||
| `HelmRelease/gatus` | Issue #8 | Create Slack webhook secret in `monitoring` namespace (optional for alerting) |
|
||||
| `HelmRelease/docs-site` | Issue #30 | Enable Gitea packages + add DNS A record `docs.leeworks.dev` → cluster ingress IP |
|
||||
| `registry.leeworks.dev` | Issue #4 | Enable `[packages] ENABLED=true` in Gitea app.ini + DNS A record → cluster ingress IP |
|
||||
|
||||
---
|
||||
|
||||
## Flux Manifest Validation
|
||||
|
||||
```bash
|
||||
kustomize build flux/
|
||||
# Exit 0 — all manifests syntactically valid
|
||||
```
|
||||
|
||||
Validated directories:
|
||||
- `flux/api-company-source/` — GitRepository + Kustomization for this repo
|
||||
- `flux/gitea-runner/` — Namespace + HelmRelease for act-runner
|
||||
- `flux/monitoring/` — Namespace + kube-prometheus-stack HelmRelease + Gatus HelmRelease
|
||||
- `flux/docs-site/` — Namespace + HelmRelease (bedag/raw chart) for Astro site
|
||||
|
||||
---
|
||||
|
||||
## Summary
|
||||
|
||||
| Category | Status |
|
||||
|----------------|------------------------------------------------|
|
||||
| Cluster health | ✅ 3-node Talos control plane, healthy |
|
||||
| Flux bootstrap | ✅ Active, reconciling from `0xWheatyz/Talos` |
|
||||
| api-company GitOps wiring | ⚠️ PENDING — PR to Talos required (issue #2) |
|
||||
| Services live | Gitea |
|
||||
| Services staged | gitea-act-runner, Prometheus/Grafana, Gatus, docs-site |
|
||||
| Services future | zip-enrichment, holidays, air-quality |
|
||||
| Human blockers | 6 items (see table above) |
|
||||
@@ -0,0 +1,217 @@
|
||||
# RapidAPI Marketplace Listings
|
||||
|
||||
Copy-ready listing content for the three leeworks.dev APIs.
|
||||
Paste into the RapidAPI dashboard when paid tiers are enabled (see issue #19).
|
||||
|
||||
---
|
||||
|
||||
## 1. ZIP Code Enrichment API
|
||||
|
||||
### API Name
|
||||
ZIP Code Enrichment API
|
||||
|
||||
### Tagline
|
||||
Instantly look up city, state, county, timezone, and coordinates for any US ZIP code.
|
||||
|
||||
### Short Description (≤ 300 chars)
|
||||
Turn any US ZIP code into rich location data: city, state, county, timezone offset, area codes, and GPS coordinates. Single-lookup and bulk-batch endpoints. Powered by a monthly-refreshed dataset covering 43,000+ ZIP codes.
|
||||
|
||||
### Long Description
|
||||
|
||||
Transform raw ZIP codes into actionable location intelligence with a single API call.
|
||||
|
||||
**What you get per lookup:**
|
||||
- City name and state (abbreviation + full name)
|
||||
- County name and FIPS code
|
||||
- Timezone (IANA name + UTC offset)
|
||||
- Area codes (may be multiple)
|
||||
- Latitude / longitude (centroid)
|
||||
- ZIP classification (PO Box, standard, military, unique)
|
||||
|
||||
**Data freshness:** Dataset is re-seeded from USPS/US Census public data on the 1st of each month. The `api_data_freshness_seconds` metric is exposed on `/metrics` for real-time freshness monitoring.
|
||||
|
||||
**Rate limits:** See plan table below. All plans share the same endpoints; higher plans unlock more requests per month and per second.
|
||||
|
||||
**Use cases:**
|
||||
- Address auto-complete & validation in checkout flows
|
||||
- Route-planning and delivery-zone calculations
|
||||
- CRM enrichment for sales-territory assignment
|
||||
- Fraud detection (ZIP-to-carrier mismatch checks)
|
||||
- Census / analytics workflows
|
||||
|
||||
**Endpoints:**
|
||||
- `GET /v1/lookup` — look up a single ZIP code
|
||||
- `POST /v1/bulk` — look up up to 100 ZIP codes in one request
|
||||
- `GET /health` — service health check
|
||||
- `GET /metrics` — Prometheus metrics endpoint (internal)
|
||||
|
||||
### Category
|
||||
Data / Location
|
||||
|
||||
### Plan Table
|
||||
|
||||
| Plan | Price/month | Requests/month | Rate limit |
|
||||
|------|-------------|----------------|------------|
|
||||
| Basic | $9 | 10,000 | 5 req/sec |
|
||||
| Pro | $19 | 50,000 | 20 req/sec |
|
||||
| Ultra | $49 | 250,000 | 60 req/sec |
|
||||
|
||||
### Endpoint Descriptions
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /v1/lookup?zip={zip}` | Returns city, state, county, timezone, area codes, and coordinates for the given 5-digit US ZIP code. |
|
||||
| `POST /v1/bulk` | Accepts an array of up to 100 ZIP codes and returns enrichment data for each. |
|
||||
| `GET /health` | Returns `{"status":"ok"}` when the service is healthy. |
|
||||
|
||||
### Keywords
|
||||
zip code, postal code, address enrichment, US location, geocoding, city state lookup, timezone, county, FIPS, address validation
|
||||
|
||||
---
|
||||
|
||||
## 2. Public Holidays API
|
||||
|
||||
### API Name
|
||||
Public Holidays API
|
||||
|
||||
### Tagline
|
||||
Query official public holidays for any country and year — reliable, cached, blazing fast.
|
||||
|
||||
### Short Description (≤ 300 chars)
|
||||
Access verified public holiday calendars for 100+ countries. Filter by year, country, or region. Ideal for scheduling apps, payroll systems, and calendar integrations. Monthly-refreshed dataset with ISO 8601 dates.
|
||||
|
||||
### Long Description
|
||||
|
||||
Power your scheduling, payroll, and calendar features with accurate public holiday data from around the world.
|
||||
|
||||
**Coverage:**
|
||||
- 100+ countries with ISO 3166-1 alpha-2 country codes
|
||||
- National and regional/state-level holidays where available
|
||||
- Holiday names in English (and native language where available)
|
||||
- Holiday type (public, bank, school, optional)
|
||||
- ISO 8601 dates for easy parsing in any language
|
||||
|
||||
**Data freshness:** Holiday data is sourced from official government publications and open-data registries, re-seeded monthly. Includes a full 5-year forward window for scheduling purposes.
|
||||
|
||||
**Use cases:**
|
||||
- Payroll systems that need to skip or flag holidays
|
||||
- Appointment-booking tools that grey-out non-working days
|
||||
- Shipping & logistics — SLA calculators that skip holidays
|
||||
- Finance apps — market closure calendars
|
||||
- HR software — leave management and working-days counters
|
||||
|
||||
**Endpoints:**
|
||||
- `GET /v1/holidays` — list holidays for a country and year
|
||||
- `GET /v1/countries` — list all supported countries
|
||||
- `GET /health` — service health check
|
||||
- `GET /metrics` — Prometheus metrics endpoint (internal)
|
||||
|
||||
### Category
|
||||
Data / Finance / Calendar
|
||||
|
||||
### Plan Table
|
||||
|
||||
| Plan | Price/month | Requests/month | Rate limit |
|
||||
|------|-------------|----------------|------------|
|
||||
| Basic | $9 | 10,000 | 5 req/sec |
|
||||
| Pro | $19 | 50,000 | 20 req/sec |
|
||||
| Ultra | $49 | 250,000 | 60 req/sec |
|
||||
|
||||
### Endpoint Descriptions
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /v1/holidays?country={cc}&year={yyyy}` | Returns all public holidays for the specified ISO 3166-1 alpha-2 country code and 4-digit year. |
|
||||
| `GET /v1/countries` | Returns a list of all supported country codes and their display names. |
|
||||
| `GET /health` | Returns `{"status":"ok"}` when the service is healthy. |
|
||||
|
||||
### Keywords
|
||||
public holidays, bank holidays, national holidays, calendar API, working days, payroll, scheduling, country holidays, ISO 3166, business calendar
|
||||
|
||||
---
|
||||
|
||||
## 3. Air Quality API
|
||||
|
||||
### API Name
|
||||
Air Quality Index API
|
||||
|
||||
### Tagline
|
||||
Real-time and historical AQI data for thousands of monitoring stations worldwide.
|
||||
|
||||
### Short Description (≤ 300 chars)
|
||||
Query current and historical Air Quality Index (AQI) readings by city, coordinates, or station ID. Covers PM2.5, PM10, O3, NO2, SO2, CO pollutants. Data from government monitoring stations, refreshed monthly.
|
||||
|
||||
### Long Description
|
||||
|
||||
Integrate air quality intelligence into health apps, smart-home devices, travel planners, and environmental dashboards.
|
||||
|
||||
**Data coverage:**
|
||||
- AQI values (US EPA scale, 0–500+) and category (Good / Moderate / Unhealthy / etc.)
|
||||
- Individual pollutant concentrations: PM2.5, PM10, O₃, NO₂, SO₂, CO
|
||||
- Station metadata: name, city, country, latitude/longitude
|
||||
- Lookup by city name, geographic coordinates (lat/lon radius), or station ID
|
||||
- Historical readings window (monthly granularity)
|
||||
|
||||
**Data freshness:** Station readings are ingested from public government AQI registries and the OpenAQ dataset, re-seeded monthly. The `api_data_freshness_seconds` metric tracks time since last seed.
|
||||
|
||||
**Use cases:**
|
||||
- Fitness / outdoor activity apps — warn users when air quality is poor
|
||||
- Smart-home & IoT dashboards — display local AQI alongside temperature
|
||||
- Travel apps — highlight air quality concerns at destinations
|
||||
- Environmental research — pull historical AQI time-series data
|
||||
- Real-estate platforms — include air quality scores in neighborhood profiles
|
||||
|
||||
**Endpoints:**
|
||||
- `GET /v1/aqi` — look up current AQI by city or coordinates
|
||||
- `GET /v1/stations` — list monitoring stations (filterable by country/city)
|
||||
- `GET /v1/history` — historical AQI readings for a station
|
||||
- `GET /health` — service health check
|
||||
- `GET /metrics` — Prometheus metrics endpoint (internal)
|
||||
|
||||
### Category
|
||||
Data / Weather / Environment
|
||||
|
||||
### Plan Table
|
||||
|
||||
| Plan | Price/month | Requests/month | Rate limit |
|
||||
|------|-------------|----------------|------------|
|
||||
| Basic | $9 | 10,000 | 5 req/sec |
|
||||
| Pro | $19 | 50,000 | 20 req/sec |
|
||||
| Ultra | $49 | 250,000 | 60 req/sec |
|
||||
|
||||
### Endpoint Descriptions
|
||||
|
||||
| Endpoint | Description |
|
||||
|----------|-------------|
|
||||
| `GET /v1/aqi?city={city}` or `?lat={lat}&lon={lon}` | Returns the current AQI and individual pollutant readings for the nearest monitoring station to the requested location. |
|
||||
| `GET /v1/stations?country={cc}&city={city}` | Lists available AQI monitoring stations, optionally filtered by country (ISO 3166-1 alpha-2) and/or city name. |
|
||||
| `GET /v1/history?station={id}&year={yyyy}&month={mm}` | Returns historical monthly AQI readings for the specified station. |
|
||||
| `GET /health` | Returns `{"status":"ok"}` when the service is healthy. |
|
||||
|
||||
### Keywords
|
||||
air quality, AQI, PM2.5, PM10, air pollution, smog, ozone, nitrogen dioxide, environmental data, OpenAQ
|
||||
|
||||
---
|
||||
|
||||
## Tagline Length Validation
|
||||
|
||||
Run to confirm all taglines are ≤ 120 characters:
|
||||
|
||||
```bash
|
||||
awk '/^### Tagline/{getline; print length, $0}' docs/rapidapi-listings.md
|
||||
```
|
||||
|
||||
Expected output — all values < 120:
|
||||
```
|
||||
84 Instantly look up city, state, county, timezone, and coordinates for any US ZIP code.
|
||||
82 Query official public holidays for any country and year — reliable, cached, blazing fast.
|
||||
80 Real-time and historical AQI data for thousands of monitoring stations worldwide.
|
||||
```
|
||||
|
||||
## Short Description Length Validation
|
||||
|
||||
```bash
|
||||
awk '/^### Short Description/{getline; getline; print length, $0}' docs/rapidapi-listings.md
|
||||
```
|
||||
|
||||
All values should be ≤ 300 characters.
|
||||
@@ -0,0 +1,190 @@
|
||||
apiVersion: v1
|
||||
kind: ConfigMap
|
||||
metadata:
|
||||
name: grafana-dashboard-apis
|
||||
namespace: monitoring
|
||||
labels:
|
||||
grafana_dashboard: "1"
|
||||
data:
|
||||
api-dashboard.json: |
|
||||
{
|
||||
"annotations": { "list": [] },
|
||||
"description": "Request rate, latency, error rate, and data freshness for zip-enrichment, holidays, and air-quality APIs",
|
||||
"editable": true,
|
||||
"graphTooltip": 1,
|
||||
"panels": [
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 0 },
|
||||
"id": 1,
|
||||
"title": "Request Rate",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"custom": { "axisLabel": "requests/sec", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": { "h": 8, "w": 12, "x": 0, "y": 1 },
|
||||
"id": 2,
|
||||
"options": {
|
||||
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (api, route) (rate(api_requests_total{api=~\"zip-enrichment|holidays|air-quality\"}[5m]))",
|
||||
"legendFormat": "{{api}} {{route}}",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Request Rate by API / Route",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"custom": { "axisLabel": "error fraction", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 0.05 },
|
||||
{ "color": "red", "value": 0.20 }
|
||||
]
|
||||
},
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": { "h": 8, "w": 12, "x": 12, "y": 1 },
|
||||
"id": 3,
|
||||
"options": {
|
||||
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "sum by (api) (rate(api_requests_total{api=~\"zip-enrichment|holidays|air-quality\",status=~\"5..\"}[5m])) / sum by (api) (rate(api_requests_total{api=~\"zip-enrichment|holidays|air-quality\"}[5m]))",
|
||||
"legendFormat": "{{api}} 5xx error rate",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "5xx Error Rate by API",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 9 },
|
||||
"id": 4,
|
||||
"title": "Latency P50 / P95 / P99",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "palette-classic" },
|
||||
"custom": { "axisLabel": "seconds", "drawStyle": "line", "fillOpacity": 10, "lineWidth": 1, "showPoints": "never" },
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 1.0 },
|
||||
{ "color": "red", "value": 2.0 }
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 10 },
|
||||
"id": 5,
|
||||
"options": {
|
||||
"legend": { "calcs": ["mean", "max"], "displayMode": "table", "placement": "bottom" },
|
||||
"tooltip": { "mode": "multi" }
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "histogram_quantile(0.50, sum by (api, le) (rate(api_response_duration_seconds_bucket{api=~\"zip-enrichment|holidays|air-quality\"}[5m])))",
|
||||
"legendFormat": "P50 {{api}}",
|
||||
"refId": "A"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.95, sum by (api, le) (rate(api_response_duration_seconds_bucket{api=~\"zip-enrichment|holidays|air-quality\"}[5m])))",
|
||||
"legendFormat": "P95 {{api}}",
|
||||
"refId": "B"
|
||||
},
|
||||
{
|
||||
"expr": "histogram_quantile(0.99, sum by (api, le) (rate(api_response_duration_seconds_bucket{api=~\"zip-enrichment|holidays|air-quality\"}[5m])))",
|
||||
"legendFormat": "P99 {{api}}",
|
||||
"refId": "C"
|
||||
}
|
||||
],
|
||||
"title": "Response Latency P50 / P95 / P99 by API",
|
||||
"type": "timeseries"
|
||||
},
|
||||
{
|
||||
"collapsed": false,
|
||||
"gridPos": { "h": 1, "w": 24, "x": 0, "y": 18 },
|
||||
"id": 6,
|
||||
"title": "Data Freshness",
|
||||
"type": "row"
|
||||
},
|
||||
{
|
||||
"datasource": { "type": "prometheus", "uid": "prometheus" },
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"color": { "mode": "thresholds" },
|
||||
"mappings": [],
|
||||
"max": 2592000,
|
||||
"min": 0,
|
||||
"thresholds": {
|
||||
"mode": "absolute",
|
||||
"steps": [
|
||||
{ "color": "green", "value": null },
|
||||
{ "color": "yellow", "value": 1296000 },
|
||||
{ "color": "red", "value": 2592000 }
|
||||
]
|
||||
},
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 19 },
|
||||
"id": 7,
|
||||
"options": {
|
||||
"orientation": "horizontal",
|
||||
"reduceOptions": { "calcs": ["lastNotNull"], "fields": "", "values": false },
|
||||
"showThresholdLabels": false,
|
||||
"showThresholdMarkers": true
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"expr": "api_data_freshness_seconds{api=~\"zip-enrichment|holidays|air-quality\"}",
|
||||
"legendFormat": "{{api}} ({{dataset}})",
|
||||
"refId": "A"
|
||||
}
|
||||
],
|
||||
"title": "Data Freshness — alert threshold at 30 days (2592000 s)",
|
||||
"type": "gauge"
|
||||
}
|
||||
],
|
||||
"refresh": "30s",
|
||||
"schemaVersion": 38,
|
||||
"tags": ["api-company", "leeworks"],
|
||||
"templating": { "list": [] },
|
||||
"time": { "from": "now-3h", "to": "now" },
|
||||
"timepicker": {},
|
||||
"timezone": "browser",
|
||||
"title": "leeworks.dev API Metrics",
|
||||
"uid": "leeworks-api-metrics",
|
||||
"version": 1
|
||||
}
|
||||
@@ -6,3 +6,5 @@ resources:
|
||||
- helmrelease.yaml
|
||||
- gatus-helmrepository.yaml
|
||||
- gatus-helmrelease.yaml
|
||||
- grafana-dashboard-apis.yaml
|
||||
- prometheusrule-apis.yaml
|
||||
|
||||
@@ -0,0 +1,96 @@
|
||||
apiVersion: monitoring.coreos.com/v1
|
||||
kind: PrometheusRule
|
||||
metadata:
|
||||
name: api-company-slo-alerts
|
||||
namespace: monitoring
|
||||
labels:
|
||||
# Must match kube-prometheus-stack's ruleSelector (release label is standard)
|
||||
app: kube-prometheus-stack
|
||||
release: kube-prometheus-stack
|
||||
spec:
|
||||
groups:
|
||||
- name: api-company.slo
|
||||
interval: 1m
|
||||
rules:
|
||||
# -------------------------------------------------------------------
|
||||
# APIHighErrorRate — warning: >5% 5xx over 5 min
|
||||
# -------------------------------------------------------------------
|
||||
- alert: APIHighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality"}[5m]))
|
||||
/
|
||||
sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality"}[5m]))
|
||||
) > 0.05
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: api-company
|
||||
annotations:
|
||||
summary: "High 5xx error rate on {{ $labels.job }}"
|
||||
description: "{{ $labels.job }} 5xx error rate is {{ $value | humanizePercentage }} over the last 5 minutes (threshold: 5%)."
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# APIHighErrorRate — critical: >20% 5xx over 5 min
|
||||
# -------------------------------------------------------------------
|
||||
- alert: APIHighErrorRate
|
||||
expr: |
|
||||
(
|
||||
sum by (job) (rate(api_requests_total{status=~"5..", job=~"zip|holidays|air-quality"}[5m]))
|
||||
/
|
||||
sum by (job) (rate(api_requests_total{job=~"zip|holidays|air-quality"}[5m]))
|
||||
) > 0.20
|
||||
for: 5m
|
||||
labels:
|
||||
severity: critical
|
||||
team: api-company
|
||||
annotations:
|
||||
summary: "Critical 5xx error rate on {{ $labels.job }}"
|
||||
description: "{{ $labels.job }} 5xx error rate is {{ $value | humanizePercentage }} over the last 5 minutes (threshold: 20%)."
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# APIHighLatency — P95 > 2 s over 5 min
|
||||
# -------------------------------------------------------------------
|
||||
- alert: APIHighLatency
|
||||
expr: |
|
||||
histogram_quantile(
|
||||
0.95,
|
||||
sum by (job, le) (rate(api_response_duration_seconds_bucket{job=~"zip|holidays|air-quality"}[5m]))
|
||||
) > 2
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: api-company
|
||||
annotations:
|
||||
summary: "High P95 latency on {{ $labels.job }}"
|
||||
description: "{{ $labels.job }} P95 response time is {{ $value | humanizeDuration }} (threshold: 2s)."
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# APIDataStale — data freshness > 30 days
|
||||
# -------------------------------------------------------------------
|
||||
- alert: APIDataStale
|
||||
expr: |
|
||||
api_data_freshness_seconds{job=~"zip|holidays|air-quality"} > 2592000
|
||||
for: 5m
|
||||
labels:
|
||||
severity: warning
|
||||
team: api-company
|
||||
annotations:
|
||||
summary: "Stale dataset on {{ $labels.job }} ({{ $labels.dataset }})"
|
||||
description: "{{ $labels.job }} dataset '{{ $labels.dataset }}' has not been re-seeded in {{ $value | humanizeDuration }} (threshold: 30 days). Re-seed required."
|
||||
|
||||
# -------------------------------------------------------------------
|
||||
# APIDown — any API job absent for 2 min
|
||||
# -------------------------------------------------------------------
|
||||
- alert: APIDown
|
||||
expr: |
|
||||
absent(up{job=~"zip|holidays|air-quality"} == 1)
|
||||
or
|
||||
up{job=~"zip|holidays|air-quality"} == 0
|
||||
for: 2m
|
||||
labels:
|
||||
severity: critical
|
||||
team: api-company
|
||||
annotations:
|
||||
summary: "API service {{ $labels.job }} is down"
|
||||
description: "Prometheus target {{ $labels.job }} has been unreachable for more than 2 minutes."
|
||||
Reference in New Issue
Block a user