diff --git a/docs/cluster-audit.md b/docs/cluster-audit.md new file mode 100644 index 0000000..86de66b --- /dev/null +++ b/docs/cluster-audit.md @@ -0,0 +1,161 @@ +# Cluster State Audit + +**Date:** 2026-05-19 +**Cluster:** testing1/first-cluster (Talos Linux) +**Audited by:** Repo Manager (agent) + +--- + +## Nodes + +3-node control-plane cluster. No dedicated worker nodes. + +``` +NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME +cp-0 Ready control-plane 84d v1.33.0 10.0.1.3 Talos (v1.11.5) 6.12.57-talos containerd://2.1.5 +cp-1 Ready control-plane 84d v1.33.0 10.0.1.4 Talos (v1.11.5) 6.12.57-talos containerd://2.1.5 +cp-2 Ready control-plane 84d v1.33.0 10.0.1.54 Talos (v1.11.5) 6.12.57-talos containerd://2.1.5 +``` + +**Per-node capacity:** 4 CPU, ~7.7 GiB memory, 110 pods max +**Cluster totals:** 12 CPU, ~23.1 GiB memory, 330 pods max + +--- + +## Namespaces + +21 namespaces total. Workload namespaces (excluding system): + +| Namespace | Age | Purpose | +|---|---|---| +| agent-company | 59d | Agent company workloads | +| authentik | 64d | Identity provider (SSO) | +| cert-manager | 64d | TLS certificate management | +| coredns | 64d | DNS | +| gatus | 64d | Uptime monitoring | +| gitea-actions-runner | 48d | CI runner for Gitea | +| gitea-mobile | 53d | Gitea mobile app | +| logging | 52d | Logging stack (Grafana) | +| mail | 62d | Mail services | +| metallb-system | 64d | Bare-metal load balancer | +| monitoring | 51d | Monitoring stack | +| nfs-provisioner | 64d | NFS storage provisioner | +| nixos-dev | 64d | NixOS dev environment | +| sealed-secrets | 55d | Sealed secrets controller | +| sparc | 64d | Sparc application | +| traefik | 64d | Ingress / reverse proxy | + +--- + +## Ingress Setup + +No standard Kubernetes `Ingress` resources found. The cluster uses **Traefik IngressRoutes** (CRD-based): + +| Namespace | IngressRoute | Age | +|---|---|---| +| authentik | authentik | 64d | +| authentik | authentik-http | 64d | +| gatus | gatus | 64d | +| gatus | gatus-http | 64d | +| gitea-mobile | gitea-mobile | 53d | +| gitea-mobile | gitea-mobile-http | 53d | +| logging | grafana | 52d | +| logging | grafana-http | 52d | +| sparc | sparc | 64d | +| sparc | sparc-http | 64d | +| traefik | traefik-dashboard | 64d | + +--- + +## Resource Usage (Headroom) + +> **Note:** `kubectl top nodes` is unavailable -- Metrics API (metrics-server) is not installed. Resource requests/limits from `kubectl describe nodes` are used instead. + +### Per-Node Allocated Resources (Requests) + +| Node | CPU Requests | CPU % | Memory Requests | Memory % | +|---|---|---|---|---| +| cp-0 | 1710m | 43% | 2738Mi | 37% | +| cp-1 | 3360m | 85% | 5950Mi | 81% | +| cp-2 | 2100m | 53% | 4018Mi | 54% | + +### Per-Node Limits (for overcommit awareness) + +| Node | CPU Limits | CPU % | Memory Limits | Memory % | +|---|---|---|---|---| +| cp-0 | 7300m | 184% | 8320Mi | 113% | +| cp-1 | 9700m | 245% | 13908Mi | 190% | +| cp-2 | 6500m | 164% | 11008Mi | 150% | + +## Headroom + +**WARNING -- The following nodes have less than 20% free capacity by requests:** + +- **cp-1 CPU: 85% requested** -- only 15% headroom. This node is near capacity for CPU requests. +- **cp-1 Memory: 81% requested** -- only 19% headroom. This node is near capacity for memory requests. + +**ADVISORY -- Overcommit risk:** + +All three nodes have CPU and memory limits exceeding 100%. This means actual usage spikes could cause OOM kills or CPU throttling. This is common in non-production clusters but should be monitored. + +- cp-0: CPU limits at 184%, memory limits at 113% +- cp-1: CPU limits at 245%, memory limits at 190% +- cp-2: CPU limits at 164%, memory limits at 150% + +**Recommendation:** Install metrics-server to enable `kubectl top` and real-time resource monitoring. Consider adding a dedicated worker node if workloads continue to grow, as cp-1 is already heavily loaded. + +--- + +## Flux Kustomizations + +All kustomizations are reconciled and ready. + +``` +NAMESPACE NAME REVISION SUSPENDED READY MESSAGE +flux-system apps main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system authentik main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system cert-config main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system flux-config main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system infrastructure main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system traefik main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +flux-system traefik-config main@sha1:c48d8414 False True Applied revision: main@sha1:c48d8414 +``` + +--- + +## Flux Helm Releases + +All Helm releases are reconciled and ready. + +``` +NAMESPACE NAME REVISION SUSPENDED READY MESSAGE +authentik authentik 2026.2.3 False True Helm upgrade succeeded +cert-manager cert-manager v1.14.7 False True Helm install succeeded +sealed-secrets sealed-secrets 2.18.5 False True Helm upgrade succeeded +traefik traefik 28.3.0 False True Helm upgrade succeeded +``` + +--- + +## Storage Classes + +``` +NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE +nfs-client (default) k8s-sigs.io/nfs-subdir-external-provisioner Delete Immediate false 64d +``` + +Single storage class using NFS. Volume expansion is **not** enabled. Reclaim policy is **Delete** (PVCs are cleaned up on release). + +--- + +## Summary + +| Item | Status | +|---|---| +| Nodes | 3x control-plane, all Ready, Talos v1.11.5, K8s v1.33.0 | +| Namespaces | 21 total (16 workload, 5 system) | +| Ingress | Traefik IngressRoutes (11 routes across 6 namespaces) | +| Flux | 7 kustomizations, 4 Helm releases -- all healthy | +| Storage | NFS-backed default StorageClass | +| Headroom | cp-1 is near capacity (85% CPU, 81% memory requests) | +| Metrics | metrics-server NOT installed -- no real-time usage data |