An operational control plane
A private, read-only service that shows what is actually running, what is backed up, and which facts have gone stale.
- Problem
- Dozens of services share one environment, and documentation and dashboards can each be wrong in different ways. I wanted one view that kept documented expectations apart from live observations, and said how old each observation was.
- Architecture
- A typed Python service with SQLite and Alembic migrations, run in Docker Compose on a dedicated VM behind private HTTPS. Read-only collectors query the Proxmox API, Prometheus, service health checks, and the UPS through an API role scoped to auditing. Every fact carries an evidence class.
- Operations
- Releases deploy only from reviewed commits with a recorded rollback target. The container runs as a non-root user with a read-only filesystem and no Linux capabilities. The deploy tool keeps a crash-safe journal; on recovery it checks that journal against the running container, image, and database, and stops if they disagree.
- Result & learning
- A database-lock incident first looked like a timeout. Raising the timeout did not hold; profiling showed a retention job stuck deleting rows through unindexed relationships. New indexes and small committed batches fixed it, verified live before the incident was closed. The first plausible fix was only a hypothesis.