Known blind spots

This lab borrows production patterns, but it is still one person's home lab on a residential connection. Some failure modes are deliberately not covered, because closing them would cost more than an outage does here. This page lists them.

The monitoring runs on the thing it monitors

Prometheus, Alertmanager, and the pipeline that pushes alerts to my phone all run inside the Kubernetes cluster, and the blackbox probe that checks https://khider.fr runs inside the same network as the site. If the cluster or the connection goes down, nothing outside the lab can page me. I find out by noticing the outage myself, and for these workloads, discovering a full outage a few hours late costs nothing.

One site, one ISP, one power feed

Everything lives in one place, behind one residential connection and one public IP, with no failover. Cloudflare hides the origin, but a power cut or an ISP outage takes everything down until it comes back.

Every byte of the cluster's state sits on one NAS

The Kubernetes cluster has three nodes, but all of their persistent volumes are served over iSCSI by a single Synology NAS, already the root cause of one postmortem and a contributing factor in another. RAID1 covers a dead drive, not a dead power supply, a bad DSM update, or a controller bug. Nightly off-site backups to R2 cap the damage at restore time plus the last day of data. This is largely a Kubernetes-specific risk, not a whole-lab one: several services run as plain Docker containers outside the cluster, with their own storage, independent of the NAS. Not every one of them is fully independent, though: some still depend on the NAS indirectly, through a database or shared storage that lives there, so a NAS failure can reach outside the cluster too, not just through Kubernetes.

One hypervisor for the whole lab

The three Kubernetes node VMs, a separate Docker host running Nextcloud and the DNS resolver behind two reverse proxies (one internet-facing, one internal-only for admin tools), and the CI runner in its own LXC container all live as guests on the same single Proxmox server. Kubernetes tolerates losing a node; it does not tolerate losing the machine all three run on, and that failure is not contained to the cluster: the same hypervisor going down takes every one of those other guests with it too. A second server used to spread this risk, but I retired it because the heat and noise at home outweighed the redundancy.

On-call is one person

There is no rotation and no escalation. If something breaks while I am asleep or away, it stays broken until I get to it. Alerts re-notify every twelve hours, but there is no bound on time to recovery.

The deploy path assumes the lab is healthy

Flux runs inside the cluster it reconciles, and the CI runner that drives the Docker hosts is self-hosted in the lab. GitOps can ship a fix to a degraded system, but not to a dead one. If the cluster or the runner host is down, recovery starts with manual intervention, not a git push.

Why publish this

Every infrastructure has a list like this; most keep it implicit. Writing it down forces me to decide whether each risk is accepted or just ignored. The ones that stop being acceptable end up on the roadmap of the repo, and the ones that cause incidents end up in the postmortems.