SSD failure takes down my homelab production server

My Proxmox host kept all of its VM storage on a single spare 256GB SSD: ten VMs on one consumer drive, no RAID, no backups. The drive started failing writes, the LVM thin pool holding every VM disk went partial, and the entire lab was down for over 12 hours. I salvaged the files that mattered by mounting the QEMU disk images by hand. Some data was permanently lost; nothing mission critical.

Impact

Down: all ten VMs on the host, including LAN DNS, the Traefik reverse proxy, GitLab, and the CI runner.
Duration: at least 12 hours from the first write errors to services back up.
Data loss: yes, permanent. Most files were recovered from the disk images; what could not be recovered was not mission critical.

Root cause

The pve volume group, including the thin pool pve/data that backed every VM disk, lived on a single 256GB SSD with no redundancy. Under load the drive began failing writes. Inside the guests the kernel logged I/O error, dev sda, op 0x1:(WRITE) and long runs of Buffer I/O error on device sda1. Every failing operation was a write; reads were still served from cache, so services stayed partially responsive while nothing was actually being persisted.

Kernel log inside a VM showing repeated buffer I/O errors on sda1 and write I/O errors on sda — Inside a guest VM: every failing operation is a write. Reads were still served from cache.

Then the drive dropped out entirely. The physical volume went missing from the volume group, and every VM start failed with Refusing activation of partial LV, including pve/data itself. With the thin pool unavailable, no VM disk could be activated.

Proxmox task log showing LVM refusing activation of partial LVs for every VM, including pve/data — LVM refusing to activate the partial LVs, including the thin pool pve/data itself.

Detection

There was no disk-health alerting, so the first signal was services misbehaving. I opened the console of the VM hosting DNS and the Traefik reverse proxy and found the kernel logging a continuous stream of I/O errors, with the same sectors failing repeatedly.

Rebooting did not help. The VM dropped to the initramfs shell: fsck exited with status code 4, UNEXPECTED INCONSISTENCY: RUN fsck MANUALLY. A manual fsck repaired the inode counts and reported FILE SYSTEM WAS MODIFIED, but the write errors resumed as soon as the system was back up. Once the host itself was restarted, the thin pool never came back.

Resolution

The pool was gone as running storage, so recovery was file salvage, not repair:

Pulled the VM disk images off the dying drive and attached them read-only on a healthy machine.
Mounted the guest filesystems inside the images by hand and copied out what mattered: compose stacks, configs, application data.
Rebuilt the VMs from the cloud-init template and redeployed the services on fresh storage.

Most of the files I needed were retrieved this way. Some were not, and that data is gone for good. Nothing in that category was mission critical.

What went well

The guest kernel logs pointed straight at dying storage, not at software: every failure was a write.
Mounting the QEMU images by hand recovered most of what mattered, even after LVM refused to activate anything.
Nothing mission critical was lost.

What went wrong

The host storage was a 256GB SSD I had laying around and decided to reuse for this host. That one device was a single point of failure for ten VMs.
No backups of any VM. Recovery meant forensics on a dying disk instead of a restore.
LAN DNS lived on the same failing host, so the outage affected the whole network, not just the lab.

Follow-ups

Replace the dead SSD and rebuild the node.
Run production homelab storage on a RAID1 mirror.
Back up VMs on a schedule: weekly for everything, daily for the critical ones.