A dying PoE port corrupts packets and crashes my storage

For three days my cluster lost its storage roughly every 12 hours. Sessions to the NAS dropped, filesystems shut themselves down, and pods died with I/O errors, while every network status light stayed green. The cause was the PoE port on a Ubiquiti USW Flex Mini going bad, silently corrupting data in one direction only. No data loss.

Impact

• Down repeatedly over three days: authentik (Postgres), prometheus, grafana, alertmanager, sonarr, prowlarr. One full-cluster storage outage on day three, which also force-shut-down the filesystem on a seventh, orphaned volume left over from a deleted backup clone; no live workload used it.
• Degraded: writes to the NAS from any machine crawled at roughly 160 KB/s. Backups failed on the worst day.
• Data loss: none. Every filesystem replayed its journal after remount.

Timeline

• Jul 16, 04:00 UTC: the first storage connection error ever logged on a cluster node, during the nightly backup window. Nobody noticed.
• Jul 17 to 18: a handful of errors per day, always during backups. Still unnoticed.
• Jul 19: 690 connection errors in the backup window. Sessions collapsed cluster-wide, filesystems went read-only, most stateful workloads down. I blamed the NAS itself, force-rebooted it, and recovered the volumes by hand. It relapsed the same evening.
• Jul 20, ~00:30: another spontaneous drop killed seven of nine volumes overnight.
• Jul 20, morning: I tried to grab a file by torrenting it directly onto the NAS over NFS, and the transfer speed was pitiful, far below what my internet connection or the NAS should allow. That is what sent me looking at the network instead of the NAS again: testing from more than one machine ruled out anything specific to my PC, and by the afternoon, methodical A/B testing had pinned the fault to one switch port.

Root cause

The NAS was plugged into a Ubiquiti USW Flex Mini, itself powered over PoE from the main switch upstream: USW 24 PoE → USW Flex Mini (PoE powered) → NAS. Its uplink port, carrying both power and data, had degraded internally: whenever 48 volts of PoE power was actually present, some of the data passing through that same port got corrupted too, worse while the switch was actively drawing power, milder if the voltage was merely present but unused.

Bigger packets failed far more often than small ones: a basic connectivity check lost about 2%; the same check with NAS-sized packets lost 22%. Storage traffic is made almost entirely of these larger packets, and the nightly backups, the heaviest sustained transfers of the day, reliably pushed the loss past the breaking point. As the port kept degrading, even ordinary background traffic became enough to trigger it, and the failures went from "only during backups" to "any time".

Here is how that turned into dead containers, confirmed step by step in the logs:

1. Corrupted data gets thrown away and resent instead of delivered: a bulk transfer toward the NAS showed a 16% resend rate, capping its throughput almost to a halt.
2. The NAS and the cluster nodes check on each other every few seconds over iSCSI; when those checks get lost too, the connection is declared dead after a 5-second timeout, logged repeatedly as ping timeout of 5 secs expired followed by a connection error.
3. The connection drops mid-write, and the filesystem shuts itself down to avoid corrupting an incomplete write.
4. The connection reconnects, but the filesystem stays shut down: every read or write to it fails, so the container crashes, sometimes while Kubernetes still shows the pod as Running.

Why it was hard to see

• Every status light was green. UniFi reported a healthy 1 Gbps link on that port the entire time. A link being "up" only means the two ends can hear each other at all, not that every bit of data they exchange arrives intact.
• Every error counter was zero. The NAS's network card reported no errors whatsoever; the corrupted data was discarded before it ever got far enough to be counted.
• The problem only showed up in one direction. Reading from the NAS was fast and clean the entire time; only writing to it was affected. Half of any given test looked completely normal.
• A previous, similar-looking incident primed the wrong suspect. Two weeks earlier, a failed NVMe drive in the NAS had dropped iSCSI sessions mid-write and left several filesystems corrupted, the same symptoms I was now seeing. So the NAS itself was the natural first suspect again.
• The NAS's own event log seemed to back that suspicion up. Its SAN event log showed a burst of warnings right after every night's backup software took its LUN snapshots, which looked exactly like a backup-driven storage problem. It turned out to be a real bug, just an unrelated one: stale login attempts retrying target IDs that no longer existed once each night's temporary backup clones were deleted. Noisy, cosmetic, and coincidentally timed with the actual outages, not the cause of them.
• An early test on a workbench passed cleanly. Tested with a short cable on a desk, the Flex Mini worked fine, PoE and all. The short cable had enough slack to paper over the damage; it only failed once the longer cable run to the NAS was reattached.

Isolation

One variable at a time, using a 200-packet test with NAS-sized packets as the yardstick:

Same cables, same physical ports, same everything else; only the presence of PoE voltage on that one port decided whether the data got through intact. That cleared the cables, the USW 24 PoE, and the NAS itself, and left the USW Flex Mini's uplink port unambiguously at fault.

Resolution

• The NAS now plugs directly into the USW 24 PoE. The USW Flex Mini is retired.
• The seven affected volumes were recovered without any reboots, by remounting each one from inside the storage driver's own pods and letting the filesystem replay its journal.
• Write throughput went from roughly 160 KB/s to 77.5 MB/s.

What went well

• No data loss across the three separate session drops in three days. Filesystem journaling plus databases replaying their own transaction logs held the line.
• Off-site backups kept working almost the whole time, even mid-degradation.
• The logs on both ends preserved a three-day error timeline that pinned down the exact onset and disproved the software theories.
• Having already worked through the same kind of volume recovery during the earlier NVMe incident meant day three's recovery took minutes, not hours, no relearning it under pressure.

What went wrong

• The "link up" indicator isn't proof the network is fine, it stayed green for three days while almost a quarter of traffic was being corrupted, and nothing in the stack checks for that on its own.
• Confirmation bias: the previous incident made the NAS the default suspect, and I spent day one troubleshooting it instead of the network path in front of it.
• Nothing alerted on the early warning signs: connection errors in the cluster's own logs, and a failed backup, both showed up days before the outage got bad.

Follow-ups

• Remove the USW Flex Mini from the storage path; NAS attached directly to the USW 24 PoE.
• Alert on storage connection errors in the cluster logs; they preceded the outage by three days.
• Add a periodic large-packet loss check toward the NAS to the monitoring stack; small-packet checks miss exactly this failure.