A dying PoE port corrupts packets and crashes my storage

For three days my cluster lost its storage roughly every 12 hours. Sessions to the NAS dropped, filesystems shut themselves down, and pods died with I/O errors, while every network status light stayed green. The cause was the PoE port on a Ubiquiti USW Flex Mini going bad, silently corrupting data in one direction only. No data loss.

Impact

• Down repeatedly over three days: authentik (Postgres), prometheus, grafana, alertmanager, sonarr, prowlarr. One full-cluster storage outage on day three, which also force-shut-down the filesystem on a seventh, orphaned volume left over from a deleted backup clone; no live workload used it.
• Degraded: writes to the NAS from any machine crawled at roughly 160 KB/s. Backups failed on the worst day.
• Data loss: none. Every filesystem replayed its journal after remount.

Timeline

• Jul 16, 04:00 UTC: the first storage connection error ever logged on a cluster node, during the nightly backup window. Nobody noticed.
• Jul 17 to 18: a handful of errors per day, always during backups. Still unnoticed.
• Jul 19: 690 connection errors in the backup window. Sessions collapsed cluster-wide, filesystems went read-only, most stateful workloads down. I blamed the NAS itself, force-rebooted it, and recovered the volumes by hand. It relapsed the same evening.
• Jul 20, ~00:30: another spontaneous drop killed seven of nine volumes overnight.
• Jul 20, morning: I tried to grab a file by torrenting it directly onto the NAS over NFS, and the transfer speed was far below what my internet connection or the NAS should allow. That sent me looking at the network instead of the NAS. Testing from more than one machine ruled out anything specific to my PC, and by the afternoon, A/B testing had pinned the fault to one switch port.

Trigger

The nightly backups, the heaviest sustained transfers of the day, pushed packet loss toward the NAS past the point where iSCSI sessions survived. As the port kept degrading, ordinary background traffic became enough, and the failures went from "only during backups" to "any time".

Root cause

The NAS was plugged into a Ubiquiti USW Flex Mini, itself powered over PoE from the main switch upstream: USW 24 PoE → USW Flex Mini (PoE powered) → NAS. Its uplink port, carrying both power and data, had degraded internally. Whenever 48 volts of PoE power was present, some of the data passing through that same port got corrupted. It was worse while the switch was drawing power, and milder if the voltage was present but unused.

Bigger packets failed far more often than small ones. A basic connectivity check lost about 2%, and the same check with NAS-sized packets lost 22%. Storage traffic is made almost entirely of these larger packets.

Here is how that turned into dead containers, confirmed step by step in the logs:

1. Corrupted data gets thrown away and resent instead of delivered. A bulk transfer toward the NAS showed a 16% resend rate, which brought its throughput almost to a halt.
2. The NAS and the cluster nodes check on each other every few seconds over iSCSI. When those checks get lost too, the connection is declared dead after a 5-second timeout, logged repeatedly as ping timeout of 5 secs expired followed by a connection error.
3. The connection drops mid-write, and the filesystem shuts itself down to avoid corrupting an incomplete write.
4. The connection comes back, but the filesystem stays shut down. Every read or write to it fails, so the container crashes, sometimes while Kubernetes still shows the pod as Running.

Contributing factors

• Every status light was green. UniFi reported a healthy 1 Gbps link on that port the entire time. A link being "up" only means the two ends can hear each other. It says nothing about whether the data arrives intact.
• Every error counter was zero. The NAS's network card reported no errors. The corrupted data was discarded before it got far enough to be counted.
• The problem only showed up in one direction. Reading from the NAS was fast and clean the entire time, and only writing to it was affected. Half of any given test looked normal.
• A previous, similar-looking incident primed the wrong suspect. Two weeks earlier, a failed NVMe drive in the NAS had dropped iSCSI sessions mid-write and left several filesystems corrupted, the same symptoms I was now seeing. So the NAS itself was the natural first suspect again.
• The NAS's own event log seemed to back that suspicion up. Its SAN event log showed a burst of warnings right after every night's backup software took its LUN snapshots, which looked exactly like a backup-driven storage problem. It turned out to be a real but unrelated bug. Stale login attempts kept retrying target IDs that no longer existed once each night's temporary backup clones were deleted. The warnings were cosmetic and only happened to line up with the outages.
• An early test on a workbench passed cleanly. Tested with a short cable on a desk, the Flex Mini worked fine, PoE and all. The short cable was enough to hide the damage, and it only failed once the longer cable run to the NAS was reattached.

Isolation

I changed one variable at a time and used a 200-packet test with NAS-sized packets to compare.

With the same cables and ports, only the PoE voltage on that one port decided whether the data got through intact. That cleared the cables, the USW 24 PoE, and the NAS, and left the USW Flex Mini's uplink port.

Resolution

• The NAS now plugs directly into the USW 24 PoE. The USW Flex Mini is retired.
• The seven affected volumes were recovered without any reboots, by remounting each one from inside the storage driver's own pods and letting the filesystem replay its journal.
• Write throughput went from roughly 160 KB/s to 77.5 MB/s.

What went well

• No data loss across the three separate session drops in three days. Filesystem journaling and the databases replaying their own transaction logs covered it.
• Off-site backups kept working almost the whole time, even while the port was degrading.
• The logs on both ends kept a three-day error timeline that showed the exact onset and ruled out the software theories.
• I had already done the same kind of volume recovery during the earlier NVMe incident, so day three's recovery took minutes instead of hours.

What went wrong

• Nothing in the stack checks packet loss on the storage path. The link stayed green for three days while almost a quarter of large packets were being lost.
• The previous incident made the NAS the default suspect, and day one went into troubleshooting it instead of the network path in front of it.
• Nothing alerted on the early warning signs. Connection errors in the cluster's own logs and a failed backup both showed up days before the outage got bad.

Follow-ups

• Remove the USW Flex Mini from the storage path and connect the NAS directly to the USW 24 PoE.