The $10k Pop: How a Rogue UPS Fried an Entire Homelab

Life in IT is full of cruel ironies. We buy UPS units to sleep soundly at night, knowing our precious servers will gracefully shut down during a power outage. But sometimes, the hardware gods wake up and choose violence. A user on Reddit just shared a horror story that's basically every homelabber's worst nightmare: The protector became the executioner.

The "Pop" Heard Around the Rack

Our protagonist (let's call him "The Devastated One") had a setup that would make any geek drool: Two RX300S7s, two DL360 Gen 10s, and a DL380 Gen 8. We're talking dual CPUs across the board and over 300GB of RAM. This wasn't a Raspberry Pi cluster; this was serious iron.

Then, catastrophe struck. The UPS didn't just fail; it went "pop," tripped the breaker, and committed seppuku. When the user tried to bypass the dead UPS and plug the servers directly into the mains... silence. Or rather, the blink-and-you-miss-it boot loop of death.

After a full day of troubleshooting—CMOS resets, PSU swaps, dip switch voodoo—the verdict was in: The motherboard power delivery subsystems were toast. Fried. Kaput. While the CPUs and RAM might have survived (fingers crossed), the boards are essentially expensive e-waste now. To add salt to the wound, the user was so distraught they threw the UPS into the trash before even checking the model number. Rage-quitting IRL.

The Reddit Tribunal: Sympathy and Reality Checks

Naturally, the r/homelab community gathered to pay respects and offer some hindsight wisdom:

• The Technical Autopsy: User PutridMeasurement522 pointed out the brutal truth: When a UPS fails "excitingly," it can act as a surge generator for a split second—literally doing the opposite of its one job. They suggested a "bare minimum" boot test (1 CPU, 1 DIMM) to see if the BMC/iLO is still alive. If the management chip is dead, the board is likely a lost cause.
• The Insurance Trap: While some suggested claiming this on homeowner's insurance, Significant-Gap-1822 dropped a reality bomb. Claiming small amounts for electronics can lead to dropped policies or skyrocketing premiums later. Sometimes, eating the cost is cheaper than being blacklisted by insurers.
• The Irony Department: As one commenter put it, "It feels cruel that the proactive redundancy was the thing that caused the issue." It’s like wearing a seatbelt that strangles you.

The C4F Takeaway: Don't Trust the Brick

This story hurts to read, but it's a good reminder for us coding monkeys:

1. Trust No One (Not Even Your UPS): Consider putting a sacrificial surge protector between the wall and the UPS, or even between the UPS and the servers if you're paranoid (though that has its own grounding debates). Point is: electricity is fickle.
2. Check Your Gear: Don't ignore that "Replace Battery" light. Old capacitors and batteries are ticking time bombs.
3. Data > Hardware: Hardware can be replaced (painfully). Data cannot. If this guy's drives are fried too, that's the real tragedy. Always follow the 3-2-1 backup rule.
4. Don't Rage-Trash Evidence: If something blows up, keep it! You might need the serial number for a warranty claim or a lawsuit. Throwing it away is throwing away your leverage.

Pour one out for those Gen 10 servers, folks. Check your backup power today.

Source

Reddit: So my UPS blew up and fried all my server’s motherboards