How to survive a data center cooling failure mid-heatwave

How to survive a data center cooling failure mid-heatwave

The afternoon the chiller plant tripped, the ambient temperature outside was 35°C and climbing. Inside the data center hall, the readings started climbing within minutes — 28°C, then 32°C, then toward the redline where servers begin throttling or shutting down entirely. That kind of failure doesn’t announce itself with a siren. It shows up first as an email from a monitoring system, then as a rising graph that looks like a heartbeat accelerating into arrhythmia.

For the operations manager at a mid-sized Singapore colocation facility that afternoon — call him Ravi, which isn’t his real name — the first sign was a temperature alarm on row C, which housed a financial services client’s trading infrastructure. Within six minutes, three more rows crossed the warning threshold. The facility had N+1 redundancy on its cooling, but the failure cascaded faster than the standby unit could compensate. These things happen in a heatwave because the design assumptions about ambient temperature get stress-tested in ways the original load calculations didn’t anticipate.

The condenser water delta nobody planned for

The most common architecture in Singapore data centers uses chilled water systems with outdoor condenser units. On a normal day — say, 28°C ambient — the system rejects heat without much trouble. But during a heatwave, when the wet-bulb temperature rises, the cooling towers lose efficiency. Condenser water leaving at 38°C, which is standard, suddenly enters at 35°C instead of 30°C. The delta shrinks. The compressors run longer. The whole system approaches its design limit.

Ravi’s facility had a backup chiller rated for 500kW. What it didn’t have was a direct expansion unit that could handle the full load independently. The standby could cover about sixty percent of the peak load — enough for a single chiller failure in normal conditions, not enough for the cascade that happened when the primary failed and the secondary couldn’t ramp fast enough to prevent a temperature spike across half the floor.

That’s the first thing to understand about cooling redundancy: N+1 means different things depending on whether you’re talking about the chiller itself, the pumps, the piping, or the CRAC units in the hall. A facility can have redundant chillers but share a single condenser water loop. If that loop fails, the redundancy on paper doesn’t translate to real-world resilience.

The difference between 29 and 34 degrees

One lesson that emerged from that afternoon: the temperature sensors inside the server racks told a different story than the facility’s room-level sensors. The room sensors showed 29°C, which is warm but within spec. The intake sensors on the servers at the top of the racks showed 34°C. The difference came from hot-aisle containment that had degraded over time — a few missing ceiling tiles, a misaligned partition, the kind of small leak nobody flags until the airflow path breaks down under load.

“The room was fine on paper,” Ravi said later. “The racks were not.”

The practical takeaway: if you colocate or run your own infrastructure, ask for the rack-level temperature history, not just the facility’s environmental compliance report. A facility that passes an ASHRAE audit at room level can still have localized hot spots that kill drives and degrade processor performance over time. During a cooling failure, those hot spots become critical first.

GPU servers go first, storage arrays drag

Not all equipment handles heat the same way. In the affected facility, the first systems to go into thermal shutdown were blade servers with high-density GPU configurations. Their intake fans spun up to maximum and still couldn’t pull enough cool air through the chassis. The older 1U servers, which generate less heat per unit volume, stayed online longer. The storage arrays held steady — the disk vendor’s firmware was set to throttle rather than shut down, so performance degraded gradually instead of dropping abruptly.

That distribution matters when you’re planning which workloads to migrate or shut down manually. If the cooling fails, the high-density compute nodes will be the first to go. The networking gear, with its more conservative thermal envelope, might last another twenty minutes. Knowing those relative timelines means you can make smarter decisions about what to fail over before the automatic protections kick in.

Ravi’s operations team had a priority list — the trading infrastructure, then the database cluster, then everything else. But the list assumed a graceful shutdown sequence of ten minutes per group. The actual cascade happened in four. Two systems that weren’t on the priority list — a backup DNS cluster and a log aggregation server — ended up staying online simply because they were physically located at the cool end of the row, near an intact cold-aisle seal. Geography within the hall mattered more than the priority list.

Eleven minutes to open the valves

The facility’s BMS (building management system) automatically activated the secondary chiller when the primary’s return water temperature exceeded 32°C. That part worked. What the BMS didn’t do was adjust the chilled water valve positions in the air handlers to compensate for the pressure drop. The standby chiller was running, but the cooling wasn’t reaching the farthest rows because the valves hadn’t throttled up.

A technician had to log into the BMS interface and manually set the valve positions to 100 percent open. That took eleven minutes — nine minutes to diagnose the issue, two minutes to execute the change. The temperature in the far rows had already reached 36°C by then.

“The manual override procedure was documented,” said Ravi. “Nobody had practiced it during a heatwave.”

Testing matters not just for failover but for the cognitive load of executing an unfamiliar procedure under time pressure. A drill at 10 AM on a Tuesday, when nothing is wrong, lets the team discover that the BMS interface is slow, or the valve position number isn’t labeled clearly, or the first person to try it can’t remember the admin credentials because they’re stored in a password manager that requires 2FA and nobody thought to cache that session. Those are the small frictions that turn a recoverable event into an outage.

The gap above the ceiling tiles on row F

Most operators know their power path — which PDU feeds which rack, which transformer feeds which PDU, which substation feeds which transformer. Fewer operators can trace the heat path with the same precision: where does the warm air from row D go after it leaves the hot aisle? Does it flow directly to a CRAC unit return, or does it mix with supply air from an adjacent row because of a recirculation path?

In the heatwave incident, the recirculation problem was worse than anyone had modeled. The hot aisle containment at the end of row F had a gap above the ceiling tiles where warm air was spilling back into the cold aisle. Under normal loads, the extra heat was negligible — maybe 0.5°C above ambient. Under the combined stress of the chiller failure and the elevated ambient temperature, that 0.5°C delta became part of a compound effect that pushed the supply air temperature above the ASHRAE recommended maximum of 27°C.

A thermal imaging walkthrough during a normal operating day, before any failure, would have caught that gap. Many facilities only do thermal surveys during commissioning or after an incident. Doing it annually, and especially before the heatwave season (May through August in Singapore), is the kind of maintenance step that prevents an emergency from becoming a crisis.

Powering down the green systems to save the red

Once the chiller was running again and the valves were fully open, the temperature in the hall still didn’t return to normal for another forty minutes. The thermal mass of the equipment and the room structure had absorbed enough heat that recovery was slower than the initial climb. During that period, the operations team made a pragmatic call: they started gracefully powering down non-critical workloads to reduce the heat load in the room, accelerating the recovery.

During a cooling failure, the instinct is to keep everything running and hope the temperature stabilizes. But reducing the heat load, even by fifteen or twenty percent, can drop the room temperature enough to keep critical systems within their operating range. The decision requires knowing which workloads are truly non-critical and having a documented procedure for shutting them down in order. Without that documentation, the team wastes time debating which server to turn off first.

Ravi’s team had a red/yellow/green classification on their asset inventory. Red systems stayed online. Yellow systems could be shut down after a fifteen-minute grace period. Green systems could be shut down immediately. The classification had been written three years earlier during a disaster recovery planning session and never revisited. Two yellow systems had become critical in the interim — one hosted a customer-facing API that had been added without updating the inventory. That system stayed online by accident, not by plan.

The inventory review, in hindsight, was the detail that mattered most. A cooling failure exposes not just the physical infrastructure but the accuracy of the documentation. If the asset list is wrong, the response plan is wrong too.

Fixing the capacitor was the easy part

The immediate fix was straightforward: the primary chiller’s compressor had tripped on an overcurrent condition caused by a failed capacitor in the drive. Replacing the capacitor took three hours. But the failure sequence revealed deeper issues — the standby chiller’s valve position logic, the gap in the hot aisle containment, the outdated asset classification, the fact that nobody had tested the manual override during a heat load simulation.

Each of those is a separate corrective action. The mistake is to stop after fixing the capacitor and declare the incident resolved. The capacitor was the symptom. The uninsulated thermal path and the untested procedure and the stale documentation were the causes. A facility that addresses all three will handle the next heatwave better. One that only replaces the capacitor will repeat the same failure when the outdoor temperature hits 36°C again, which in Singapore happens roughly every two to three years now.

“We got lucky,” Ravi said. “The financial client didn’t lose any trades because their systems throttled instead of crashing. But you don’t want to find out your limits by hitting them.”

The temperature outside that afternoon had reached 34.8°C, close to Singapore’s all-time record. Inside the data center, the temperature peaked at 37.2°C at the top of row F. Fifty-two servers had throttled their clock speeds. Two had entered thermal shutdown. The facility’s PUE that afternoon was over 2.0, more than double its normal efficiency. The client who lost the most was the one whose trading infrastructure sat right in the path of the recirculating hot air — a positioning decision that made sense during commissioning, when the CFD modeling showed uniform airflow, but that the real-world gap in the containment later contradicted.

Geography, it turned out, was the detail the models never captured. The next heatwave would arrive eventually. The question was whether the facility would be ready for it or would learn the same lesson again.

📷 Photos: Albert Stoynov (Unsplash), Taylor Vick (Unsplash)