You've been painting a room. Careful work, every stroke deliberate. New locks on the doors. A stronger process for who gets a key. A gate in front of the gate. Then you step back to admire it and realize you're standing in the far corner, and the whole floor between you and the exit is wet paint.
You didn't decide to trap yourself. Nobody does. Every stroke was the right call in the moment — a hardening step, a locked-down account, a fix for the CVE that dropped last week. You did all of it correctly. And the sum of all that correct work is a corner you can't walk out of.
That's the shape: every control you add is a door, and some of those doors only open on the days you don't need them. This isn't about any single control. It's about the way they stack, quietly, until the room is painted and you're standing in the wrong part of it.
We chase the security envelope. We move forward. Technology, mandates, process, procedures, regulations. We tighten, and we tighten fast, because the pressure to tighten is real and it never lets up. Somewhere in that speed we stop asking one small question: what if this fails? We ask "is this secure" constantly. The one we skip is: if this one thing goes down, does it take with it the thing I'd use to fix it?
Think of it as building a ladder to the sky. To climb higher you reach down, pull out the bottom rung, and fasten it above your head. Do that enough times and you're a long way up, still climbing, with nothing beneath you. Every rung was progress. There's just no way back to the ground anymore.
Redundancy you haven't checked isn't redundancy. Say you run an anycast service — the same address announced from a handful of locations, traffic landing at the nearest healthy one. It's your safety net. It's the thing that lets you take a node down for maintenance without anyone noticing. Now say three of those locations went silent weeks ago. The route stopped being announced from them and you never got the page. From the outside everything looks fine, because the one surviving location is quietly carrying all of it. So you do what the design says you're allowed to do: you take a node down for maintenance. Except it isn't one of many anymore. It's the last one. And now the service is gone, during a change you were told was safe.
The redundancy didn't fail you. The belief in the redundancy failed you. You never checked that the net was there before you jumped.
Now the worst kind — when the thing that breaks is the very thing you'd use to fix it. DNS is the one that keeps me up. So much rides on name resolution that we forget it's a dependency at all. When it goes, it doesn't go alone — it takes the tools you'd reach for to fix it.
In October 2021, Facebook ran a routine command during maintenance and accidentally pulled the routes to their own name servers off the internet. DNS went dark. Then the corner closed in. They couldn't reach their data centers the normal way, because the network they'd use to get there was the thing that was down. The internal tools they'd normally troubleshoot with were broken too, because those tools leaned on the same DNS that had just vanished. It ended with people physically getting to a building and putting hands on the machines.
They got lucky. That's the lesson I keep. A physical door can be pried open. Somebody got into a room they could no longer badge into, and the machines were reachable by hand. Luck closed the loop — and luck is not an architecture. We should be able to engineer the luck out, so recovery doesn't ride on someone happening to force a door.
Which is why some paths should be hard-coded. If your whole world insists that every destination be a fully-qualified name, resolved through a proxy, checked against a policy — fine, right up until name resolution is the thing that broke. Then you can't get there from here. A few things should be reachable by raw address, on a path that doesn't run through the machinery most likely to be on fire.
And then the hard one — authentication. The nastiest version is login. This tool has to call that service to let you in. That service is down. So the tool can't do its job, so you can't authenticate, so you can't get into anything — including the console you'd use to fix the service that's down. One dependency, one bad Tuesday afternoon, and the whole building locks at once — everybody standing in the hallway holding a badge that suddenly means nothing.
I've stood in a version of this. The thing I needed to fix the problem was on the far side of the thing that broke. You don't forget that feeling — staring at a screen that wants you to log in through a service lying flat on its back, knowing the fix is right there and you can't reach it.
Break-glass is supposed to be the answer here, and it can be a real one. But walk it through honestly. If your emergency access rides on the same directory, the same network, the same name resolution that just failed, then the glass is already broken. It's a normal door with a red frame around it. It only opens on the days you don't need it.
Nobody is watching the whole chain. In a real organization this gets worse, because the pieces don't come from one place. One team owns the proxy. Another owns identity. Another owns the backup platform, the jump path, the change process. Each group does solid work inside its own box. The dependency that ties their boxes together — the one where all of it funnels through a single service on a Tuesday afternoon — is an orphan. It belongs to nobody. So nobody looks at it, because looking at it was never anyone's job.
So how do you find the corner before you're standing in it? This is the part I actually care about, and it's the part nobody writes down. We've got shelves of examples. We've got almost no habits for testing whether we're already cornered. A few that work:
- Fire the break-glass on a calm Tuesday. Not during an incident — on an ordinary morning when nothing's wrong. If it only works when everything else is healthy, it was never break-glass. It was a normal door you painted red.
- Take DNS away in a controlled window and write down what you can still reach. Anything you can't get to without a name is a rung you've already pulled out from under yourself. The things that have to survive get a hard-coded path.
- Restore a backup from the locked-down account, not the admin one. The integrity of the backup is the easy half. Prove that the on-call person, at three in the morning, with the permissions they actually hold, can reach it and is allowed to pull it back. A perfect backup you aren't authorized to download is a corner with a view.
- Trace the login path for your two or three most important admin tools. Whatever all of them quietly call to let you in is a single corner for all of them at once. Find it before it finds you.
You can't repaint the room overnight. I won't pretend there's a tidy fix. When one of these finally breaks, you'll be looking across all that wet paint knowing you can't redesign everything by morning. That's the trap — the corner is easy to walk into and slow to walk out of. So the work happens before the paint's down: walk the room while it's dry. Find the rungs you're about to pull. Keep at least one path that doesn't depend on the thing it's meant to rescue.
Every control you add is a door. Just make sure you're never standing on the wrong side of one you can't open. Engineer the luck out, while it's still cheap.