You're paying for bandwidth you're not using. If you run an enterprise network with dozens or hundreds of Internet-connected sites — static addressing, BGP to your upstreams, maybe MPLS humming alongside — then at most of those sites there's a fat Internet pipe sitting mostly idle, bought for inbound traffic, redundancy, or reasons nobody remembers. That idle capacity is an asset — you're just not treating it like one. This piece is about putting it to work deliberately: an enterprise anycast CDN, built at your own edge, on infrastructure you already own.
Before we go anywhere: this is a theory — my theory. Parts of it are well-understood practice; parts I've reasoned through and tested where I could — not at planetary scale. Poke at it, steal what works, throw out what doesn't. What I can promise is that I've tried to think clearly about what's possible to operate, not just what's possible to build.
And it's not for everyone. If your edge is a managed platform where BGP is somebody else's problem and traffic just "arrives where it should," most of what follows will read as unnecessary complexity — and for you, it probably is. This is for the narrower crowd that already operates the edge, already speaks BGP, and wants deterministic control over how traffic enters and moves through the network instead of renting that decision from someone else's control plane.
The problem is a familiar tension. Customers expect low latency and local responsiveness. Services have to survive partial failures and regional outages. Internet paths are unpredictable. And centralized edge platforms — however good — are external dependencies with opaque failure modes. Meanwhile you're sitting on something that already looks suspiciously like a global edge: sites in the same regions as your customers, existing load balancers and health systems, and a private WAN that often outperforms the public Internet during an incident.
So the question was never whether an enterprise could build something CDN-shaped. It's whether you can do it safely, deliberately, and without creating new failure domains. That's the whole game. Nothing trusted implicitly, every routing decision scoped, every optimization optional, every failure contained. You're not chasing maximum cleverness. You're building a system that stays understandable under stress.
Anycast answers one question
Anycast gets described as a clever trick, or as something mysterious only the giant providers can run safely. It's neither. Anycast is a very simple idea with very specific properties, and most of the problems people blame on it come from asking it to do things it was never designed to do.
Here's all of it, in plain terms: multiple edge nodes advertise the same public IP prefix — typically a /24 — into BGP, each independently, from its own location. From the Internet's point of view the advertisements are indistinguishable, so standard BGP attributes — path length, policy, topology — deliver each client to the nearest reachable node. If a node dies, its route is withdrawn and the Internet converges around the loss without coordination, without signaling, without the nodes even knowing about each other. No magic synchronization. No special protocol beyond BGP you already run. No DNS games, no client awareness, no application logic.
That's it. Anycast answers exactly one question:
Which node is reachable from here, right now?
It does not answer whether the service is healthy, whether the cache is warm, whether the backend is slow — or what your business intends for the traffic. All of that is somebody else's job, and the most common mistake in these designs is trying to make it BGP's. "If the service is unhealthy, anycast should stop sending traffic." "If the cache is cold, pick another site." BGP has no language for any of that, and forcing application state into the routing layer buys you flapping advertisements, churn, and failure modes you can't diagnose at 2 a.m.
So in this architecture, anycast is deliberately confined to a single responsibility: expose the edge to the Internet, and withdraw it when the node itself is gone. Nothing more. Anycast is powerful precisely because it is limited.
Reachability and correctness are separate concerns, and they get separate machinery. That one sentence is the foundation everything else stands on.
The second question needs a second routing plane
Traffic lands on an edge node. The Internet has made its coarse, reliable decision. Now what? Should this node serve the request locally, forward it to a site that can, or decline entirely?
If anycast were the only routing mechanism you had, every node would need to be equally capable of serving every request — and that's never true in practice. Not every site hosts every service. Not every cache is warm. Not every backend is healthy. Failures are usually partial, not total. BGP sees a prefix and assumes reachability; it's blind to all of this.
So the architecture introduces a second routing domain — and I want to be precise about what it is, because this is where designs go sideways. It is not your enterprise routing table. Not a replacement for your IGP, not MPLS, not data-center routing. It carries no corporate subnets, no user networks, no east-west application traffic. It's a narrowly scoped overlay between edge nodes, invisible to the Internet and converging on its own schedule — Internet BGP churn doesn't shake it, and its reconvergence never touches the global table. It exists to answer the second question:
Given that traffic arrived here, where should it go next?
And it's intentionally starved of vocabulary. The overlay carries node identities — /32 loopbacks — and service reachability signals, also /32s. It does not carry default routes, summaries, arbitrary internal prefixes, or Internet reachability. By limiting what the overlay can express, you limit what it can break. (Past the overlay's decision, your existing load balancers and caches enforce application-level behavior the way they always have — the overlay hands traffic off, it doesn't micromanage what happens inside a site.)
Internet BGP decides who gets the traffic. The overlay decides what happens next. Neither plane attempts the other's job — and that separation is the core safety property of the whole system, not an implementation detail.
Discovery is not trust
Two planes, fine. But how does the overlay come to exist without quietly torching the safety story? This is where a lot of designs fail — the assumption that because systems live "inside" the enterprise, they can just trust each other. At scale, that assumption doesn't hold.
The first move is to split two questions people love to conflate. Discovery asks: who might be out there? Trust asks: who am I willing to talk to? Deliberately separate steps, in a deliberate order.
Bootstrap happens over the Internet — yes, the untrusted side, and that's not an accident. The Internet-facing interface is universally reachable, already required for operation, treated as hostile by default, and therefore already wrapped in explicit authentication. Your private WAN is the opposite: powerful, and dangerous to trust as a bootstrap medium, because a mistake there inherits the WAN's reach as its blast radius.
Each edge node exposes a minimal control-plane endpoint on its Internet interface. It carries no user traffic, installs no routes, forwards no packets. It exists to answer one question: is this peer who it claims to be? Only after both sides answer yes does anything else happen — an overlay adjacency, an exchange of node identities, routing protocols bound to that adjacency. Every step explicit. If authentication fails, the process stops clean: no routes installed, no partial state lying around.
The ordering is the safety mechanism: discover, authenticate over the untrusted medium, form explicit adjacencies, exchange narrowly scoped routes, and only then optimize transport. Reverse any of it and you've built implicit trust with unpredictable failure modes.
Services declare their own truth
So traffic gets in, and the nodes trust each other. What's left: what do the routing decisions actually run on? This is where complexity usually creeps in — engineers building ever-cleverer selection logic to pick the "best" site, the "fastest" backend, the "least loaded" cache. It works in tightly controlled environments and fails ugly when distributed across many sites. This design goes the other way: don't be clever about selection. Be precise about truth.
Services advertise their own availability into the overlay. Healthy service at a site? That site advertises the service's identity — a /32 — into the overlay. Service goes unhealthy? The advertisement is withdrawn. No central controller inferring state, no external system guessing. The component closest to the service declares the truth, and everything else reacts to explicit signals.
That sentence is doing more work than it looks like, so let's name the machinery — because you already own it. Your load balancers have been health-checking these services forever: probing pool members, marking them up and down, holding the one continuously tested opinion on whether a service actually answers. In this design, that opinion finally goes somewhere beyond the load balancer's own bookkeeping. When the checks pass, the LB injects the VIP it fronts — the service's /32, allocated out of the reserved block — into routing at its site. Route health injection; your vendor has shipped it for years under one name or another. The edge node picks that /32 up, runs it through the same boundary filter as everything else, and advertises it into the overlay. When the checks fail, the LB pulls the route, the /32 disappears from the overlay, and every edge learns the truth at routing speed. No new agent to deploy, no health API to invent, no controller to keep synchronized. The component that already knew is the component that tells.
Here's the part I'd underline twice: withdrawal matters more than selection. Everyone thinks the hard problem is choosing the best destination. The problem that actually hurts you is knowing when a destination shouldn't be used at all. Withdrawal is unambiguous. It converges fast. It leaves no stale state. It's native to how routing already behaves — a withdrawn route simply stops being a candidate, no special logic anywhere. Ranking and scoring, by contrast, need shared assumptions, synchronized metrics, and constant tuning, all of which turn brittle at scale.
Partial failure becomes a first-class citizen for free. A node stays reachable via anycast, its other services stay advertised, and one sick service quietly withdraws — traffic for it flows elsewhere without disturbing anything unrelated. Try getting that behavior from a system that infers health.
From the overlay's perspective there is no "load," no "preference." Routes exist or they don't. Multiple sites advertise the same service? All valid; topology and cost pick among them. Nobody advertises it? The service is down, and that fact is explicit and visible instead of buried in a metrics pipeline. Reducing service state to present-or-absent kills a whole family of subtle failures — stale health data, split-brain decisions, oscillation on marginal metrics. Routing works best when it reflects reality, not when it tries to predict it.
The private WAN has to earn its place
Notice that nothing so far required MPLS. That's the test, actually — I'd go as far as making it the acceptance criterion: if all private transport disappeared right now, would the system still be correct? The answer has to be yes. Private transport improves performance. It must never be required for correctness.
Private WANs are cleaner, faster, and more predictable than the public Internet, which is exactly what makes them dangerous as a foundation. Build on them and failures get harder to reason about, bootstrap assumptions leak into steady state, implicit trust replaces explicit verification, and your recovery paths go unexercised until the day they're load-bearing. The system becomes correct only while everything is working — which is no correctness at all.
So private transport gets earned use, never foundational use. Only after nodes have discovered each other, authenticated over the untrusted medium, and stabilized their overlay adjacencies may private links enter — as an alternative underlay for adjacencies that already exist. No new peers, no new routes, no new trust relationships. And the underlay routes that private transport does introduce stay narrow and peer-specific — no defaults ride in with them, and no transitive reachability is assumed. A path to one peer is a path to that peer, nothing more. The overlay above doesn't change: same service truth, same withdrawal semantics, same boundaries. The only difference is that traffic between already-trusted nodes takes a better path. A better road between places that already trust each other — never a side door around authentication or policy.
This inverts the traditional enterprise posture, where the private WAN is primary and the Internet is the fallback. Here the Internet is the common denominator, and losing MPLS degrades performance, not correctness. Traffic falls back to Internet paths automatically, no special re-convergence logic, and — because the Internet path is the normal path for bootstrap and a working underlay every day — your fallback is permanently exercised instead of rotting in a runbook.
One /23, and nothing else exists
All of this is separation of concerns, and separation reduces complexity. It doesn't eliminate mistakes — people fat-finger configs, automation has bugs, systems evolve. So the last layer doesn't try to prevent mistakes. It decides, in advance and mechanically, how far one can travel.
The mechanism is almost embarrassingly blunt: a hard prefix limit enforced by routing policy. Every identity in this system — every node loopback, every service /32 — lives inside one well-defined block. Say a /23, reserved exclusively for this. At the boundary where overlay routing information is consumed, the filter enforces a single rule:
If it is not explicitly allowed, it does not exist.
Outside that /23, no route is ever accepted. Not a default that wandered in from a lab. Not a route leaked out of your internal network. Not a misconfigured summary. Not an arbitrary prefix from a compromised or just plain misbehaving peer. One blunt constraint eliminates the entire class — even a node gone fully wrong cannot escape the sandbox the policy defines.
And here's the payoff that makes the whole design breathe: bounded blast radius is what makes automation safe. New nodes can join gradually, peering can establish automatically, service advertisements can change dynamically — the code doesn't have to be perfect, because the consequences are mechanically capped. That's the honest prerequisite for automating a distributed system: not perfect software. Bounded outcomes. The policy doesn't try to understand intent or infer correctness; combined with the authentication, adjacency control, and overlay scoping above it, the system fails closed instead of failing creatively.
Who this is for, and what it trades away
I said at the top this isn't for everyone, and having walked the whole shape, you can now see why.
It fits organizations that operate many Internet-connected sites, already run BGP and know its failure modes, have predictable regional customer presence, hold private transport that can complement the Internet rather than replace it, and value determinism and fault isolation over abstraction. For them, the edge isn't a black box — it's an asset, and probably an underused one.
It's a poor fit if your edge lives entirely on managed platforms, you don't control routing policy, you expect the network to infer application intent, you prefer decisions made in one central place, or you can't carry operational responsibility at the edge. No judgment in that — those are just different constraints, honestly held.
The tradeoffs, stated plainly because that's the whole point of a theory: you give up abstraction, convenience, and the automatic optimization a managed platform would hand you. You get back predictable failure behavior, bounded blast radius, transparency under stress, and nobody else's control plane in your critical path. None of that is universally desirable — it's situationally valuable, and you know your situation better than I do.
Is it complex? At first glance, sure: two routing planes, explicit trust ceremonies, careful ordering, a fenced prefix block. But the complexity is structural — each piece has one responsibility, tight limits, and a predictable direction to fail in. That discipline, not size, is what keeps the cleverness out.
Networks age, assumptions harden, and convenience quietly turns into dependency. Every so often it's worth stepping back and asking not what's easiest to consume — but what's possible to operate. That idle bandwidth at your edges has been waiting years for you to ask.